Information processing system, program and the like

Abstract

To provide a system and the like better than a prior art, such as an information processing system and program and the like for imaging a vehicle interior from a rear side.SOLUTION: An information processing system comprises a control unit which processes video data of a vehicle interior imaged from a vehicle rear toward a vehicle front, analyzes the video data of the vehicle interior, and determines a situation based on a result of the analysis. For example, the control unit has functions of determining the state of an installed object on the ceiling of the vehicle and the state of a person within the vehicle interior projected in the video data of the vehicle interior and outputting result information based on a result of the determination of the state of the installed object and the state of the person.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a program, and the like. [Background technology]

[0002] JP 2018-160135 A (Patent Document 1) is a background technology in this technical field. This publication states that "the vehicle attention warning device 10 is a system mounted on an autonomous vehicle, and includes a microphone 22 for capturing sound outside the vehicle in front, a microphone 23 for capturing sound outside the vehicle in rear, a front camera 24, a rear camera 25, a camera 26, a front speaker 31 for sound outside the vehicle, a rear speaker 32 for sound outside the vehicle, a first display 33, and a second display 34, and is used to warn passengers inside or outside the vehicle when they exit the vehicle" (see Abstract). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-160135 Summary of the Invention [Problem to be solved by the invention]

[0004] The conventional system has various problems, for example, Patent Document 1 does not consider a mechanism for capturing images of the interior of the vehicle from the rear side. Therefore, the present invention provides a system that is superior to conventional systems, such as a mechanism for photographing the interior of a vehicle from the rear side.

[0005] The object of the present invention is not limited to this, and the applicant intends to obtain rights for configurations that aim to achieve the effects achieved by parts of the configuration disclosed in the specification and drawings, etc., through divisional applications, amendments, etc. For example, this specification discloses problems in which the phrases "can" and "is possible" are read as "the problem is." Each problem is described as an independent problem, and the applicant intends to obtain rights for configurations that solve each problem separately through divisional applications, amendments, etc. Even if the problem is implicitly understood from the description in the specification, the applicant intends to claim part of the configuration described in this specification through amendments or divisional applications. Furthermore, the applicant has disclosed configurations that solve problems that combine these independent problems, and the applicant intends to obtain rights for them. [Means for solving the problem]

[0006] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes multiple means for solving the above-mentioned problems, and one example is characterized by processing video data of the interior of the vehicle taken from the rear of the vehicle toward the front of the vehicle, for example, by processing video data of the interior of the vehicle taken from the rear of the vehicle toward the front of the vehicle and video data of the exterior of the vehicle taken from the rear of the vehicle. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a system that is superior to conventional systems, such as a system for photographing the interior of a vehicle from the rear side. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments.

[0008] The effects of the present invention are not limited to these, and effects achieved by the configuration disclosed in the present specification and drawings, etc. are also disclosed, and the applicant intends to obtain rights to the configuration achieving such effects through divisional applications, amendments, etc. For example, in this specification, phrases such as "can" and "is possible" are descriptions that clearly indicate the effects achieved, and there are also parts that demonstrate effects even without the phrases "can" and "is possible." Furthermore, there are effects that can be understood from the configuration even without such phrases. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the installation positions and shooting directions of a photographing device 101R and a rear camera 102 for photographing inside the vehicle 10 according to this embodiment. [Figure 2] FIG. 2 is an example of a connection diagram of the image capturing devices 101F and 101R, the rear camera 102 for capturing images inside the vehicle, the user terminal 103, and the external server 104. In FIG. [Figure 3] FIG. 3 shows an example of the hardware configuration of the image capturing device 101R connected to the rear camera 102 for capturing images inside the vehicle. [Figure 4] FIG. 4 shows an example of a processing flow 400 for images captured by the rear camera 102 for capturing images inside the vehicle. [Figure 5] FIG. 5 shows an example in which two cameras are installed. [Figure 6] FIG. 6 shows an embodiment in which the shooting direction of the camera in the embodiment of FIG. 1 can be changed. [Figure 7] FIG. 7 shows an embodiment in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle in FIG. 1 are configured as a single wide-angle camera. [Figure 8] FIG. 8 shows an embodiment in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are attached to the rear hatch of an ambulance vehicle. [Figure 9] FIG. 9 shows an embodiment in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are attached to the rear hatch of a welfare vehicle. [Figure 10]FIG. 10 is a diagram illustrating an image captured by the rear camera 102 for capturing images inside the vehicle. [Figure 11] FIG. 11 is a diagram illustrating an image captured by the rear camera 102 for capturing images inside the vehicle. [Figure 12] FIG. 12 is an example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture. [Figure 13] FIG. 13 is another example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture. [Figure 14] FIG. 14 is yet another example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture. [Figure 15] FIG. 15 shows an example of a variation of the mounting fixture. [Figure 16] FIG. 16 shows an example of another variation of the mounting fixture. [Figure 17] FIG. 17 shows an example of a rear camera 102 for photographing the interior of a vehicle, which has a pin plug. [Figure 18] FIG. 18 is an example of an enlarged view of another anti-rotation structure for the camera of FIG. [Figure 19] FIG. 19 shows an embodiment in which the camera of FIG. 17 is attached to a cylindrical photographing device. [Figure 20] FIG. 20 is an explanatory diagram illustrating an example of a procedure for attaching the camera of FIG. 17 to a digital camera type photographing device. [Figure 21] FIG. 21 shows an embodiment in which two cameras shown in FIG. 17 are attached to a digital camera type photographing device using a branching cable 1804. [Figure 22] FIG. 22 shows another example of a wide-angle camera 100. [Figure 23] FIG. 23 shows an example of a camera that blocks the light from a projector. [Figure 24] FIG. 24 is a diagram showing an example of the external configuration of the photographing device 101F. [Figure 25] FIG. 25 is a diagram showing the configuration of the bracket 600. As shown in FIG. [Figure 26]FIG. 26 is a block diagram showing the electrical configuration of the photographing device 101F. [Figure 27] FIG. 27 is an explanatory diagram of a screen displayed by the photographing device 101F. [Figure 28] FIG. 28 is an explanatory diagram of an example of a viewer. [Figure 29] FIG. 29 is an explanatory diagram of display elements included in the viewer of FIG. [Figure 30] FIG. 30 is an explanatory diagram of display elements included in the viewer of FIG. [Figure 31] FIG. 31 is an explanatory diagram of display elements included in the viewer of FIG. [Figure 32] FIG. 32 is an explanatory diagram of an example of a viewer. [Figure 33] FIG. 33 is a diagram showing another example of the rear camera 102 for photographing the interior of the vehicle, different from that shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment shown below is one example of the present invention, and the content of the present invention should not be construed as being limited based on the following description. In addition, the following explanation will specifically describe the individual components of an information processing system that processes video data of the interior of the vehicle taken from the rear of the vehicle toward the front of the vehicle and video data of the exterior of the vehicle taken from the rear of the vehicle. However, the components of this system are not limited to these, and other configurations and forms may be used as long as they can perform the functions described in relation to the individual components. Furthermore, the video data may be data that constitutes either a still image or a moving image, but moving images are particularly preferable. In the following explanation, the term "video" may refer to a still image (also called a still image) or a moving image (also called a moving image) that can be seen by the human eye, but it may also be interpreted as "video data," "image," or "image information" as long as it is consistent with the technical content.

[0011] FIG. 1 is an explanatory diagram illustrating an example of the installation positions and shooting directions of a photographing device 101R and a rear camera 102 for photographing inside the vehicle 10 according to this embodiment. FIG. 1(a) is a side view of the vehicle 10, and FIG. 1(b) is a top view of the vehicle 10. As shown in FIG.

[0012] 1(a) and 1(b), the vehicle 10 has a photographing device 101R at the rear of the passenger compartment. The photographing device 101R is an electronic device with a photographing function, and may be a camera or the like provided for security, surveillance, or other sensing purposes, but it is preferable that the photographing device 101R be a rear camera that cooperates with a drive recorder, and most preferably, it is the drive recorder itself.

[0013] In the example of FIG. 1 , the photographing device 101R is shown with a rear camera 102 for photographing the interior of the vehicle 10 from behind the vehicle 10, mounted side by side in the left-right direction of the vehicle. The rear camera 102 for photographing the interior of the vehicle 10 is mounted on the rear side of the vehicle 10, for example, preferably mounted rearward of the center of the interior of the vehicle 10 in the longitudinal direction of the vehicle 10. It is particularly advantageous to position the rear side of the interior of the vehicle 10 between the headrests and backrests of the rear seats and the rear window, and a position attached to the rear window is particularly preferable. However, the method of mounting the rear camera 102 for photographing the interior of the vehicle 101R is not limited to this. The photographing device 101R and the rear camera 102 for photographing the interior of the vehicle 10 can be mounted in various manners, as described below. For example, the rear camera 102 for photographing the interior of the vehicle 10 and the photographing device 101R may be connected by a mounting fixture.

[0014] In the example of Figure 1, the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are shown directly adjacent to each other, but they may also be connected via a mounting bracket.In this case, by pulling out the cables of both the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle from between the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using, for example, an L-shaped cable, and combining them into a single cable, and then connecting it to an existing line that connects to a power source, etc., the rear camera 102 for photographing the interior of the vehicle can be added without the wiring being noticeable and without detracting from the aesthetic appearance of the interior of the vehicle. Similarly, even if there is a front-side camera device inside the vehicle and a cable connection for power and data communication exists between the rear-side camera device 101R and the front-side camera device, it is possible to integrate both cables pulled out from the space and connect them to the connection cable with the front-side camera device.

[0015] The two cameras, one for the rear rear and one for the interior of the vehicle, may be integrated. With this configuration, the photographing device 101R and the interior camera 102 can be handled as a single unit, which has the advantage of being easy to install and convenient for the user. For example, it is preferable to position the lens for the interior of the vehicle higher than the lens for the rear rear (e.g., vertical camera arrangement). In particular, when the lens for the rear rear is installed in a position where it can photograph the rear view through the rear window, it is preferable to install the lens for the interior of the vehicle using a mounting fixture that is installed in a position where its back side is in the non-glass part of the upper part of the rear window, particularly in the part where it cannot photograph the rear view through the rear window. Furthermore, in such a configuration, the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle may be configured so that the central axes of the respective photographed images are aligned in advance, for example, during the design and manufacturing stages. In this way, it is easy to perform processes such as comparing the images from each camera. The photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are arranged vertically and integrated, and the central axes of the two are aligned in advance, so that the user can obtain images of the interior of the vehicle and images of the exterior of the vehicle behind the vehicle with their centers aligned with each other simply by placing one camera with an integrated vertical camera arrangement.

[0016] In another embodiment of the two-camera integrated type, the interior rear camera 102 for photographing the interior of the vehicle may be rotated vertically to change the vertical angle of the photographing direction. Furthermore, if the interior rear camera 102 for photographing the interior of the vehicle is rotated horizontally to change the angle of the photographing direction up, down, left, and right, the desired photographing direction can be set over a wider range. Furthermore, the user may use his / her own user terminal, a user terminal installed in the vehicle, a remote control, etc. to adjust the angle of the shooting direction of the rear camera 102 for shooting inside the vehicle while checking the image on the display of the user terminal, etc.

[0017] In another embodiment, the image capturing device 101R and the interior rear camera 102 may be configured as a single wide-angle camera, and the image captured by this configuration may capture both the interior of the vehicle and the rear of the vehicle from rear to front. Such an embodiment will be described in more detail with reference to FIG. 7.

[0018] In another embodiment, the photographing device 101R and the interior-photographing rear camera 102 may be configured as two physically independent rear cameras (for example, configured as separate housings) and may be configured to be mounted at different positions on the rear side of the vehicle 10. In particular, the two rear cameras may be mounted near both sides of the interior of the vehicle, and in particular, one may be mounted at the position of one pillar on the rear side of the vehicle 10, and the other may be mounted at the position of the other pillar on the rear side of the vehicle 10. The position of the pillar may be on the interior side of the pillar. In this way, the camera is less likely to be in the field of view of the occupants of the vehicle 10, and also less likely to be in the field of view of the occupants of the vehicle 10, such as the driver, of a vehicle behind the vehicle 10.

[0019] In any of the embodiments, it is advantageous to configure the system to record both the image of the interior of the vehicle from the rear to the front and the image of the rear of the vehicle by using the image capturing device 101R and the rear camera 102 for capturing images inside the vehicle. It is also preferable to configure the system to record images of the outside to the side of the vehicle and the outside in the distance behind the vehicle.

[0020] The above describes examples of the placement and shooting direction of the image capturing device 101R and the vehicle interior rear camera 102. Below, examples of the functions and roles of the image capturing device 101R and the vehicle interior rear camera 102 will be described.

[0021] The interior rear camera 102 is connected to a photographing device 101R installed at the rear of the vehicle, which performs processing and compression using an ISP (Image Signal Processor). Using this photographing device 101R as the main unit is significantly more advantageous than using a front-facing photographing camera attached to the front of the vehicle, for example, the windshield, as the main unit. The front-facing photographing camera is installed at the front of the vehicle 10 and photographs the area ahead of the vehicle 10. Due to its mounting location, the front-facing camera is susceptible to the effects of direct sunlight and the resulting heat. On the other hand, because the front-facing camera is installed on the windshield, etc., it is functionally limited by the need to have an open field of view in front of the vehicle, making it difficult to adopt effective shading measures.

[0022] In contrast, a rear rear camera attached to the rear window of a vehicle has the advantage of being able to more easily reduce the effects of direct sunlight and the heat it generates at the installation position than a front camera. Also, compared to the windshield, the rear window can be shielded to a certain extent, further reducing the effects of direct sunlight and heat.

[0023] In addition, in many vehicles, such as minivans, the rear window is closer to vertical than the windshield. Therefore, there is an advantage that direct sunlight (especially direct sunlight during hot hours when the sun is high in the sky) is less likely to enter the drive recorder when positioned on the rear window than when positioned on the windshield. The rear rearview camera is installed on the rear side of the vehicle 10 and is used to capture images behind the vehicle 10. The rear rear camera, which is less affected by direct sunlight and heat, is used as the main unit, and this rear rear camera processes and records images from the additionally installed camera for photographing inside the vehicle and the front front camera, making it possible to make the entire system less susceptible to breakdowns and to reduce malfunctions due to heat.

[0024] By using a rear camera 102 for capturing images inside a vehicle as shown in FIG. 1, the following types of images can be captured. The rear camera 102 for photographing inside the vehicle may be configured to photograph the interior of the vehicle from behind, so that it can photograph, analyze, and detect people's actions while protecting their privacy and not photographing their faces. The rear camera 102 for photographing inside the vehicle may be configured to photograph the outside of the vehicle through the side window glass from the rear of the vehicle to detect the situation around the vehicle. The rear camera 102 for photographing the interior of the vehicle may be configured to detect the state of an object installed on the ceiling of the vehicle. The rear camera 102 for photographing inside the vehicle may be configured to detect luggage on the luggage rack, on the seats, etc. The rear camera 102 for photographing inside the vehicle may be configured to detect the state of a display unit of the vehicle, such as a monitor provided on the back of a seat or a monitor provided on the ceiling. The rear camera 102 for photographing the interior of the vehicle may be configured to detect whether or not there is a person inside the vehicle, particularly in the space behind the vehicle. The rear camera 102 for photographing inside the vehicle may be configured to detect the parking state based on whether or not there is a person inside the vehicle. These points are explained in more detail in connection with FIG.

[0025] When using an infant-only or rear-facing child car seat, conventional front-facing cameras are unable to see the front seats, making it extremely difficult to check the condition of the infant in the child car seat. However, the rear camera 102, which captures the interior of the vehicle from the rear, can capture excellent images of the infant in the child car seat. In fact, by adjusting the capture direction of the image captured by the rear camera 102 while checking the image captured by the rear camera 102 on a handheld device, as described above, it is even possible to place the infant in the center of the image captured by the rear camera 102. This allows the control unit 303 of the photographing device 101R to accurately identify the baby in the child car seat based on the image from the rear camera 102 for photographing inside the vehicle interior, and to reliably detect the occurrence of abandonment, etc. Such abandonment detection and associated warning can be more reliably performed by a configuration including a human detection sensor, which will be described later.

[0026] The drive recorder may also be provided with a function for combining the video from the rear camera 102 for capturing images inside the vehicle with video from other cameras, connecting the video from the rear rear, the left and right sides of the rear interior, the left and right sides of the front interior, and the front of the vehicle to generate video that can be rotated around the vehicle (for example, a full circle). Alternatively, a viewer program for a personal computer or an information processing device such as a smartphone or tablet that plays back data from a recording medium on which these videos are recorded may be provided. The drive recorder may also be provided with a function for generating video that can be rotated around the vehicle (for example, a full circle) from the video from the rear interior and the video from the front interior. For example, in a configuration in which the rear rear camera 101R and the rear camera 102 for interior photography each cover a 180° photography range, it is preferable to configure the camera 101R to continuously record sudden approaches from behind and rearward tailgating, as photographed by the rear camera 101R, and side tailgating and dangerous overtaking, as photographed through the window glass of the vehicle interior by the rear camera 102.

[0027] The video captured by the interior rear camera 102 is preferably configured with a viewing angle and installation position that not only allows the interior of the vehicle to be seen from the rear, but also captures the driver's seat and passenger seat, making it possible to record acts of aggressive driving from the side, or acts of someone getting out of the car and committing intimidation or violence. In such situations, if it becomes known that recording has started, the situation may worsen. Therefore, it is advantageous to configure the camera to automatically start recording when it detects that such a situation has occurred, or to perform event recording using a weak remote control, IR remote control, or the like so that it is not easily noticed that the user has started recording.

[0028] Because the interior camera 102 photographs the interior of the vehicle from the rear (rear side) of the vehicle, compared to conventional interior cameras that photograph the interior from the front (front side) of the vehicle, the opportunity to photograph people in the driver's seat, passenger seat, and other seats from the front is significantly reduced. This makes it possible to protect the privacy of people in the vehicle. This is particularly advantageous when used in taxis, buses, etc.

[0029] High-precision detection of abandoned vehicles, seamless recording of dangerous driving by other vehicles, and privacy protection can also be achieved by installing a wide-angle camera, such as a 360° camera, on the rear. For example, if a vehicle behind approaches from the rear, changes lanes, and cuts to the side of the vehicle, it is advisable to use a 360° camera to record the changing situation without interruption. In addition, while 360° cameras typically have a large viewing angle and small images, it is advisable to configure the camera to have a viewing angle that is as wide as possible, so that it matches the interior of the vehicle (enables viewing outside the driver's seat and passenger seat).

[0030] For example, with conventional front-side interior cameras installed near the front drive recorder, there is a problem in that the front mirror occupies an unnecessarily large area of ​​the screen, which sometimes makes it difficult to use interior cameras with wide-angle or ultra-wide-angle shooting ranges. In the above embodiment, the rear camera 102 for capturing video data inside the vehicle cabin and the photographing device 101R for capturing video data outside the vehicle can be designed separately and more flexibly depending on the purpose of the photographing, etc.

[0031] The rear camera 102 for photographing inside the vehicle may also be configured to capture images of the left and right front and rear of the vehicle through the windows. The photographing device 101R may be configured to perform processing by combining these images with images from other cameras. For example, when a vehicle behind approaches or overtakes dangerously, it is possible to perform a risk assessment from more diverse angles than before based on images from the perspective of a rear seat passenger and combining them with images from the photographing device 101R that photographs the rear from the rear side. In particular, unlike cameras for monitoring the exterior of the vehicle that are attached to the left and right sides of the vehicle outside, the rear camera 102 for photographing the interior of the vehicle can photograph both the state of tailgating from the side and the state of the interior of the vehicle at that time.

[0032] The user terminal 103 in FIG. 1 may be, for example, a portable terminal (mobile terminal) such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or may be a wearable terminal such as glasses, a wristwatch, or clothing. It may also be a stationary or portable computer, or a server located on the cloud or a network. It may also function as a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Alternatively, it may be a combination of multiple of these terminals. For example, a combination of one smartphone and one wearable terminal can logically function as one terminal. It may also be other information processing terminals.

[0033] The user terminal 103 includes a processor that executes an operating system, applications, programs, etc., a main storage device such as RAM (Random Access Memory), an auxiliary storage device such as an IC card, hard disk drive, SSD (Solid State Drive), flash memory, etc., a communication control unit such as a network card, wireless communication module, or mobile communication module, input devices such as a touch panel, keyboard, mouse, voice input, and input based on motion detection captured by a camera unit, and output devices such as a monitor, display, and speaker. The output device may also be a device or terminal that transmits information to be output to an external monitor, display, speaker, printer, device, etc.

[0034] 1, the rear camera 102 for photographing inside the vehicle is electrically connected to the photographing device 101R at the rear of the vehicle cabin, and transmits video data photographed from the rear of the vehicle toward the front of the vehicle to the photographing device 101R. The photographing device 101R and the rear camera 102 for photographing inside the vehicle may be further connected to the photographing device 101F at the front and the front cabin camera.

[0035] In such a configuration, the rear camera 102 for photographing the interior of the vehicle may also be configured to have a different aspect ratio from the front and rear photographing devices 101F, 101R. For example, the front and rear image capturing devices 101F, 101R may be arranged horizontally (horizontal position in camera terminology) to capture images, and the rear camera 102 for capturing images inside the vehicle may be arranged vertically (vertical position in camera terminology) to capture images. In other words, the rear camera 102 for capturing images inside the vehicle may be configured to capture images so that the long side of the image is the vertical direction of the image (i.e., the vertical direction) and the short side is perpendicular to the vertical direction. For example, the rear camera 102 for capturing images inside the vehicle may be configured to capture images of the vehicle interior from the center of the rear left and right directions.

[0036] On the other hand, the rear exterior, front exterior, and front interior of the vehicle should preferably be configured in a horizontal arrangement, i.e., the cameras capturing the rear exterior, front exterior, and front interior of the vehicle should capture images such that the longitudinal direction of the images is the horizontal direction of the images. The rear camera 102 for capturing images inside the vehicle interior is preferably configured to capture images in a vertically positioned manner using an image sensor with a 4:3 aspect ratio for a standard vehicle, or a 16:9 aspect ratio for a vehicle with a long interior such as a van. The front interior camera, which takes pictures of the interior from the front, can be positioned vertically. However, a camera that takes pictures of the interior from the rear is preferable because it is easier to install in the center. By installing it in the center, it is possible to configure a seven-seater vehicle so that it can take pictures of a wide range all the way to the front through the center aisle.

[0037] In a configuration where both front and rear interior cameras are installed, the front interior camera may be configured to record images that are less clear than the rear interior camera, thereby protecting privacy and making it easier to understand the state of the interior of the vehicle by viewing the recording. This is an example: The rear camera for taking pictures inside the car has a higher resolution (more pixels) than the front camera for taking pictures inside the car. The front interior camera is a thermal imaging camera, and the rear interior camera is a visible light camera. The front interior camera is a black and white camera (infrared camera), and the rear interior camera is a visible light camera. It is advisable to adopt measures such as the above to achieve this. The in-car camera may also be configured to record (also called audio recording) sounds inside the car.

[0038] When using an interior camera mounted at the front of a vehicle to capture the interior of the vehicle from front to rear, part of the field of view is blinded by the rearview mirror, making it impossible or very difficult to check the condition of luggage piled up behind the driver's seat or passenger seat. The rear camera 102 for photographing the interior of the vehicle photographs the interior of the vehicle from the rear to the front, so it is desirable to configure it to photograph areas of the interior of the vehicle that are blind spots for conventional front interior cameras such as those behind the driver's seat and passenger seat, or for the rearview mirror. It is also preferable to configure the interior rear camera 102 so that its shooting direction can be changed to the rear of the vehicle. Such an embodiment will be described later with reference to FIG.

[0039] FIG. 2 is an example of a connection diagram of the image capturing devices 101F and 101R, the rear camera 102 for capturing images inside the vehicle, the user terminal 103, and the external server 104. In FIG. The photographing devices 101F and 101R, the user terminal 103, and the external server 104 are each connected via a network 105. The network 105 may be wired or wireless, and each user terminal 103 may be configured to send and receive information via the network 105. The network 105 is not limited to the Internet, and may also be a combination of networks with different protocols.

[0040] The image capturing devices 101F and 101R are devices that are mounted on, for example, a vehicle and record (also called video recording) images while driving. However, although the vehicle to be installed is a four-wheeled automobile in this embodiment, it is not limited to a four-wheeled automobile, and any vehicle capable of installing the image capturing devices 101F and 101R may be used. For example, large transport vehicles with four or more wheels, such as buses and trucks, two-wheeled vehicles such as motorcycles and bicycles, and other vehicles may be suitable for installation. Furthermore, for example, transportation vehicles such as trains, monorails, and linear motor cars may also be suitable for installation. The photographing devices 101F and 101R may be configured to transmit the acquired various information and information relating to the results of event determination (described later) to the user terminal 103 via the network 105.

[0041] The user terminal 103 is a terminal used by, for example, a user who drives a car. The user terminal 103 is preferably configured to be connected to the image capturing devices 101F, 101R via a mobile data communication network or an in-vehicle network such as Wi-Fi (registered trademark) of the car, and to operate the image capturing devices 101F, 101R. Furthermore, the user terminal 103 may be configured to be connected to an external server 104 via these networks 105 and transmit various types of information to the external server 104 .

[0042] FIG. 3 shows an example of the hardware configuration of the image capturing device 101R connected to the rear camera 102 for capturing images inside the vehicle. The photographing device controller 304 (DVR controller 304) is connected to and controls a camera 302 that mainly captures rearward video outside the vehicle, including the area immediately behind the vehicle, an acceleration sensor 305, an SD card 306, a GNSS sensor 307, an input / output unit 315, and other sensors (not shown). Based on information received from these elements, the photographing device controller 304 generates status information indicating the state of the photographing device controller 304. The photographing device controller 304 also transmits the generated status information to the microcomputer 301. The photographing device controller 304 is connected via the input / output unit 315 to a rear camera 102 for photographing the interior of the vehicle, which is different from the camera 302 and photographs the interior of the vehicle from the rear toward the front.

[0043] The microcomputer 301 performs the following various processes, for example. Controls the communication processing unit 314. The communication processing unit 314 notifies the photographing device controller 304 of the acquired location information. The status information acquired from the photographing device controller 304 and the location information acquired from the communication processing unit 314 are converted into, for example, 16-byte data. - Performs sleep control when powered by secondary battery. - Execute test mode during manufacturing and testing. The microcomputer 301 is programmable, and stores programs for executing the various processes described above. The various processes described above are realized by executing these programs with a processing unit of the microcomputer 301. In this embodiment, the processing unit of the microcomputer 301 or the microcomputer 301 itself may also be referred to as a control unit.

[0044] The photographing device controller 304, the microcomputer 301, and the communication processing unit 314 can each be implemented as an integrated circuit (chip) such as an SoC (System on a Chip), or a plurality of these may be implemented together on a single chip. In particular, in this embodiment, the photographing device controller 304 and the microcomputer 301 operate in cooperation with each other, so they may be implemented together on a single chip.

[0045] Also, the microcomputer 301 may be configured to execute some of the functions of the photographing device controller 304, or the microcomputer 301 may be configured to execute some of the functions of the photographing device controller 304. In this embodiment, the photographing device controller 304 and the microcomputer 301 are collectively referred to as the control unit 303. The control unit 303 may be a single chip, or may be configured to include multiple chips in which the functions of the photographing device controller 304 and the microcomputer 301 are separated.

[0046] It is advisable to equip the image sensor of the camera 302 with an AI processing function that performs preprocessing, so that the following output can be performed in addition to or instead of generating a normal captured image. The AI ​​processing function has the function of performing image recognition based on the video captured by these cameras. -Export objects from images as metadata. - Outputs images in ISP (Image Signal Processor) output format, such as YUV or RGB. - Output an image cropped from a specific area. The AI ​​processing function may be configured to perform similar processing on the video data transmitted from the rear camera 102 for capturing images inside the vehicle.

[0047] By performing preprocessing on the image sensor side in this way, it is possible to reduce the amount of data and enable real-time tracking of objects using high-speed AI processing. For example, if the photographing device 101R is equipped with an AI processing function that performs preprocessing on the image sensor of the camera 302, the photographing device 101R may be configured to independently perform video processing and image processing, and to create image information with ideal image quality. Similar processing can also be performed on video data transmitted from the rear camera 102 for recording inside the vehicle interior, and the imaging device 101R may be configured to be updated to add processing according to the model and type of the connected rear camera 102 for recording inside the vehicle interior.

[0048] The acceleration sensor 305 detects acceleration and generates acceleration information. For example, when a sudden change in acceleration occurs due to an impact, sudden steering, sudden stop, etc., the control unit 303 detects the occurrence of an event. The control unit 303 may be configured to analyze the change in acceleration information and estimate the circumstances of an accident, etc., that has occurred to a vehicle, etc. The SD card 306 is a storage device that records various information such as image information and recording information corresponding to that image information. Note that the SD card is not limited to an SD card and any configuration that can store data will suffice, but a flash memory with high vibration resistance is preferable for in-vehicle use.

[0049] The communication processing unit 314 transmits information about the photographing device 101R to the user terminal 103 via the wireless communication antenna 308 and the network. The GNSS sensor 307 has a location information acquisition unit implemented as an integrated circuit or the like, and acquires location information based on a signal received from a GNSS antenna compatible with a GNSS (Global Navigation Satellite System) such as a GPS (Global Positioning System). The location information acquisition unit may be configured to acquire time information, moving speed information, and moving direction information.

[0050] The photographing device 101R has a power supply 311 connected to a vehicle power supply 310. The power source 311 supplies power received (or sometimes referred to as power reception) from the vehicle power source 310 to the photographing device controller 304, and also charges the secondary battery 312. Note that the input / output unit 315, which will be described later, may also be configured to supply power. In this case, the photographing device 101R receives power from the vehicle power source 310 via the input / output unit 315.

[0051] The secondary battery 312 is a rechargeable battery such as a lithium-ion battery, a nickel-cadmium battery, a nickel-metal hydride battery, or a lead-acid battery. However, a non-rechargeable primary battery may also be used. When the accessory power supply (ACC power supply) is turned off, or when the power supply from the vehicle power supply 310 to the power supply 311 of the photographing device 101R is interrupted due to a fault such as a broken wire or a disconnected wire caused by an accident, or when the power supply 311 is unable to supply power due to a malfunction or the like, the secondary battery transmits (or may also be called power transmission or power supply) power to the microcomputer 301, the communication processing unit 314, etc., and maintains at least some of the functions of the photographing device 101R.

[0052] The control unit 303 acquires image information from the camera 302 of the photographing device 101R and stores it in a storage device such as an SD card 306. Note that in this specification, "storage" is sometimes referred to as "recording."

[0053] The input / output unit 315 inputs and / or outputs video data captured by the camera 402 or the rear camera 102 for capturing images inside the vehicle interior, and / or video data stored in a storage unit such as the SD card 306. The input and / or output of video data is basically performed in a digital format, but is not limited to this. The input / output unit 315 is connected to the photographing device controller 304, but may be connected to the microcomputer 301 instead. The input / output unit 315 is directly connected to the network 105, but may be configured to be connected to the network 105 via the rear camera 102 for capturing images inside the vehicle. As described above, the input / output unit 315 may be configured to be able to input and output power from the vehicle power supply 310 .

[0054] The rear camera 102 for photographing inside the vehicle is an additional camera connected to the photographing device 101R via the input / output unit 315. The vehicle interior rear camera 102 transmits the video data captured by the vehicle interior rear camera 102 to the imaging device controller 304 of the imaging device 101R via the input / output unit 315.

[0055] In addition, the rear camera 102 for photographing inside the vehicle interior may be configured to output the video data input from the photographing device 101R to the network 105. In addition, the rear camera 102 for photographing inside the vehicle interior may be configured to output the power input from the vehicle power supply 310. By configuring the rear camera 102 for taking photographs inside the vehicle interior to be able to input and output video data and power, the video data and power can be passed through to the outside simply by connecting the rear camera 102 for taking photographs inside the vehicle interior to the photographing device 101R having the input / output unit 315, which makes it possible to add a new camera without having to change the configuration of the existing photographing device 101R, which is convenient.

[0056] FIG. 4 shows an example of a process flow 400 for processing images captured by the rear camera 102 for capturing images inside the vehicle. The control unit 303 of the photographing device 101R executes the following process in accordance with the process flow 400. In the following explanation, a description of specific algorithms and settings of the image processing and video processing by the control unit 303 of the photographing device 101R will be omitted.

[0057] The control unit 303 acquires an image (for example, video data such as a still image or a moving image) from the rear camera 102 for capturing images inside the vehicle (S410) and analyzes the image (S420). Based on the analysis results, the control unit 303 makes a determination about each of the set situations, which will be described later (S430). The control unit 303 performs output, which will be described later, based on the determination result (S430).

[0058] In the following explanation, an example will be described in which the control unit 303 is the main processing unit in each of steps S410 to S440, but each step may also be performed by, for example, an individual module within the control unit 303 or in a data connection state with the control unit 303. For example, the image acquisition module may execute the image acquisition processing step S410, the image analysis module may execute the image analysis step S420, the judgment module may execute the judgment step S430, and the output module may execute the output processing step S440, and the control unit 303 may work in conjunction with each module. Furthermore, each module may be implemented in hardware or software.

[0059] A specific example of the processing that can be executed according to the processing flow 400 will be described below. First example Possible cases It takes a picture of the inside of the car from behind, detects people, and applies a filter. Example input Images in which people can be identified Example output Images of people with video filters applied, etc.

[0060] The control unit 303 has a function of recognizing the face position (body position) of the image captured from the rear of the vehicle interior and applying an image filter. The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle in S410, analyzes the captured image in S420, and determines in S430 whether a person is captured, whether the face of the person captured can be identified, or whether only the back of the person is captured. In S440, when the control unit 303 identifies the face of a person in the image, it creates and outputs an image in which a filter with, for example, an animal's face motif is applied to the area of ​​the face.

[0061] In addition to the above process of applying a filter to the person's face in S440, if the control unit 303 recognizes that the video taken from the rear of the vehicle interior has been taken from behind the person's face (body position), it is preferable that the control unit 303 be configured to create and output video that has been processed to overlay the back of an animal's face (body) such as a cat as a filter.

[0062] In other words, the image is analyzed, and a filter of the animal's front face is applied to the image taken from the front, and a filter of the animal's back of the head is applied to the image taken from behind. It is advisable for the control unit 303 to be configured to overlay a 3D model over the entire face of a person in the video (like a mask) and change the display position depending on the direction of the face, but since such processing requires a high computational load, the computational load for processing can be significantly reduced by configuring the camera to overlay a front image on the front side for a camera from the front and a back image on the back side for a camera from the rear.

[0063] In this case, by filming the interior of the car from behind, the chances of people's faces being captured are reduced, and by applying a filter to the face area in the photograph, it is possible to film and record the actions of people inside the car while protecting their privacy more than ever before. Furthermore, by applying a filter to people's faces, etc., it is possible to create video that is even more considerate of the privacy of people inside the vehicle. This can reduce the resistance felt by passengers of buses, taxis, etc., about having their own video recorded. Such resistance can be further reduced by, for example, displaying the filtered video on an in-vehicle monitor so that passengers can check it themselves. With this configuration, it is also possible to set it so that the video without the filter can be viewed if a request from the police or other authorities is granted in the event of an incident or accident.

[0064] Second Example Possible cases It detects the conditions around the vehicle and issues notifications and warnings based on those conditions. Example input Images of the area around the vehicle Example output Notifications and warnings about the situation around the vehicle

[0065] In this case, the image capturing device 101R may be configured to capture and record the surroundings of the vehicle by capturing an image of the outside from the rear of the vehicle through a side window (on the side of the vehicle). The control unit 303 may be configured to analyze the video data captured from the rear of the vehicle interior by the rear camera 102, and perform, for example, side vehicle detection, human detection, and obstacle detection. In this configuration, the rear camera 102 may be a wide-angle camera, and may also be a 180° camera, a hemispherical camera, or a celestial sphere camera. For example, by specifying the outline of the window glass in advance, the control unit 303 can distinguish between the state inside the vehicle and the state around the vehicle (the state outside the vehicle), and by analyzing only the video data of each part depending on the situation, it is possible to identify the situation with a lower computational load.

[0066] The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured video data in S420, and determines which parts of the captured video data in S430 are images inside the vehicle and which parts are images outside the vehicle, whether other vehicles are captured in the images outside the vehicle, whether dangerous driving by other vehicles is observed, and whether there are any other situations that would result in pre-set notifications or warnings.

[0067] Examples of situations that may trigger pre-defined notifications or warnings include: When you turn on the turn signal, there are cars, people, or obstacles on the side, - When someone approaches your vehicle unnaturally while it is parked with no one inside, - Situations where such a person is looking into the car, - When starting the engine, slowing down, or driving, there are people, especially children, around the vehicle, approaching or playing. - Situations where the vehicle is being hit from behind or the side, etc.

[0068] In S440, when the determined situation occurs, the control unit 303 outputs a notification or warning by video or audio. Also, the control unit 303 may output a notification or warning about the state of the surroundings of the vehicle customized according to each determined situation. With an in-vehicle camera installed at the front, the front seats (driver's seat, passenger seat) get in the way, making it difficult to capture side views through the rear seat window glass, making it difficult to make highly accurate judgments. On the other hand, with a configuration in which the front view is captured from the in-vehicle camera 102 installed at the rear of the vehicle, as in this embodiment, the sides of the vehicle and the front side can be captured well, making it possible for the control unit 303 to judge these situations with high accuracy.

[0069] For example, when the control unit 303 determines that another vehicle is tailgating the vehicle by analyzing the video data, the control unit 303 outputs a warning sound from a speaker inside the vehicle or a voice message such as "Your vehicle is being tailgated from behind." A configuration that allows the driver to immediately recognize the occurrence of tailgating from behind the vehicle is extremely advantageous from the perspective of safe driving, as it allows the driver to have more time and psychological leeway.

[0070] The control unit 303 may be configured to analyze the video data and, when it determines that a suspicious person is approaching the parked vehicle while the driver or passengers are away from the vehicle, send a notification or warning about the approaching or suspicious person to a user terminal such as a smartphone. In particular, the control unit 303 may be configured to determine a suspicious person peering into the vehicle interior in the blind spot of the front camera located on a straight line between the front cabin camera and the front seats (driver's seat and passenger seat). This processing may be performed in combination with, for example, the seventh example described below.

[0071] In a configuration where a wide-angle rear camera 102 for capturing images inside the vehicle is installed at the rear of the vehicle, not only can the interior of the vehicle be recorded, but also the surrounding area of ​​the vehicle can be clearly recorded through the window glass on both sides. With this configuration, the view outside the driver's seat, passenger seat, and rear seat windows can be recorded without the front seats getting in the way.

[0072] Third Example Possible cases The system detects the status of items installed on the vehicle ceiling and issues notifications and warnings based on that status. Example input Images of installations on the vehicle ceiling Example output Notifications and warnings about the condition of items installed on the vehicle ceiling

[0073] The control unit 303 may be configured to analyze the captured video data to identify the status of objects installed on the vehicle ceiling, such as room lamps, nets, projectors, flip-down monitors, etc., installed on the vehicle ceiling. The control unit 303 acquires an image taken by the rear camera 102 for photographing the interior of the vehicle in S410, analyzes the taken image in S420, and determines in S430 whether an object installed on the vehicle ceiling is in the image, whether the object installed on the vehicle ceiling that is in the image is in an appropriate position and state, whether the object installed on the vehicle ceiling is in contact with passengers, particularly passenger heads, or whether there is a risk of contact. In S440, if the control unit 303 determines that an object installed on the vehicle ceiling is not in an appropriate position or state, or that the object installed on the vehicle ceiling is in contact with or at risk of contact with a passenger, particularly a passenger's head, it outputs a warning about the situation.

[0074] For example, the control unit 303 may be configured to analyze the video data to determine whether the room lamp is on and whether there is anyone in the vehicle. If the control unit 303 determines that the room lamp remains on even after the door has been closed and that there is no one in the vehicle, it issues a warning from a speaker in the vehicle or a user terminal carried by the driver. This effectively prevents the battery from running out.

[0075] In another example, the control unit 303 may be configured to analyze the video data to determine the condition of the net attached to the ceiling, for example, whether the net is sagging due to excessive luggage, whether the left and right fasteners have come loose, etc. When the control unit 303 determines that the net is sagging due to excessive luggage or that the left and right fasteners have come loose, it outputs a warning sound or a warning voice such as "The net condition is abnormal" from a speaker or the like inside the vehicle. This allows the user to adjust the net to an optimal condition in advance before the net is damaged, before luggage falls from the net, or before the net comes loose unexpectedly.

[0076] In yet another example, the control unit 303 may be configured to analyze the video data to determine the orientation and state of a projector mounted on the ceiling, the position and state of the projector's image, the relative positions of the projector and passengers in the rear seats, etc. For example, the control unit 303 - The projector is adjusted to the desired position or direction. - Check whether the projector is running when there are no passengers on board, or conversely, whether the projector is running when there are passengers on board, When the image is being projected, is the position of the image relative to the screen appropriate? - Check whether passengers are hitting their heads on the projector. It is preferable to have a configuration that determines the above. When the control unit 303 determines that one of these situations has occurred, the control unit 303 may output a warning sound or a warning voice such as "The projector is in an abnormal state" or a warning image from a speaker, monitor, or the like in the vehicle.

[0077] The projector in this example may be configured to project advertisements or the like onto a screen placed on the side window glass of a vehicle such as a taxi. Based on these warnings, the driver can adjust the projector's position, status, settings, etc. as appropriate. For example, when there are no passengers, the projector can be used to reliably project advertising images toward the outside of the vehicle, and when there are passengers inside, the projector can be stopped to ensure passenger comfort. Also, when there are no passengers, the projector can be used to reliably project advertising images toward the outside of the vehicle, and when there are passengers inside, the projector can be used to project images aimed at those passengers, thereby increasing the effectiveness of advertising.

[0078] In yet another example, the control unit 303 may be configured to analyze the video data to determine the open / closed state of a flip-down monitor mounted on the ceiling, its activation state (whether or not video is being displayed), and the like. For example, the control unit 303 - The flip-down monitor is in the stopped or closed position when passengers are on board. Conversely, if the flip-down monitor is playing video when there are no passengers on board or if the flip-down monitor is in the open position, It is preferable to have a configuration that determines the above. When the control unit 303 determines that such a situation has occurred, the control unit 303 outputs a warning sound or a warning image such as "Please check the status of the flip-down monitor" from a speaker, monitor, or the like in the vehicle.

[0079] Based on such warnings, the driver can appropriately adjust the state and settings of the flip-down monitor. This allows, for example, a taxi to stop video playback on the flip-down monitor when there are no passengers, thereby reducing power consumption, while still reliably playing advertising videos and the like when there are passengers in the vehicle. In addition, the rear camera 102 for photographing the interior of the vehicle, which is installed on the rear side of the vehicle, can clearly recognize the images displayed on the flip-down monitor, so the control unit 303 can record the passenger's level of interest and concern for each advertising image by analyzing data such as the length of time that the passenger spent looking toward the flip-down monitor for each image. The control unit 303 may also be configured to use the results of analyzing such records to play advertising videos that are likely to be of greater interest to passengers, or to transmit the analysis results via the network 105 to an external server 104 of a company that provides the advertising videos.

[0080] Fourth Example Possible cases The state of luggage on the luggage rack, seats, etc. is detected, and notifications and warnings are given based on that state. Example input Images showing the condition of the luggage Example output Notifications and warnings about the stability of your luggage

[0081] In this case, the control unit 303 may be configured to detect the state of luggage placed behind the driver's seat, for example, by analyzing the video data. The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured image in S420, and determines in S430 the state of luggage placed or piled up on the luggage rack or seats, for example, the positional relationship between the luggage and the luggage rack or seats, the estimated position of the center of gravity of the luggage, the positional relationship between the centers of gravity of multiple luggage, the speed and acceleration of the vehicle and changes in the position of the luggage related to these changes, etc. The center of gravity of the luggage may be simply estimated, for example, as the center point of a two-dimensional shape recognized as luggage on the screen, but may also be estimated by other methods.

[0082] In S440, when the control unit 303 determines that there is a risk of luggage falling off the luggage rack or a seat, that there is a risk of stacked luggage collapsing, or that the luggage is slipping more than a set value on the luggage rack or a seat, the control unit 303 outputs a warning about that situation. The control unit 303 may output such a warning while the vehicle is in operation, or may detect the state of the luggage when a person gets into the vehicle and output a notification according to that state.

[0083] Images of the interior of the vehicle taken from behind can capture particularly good images of the condition of luggage placed near the back seat. Therefore, if the vehicle is involved in an accident involving another vehicle, such images of the interior of the vehicle can serve as strong evidence that the other vehicle caused the damage to the luggage, allowing for smooth and accurate calculation of damages in the traffic accident. In particular, if the control unit 303 is configured to determine whether the luggage was placed securely and stably inside the vehicle, the reliability of the evidence is improved, which is even more advantageous.

[0084] Furthermore, in combination with the second case, for example, the control unit 303 may be configured to analyze the video data to determine the correlation between the collapse of luggage and tailgating or contact by another vehicle filmed from behind through the window inside the vehicle. For example, if a driver swerves the steering wheel to avoid a collision with another vehicle when another vehicle suddenly tailgates them from behind, but the luggage loaded in the vehicle collapses and causes damage, the video from the rear camera used to record footage of the other vehicle rapidly approaching from the side, the driver's driving in response to it, the driving trajectory of the other vehicle at the time, and the timing at which the luggage collapsed will be recorded in a single piece of video data, which is extremely useful for analyzing accidents and events that have occurred.

[0085] There have been many cases in which drivers have been concerned about the collapse of purchased items and have caused accidents. For example, a case in which a person who went to buy lunch for work became concerned about the lunch and then had an accident the moment he looked at it is a very familiar scenario for many drivers. Therefore, the above configuration can detect when luggage is about to collapse and notify the user in advance, thereby providing a safer driving environment and allowing the driver to concentrate on driving.

[0086] The control unit 303 may also be configured to detect the state of the luggage and the state of the person by analyzing the video data. The control unit 303 may, for example, detect the correlation between the state of the luggage and the state of the person and output a warning, notification, or the like based on the detection result. The control unit 303 may, for example, detect that the luggage has collapsed in the direction of the person. The control unit 303 may, for example, detect that the distance between the collapsed luggage and the person is less than a predetermined distance. The control unit 303 may even better detect both of these, for example, that the distance between the collapsed luggage and the person is less than a predetermined distance and that the luggage has collapsed in the direction of the person. The control unit 303 may change the content of the output of the warning, notification, or the like based on the relative positions of the luggage and the person. For example, the higher the degree of impact on the person, such as the degree of danger posed by the luggage, the higher the level of the output of the warning, notification, or the like. The output of the warning, notification, or the like may be targeted at a person who may be hit by the collapsed luggage, so that the person can be notified of the danger and prompted to take appropriate action.

[0087] Fifth Example Possible cases The state of the vehicle's display is detected and notifications, commands, etc. are given based on that state. Example input Image of the vehicle's display Example output Notification of the display status, commands according to the display status, etc.

[0088] In this case, the display unit is a display unit disposed in the vehicle, particularly a monitor mounted on the back of a seat or a monitor mounted on the ceiling, and the control unit 303 detects the status of these units by analyzing the video data. It is also preferable to configure the system to capture good images of the display content and operation status of a monitor (hereinafter referred to as the "center monitor") mounted near the center of the front passenger seat between the driver's seat and the passenger seat. The display unit is not limited to a display unit fixedly mounted in the vehicle, but may also be a user terminal 103 that displays information related to the vehicle.

[0089] The center monitor screen tends to be in a blind spot from the position of the interior camera installed at the front of the vehicle. In contrast, the interior rear camera 102 installed at the rear of the vehicle is very suitable for capturing the center monitor screen from behind the vehicle.

[0090] The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured image in S420, and determines the state of the vehicle's display units in S430, particularly the state of the monitors installed on the backs of the seats and the monitors installed on the ceiling, the display content of the center monitor and whether or not it has been operated, whether or not there are passengers in the rear seats of the vehicle, etc.

[0091] In S440, if the control unit 303 determines that, for example, there is no passenger in the rear seat of the vehicle and a monitor installed on the back of the seat is playing video, the control unit 303 outputs a warning sound, a report voice such as "The rear monitor is on," or a warning image from a speaker or monitor inside the vehicle. In addition, in combination with a ninth example described below, the control unit 303 can also output a command to temporarily suspend or hide the display on the monitor when it is determined that a child in the back seat is asleep, for example.

[0092] This embodiment provides an even more advantageous effect when the center monitor is a touch panel display that accepts inputs related to vehicle operation and control. The control unit 303 may be configured to trigger recording of in-vehicle video when it determines that a part of a person's body is overlapping the center monitor. For example, if video is recorded for about 10 seconds before and after an operation of the touch panel display (monitor operation), even if an accident occurs, it can be proven that the monitor operation was not the cause of the accident if it can be proven that the timing of the driver's monitor operation was different from the timing of the accident.

[0093] Alternatively, the user can use the absence of remaining video as a means of proving that at least no action that could be determined as monitor operation was performed while driving at the time of the accident. In this case, it is preferable that the control unit 303 is configured to record a deletion history of recorded video, etc.

[0094] The control unit 303 is preferably configured to determine and record the specific details of the operation. In particular, in a configuration in which the center monitor also serves as an operation console for an autonomous vehicle, it is more preferable that the recorded video be configured to clearly capture and record the state in which the screen indicates whether the vehicle is being driven autonomously or manually at the time the video was recorded. Therefore, if a driver is involved in an accident while driving properly, the photographing device 101R of this information processing system: The accident did not occur while operating the device. The accident was not caused by incorrect operation of the device. Whether the accident occurred during automated or manual driving; It is advisable to provide objective evidence of the above. Furthermore, a configuration in which the above-mentioned conditions are comprehensively judged from images captured by cameras inside the vehicle at the front and rear is even more advantageous, as it can improve the accuracy of the judgment and increase the amount of objective evidence.

[0095] Furthermore, when the control unit 303 determines that an accident has occurred, it is preferable to configure the control unit 303 to transmit video data before and after the accident to an external server. This eliminates any suspicion of video data tampering, further improving the evidentiary value of the video data. It is best if the external server is operated by a neutral organization that is unrelated to the accident, such as the company that manufactured and sold the imaging equipment.

[0096] Sixth Example Possible cases The system detects when the head is too often visible in the captured video and issues notifications or warnings based on that state. Example input Images showing people, especially the head Example output Notifications and warnings about heads appearing too often in the footage

[0097] For example, if the majority of the screen is the head, the control unit 303 issues a warning to that effect, or issues a warning to change the installation position. The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured image in S420, and determines whether a head is captured on the screen of the captured image in S430, what proportion of the screen the captured head occupies, etc., or determines whether the installation state is such that such an event can be anticipated in advance, etc.

[0098] The proportion of the screen occupied by the head can be set to, for example, 50% or more of the entire screen. Also, it may be configured to issue a notification or warning if the head occupies 50% or more of the entire screen for a total of 5 minutes or more within a time interval set to, for example, 10 minutes. Furthermore, since the proportion of the screen occupied by the head tends to increase when getting in or out of a vehicle, it is desirable to configure the system to stop this determination while the vehicle door is open.

[0099] In S440, if the control unit 303 determines that a head is captured and that the proportion of the head captured is equal to or greater than a set value, it outputs a notification to warn that the proportion of the head in the captured image is large or that the rear camera 102 for capturing images inside the vehicle is installed in a state in which such an event is anticipated. This allows the control unit 303 to provide the user with an opportunity to correct the state of the image captured by the vehicle interior rear camera 102 to a desired state at an appropriate timing.

[0100] In this case, depending on the type of car and the installation location, when someone sits in the back seat, it is possible to avoid the back of the person's head being constantly displayed on the screen (the entire screen being completely black).

[0101] Seventh Example Possible cases The parking status is detected, and based on that status, commands to switch the shooting mode are issued. Example input Images of people Example output Command to switch to parking monitoring mode, etc.

[0102] The control unit 303 has a function of detecting whether a person is inside the vehicle from an image of the person, and automatically switching to parking monitoring mode when it detects that the person has left the vehicle. The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured image in S420, and determines whether there is a person in the captured image, whether the vehicle is parked, etc. in S430. In S440, when it is determined that the vehicle is in a parked state, the control unit 303 outputs a command to switch the imaging mode of the rear camera 102 for imaging inside the vehicle to a parking monitoring mode.

[0103] In the parking monitoring mode, the control unit 303 prioritizes processing to identify suspicious approaching individuals, for example, from within the area of ​​the vehicle's windshield, rear window, or side window, and stops processing to filter out people such as those in the first example above. This configuration can be combined with the eighth example of abandoned vehicle detection described below, and abandoned vehicle detection can be activated in parking monitoring mode. Also, for example, it can be combined with the second example above to activate detection of the state around the vehicle in parking monitoring mode.

[0104] As a result, when the vehicle is parked, the control unit can allocate computing power preferentially to processing to identify suspicious approaching individuals, for example, from within the area of ​​the vehicle's windshield, rear window, or side window, allowing the user to use a more efficient information processing system.

[0105] Eighth Example Possible cases It detects situations such as people being left behind in the vehicle and issues notifications and warnings based on that situation. Example input Images of people Example output Notifications and warnings about abandonment, etc.

[0106] The control unit 303 may be configured to determine whether or not a person, such as a child or a disabled person, has been left behind in the vehicle from the image of the person, and to issue a warning if such a person has been left behind. This processing may be preferably performed in conjunction with the processing in the seventh example above. The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured image in S420, and determines whether there is a person in the captured image in S430, whether the person in the image is a child or a person with a disability, whether there is a person other than a child or a person with a disability in the vehicle, whether the vehicle is parked, or whether it has been abandoned, etc.

[0107] In S440, if the control unit 303 determines that the vehicle is abandoned, it outputs an alarm or an alarm image from a device carried by the driver or from a speaker or monitor of the vehicle, notifying the driver that the vehicle is abandoned. The control unit 303 may change the output destination or output method of the alarm based on the elapsed time since the vehicle was abandoned and the state of the person who was abandoned. For example, the control unit 303 may be configured to initially output the alarm only to the device carried by the driver, and then, if the abandoned state is not resolved, to output the alarm audio or image to both the device carried by the driver and the speaker or monitor of the vehicle. This means that in situations where the driver is able to deal with the situation, the situation can be resolved with minimal warning, and in situations where the driver is unable to deal with the situation, the driver can notify those around them of the situation.

[0108] In this configuration, the rear camera 102 for photographing the interior of the vehicle may further be equipped with a sensor other than the camera that can detect people, and may be configured to detect whether children, disabled people, etc. have been left behind based on the image from the rear camera for photographing the interior of the vehicle and the sensor that can detect people.

[0109] Examples of sensors capable of detecting people include pyroelectric infrared sensors and thermosensors. A configuration using a sensor capable of detecting people can identify children, people with disabilities, and the like with a lower load than AI processing of video data. Since constantly processing video data requires a large computational load, it is advantageous from the perspective of reducing the computational load and achieving highly accurate human detection to detect the presence of people using a separate sensor capable of detecting people and to use the result of the detection of people inside the vehicle by the sensor capable of detecting people as a trigger for starting human detection processing based on images from the rear camera 102 for capturing images inside the vehicle interior. For example, it is advisable to limit the detection range of a sensor capable of detecting people to the rear area of ​​the vehicle or the vicinity of the rear seats, and to configure the system so that the driver and passengers can be distinguished with high accuracy.

[0110] With the above configuration, it is possible to detect people, distinguish between children, disabled persons, etc. and the driver, and determine whether children, disabled persons, etc. have been left behind with extremely high accuracy using images taken from the rear of the vehicle, making it possible to provide the vehicle with a highly accurate rear-end abandonment prevention function.

[0111] As already explained, the interior camera 102, which takes pictures of the interior of the vehicle from the rear, can also capture and identify babies in rearward-facing child seats behind the front seats, so in this embodiment, the control unit 303 can provide a highly accurate rear-side baby abandonment prevention function in such situations. Furthermore, if the human detection sensor is a thermal sensor, it can accurately detect the baby's body temperature and issue a warning with the utmost caution if necessary.

[0112] As a variation of the eighth example, detecting a person such as a child from an image captured from the rear side of the vehicle may be used to determine something other than whether the person has been abandoned. For example, when the control unit 303 of the image capturing device 101R recognizes a child's head protruding from a child seat based on an image captured by the interior rear camera 102, the control unit 303 may output a warning or notification. The protruding position may be, for example, a position above the child seat. Assume that the interior rear camera 102 is installed so that the child's head is not captured in the captured image when the child is sitting in the child seat. In this case, if the child's head is captured in the captured image, the child may be standing up, which is considered to be a dangerous situation for the child.

[0113] The warning or notification may be output to a passenger inside the vehicle 10, or may be output to an external terminal such as a terminal carried by the driver (for example, the user terminal 103). The warning may be output as an audible warning, or may be output using light or a display. The control unit 303 may determine whether to output a warning or notification or change the content of the warning or notification based on the direction or amount of movement of the head protruding from the child seat. According to this modification of the eighth example, an image captured from the rear side can be used to make passengers aware of the dangerous situation of a child standing up.

[0114] Ninth Example Possible cases The system detects whether people in the vehicle are dozing or sleeping, and based on that, issues notifications and warnings. Example input Images of people Example output Notifications and warnings about drowsiness and sleep

[0115] The control unit 303 may be configured to determine the state of people inside the vehicle, particularly whether they are drowsy or asleep, from the video captured from behind, and issue a warning or notification. The control unit 303 acquires an image captured by the rear camera 102 for capturing images inside the vehicle interior in S410, analyzes the captured image in S420, and determines whether there is a person in the captured image in S430, whether the person in the image is the driver or a passenger, whether the driver's head is captured, whether the state of the head shows a warning pattern for drowsiness, such as nodding, whether the face or head of a passenger in the rear seat is captured, and whether the state of the face or head shows a sleeping pattern, etc.

[0116] In the past, it was difficult to determine the sleep state of passengers in the rear seats, especially, using images captured by an in-vehicle camera installed at the front of the vehicle, because the front seats were in a blind spot. In this embodiment, the control unit 303 is configured to effectively perform these determinations by using images captured from the rear. In order to determine whether a passenger is dozing off or sleeping from the rear, the control unit 303 stores sleep patterns indicated by head movement, body movement, and the positional relationship between the body and the vehicle, such as continuous head contact with the window glass, in addition to sleep patterns indicated by images of the front of the face.

[0117] In S440, when the control unit 303 analyzes the video data and detects that the video data matches a stored sleep pattern and determines that the driver is dozing, it uses this as an event trigger and outputs an alarm from a speaker or the like of the vehicle to warn that the driver is dozing. With such a configuration, the control unit 303 can also determine with higher accuracy whether the driver is dozing in combination with a lane departure warning system (LDWS).

[0118] Furthermore, if the control unit 303 determines in S440 that the child in the back seat is asleep, it may be configured to output a notification sound such as "Your child is sleeping" from the vehicle's speaker or the like. This allows the control unit 303 to provide the driver with an opportunity to start driving carefully so as not to wake the sleeping child. In another case, if the control unit 303 determines that the child in the back seat is asleep, it may be configured to output a command to change the driving mode of the autonomous vehicle to a gentler driving mode with more gradual acceleration and deceleration.

[0119] Tenth Example Possible cases Take a photo when there are people in the vehicle Example input Information to determine whether people are in the vehicle, such as images that show people Example output Image information of the captured image

[0120] The control unit 303 has a function of capturing an image when a person is in the vehicle. In particular, the control unit 303 preferably has a function of capturing an image when a person is in the vehicle, regardless of whether the vehicle engine is on or off or whether the accessory power supply is on or off. Capturing an image regardless of whether the vehicle engine is on or off or whether the accessory power supply is on or off may be referred to as capturing an image in either case. For example, the power supply control for this purpose may be as follows: The image capturing device 101R is electrically connected to a battery unit (an example of a power supply device) (not shown) that has a charging function.

[0121] The photographing device 101R may operate by receiving power from the vehicle battery while the vehicle engine or accessory power is on. The photographing device 101R may operate by receiving power from the battery unit while the vehicle engine or accessory power is off. The battery unit is electrically connected between the photographing device 101R and the vehicle battery. The battery unit may charge based on the power supplied from the vehicle battery while receiving power from the vehicle battery.

[0122] If the battery unit continues to supply power to the photographing device 101R after the power supply from the vehicle is stopped, the battery unit may supply the stored power to the photographing device 101R. The photographing device 101R may have a function equivalent to the battery unit described above as an internal power source. If the vehicle is equipped with a continuous power source (also referred to as +B), the photographing device 101R may receive power from the continuous power source and operate regardless of whether the vehicle engine or the accessory power source is on or off. Alternatively, the photographing device 101R may have an internal power source such as a secondary battery 312 and use this internal power source to operate regardless of whether the vehicle engine or the accessory power source is on or off.

[0123] The control unit 303 may determine that a person is in the vehicle when it detects the presence of a person in the vehicle cabin. Information for detecting the presence of a person in the vehicle cabin may be, for example, an image in which a person in the vehicle cabin can be identified, for example, an image captured by the vehicle interior rear camera 102. The control unit 303 acquires image information of the image captured by the vehicle interior rear camera 102 in S410, analyzes the captured image in S420, and determines whether a person is captured in S430. The control unit 303 may detect the presence of a person in the vehicle cabin when it recognizes, for example, that a person's head, face, or part of a body is captured in the image.

[0124] In S440, when it is determined that a person is in the vehicle, the control unit 303 may output image information indicating an image captured by the interior-photography rear camera 102 to a predetermined output destination. For example, the control unit 303 may output the image information to a storage device such as the SD card 306 as an output destination and record the image information in the storage device. This recording of the image information may be considered as event recording triggered by the determination that a person is in the vehicle.

[0125] When image information is output to a predetermined output destination, audio information of audio picked up by a microphone may also be output to the output destination. For example, when recording image information (i.e., video recording), audio information may also be recorded at the same time. This also applies to other configuration examples unless otherwise specified. The microphones may be positioned or numbered to capture audio within the vehicle cabin. The image capturing device 101R may have a microphone, or a microphone may be located outside the image capturing device 101R. The control unit 303 may set an external device as the output destination. For example, the control unit 303 may use the communication processing unit 314 to transmit (i.e., transmit and output) image information to an external device. The external device may be, for example, a server device for recording image information, the user terminal 103, a computer device at a management center that comprehensively manages image information captured by multiple vehicles, or the like.

[0126] If the control unit 303 does not determine that there is a person in the vehicle, or if it determines that there is no person in the vehicle, in S440, it is preferable not to output image information showing the image captured by the rear camera 102 for capturing images inside the vehicle to a specified output destination.

[0127] This makes it easier to understand the situation when a person is in the vehicle from the captured image. Since troubles or other incidents often occur in the vehicle cabin when a person is in the vehicle, easily understanding the situation when a person is in the vehicle is useful for confirming whether or not such incidents have occurred and the circumstances under which they have occurred. The person in the vehicle may be a legitimate user of the vehicle, such as the driver or a passenger, or may be someone who has illegally entered the vehicle (e.g., a thief). In particular, when a person is in the vehicle, troubles or other incidents may occur in the vehicle cabin regardless of whether the vehicle's drive source or the vehicle's power source is on or off. Therefore, easily understanding the situation when a person is in the vehicle, regardless of whether the vehicle's engine or the vehicle's accessory power source is on or off, is useful for more reliably confirming whether or not such incidents have occurred and the circumstances under which they have occurred.

[0128] Furthermore, the following configuration may be adopted. The control unit 303 may output to a predetermined output destination only image information for the period during which it is determined that a person is in the vehicle, but it is more preferable to output image information for a period before, after, or both of these periods (hereinafter referred to as the "period before and after"). The period before and after may be a fixed time length (e.g., 5 minutes before and 5 minutes after), but the image capturing device 101R may be configured so that the user can set it. In this way, image information before and after the period during which a person is in the vehicle is also output, making it possible to grasp, for example, what it was like when a person got in or out of the vehicle from the captured image.

[0129] Information other than an image that can identify a person may be used as information for determining whether a person is in the vehicle. For example, a detection signal from the human detection sensor described above may be used. The control unit 303 may determine whether a person is in the vehicle based on this detection signal.

[0130] The control unit 303 may stop capturing images by the interior-use rear camera 102 during a period when it has not determined that a person is in the vehicle or during a period when it has determined that no person is in the vehicle. In this case, when the control unit 303 determines that a person is in the vehicle, it may start capturing images by the interior-use rear camera 102, acquire image information, and output the image information to a predetermined output destination. Furthermore, up to this point, the description has been given of a function that captures images when it has determined that a person is in the vehicle, regardless of whether the vehicle engine is on or off or whether the accessory power is on or off. However, the function may also be such that it captures images only while the vehicle engine is on or the accessory power is on.

[0131] Example 11 Possible cases Record when someone speaks Example input Information for determining whether a person has spoken, such as a person's voice Example output Image information of the captured image

[0132] The control unit 303 has a function of capturing an image when a person speaks. "A person speaks" may be interpreted as "a person speaking" or "a person's voice is detected," etc. The control unit 303 may capture an image when a person speaks regardless of whether the vehicle engine is on / off or whether the accessory power is on / off, or may capture an image only while the vehicle engine or the accessory power is on. If the image is captured regardless of whether the vehicle engine or the accessory power is on / off, the same power control as in the tenth example may be performed.

[0133] The information for determining whether a person has spoken may be, for example, audio information of a voice input to a microphone placed in the vehicle cabin. The microphone may be located in a position suitable for picking up audio within the vehicle cabin. The image capturing device 101R may have a microphone, or the microphone may be placed outside the image capturing device 101R. The control unit 303 may acquire audio information indicating the audio input to the microphone in S410, analyze the audio information in S420, and determine whether a person has spoken in S430. There is no particular limitation on the method for determining whether a person has spoken. The control unit 303 may, for example, make the determination based on the signal waveform of the audio information or based on the frequency band components of the audio information to which the human voice belongs.

[0134] The control unit 303 may determine that a person has spoken when it detects the voice of one person, such as talking to oneself, or may determine that a person has spoken when it detects the voices of two or more people, such as a conversation (dialogue). The condition for determining that a person has spoken may be the recognition of voice that can be recognized as language, or in particular the recognition of human voice in a dialogue format, for example, the detection of an utterance and a response thereto. The condition for determining that a person has spoken may also be the detection of voice (for example, voice keywords) that suggests the occurrence of harassment such as power harassment, sexual harassment, or moral harassment, the utterance of abusive language, or other trouble.

[0135] In S440, if the control unit 303 determines that a person has spoken inside the vehicle cabin, it may output image information indicating an image captured by the vehicle cabin image capturing rear camera 102 to a predetermined output destination. The predetermined output destination may be the same as in the tenth example. For example, when the control unit 303 records the image information to a storage device such as the SD card 306 as an output destination, the recording of the image information may be considered as event recording triggered by the determination that a person has spoken inside the vehicle cabin. If the control unit 303 has not determined that a person has spoken inside the vehicle cabin or has determined that a person has not spoken inside the vehicle cabin, it may not output the image information to the predetermined output destination in S440.

[0136] In this way, it is possible to easily grasp from the captured image what is happening inside the vehicle when a person is talking. For example, the inside of a vehicle can be considered a closed room, and it is thought that troubles such as harassment or some other incidents may occur. In this case, there is a possibility that the person inside the vehicle may talk about such an incident. Also, for example, when going on a trip with friends, there is a case where the person inside the vehicle wants to keep the captured image as a memory by capturing the situation during the time when the person was talking. In this way, it is possible to easily grasp what is happening inside the vehicle when a person is talking, which is useful for confirming whether or not such an incident has occurred and the circumstances under which it occurred.

[0137] Furthermore, the following configuration may be adopted. The control unit 303 may output image information of a period before, after, or both of a period when a person spoke in the vehicle cabin. The period before and after may be a fixed time length, or the image capturing device 101R may be configured so that the user can set the period. In this way, image information before and after the period when a person spoke in the vehicle cabin is also output, so that the situation before and after the speaking period can also be understood from the captured image.

[0138] Information other than human voice may be used as information for determining whether a person has spoken. For example, control unit 303 may analyze an image captured by rear camera 102 for capturing images inside the vehicle, and determine that a person has spoken when the control unit 303 recognizes a speaking action.

[0139] The control unit 303 may output image information to a predetermined output destination regardless of who speaks, or may output image information to a predetermined output destination when a specific person speaks, and not output image information to a predetermined output destination when any other person speaks. In this case, the control unit 303 may use voice analysis technology to identify the person who spoke. The specific person may be an authorized user such as the driver or his / her passenger, or may be a person other than an authorized user (for example, a person who has ridden the vehicle illegally, such as a thief). If an image of the face of the person who spoke can be obtained, the control unit 303 may analyze the image to identify the person who spoke.

[0140] During a conversation with multiple passengers, if the control unit 303 determines using the above-mentioned voice analysis technology that the voice contains specific information assigned to various types of harassment, such as voice that is semantically recognized as harassment or voice that is identified as harassment based on waveform, the control unit 303 may add information about the speaker and the assigned harassment to the image information and output it. It is also preferable that the specific information assigned to various types of harassment can be set, selected, or added by the user.

[0141] 12th Example Possible cases Take a photo if someone inside the vehicle reaches out Example input Information to determine whether a person inside the vehicle has reached out, such as an image that identifies the person Example output Image information of the captured image

[0142] The control unit 303 has a function of capturing an image when a person extends their hand inside the vehicle. A person extending their hand may be interpreted as a person extending their arm. The control unit 303 may capture an image when a person extends their hand inside the vehicle, regardless of whether the vehicle engine is on / off or whether the accessory power is on / off, or may capture an image only while the vehicle engine or the accessory power is on. If the image is captured regardless of whether the vehicle engine or the accessory power is on / off, the same power control as in the tenth example may be performed.

[0143] The information for determining that a person has reached out in the vehicle cabin may be, for example, an image captured by the vehicle interior rear camera 102. In S410, the control unit 303 acquires image information of an image in which a person can be identified, for example, an image captured by the vehicle interior rear camera 102, analyzes the captured image in S420, and determines whether a person in the vehicle has reached out in S430. The control unit 303 may determine that a person has reached out when a person sitting in the driver's seat (i.e., the driver) reaches out their hand toward the passenger seat, or may determine that a person has reached out when a person sitting in the passenger seat reaches out their hand toward the driver's seat, or may do both. As a part of a human body, a hand is a relatively long part. The control unit 303 may determine that a hand has been reached out based on, for example, the characteristics of that part of the body.

[0144] The control unit 303 may determine that a person has reached out when the person sitting in the driver's seat (i.e., the driver) reaches out their hand or something they are holding toward the passenger seat, for example, beyond the movable range of the shift lever or the range of the center console. Similarly, the control unit 303 may determine that a person has reached out when the person sitting in the passenger seat reaches out their hand or something they are holding toward the driver's seat, for example, beyond the movable range of the shift lever or the range of the center console. As described above, the range of movement of the shift lever and the range of the center console can be used as criteria for determining whether a person has reached out, but it is also desirable that the user can use images captured by the rear camera 102 for capturing images inside the vehicle to independently set criteria for determining whether a person has reached out, for example, to any position between the driver's seat and the passenger seat, and furthermore, that criteria can be changed or adjusted as needed.

[0145] In S440, when the control unit 303 determines that a person has reached out a hand inside the vehicle cabin, it outputs image information indicating an image captured by the vehicle interior shooting rear camera 102 to a predetermined output destination. The output destination may be the same as that in the tenth example. For example, when the control unit 303 records the image information to a storage device such as the SD card 306 as an output destination, the recording of this image information may be considered as event recording triggered by the determination that a person has reached out a hand inside the vehicle cabin. When the control unit 303 has not determined that a person has reached out a hand inside the vehicle cabin or has determined that a person inside the vehicle has not reached out a hand, it may be configured not to output the image information to the predetermined output destination in S440.

[0146] In this way, it is possible to easily grasp from the captured image what happens when a person inside the vehicle reaches out their hand. For example, the interior of a vehicle can be considered a closed room, and if a trouble such as harassment or some other incident occurs, it is thought that there is a possibility of violent or threatening acts. If a person is seen extending their arm, it is suspected that an act such as hitting has occurred. Making it easier to grasp what happens inside the vehicle when a person reaches out their hand is useful in confirming whether or not such an incident has occurred and the circumstances under which it occurred.

[0147] Furthermore, the following configuration may be adopted. The control unit 303 may output image information of a period before, after, or both of the period when a person in the vehicle reaches out their hand. The period before and after may be a fixed time length, or the image capturing device 101R may be configured so that the user can set the period. In this way, image information before and after the period when a person in the vehicle reaches out their hand is also output, making it possible to grasp the situation before and after the period when the person reaches out their hand.

[0148] The control unit 303 may output the image information to a predetermined output destination regardless of who extends their hand, or may output the image information to a predetermined output destination when a specific person extends their hand, and may not output the image information to the predetermined output destination when any other person extends their hand. The control unit 303 may identify the person using the method described in the eleventh example.

[0149] Furthermore, a person may reach out even when simply picking up or placing an object. Therefore, the control unit 303 may identify a situation in which a person is reaching out in the vehicle cabin using audio picked up by a microphone or other information, and may determine whether to output image information to a predetermined output destination depending on the situation. For example, the control unit 303 may output image information when it determines that a predetermined audio related to the occurrence of a problem has been detected or when it acquires other information for understanding the occurrence of a problem, and may not output image information in other cases.

[0150] A person reaching out in the vehicle cabin may be interpreted as a person reaching out from the back seat to the driver's seat or the passenger seat. For example, in a taxi, a person pretending to be a passenger may commit an act of violence or intimidation against the driver. This makes it easier to understand the situation when a problem occurs from the captured image. A person reaching out in the vehicle cabin may be interpreted as a person sitting in the driver's seat or the passenger seat reaching out to the back seat, and various variations are possible regarding the position of the person.

[0151] In the tenth, eleventh, and twelfth examples, the control unit 303 may be configured to further issue a warning or notification when performing the process of S440. The warning or notification may be, for example, by illuminating a light-emitting element provided in the photographing device 101R or illuminating a light-emitting element provided in the photographing device 101F, but the warning or notification may be issued in any manner that can be perceived by a person. However, when the photographing device 101R is used for purposes such as crime prevention, the warning or notification may be directed to the outside of the vehicle, or a higher level warning or notification may be issued.

[0152] FIG. 5 shows an example in which two cameras are installed. In the embodiment of Fig. 5(a), in addition to the photographing device 101R, two rear cameras 102a and 102b for photographing the interior of the vehicle are installed on the vehicle 10. In the embodiment of Fig. 5(b), in addition to the photographing device 101R, two rear cameras 102c and 102d for photographing the interior of the vehicle are installed on the vehicle 10.

[0153] In both the embodiments of FIG. 5(a) and FIG. 5(b), the interior rear cameras 102a, 102b or the interior rear cameras 102c, 102d capture images in a direction from the rear of the vehicle to the front of the vehicle, but one or both of them can be directed rearward. Also, one of the interior rear cameras 102 may be replaced with the imaging device 101R. Instead of two interior rear cameras 102, only one of the cameras may be used. The interior rear cameras 102c, 102d are preferably attached to the pillars of the vehicle 10 (here, on the cabin side).

[0154] Furthermore, it is advantageous to attach the two rear cameras 102a and 102b for photographing the interior of the vehicle or the rear cameras 102c and 102d for photographing the interior of the vehicle to the rear of the vehicle, particularly in the range from behind the rear seats to the rear window. One photographing device 101R and two additional rear cameras 102a, 102b or 102c, 102d for photographing the interior of the vehicle are connected by wire or wirelessly, and the video data photographed by the two additional rear cameras 102a, 102b or 102c, 102d for photographing the interior of the vehicle may be image processed by the photographing device 101R.

[0155] In the embodiment of Figure 5(b), the images from the rear cameras 102c and 102d for shooting inside the vehicle interior may each be sent directly to the photographing device 101R, or, for example, the image from the rear camera 102c for shooting inside the vehicle interior may be sent to the photographing device 101R via the rear camera 102d for shooting inside the vehicle interior.

[0156] A camera mounted on the rear of a vehicle, such as a bus, can capture clear images of passengers sitting in the rearmost seats. The rear camera may be equipped with a connector designed to extend the electrical wires from the glass. Such a connector should be positioned so that the cables are less visible inside the vehicle, and it is even more preferable to have a configuration in which the mounting position can be adjusted depending on the type and design of the vehicle.

[0157] The interior rear cameras 102c, 102d may be configured as, for example, spherical cameras (semi-spherical cameras), 180° cameras, 360° cameras, etc. In such a configuration, the interior rear cameras 102c, 102d may be configured to capture images of the interior of the vehicle from the rear of the vehicle toward the front of the vehicle, as well as images of the exterior of the vehicle from the rear of the vehicle, without changing their orientation. Images captured with such a configuration may be combined with images captured by the front camera for analysis. If this video data is transmitted using the fifth-generation mobile communication system (5G), data transmission can be carried out smoothly even if, for example, many celestial cameras are installed.

[0158] Figure 6 shows an embodiment in which the shooting direction of the rear camera 102 for shooting inside the vehicle interior of the embodiment in Figure 1 can be changed. Note that the rear camera 102e for shooting inside the vehicle interior in Figure 6(a) and the rear camera 102d in Figure 6(b) show the same camera facing different shooting directions.

[0159] 6, for example, the interior rear camera 102e can be rotated 180° around its longitudinal axis relative to the image capturing device 101R to change the capturing direction to that of the interior rear camera 102d. By changing the capturing direction of the interior rear camera 102d in this way and configuring it to capture images of the rear of the vehicle 10, it is possible to capture images of the rear of the vehicle using two cameras, the image capturing device 101R and the interior rear camera 102d, and to accurately measure the distance to an object behind, such as a vehicle behind.

[0160] In this embodiment, accurate distance measurement is possible even if the distance between the two cameras is as little as 1 mm. Furthermore, highly accurate distance measurement is possible when the distance is approximately 10 cm. In this configuration, the distance to the vehicle behind can be accurately detected using the image from the interior rear camera 102d. Therefore, based on the detected accurate distance, it is preferable to more accurately determine tailgating, dangerous lane changes by the vehicle behind, etc.

[0161] In this configuration, by adding one rear camera 102 for photographing inside the vehicle to the photographing device 101R, it becomes possible to photograph the rear of the vehicle with two cameras, providing significant scalability to the in-vehicle camera system.

[0162] The above-mentioned accurate distance measurement can also be performed for vehicles on the side by using two cameras to capture images of the side. For example, this can be used in the second example above to more accurately determine the risk of sideways collisions.

[0163] In addition, the rear camera 102 for taking photos inside the vehicle interior may be equipped with a floodlight such as an LED light (e.g., an infrared LED light) whose light projection direction is linked to the adjustment of the camera direction, and when the rear camera 102 for taking photos inside the vehicle interior is pointed outside the vehicle, the light from the floodlight may be electrically blocked (e.g., by turning off a switch) or mechanically shielded (e.g., by covering the light-emitting part with a cover) so that the light from the floodlight cannot be seen outside the vehicle.

[0164] In such a configuration, when the interior-photographing rear camera 102 is facing inward, it is preferable that the light from the photography light can capture a clear image of the interior of the vehicle even in a dark environment such as at night, or that the light from the indicator light can be used to check the operating status of the interior-photographing rear camera 102. On the other hand, when the interior-photographing rear camera 102 is facing outward, it is possible to stop emitting light from the floodlights such as the photography light and indicators, thereby preventing unnecessary light from leaking outside the vehicle.

[0165] FIG. 20 shows an example of a camera that blocks the light from a floodlight. 10(a) shows an overview of the rear camera 102 for photographing the interior of a vehicle, and includes a lens 2004 on a housing 2001, and two LED lamps 2003 attached to the left and right of the lens 2004. Note that the location of the LED lamps is not limited to this location, and they may be located in another position on the housing 2001.

[0166] 10 and 16 , the cover 2002 covers the LED lamp 2003 so as to block the light from the projector. When capturing an image of the exterior of the vehicle behind the vehicle, the cover 2002 covers the LED lamp 2003 so as to block the light from the LED lamp 2003. By mechanically covering the LED lamp 2003 in this manner, it is possible to prevent the lens 2004 from emitting light toward the exterior of the vehicle when the lens 2004 is facing the exterior of the vehicle.

[0167] (c-1) and (c-2) are diagrams showing examples of a configuration in which the lens 2004 is rotated 180 degrees relative to (b) to photograph the interior of the vehicle. (c-1) is a diagram seen from the rear of the vehicle, with the lens 2004 located at the back of the page and photographing the interior of the vehicle. (c-2) is a diagram seen from inside the vehicle, with the LED lamps 2003 on the left and right sides of the lens 2004 not blocked and projecting light into the interior of the vehicle.

[0168] Even when the rear camera 102 for photographing inside the vehicle interior is not configured to be adjustable in camera direction, the above-described floodlight blocking configuration can be applied. For example, the floodlight of the camera may be attached to face the interior of the vehicle, but not to face the exterior of the vehicle. Alternatively, even if the floodlight of the camera is attached to face the exterior of the vehicle, it may be configured not to emit light toward the rear of the vehicle 10. For example, a configuration may be adopted in which the light of the floodlight is electrically blocked or mechanically blocked.

[0169] It is preferable that the rear cameras 102e and 102d for photographing the interior of the vehicle be configured to be able to rotate by 270° or more. With such a configuration, the photographing direction can be flexibly adjusted depending on the shape of the vehicle and the installation positions of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle.

[0170] FIG. 7 shows an embodiment in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle in FIG. 1 are configured as a single wide-angle camera. The wide-angle camera 700 in FIG. 7(a) has a wide-angle lens. A wide-angle lens is a lens with a wider angle of view than a standard lens. A wide-angle lens has a shorter focal length than a standard lens. The wide-angle camera 700 has an imaging means for capturing images using the wide-angle lens. The imaging means of the wide-angle camera 700 captures images and generates image data representing the captured images.

[0171] The image may be either a moving image or a still image, but is typically a moving image. Images captured using a wide-angle lens contain distortion (also referred to as image distortion) caused by the wide-angle lens. In this image, the distortion increases as you move from the center of the image to the edges. In this embodiment, the wide-angle lens has a 240° angle of view on both sides of a vertically downward direction when wide-angle camera 700 is mounted in the orientation that it should be mounted on the rear window or rear roof of a vehicle. This allows wide-angle camera 700 to capture a wide area on both sides of a vertically downward direction.

[0172] The wide-angle camera 700 in Figure 7(a) is a hemispherical camera with a viewing angle of X°, which is 240°. This allows the hemispherical camera to capture, as a single image data, images of the interior of the vehicle taken from behind the vehicle toward the front, images immediately behind the vehicle, images in front of the vehicle, and images in the elevation angle direction in front of and behind the vehicle. By selecting a hemispherical camera with an appropriate viewing angle depending on the vehicle model and placing it in an appropriate position, it becomes possible to record the status of traffic lights in front of and behind the vehicle in the image data.

[0173] The angle of view X° is greater than 180°, and may be, for example, 220°, 270°, 360°, or another angle of view. The wide-angle lens 114 may be a fisheye lens. The angle of view of the wide-angle lens does not need to be uniform in all directions; for example, the lens may have a characteristic such that the angle of view in the longitudinal direction of the vehicle is greater than the angle of view in the lateral direction of the vehicle.

[0174] FIG. 7(b) shows a close-up view of the wide-angle camera 700 of FIG. 7(a). Wide-angle camera 700 has a projector 701 in the direction of the lens, and the orientation of projector 701 changes in conjunction with the orientation of the lens. The rear half of wide-angle camera 700 is covered with filter 702. In the embodiment shown in FIG. 7(b), filter 702 is attached to a housing cover that covers wide-angle camera 700.

[0175] The light from floodlight 701 is expected to be auxiliary light for taking bright photographs of the interior of the vehicle, or light from an indicator LED light, but when the lens is facing the rear of the vehicle, it is preferable that filter 702 be configured to block the light emitted from floodlight 701, thereby reliably preventing the light from being seen by people outside the vehicle through the rear window. If the light from the projector 701 is infrared light and the wide-angle camera 700 is an IR camera, the interior of the vehicle can be clearly photographed even at night.

[0176] Wide-angle camera 700 can also be attached to the rear or front of the vehicle. Wide-angle camera 700 can also be a hemispherical camera and installed in a diagonal corner of the vehicle rather than in the center of the windshield or the rear window, allowing a full view of the inside and outside of the vehicle.

[0177] 19 is another example of wide-angle camera 100, and is a diagram showing the positional relationship between the flat surface of operation surface 120 and optical axis 114A of wide-angle lens 114 when wide-angle camera 100 is viewed from the side. In the state shown in FIG. 19, wide-angle camera 100 is in a normal posture. The normal to the flat surface of operation surface 120 faces horizontal direction 120H, and optical axis 114A of wide-angle lens 114 is tilted at a predetermined angle θ with respect to the vertical downward direction toward the side opposite to the side where operation surface 120 exists.

[0178] In this embodiment, optical axis 114A of wide-angle lens 114 is tilted by a predetermined angle θ toward the rear of the vehicle's rear window relative to the vertically downward direction, which is a direction that a user can visually confirm. This orientation when case 116 is attached to the vehicle's rear window is the normal orientation. Furthermore, angle of view X1° is the angle of view on the side where operation surface 120 is located relative to the vertically downward direction when case 116 is in the normal orientation, and angle of view X2° is the angle of view on the side opposite to the side where operation surface 120 is located relative to the vertically downward direction when case 116 is in the normal orientation. Angle of view X°, angle of view X1°, and angle of view X2° satisfy the relationship X° = X1° + X2°. The angle of wide-angle lens 114 is fixed relative to case 116.

[0179] The angle θ satisfies the following formula (1) for the angle of view X° of the wide-angle lens 114. θ=((X-200)×0.5)° (1) By setting a predetermined angle θ to satisfy equation (1), wide-angle camera 100 tilts optical axis 114A of wide-angle lens 114 by ((X-200) × 0.5) degrees relative to the downward vertical direction on the side opposite to the side on which operation surface 120 is located, and can capture a wide area both vertically and horizontally, including the area (X-190) degrees above horizontal direction 120H.

[0180] For example, when the angle of view X° of wide-angle lens 114 is 240°, the predetermined angle θ is 20°, the angle of view X1° is 100°, and the angle of view X2° is 140°, and wide-angle camera 100 can capture a wide area both vertically and horizontally, including an area 50° above horizontal 120H. When capturing an image with wide-angle lens 114, the edges of the captured image are distorted. Therefore, even if a smaller angle of view is set by 10° to minimize distortion, wide-angle camera 100 can capture an area up to 40° above horizontal 120H. When wide-angle camera 100 is in the upright position, the capturing area includes the area behind the vehicle and the interior of the vehicle. It is even better if the capturing area also includes areas on the sides of the vehicle.

[0181] The angle of wide-angle lens 114 may be adjustable within a predetermined variable angle range on both sides of the vertically downward direction relative to case 116. For example, in the case of wide-angle camera 100 attached to the rear window of a vehicle, the two sides of the vertically downward direction may be the front-to-rear direction of the vehicle. The predetermined variable angle range may be, for example, a variable angle range set in relation to the angle of view X° of wide-angle lens 114, centered on optical axis 114A of wide-angle lens 114, so that wide-angle camera 100 can capture a wide area both vertically and horizontally. For example, when the angle of view of wide-angle lens 114 is 240° and the center of optical axis 114A of wide-angle lens 114 is tilted by 20°, the variable angle range may be a range of -20°<variable angle≦20°. In this case, wide-angle lens 114 is adjustable within a range of 0° to 40° on the side opposite the side where operation surface 120 is located, relative to case 116, in the vertically downward direction.

[0182] The variable angle range may be any range that includes both sides of the vertically downward direction, and may be, for example, an angle range that is asymmetric between the front and rear of the vehicle with the vertically downward direction in between. Also, the variable angle range may include only the range from the vertically downward direction to the front of the vehicle, and thus, both sides of the vertically downward direction may include a case where one of the both sides is in the vertically downward direction. Implementing a single rear camera in this way reduces costs, eases installation work, and makes the device smaller than if the rear camera were to consist of two cameras, one for the interior and one for the rear.

[0183] FIG. 8 shows an embodiment in which the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are attached to the rear hatch of an ambulance vehicle. Fig. 8(a) shows the state in which the rear hatch of the ambulance is closed, and Fig. 8(b) shows the state in which the rear hatch of the ambulance is open. In the state in Fig. 8(b), the rear camera 102 for photographing inside the vehicle can clearly photograph the movement of a stretcher or gurney into the ambulance vehicle and the surrounding environment while the vehicle is stopped.

[0184] A rear hatch is an example of a rear opening / closing part that opens and closes an opening provided at the rear of a vehicle. A rear hatch is also called a hatchback door, a rear hatch gate, a back door, etc. A rear hatch closes the opening when closed, and opens the opening when open. The opening allows access between the inside and outside of a vehicle and is used, for example, for passengers to get in and out, or for loading and unloading luggage. A rear hatch such as the one shown in Figure 8 is also called a flip-up rear hatch, and is of the type that opens upward.

[0185] In FIG. 8, the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle are shown, but instead of these, a celestial camera or a wide-angle camera similar to that described in FIG. 7 may be used. It would be good to mount, for example, a spherical camera or a similar wide-angle camera on the inside of a flip-up door such as a rear hatch, so that when the hatch is open, it can take an image of the state of a stretcher or litter placed on the road from above.

[0186] It is also advisable to configure the camera so that the scene of the person (patient) being loaded into the vehicle on a stretcher or gurney is also captured as a hemispherical image. In this embodiment, for example, the rear camera 102 for taking photos inside the vehicle interior is not operated from the time the ambulance is dispatched until it arrives at the scene, and the control unit 303 may be configured to start recording using the rear camera 102 for taking photos inside the vehicle interior as a trigger when it detects that the hatch has been opened, or to change the shooting settings or the lens to be used.

[0187] When loading the patient, the patient's identity and accident / condition are confirmed on the road or on the stretcher, but this information is often not communicated accurately, and in many cases, the accident situation is often communicated incorrectly until the very end. By transmitting the video captured by the rear camera 102 for capturing images inside the vehicle to a hospital or fire department via 5G or the like, it becomes possible to convey accurate information in the form of video and audio. Automatic transcription using voice recognition is also effective.

[0188] It is further advantageous if the interior camera 102 has sensors for detecting various biological reactions. In such a configuration, the control unit 303 may be configured to quickly transmit the patient's body temperature data detected by, for example, a thermosensor to the medical monitor 802, even before various medical monitor devices are attached to the patient. It is also further advantageous if the control unit 303 stores the biological reaction data in synchronization with the video data.

[0189] It is preferable to use the rear camera 102 for photographing inside the vehicle to photograph the entire process from the state in which the hatch is closed, to the state in which the hatch is opened and the stretcher is lowered, the patient is placed on the stretcher, the patient is carried into the vehicle, the hatch is closed, and the vehicle is started to transport the patient. This makes it possible to continuously photograph the entire situation, even if questions are asked while the patient is in the vehicle, for example. In addition, it can also capture images of people who were nearby at the time of the accident, which can be used to identify the cause of the accident and the culprit.

[0190] When the ambulance arrives at the hospital and the patient is removed from the vehicle, it can be verified later whether the patient was removed properly, and it can also be verified whether information has been passed on to the hospital appropriately, making it easier to identify the party responsible if any problems arise later.

[0191] In a configuration in which the celestial sphere camera is attached to a flip-up hatch, it is preferable that when the hatch is closed, it photographs a 180-degree celestial sphere with the interior of the vehicle as its central axis, and when the hatch is open, it photographs a 180-degree celestial sphere with the downward direction from the ceiling as its central axis. The control unit 303 should have a function to determine whether the hatch is open or closed and record that point in time. It is also desirable to have a configuration in which it determines whether the hatch is in the process of being opened or closed and records that information.

[0192] Furthermore, it is preferable to configure the camera so that the recording method and camera settings are switched depending on whether the hatch is open or closed. In other words, it is preferable that the outdoor recording method and camera settings are applied when the hatch is open, and the indoor recording method and camera settings are applied when the hatch is closed. This makes it possible to reliably record good video data even when the shooting environment and shooting conditions change significantly.

[0193] In the case of a spherical camera, it is best to mount it on the hatch so that one camera is in the transparent window that can capture the outside from the window of the hatch, for example, the main body should be in the car body part just below the window and the camera should pop out above and be in the transparent window part.

[0194] In this way, when the hatch is closed (Fig. 8(a)), one camera will take pictures of the inside of the vehicle, while the other camera will take pictures of the outside of the vehicle, such as the rear. In this case, the thickness of the hatch will be close to the boundary between the two cameras, making it easy to capture clear images from each camera, for example, with appropriate exposure. In this case, it is best not to match the exposure of the two cameras.

[0195] Also, when the hatch is open, one of the cameras can capture the surroundings, including what is happening below the hatch. The other camera will take pictures of the sky, so in the event of a fire or jumping, it can capture the situation at the scene and weather conditions in the air.

[0196] In the configuration of FIG. 8, it is also possible to use a hemispherical camera. In the case of a hemispherical camera, it is recommended that it be attached to the hatch just below the hatch window, in the center in the left-right direction. In this way, when the hatch is open, it is less susceptible to the effects of wind and rain, and it is possible to photograph the surroundings, including what is happening under the hatch.

[0197] Data from the rear camera 102 for taking photos inside the vehicle can be synchronized with other cameras (cameras installed on the ceiling of the vehicle, on wearable devices worn by team members, on stretchers, etc.) and recorded in a reproducible manner or transmitted via 5G or the like.

[0198] Although the interior-use rear camera 102 may also be used to photograph the exterior rear of the vehicle, it is referred to as the "interior-use rear camera" because it is used at least to photograph the interior of the vehicle. Such an interior-use rear camera may be understood as a "combined use camera for both interior and exterior rear of the vehicle." Furthermore, the interior-use rear camera 102 may be understood as the first camera, and the camera 402 may be understood as the second camera. The camera of the photographing device 101F may be understood as the third camera.

[0199] The above description has been given assuming that the vehicle is an ambulance, but the vehicle in this embodiment may also be, for example, a commercial vehicle that has a flip-up door such as a rear hatch to carry luggage or passengers, or a caravan. Commercial vehicles include, for example, welfare vehicles used to transport users (e.g., elderly people or people with disabilities) of designated facilities such as day care centers (day care centers), emergency vehicles such as police cars, and postal vehicles. Furthermore, rear hatches may also be installed in vehicles for general users, as exemplified by caravans.

[0200] The rear hatch may be installed in vehicles such as minivans, hatchbacks, and station wagons. In such cases, it is advisable to replace "patient" with "luggage" or "passengers other than patients, such as the driver." For example, if the vehicle is a welfare vehicle, "passengers other than patients, such as the driver," may be users of a designated facility. The attributes of the passengers vary depending on the use of the vehicle. Even in these configurations, it is advisable to record the situation when luggage is loaded, changes in the state of the loaded luggage, the situation when passengers other than the patient, such as the driver, are on board, and the state of work being done with the rear hatch open.

[0201] <Another embodiment of the photographing device provided in the rear hatch> For example, the following is an embodiment based on a concept common to the embodiment described in Fig. 8. Next, referring to Fig. 9, an embodiment in which a vehicle 500, which is a welfare vehicle, is used to transport a day care service user U1 will be described. In Fig. 9, the vehicle 500 is a welfare vehicle adapted for wheelchairs. The camera of the image capture device 101R is a hemispherical camera or a wide-angle camera equivalent thereto, and the case where the capture range is the same as that shown in Figs. 8(a) and (b) will be described as an example. It is preferable that the vehicle 500 has a mechanism for smoothly loading a wheelchair onto the vehicle using a ramp or lift, but this is not shown in the figure.

[0202] FIG. 9(a) is a diagram showing the user U1 before getting into the vehicle 500. The user U1 is sitting in a wheelchair 501. The supporter U2 is, for example, a staff member of a facility such as a nursing home or a hospital. The supporter U2 is, for example, a caregiver, an attendant, or a nurse. The supporter U2 supports the user U1 by pushing the wheelchair 501 or by carrying the user U1 into the vehicle 500 together with the wheelchair 501. For example, the supporter U2 moves from the user U1's home to the rear of the vehicle 500 while pushing the wheelchair 501.

[0203] As explained in FIG. 8, the photographing device 101R is attached to the cabin side of the rear window 11 provided on the rear hatch of the vehicle 500. When the rear hatch is closed, the interior photographing rear camera 102 of the photographing device 101R faces the interior of the vehicle 500, specifically photographing the front of the vehicle from the rear of the vehicle. By photographing the interior of the vehicle from the rear, the interior photographing rear camera 102 photographs people's actions while protecting their privacy without photographing their faces, which is also an aspect that takes privacy into consideration. As explained in FIG. 8, this embodiment is also used to actively photograph and record people's behavior.

[0204] FIG. 9(b) is a diagram showing a state when the supporter U2 is trying to get the user U1 into the vehicle 500 (i.e., getting in). When the rear hatch of the vehicle 500 is opened, the shooting direction of the interior camera 102 changes. When the rear hatch is open, the interior camera 102 shoots an image of the rear exterior of the vehicle 500 facing downward. Therefore, the interior camera 102 shoots an image of the user U1 and the supporter U2, who are located just behind the vehicle 500, from above. In this way, when the rear hatch is open when the supporter U2 is getting the user U1 into the vehicle 500, it is possible to automatically shoot an image of the situation at the time of getting in, and furthermore, capture an appropriate range for recording. Such a captured image is useful, for example, for verifying whether the user U1 struggled and fell, or whether the supporter U2's response was appropriate.

[0205] 9(a) and 9(b) illustrate the case where the door hatch is opened when the supporter U2 tries to get the user U1 into the vehicle 500, but there are also cases where the rear hatch is already in an open state at the stage shown in Fig. 9(a). In this case, the rear camera 102 for photographing inside the vehicle may be able to photograph the user U1 and the supporter U2 at an earlier stage.

[0206] FIG. 9(c) is a diagram showing the state after user U1 has been placed in vehicle 500 while sitting in wheelchair 501. Wheelchair 501 is placed on platform 502. Platform 502 is provided with a mechanism (not shown) for locking (i.e., fixing) the wheelchair. Wheelchair 501 is fixed to platform 502 by the mechanism so that it does not move while vehicle 500 is moving. After user U1 has been placed in vehicle 500, the rear hatch of vehicle 500 is closed and put into a closed state.

[0207] As described above, when the rear hatch is closed, the interior camera 102 takes pictures toward the interior of the vehicle 500. This photographing range includes at least the user U1. Therefore, the photographing device 101R can use the interior camera 102 to photograph and record the state of the user U1 while traveling. For example, by photographing and recording the state inside the vehicle 500 while traveling to a facility (e.g., a nursing home or a hospital), it is possible to record interactions such as conversations between the user U1 and the supporter U2. Therefore, even if some kind of trouble occurs during travel in the vehicle 500, such as abusive language or harassment, the photographed images can be used to investigate the trouble later.

[0208] 9, the interior rear camera 102 of the photographing device 101R attached to the rear hatch can capture an appropriate range depending on the situation at the time. Here, the control when the user U1 gets into the vehicle 500 has been described, but the same applies when the user U1 gets off the vehicle 500 after arriving at a facility, for example. In this case, the photographing device 101R can use the interior rear camera 102 to capture and record the situation when the user U1 gets off the vehicle.

[0209] Furthermore, the following configuration may be adopted. 9(b) to 9(c), that is, during the period when the user U1 and the wheelchair 501 are in the vehicle 500, the engine of the vehicle 500 may be on or off. Even in such a case, for example, by configuring the power supply control described in the above-mentioned tenth example to be performed, the photographing device 101R can photograph and record using the rear camera 102 for photographing inside the vehicle interior, regardless of whether the engine of the vehicle 500 is on or off or whether the accessory power supply is on or off.

[0210] (B) For example, as in the case where the image capturing device 101R continuously records, the image capturing and recording may be performed over the period described in Figures 9(a) to 9(c), or as in the case described in Figures 9(b) to 9(c), the image capturing and recording may be performed from the time when the driver attempts to get into the vehicle 500 onwards. In the latter case, the control unit 303 in the image capturing device 101R may record image information of the image captured by the rear camera 102 for capturing images inside the vehicle interior when the rear hatch is changed from a closed state to an open state.

[0211] If the photographing device 101R is powered off before the start of recording, or is powered on but is in a state where photographing and recording cannot be started immediately (for example, in a power saving mode), the control unit 303 may first power on the photographing device 101R or transition the photographing device 101R to a state where photographing and recording can be started (for example, in a normal operation mode).The control unit 303 may then start photographing and recording.If the photographing device 101R has already been powered on and is in a state where recording can be started immediately, the control unit 303 may start the recording immediately.

[0212] A method for determining whether the rear hatch has changed from a closed state to an open state will now be described. The control unit 303 may determine that the rear hatch has changed from a closed state to an open state based on, for example, a rear hatch door open detection signal. The rear hatch door open detection signal is a signal output from the OBDII ("II" is the Roman numeral for "2") connector when the rear hatch is detected to be open while the accessory power of the vehicle 500 is turned on. Alternatively, the control unit 303 may determine that the rear hatch has changed from a closed state to an open state by, for example, configuring a predetermined circuit outside or inside the control unit 303.

[0213] The control unit 303 may determine that the rear hatch has changed from a closed state to an open state, for example, based on acceleration detected by the acceleration sensor 305. The control unit 303 may acquire acceleration information and determine that the rear hatch has changed from a closed state to an open state if the acceleration information indicates the acceleration when the rear hatch is in an open state or if the acceleration information indicates a temporal change in acceleration when the rear hatch changes from a closed state to an open state. By using a method that does not rely on a door open detection signal in this way, the control unit 303 can photograph and record when the rear hatch changes from a closed state to an open state, even if the accessory power is turned off. For example, even if the accessory power is turned off, it is possible to photograph and record the state of user U1 entering the rear hatch before the rear hatch is opened.

[0214] The following configuration may be employed to determine whether the rear hatch has changed from a closed state to an open state. A G sensor is provided in the rear camera 102 for photographing the interior of the vehicle, and a G sensor (e.g., acceleration sensor 305) is provided in the photographing device 101R, which is a rear-facing camera attached to the same rear hatch. The control unit 303 may then determine whether the rear hatch is open or closed based on the degree of agreement between the direction of gravitational acceleration (e.g., measured assuming the vehicle is stopped) of the G sensor when the rear hatch is closed and when it is open, and the time during which this agreement is maintained (i.e., the stabilization time). Alternatively, the control unit 303 may determine whether the rear hatch is open or closed based on a change in a characteristic G sensor value when the rear hatch is opened or closed. The characteristic G sensor value may be a three-dimensional value, such as acceleration in three axes.

[0215] In the configurations described in (C) and (B), when the control unit 303 determines that the rear hatch has changed from a closed state to an open state, the control unit 303 may start by turning on the power of the photographic device 101R. However, it is preferable to have the photographic device 101R turned on in advance, since this allows for quicker recording. To achieve this, the photographic device 101R may use a recording standby function. The recording standby function is a function that waits for the photographic device 101R to start recording captured images (hereinafter sometimes referred to as "recording during the designated time period") when the photographic device 101R is powered on during a designated time period (hereinafter referred to as the "designated time period") and a predetermined trigger is detected.

[0216] When the recording standby function is used, if the accessory power supply changes from on to off, the recording that was being performed up until then (for example, continuous recording) stops, but the power supply of the photographing device 101R does not turn off, at least within the specified time period. To keep the power supply on even when the accessory power supply is off, the photographing device 101R should be supplied with power from a continuous power supply (i.e., +B operation), but an internal power supply such as the secondary battery 312 or an external battery unit may be used instead, or both may be used.

[0217] In this embodiment, the designated time period for the recording standby function is preferably designated (i.e., set) in advance to include the time period during which user U1 is picked up or dropped off. Then, the control unit 303 operates the recording standby function, and when it determines that the rear hatch has changed from a closed state to an open state within the designated time period, it preferably starts recording captured images. In this way, the change of the rear hatch from a closed state to an open state is used as a trigger to promptly start recording captured images, thereby making it possible to more reliably capture the situation during the pick-up or drop-off.

[0218] Instead of or in addition to the configurations (D) and (C), the following configuration may be used to more reliably capture and record the situation during pick-up and drop-off. When the control unit 303 is recording captured images while the accessory power is on (for example, continuous recording, etc.), if the accessory power is turned off, the control unit 303 may extend the recording of images captured by the interior rear camera 102 by a specified time length rather than immediately ending the recording of the images (hereinafter referred to as "specified time extended recording"). The starting point of the specified time may be, for example, the time when the accessory power is turned off. The inventors have considered that this method may be advantageous in reducing adverse effects on the battery of the vehicle 500.

[0219] Next, an actual usage scenario will be described. When the vehicle 500 arrives at the home of the user U1, the accessory power of the vehicle 500 is changed from on to off. In this case, the control unit 303 of the image capturing device 101R is triggered by the accessory power being turned off to start extended recording for a specified time. In this way, the situation of the user U1 and the supporter U2 who are about to get in or out of the vehicle can be captured and recorded using the rear camera 102 for capturing images inside the vehicle. The specified time may be specified in advance, for example, at the time of product shipment.

[0220] For example, the specified time may be specified taking into consideration the time required from when the accessory power is turned off, to when the user U1 gets into the vehicle 500, and until the accessory power is turned on again. The specified time may be specified taking into consideration the time required for standard boarding and disembarking (for example, the time required to stop the vehicle with the engine turned off). For example, the stop time for day care pick-up and drop-off may be approximately 5 to 10 minutes. However, there may be various factors that may extend the time, such as creating a wheelchair ramp and operating the lock to secure the wheelchair after boarding and disembarking, or talking with the user UI's family. Therefore, a configuration may be adopted in which the specified time is specified to the imaging device 101R by a facility staff member such as a supporter U2.

[0221] The specified time extended recording may be performed as follows. If the control unit 303 determines that the rear hatch has changed from a closed state to an open state within the extended specified time, it may perform the specified time extended by a first specified time length. If the rear hatch remains closed and does not change to an open state, it may perform the specified time extended by a second specified time length that is shorter than the first specified time. The specified time extended recording performed for the second specified time may be performed by interrupting the specified time extended recording for the first specified time. This is because if the rear hatch does not change to an open state, it is assumed that no passengers are getting in or out of the vehicle using the opening at the rear of the vehicle, and it can be said that this is a situation in which it is not necessary to record images captured by the interior camera 102.

[0222] In the configurations (E), (C), or (D), the control unit 303 may record, when recording within a specified time period or with an extended recording time, identification information for identifying whether the recording was within a specified time period or with an extended recording time, in association with (e.g., added to) the image information of the captured image. Software capable of playing back recorded video, such as a viewer, may have a function for outputting information to notify the user whether the recording was within a specified time period or with an extended recording time, based on the identification information. The information for notifying the user may be, for example, display information, which may be displayed overlaid on or on the same screen as the video being played back. Alternatively, the software may have a function for enabling a search for images recorded within a specified time period or with an extended recording time.

[0223] In the configurations (F) and (D), the image capturing device 101R may be configured to allow the user to set whether or not to perform extended recording for a specified time after the accessory power is turned off. The control unit 303 controls whether or not to perform extended recording for a specified time after the accessory power is turned off according to the setting. Setting this in advance reduces the burden on the user, but the user may also be able to set it each time the accessory power is turned off.

[0224] In the configurations (G) and (D), when the rear hatch is changed from a closed state to an open state and the control unit 303 determines that a person is getting off the vehicle through the opening, the control unit 303 may perform extended recording for a specified time using the rear camera 102 for shooting inside the vehicle. In the case of picking up or dropping off, this is useful because it allows the supporter U2 to use the rear camera 102 for shooting inside the vehicle to shoot the state of pushing the wheelchair 501 in which the user U1 is sitting. This is based on the idea that the supporter wants to continue shooting (i.e., want to see) the object that was being shot inside the vehicle by the rear camera 102 for shooting inside the vehicle in the rearward, outside direction.

[0225] The control unit 303 may determine that a person has exited the vehicle through the opening when the rear hatch changes from a closed state to an open state or from an open state to a closed state while the vehicle 500 is stopped with the accessory power on. Furthermore, as will be described later, the control unit 303 may determine that a person has exited the vehicle through the opening based on a signal from a mechanism provided in the vehicle 500, which is a welfare vehicle.

[0226] The control unit 303 may set different modes of the specified time extension recording depending on whether it is determined that a person has exited the vehicle through the opening or whether it is determined that a person has entered the vehicle. For example, the control unit 303 may set different recording times for both cases, and in particular, may set a shorter recording time for when a person has entered the vehicle than when a person has exited the vehicle. This is based on the idea that, since there are people outside the vehicle when a person has exited the vehicle and the possibility of an accident or the like is relatively high, recording such an event should be made more reliable. Furthermore, the control unit 303 may record higher quality images when a person has entered the vehicle than when a person has exited the vehicle. Examples of quality include frame rate and resolution.

[0227] The door that is determined to have changed from a closed state to an open state when disembarking, or from an open state to a closed state when disembarking, may be a door other than the rear hatch, such as the driver's door or the passenger's door. In this way, extended recording for a specified time can be performed based on the latter of the supporter U2 who is a passenger, which may more reliably capture the state of the user U1 getting in and out of the vehicle.

[0228] In the configurations (H) and (D), the control unit 303 may have a function for setting a designated time. The setting of the designated time here is so-called automatic setting. The control unit 303 may, for example, identify the situation inside or around the vehicle when the user dismounts by recognizing an image captured by the vehicle interior rear camera 102. For example, the control unit 303 may determine whether the rear hatch of the vehicle 500 is open or closed by performing image recognition on the image captured by the vehicle interior rear camera 102. The control unit 303 may also determine whether the user U1 and the supporter U2 are getting in and out of the vehicle, or whether the action has been completed, by performing image recognition on the image captured by the vehicle interior rear camera 102. The control unit 303 may set the designated time for the designated time extended recording based on the results of these determinations.

[0229] (I) The control unit 303 may recognize images captured by the interior-photography rear camera 102 and record information about the presence or absence of a passenger in the vehicle 500 in association with (for example, by adding) the information to be recorded. The control unit 303 may add information indicating the position of the passenger in the image to the recorded image information. The information indicating the position may be, for example, an arrow pointing to the passenger or a graphic such as a circle surrounding the passenger's position. Furthermore, even if the passenger is in a fixed position, for example, because the wheelchair 501 is fixed to the base 502, the position of the passenger in the image differs depending on whether the rear hatch is open or closed. Therefore, if information indicating the actual position of the passenger (the position of the wheelchair 501) is recorded or displayed, the user can easily understand the situation of the passenger from the image. The position of the passenger may be detected using the human detection sensor described above.

[0230] (J) The condition for ending recording of the specified time period or the specified extended time period in the imaging device 101R may be primarily that the specified time period has elapsed or that the specified extended time period has elapsed, but other conditions may also be included. For example, a condition may be that the door has changed from an open state to a closed state within the specified time period or the extended specified time period. This type of operation may be used when the interior of the vehicle is not being photographed or recorded. Furthermore, the recording time for the specified time period may be specified taking into account the time required for the user U1 to get into the vehicle 500 and for the accessory power to be turned on again. For example, the recording time may take into account the time required for standard entry and exit (e.g., the time required to stop the vehicle with the engine turned off).

[0231] Regarding the recording of the designated time period described in (K)(C), the control unit 303 may set the designated time period. In cases where the pattern of round trips of the vehicle between different bases, such as between the user UI's home and a facility, and the vehicle operation schedule is determined, or reservation data for operation reservations exists, the control unit 303 may set the designated time period to include the estimated time of arrival at the base, based on the schedule or reservation data.

[0232] The recording of designated time periods described in (L) and (C) can also be applied to applications other than the transportation mentioned above. For example, it may be used in applications where the vehicle stops at a predetermined location (e.g., a facility) at a fixed time or an estimated time. For example, a delivery person who delivers parcels may make deliveries along a set route. In this case, the rear hatch is opened when the delivery person retrieves the parcel. Even if the delivery person leaves the rear hatch open while delivering the parcel, the situation behind the vehicle can be captured and recorded, which is desirable for the security of the parcel. Note that the vehicle in this case may be a cart or a bicycle with a carrier that the delivery person pushes. In such a configuration, the rear hatch may be interpreted as a cover or lid attached to the cart or carrier.

[0233] (M) Figures 10 and 11 are diagrams illustrating images captured by the interior camera 102. Figures 10(a) and 11 show examples of images captured when the interior camera 102 is attached to the rear hatch of a light wagon as described in Figures 8 and 9. Figure 10(a) is a diagram illustrating an image captured when the rear hatch is closed. Figure 10(b) is a diagram illustrating the capture range when the rear hatch is closed.

[0234] As shown in FIG. 10(a), when the rear hatch is closed, the interior of the vehicle is photographed from the rear toward the interior, allowing the person's actions to be captured while protecting their privacy without capturing their face. Since the head of the person protruding above the driver's seat is visible in the captured image, the situation of the person inside the vehicle can be understood from the captured image. As shown in FIG. 10(b), when the rear hatch is open, the optical axis direction of the interior-photographing rear camera 102 extends in a substantially vertical direction, ensuring a substantially conical photographing range. However, because the vehicle floor is higher than the ground inside the vehicle, the photographing range is limited to floor height. As can be seen from FIG. 10, even when the rear hatch is open, the photographing device 101R is less susceptible to wind and rain, and can capture the exterior rear view of the vehicle, including the view below the rear hatch, using the interior-photographing rear camera 102.

[0235] FIG. 11 shows an image captured by the rear camera 102 for capturing images inside the vehicle interior when the rear hatch is open. FIG. 11(a) shows the rear hatch when it is open (horizontal dimension), FIG. 11(b) shows the rear hatch when it is open (vertical dimension outside the vehicle), and FIG. 11(c) shows the rear hatch when it is open (vertical dimension inside the vehicle). As can be seen, in the open state, the area below the rear hatch and the area toward the rear of the vehicle interior are included in the capture range. In this way, the rear camera 102 for capturing images inside the vehicle interior can be used to capture images of the area below the rear hatch outside the vehicle and the area toward the rear of the vehicle interior.

[0236] With interior cameras and hemispherical cameras that take pictures from the front of the vehicle, the images are taken from in front of the driver, which can make passengers feel as if they are being watched. Also, because vehicles are equipped with rearview mirrors, the images are not visible or are difficult to see through the side windows. To solve this problem, the interior of the vehicle is photographed from the rear. With the interior rearview camera 102, passengers are photographed from behind, so they do not feel like they are being watched, and images from the inside of the vehicle can be recorded, and images from the side windows can be clearly captured with little interference from other objects.

[0237] 10 and 11, the viewing angle of the rear camera 102 for capturing images inside the vehicle interior may be narrower than that of a 360-degree camera, etc. Therefore, the images outside the driver's seat and passenger seat windows captured by the rear camera 102 for capturing images inside the vehicle interior are larger and have higher resolution than when a 360-degree camera, etc. is used, which may make it easier for the user to see the contents of the captured image.

[0238] (N) Regarding the appeal regarding the rear camera 102 for taking pictures inside the vehicle on the rear hatch For example, the appeal regarding the rear camera 102 for photographing the interior of the vehicle on the rear hatch is as follows:

[0239] Vehicles equipped with the photographing device 101R (in-vehicle photographing rear camera 102) may include emergency vehicles such as police cars, parcel delivery vehicles (delivery vehicles), and postal vehicles, in addition to the ambulances and welfare vehicles mentioned above. In this way, when the rear hatch of a vehicle unloading people or cargo at the rear of the vehicle is opened, the unloading of the luggage or people is photographed. Furthermore, even if the rear hatch is opened when a parcel delivery person retrieves a package, and the delivery person leaves the hatch open while delivering the package, the situation behind the vehicle can be photographed and recorded, which is desirable for package security. Furthermore, even if a postal worker is snatched while unloading a package, the photographs can be recorded, which can contribute to the arrest of the perpetrator.

[0240] Similarly, general users can also capture images in the event of a snatch theft while unloading luggage. For example, when carrying luggage in a supermarket parking lot, a user may leave the rear hatch open and go to a remote location. For example, a general user may go to a remote location to pick up luggage, go to dispose of cardboard boxes brought from home, or return from the supermarket and carry luggage to their vehicle. In such cases, the rear hatch is often left open for several minutes. Even in such cases, the presence of the imaging device 101R can be expected to provide a crime prevention effect, and so the benefits of so-called consumer use are also fully anticipated.

[0241] (O) Furthermore, the following configuration may be adopted. (O-1) The camera 402 of the image capturing device 101R, the rear camera 102 for capturing images inside the vehicle, and the camera (not shown) of the image capturing device 101R may each record images at a resolution of 1080P, for example, as FULL HD image quality. In this way, the three cameras can record uniform images, for example, high-resolution images.

[0242] (O-2) In a viewer having the function of time-synchronizing and playing back images from cameras with different angles of view, such as the camera 402 of the photographing device 101R, the rear camera 102 for photographing inside the vehicle, and the photographing device 101R, it is desirable to provide a viewer having a button that adjusts the angle of view when playing back images from other cameras to the angle of view of the camera with the narrowest angle of view.

[0243] If the viewer assumes that camera 402 (for photographing the rear of the vehicle) has the narrowest angle of view among these three cameras, then by pressing a button, the angle of view of the images from photographing device 101F and rear camera 102 for photographing the interior of the vehicle can be changed to the angle of view of camera 402. This solves the problem of it being difficult to understand the correspondence between the ranges visible by each camera.

[0244] (O-3) It is preferable that the viewer is a viewer having a function to change the angle of view of the video being played, and a viewer having a function to change the angle of view of the video from another camera to the angle of view of the video from the camera that is currently set to the narrowest angle of view with one button.

[0245] (O-4) In the above explanation, the word "viewer" may be read as a rearview mirror type drive recorder body that is attached to a rearview mirror for use, and (O-1) to (O-3) are applicable to various devices.

[0246] (P) Changing the settings for taking or recording images triggered by the opening or closing of the vehicle's door The following configuration may be adopted for changing the settings for capturing or recording video using the opening / closing operation of the vehicle door as a trigger, as described above: The settings for capturing or recording video include whether to record or not, i.e., which camera to record images from, and the quality of the recording, i.e., which camera to record from and at what quality, such as frame rate and resolution.

[0247] Situations include (P-1) opening and closing the rear hatch, (P-2) the vehicle is moving, (P-3) when you want to record the front of the vehicle, (P-4) when the vehicle engine is on or the accessory power is on, (P-5) when the vehicle engine is off or the accessory power is off but you want to record within a specified time period, (P-6) when recording for a specified extended period after the vehicle engine is off or the accessory power is off, and (P-7) when the vehicle engine is on or the accessory power is on but you want to record within a specified time period.

[0248] The control unit 303 may have a function for setting the image capture or recording for each of all or some of these situations. The control unit 303 may also have a function for setting a common setting for two or more of these situations. The image capture or recording setting may be set manually by the user, or may be set in advance or after product shipping by a method other than manual setting by the user (for example, automatic setting).

[0249] When the settings are made using a method other than manual user settings, the settings for capturing or recording video may be made, for example, as follows. In the following description, if there are two cameras that record at a relatively high quality, or if there are two cameras that record at a relatively low quality, the quality should be the same, but may be different. In the following, the control unit 303 is responsible for control of recording, such as determining the quality, but this may also be performed by a control unit of the image capturing device 101F (for example, the control unit 5511 described below) or another entity.

[0250] (P-1) Rear hatch opening and closing operation When the rear hatch is open, there may be a situation where you want to check the area around the rear hatch later, so it is preferable to record the area around the rear hatch with high quality. To achieve this, it is preferable to record with a relatively high quality using the rear camera 102 for shooting inside the vehicle. For example, when the rear hatch is open, it is preferable to have the camera 101F capture the front and the camera 101R capture the rear at a relatively low quality (e.g., coarse), and the rear camera 102 for shooting inside the vehicle capture at a relatively high quality (e.g., detailed). If the cameras 402 of the camera devices 101F and 101R record at a lower quality than the rear camera 102 for shooting inside the vehicle, the situation around the vehicle can be checked over a wider area.

[0251] At least one of the image capturing devices 101F and 101R may stop recording. Since the camera 402 of the image capturing device 101R is a device that captures images of the exterior rear of the vehicle, the image capturing may continue to be recorded at a lower quality than the rear camera 102 for capturing images inside the vehicle. Alternatively, since the image capturing direction of the image capturing device 101R is approximately directly above, the image capturing by the camera 402 of the image capturing device 101R may be stopped, and the image capturing device 101F may continue recording.

[0252] (P-2) While the vehicle is moving While the vehicle is traveling, there is a high need to later check the outside of the vehicle, in this embodiment, the front, rear, or both of the vehicle, so it is preferable to record the outside of the vehicle with high quality. To achieve this, it is preferable to record images captured by the cameras 402 of the image capturing devices 101F and 101R with relatively high quality. For example, while the vehicle is traveling, it is preferable to have the image capturing device 101F capture the front and the camera 402 of the image capturing device 101R capture the rear with relatively high quality (e.g., detailed).

[0253] Typically, the rear hatch is closed while the vehicle is traveling. Therefore, the rear camera 102 for capturing images of the interior of the vehicle may capture images of the interior of the vehicle at a relatively low quality (for example, coarse). The control unit 303 may determine that the vehicle is traveling based on the vehicle speed. For example, the control unit 303 may determine the vehicle speed based on changes in position information acquired using the GNSS sensor 307, and may determine that the vehicle is traveling if the vehicle speed is equal to or greater than a predetermined speed (for example, 8 km). Furthermore, if information can be acquired from the vehicle, the control unit 303 may acquire the vehicle speed via the OBDII connector.

[0254] (P-3) When you want to take a photo of the front of the vehicle The front side of the vehicle may be the front area inside the vehicle cabin, the front area outside the vehicle, or an area including both. For example, there may be cases where you want to check the situation around the driver's seat and passenger seat of the vehicle, or / and the situation in the front area outside the vehicle later. Therefore, it is desirable to record the front side of the vehicle with high quality. Therefore, if you want to record the front area inside the vehicle cabin with relatively high quality, it is desirable that the interior rear camera 102 record images with higher quality than the cameras 402 of the image capturing devices 101F and 101R.

[0255] When it is desired to record the area outside the vehicle forward with relatively high quality, the image capturing device 101F may record images of higher quality than the rear camera 102 for capturing images inside the vehicle and the camera 402 of the image capturing device 101R. When it is desired to record the area outside the vehicle forward with relatively high quality, the image capturing device 101F and the rear camera 102 for capturing images inside the vehicle may record images of higher quality than the camera 402 of the image capturing device 101R. The image capturing device 101R may stop recording, but may record at a relatively low quality.

[0256] (P-4) When the vehicle engine is on or the accessory power is on When the vehicle engine is on or the accessory power is on, and the vehicle is not running, it is likely that the vehicle is stopped, for example, parked. Therefore, it is desirable to record the exterior of the vehicle with high quality. To achieve this, it is desirable to record images captured by the cameras 402 of the image capturing devices 101F and 101R at a relatively high quality. The rear camera 102 for capturing images inside the vehicle interior may stop recording, but it is desirable to capture images of the interior of the vehicle at a relatively low quality.

[0257] (P-5) When the vehicle engine or accessory power is off, but recording is to be performed within the specified time period When recording within a specified time period while the vehicle engine or accessory power is off, as described above, there may be cases where the driver wants to check the situation around the rear hatch later for purposes such as transportation, so it is preferable to record high-quality images of the area around the rear hatch. To achieve this, it is preferable to record images captured by the rear camera 102 for capturing images inside the vehicle at a relatively high quality. For example, it is preferable that the rear camera 102 for capturing images inside the vehicle captures images at a relatively high quality, and the image capturing device 101F and the camera 402 record images at a relatively low quality. Recording of images captured by at least one of the image capturing device 101F and the camera 402 may be stopped.

[0258] Also, since the vehicle engine or accessory power is off, in order to reduce power consumption, it is possible to achieve lower power consumption by recording at a lower quality for at least one camera than in the case described in (P-4).

[0259] (P-6) In the case of extended recording for a specified time after the vehicle engine or accessory power is turned off Even when recording for a specified extended time after the vehicle's drive source or accessory power is turned off, as mentioned above, there may be cases where you want to check the situation around the rear hatch later for purposes such as transportation, so it is advisable to record the area around the rear hatch with good quality. The recording control for this may be the same as (P-5), for example, but may be different.

[0260] (P-7) When the vehicle engine or accessory power is on, but recording is performed within a specified time period Even when recording within a designated time period while the vehicle engine or accessory power is on, as mentioned above, there may be cases where the driver wants to check the situation around the rear hatch later for purposes such as transportation, so it is advisable to record the area around the rear hatch with high quality. In this case, the recording control may be the same as in (P-5), for example, but may be different. Alternatively, if the vehicle's drive source or accessory power is on during the designated time period, power can be supplied from the vehicle, so at least some of the cameras may record with higher quality than in (P-6).

[0261] In this way, the control unit 303 may have a function to vary the video quality according to various combinations. Note that when two or more of the multiple situations described in (P-1) to (P-7) are met simultaneously, the control unit 303 may perform control according to one of the situations, with or without a user specification. Alternatively, when two or more situations are met simultaneously, the control unit 303 may prioritize recording of higher quality.

[0262] (Q) It is preferable that a microphone for picking up sound inside the vehicle cabin is mounted on the photographing device 101F, which is the front body, and one or both of the photographing device 101R and the rear camera 102 for photographing inside the vehicle cabin. The control unit 303 may then set the sound to be recorded for each photographing device or each camera. The setting for the sound to be recorded may be set manually by the user, or may be set in advance or after the product shipping stage by a method other than manual setting by the user (for example, automatic setting). When setting in advance or after the product shipping stage by a method other than manual setting by the user, the setting may be performed, for example, as follows.

[0263] (Q-1) When the control unit 303 determines that a person is in the vehicle, it is preferable to set the recording to be of higher quality than when it is not determined that a person is in the vehicle or when it is determined that no person is in the vehicle (for example, it is preferable to record in detail). In this way, human voices can be recorded with high quality, making it easier to check the content of the voices later.

[0264] In the configurations of (Q-2) and (Q-1), the control unit 303 may be configured to record higher quality audio when it determines that a person is present within a predetermined range from the position of the camera or microphone than when it determines that no person is present within the predetermined range or when it determines that no person is present. In this way, even if the vehicle interior is large, high-quality audio recording can be performed in a situation where human voices can be picked up.

[0265] (Q-3) A program such as a viewer that has the function of playing back video should have the function of simultaneously recording the audio input from all microphones and selecting which microphone to use to output the recorded audio when playing back the recorded video. The selection may be manual by the user, or it may be selected without manual selection (for example, automatic selection). In the latter case, the program should select and output the audio from the microphone that has produced the highest quality recording.

[0266] (Q-4) When the rear hatch is open, recording should be centered on the microphone of the camera 402 of the photographing device 101R or the microphone of the rear camera 102 for photographing the interior of the vehicle, and when the rear hatch is closed, particularly when the vehicle is moving, recording should be centered on the microphone of the photographing device 101F. While recording at the center may be performed using only that microphone, recording may also be performed using other microphones, and the recording level may be set so that the audio level of the center microphone is relatively high. Note that although the adjustment of the recording level has been described, the level may also be adjusted during playback.

[0267] FIG. 12 is an example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture. The mounting fixture 901 has a first mounting portion 902 , a second mounting portion 903 , and a fixing means 904 .

[0268] The first attachment portion 902 has a C-shaped cross section, and may be configured to clamp the photographing device 101R based on the elastic force of the C-shaped portion. The second mounting portion 903 is formed in a pipe shape, and its inner diameter is formed to be slightly larger than the outer diameter of the interior-of-vehicle interior rear camera 102. The direction of the interior-of-vehicle interior rear camera 102 inserted into the second mounting portion 903 can be easily adjusted in the circumferential direction and the up-down direction, and the user can fix the interior-of-vehicle interior rear camera 102 in a desired state using fixing means 904 such as a screw.

[0269] In this configuration, the user can very easily attach and fix the rear camera 102 for capturing images inside the vehicle to the existing image capturing device 101R. Furthermore, since the mounting portion 902 is formed in a pipe shape, it is possible to change the orientation of the mounting fixture 901 by rotating the mounting fixture 901 around the axis of the pipe, for example, when the photographic device 101R is attached to the rear window. Furthermore, since the attachment portion 903 is formed in a pipe shape, the photographing direction can be changed by rotating the rear camera 102 for photographing the interior of the vehicle relative to the attachment 901 about the axis of the pipe.

[0270] FIG. 13 is another example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture. The illustrated pipe-shaped mounting fixture 1001 has three joint portions (openings) 1002, and it is advantageous to set the outer diameter of at least one end of each rear camera, which has an arc portion, to match or be slightly larger than the inner diameter of the pipe joint so that a cylindrical camera can be inserted into each of these joint portions 1002. This allows the rear camera to fit into the hole. For example, two rear cameras can be attached to a pipe-shaped mounting fixture with two holes, and two or three rear cameras can be attached to a pipe-shaped mounting fixture with three holes. As in the case of FIG. 12, the position and direction of the rear camera 102 for photographing the interior of the vehicle can be easily adjusted in the circumferential direction and the up-down direction.

[0271] 13, a so-called pipe joint method can be used to attach the interior-of-vehicle rear camera 102 to the image capturing device 101R, allowing the user to easily and intuitively attach the additional interior-of-vehicle rear camera 102 to the information processing system. Also, by changing the relative position of the additional interior-of-vehicle rear camera 102 with respect to the image capturing device 101R or the capturing direction of the additional interior-of-vehicle rear camera 102, it becomes possible to easily adjust the camera so that a desired image can be captured.

[0272] A cable that provides electrical connection between the cameras may be housed inside the pipe-shaped mounting fixture 1001. This makes it possible to prevent the cable from coming loose due to accidental contact, etc., compared to a configuration in which the cable is exposed. The fixture 1001 can be designed with various shapes and positional relationships of the openings, and an extension pipe can also be used depending on the desired installation state. A preferred shape of the fixture 1001 is shown in FIG.

[0273] The pipe-shaped mounting fixture 1001 may be configured to be fixed as is with screws or adhesive, but it is better to configure it so that a part (for example, a ball joint) is provided that connects to the mounting member from a place other than the opening for the joint. For example, a ball joint can be provided at the part with the screw in the figure.

[0274] FIG. 14 is yet another example of an explanatory diagram illustrating the connection of the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle using a mounting fixture. 14 is a pan-tilt-shaped mounting fixture 1101, and it is particularly advantageous to have a configuration in which the photographing device 101R and the rear camera 102 for photographing inside the vehicle are connected to each other via the pan-tilt-shaped mounting fixture 1101. In this way, it is possible to have a camera configuration in which only the photographing device 101R and the rear camera 102 for photographing inside the vehicle are rotated around.

[0275] The mounting fixture 1101 has dowel screw portions 1102 and 1103, and the photographing device 101R, the vehicle interior photographing rear camera 102, and the mounting fixture 1101 can be connected to each other by screwing the dowel screw portions 1102 and 1103 into dowel screw holes 1105 and 1106 provided in the housings of the photographing device 101R and the vehicle interior photographing rear camera 102. After adjusting the positional relationship between the photographing device 101R and the vehicle interior photographing rear camera 102, the two can be fixed together by tightening a fixing portion 1104.

[0276] This embodiment makes it easy to adjust the shooting direction by rotating the vehicle interior rear camera 102, and to adjust the shooting direction in the elevation angle direction or depression angle direction. Even if the vehicle interior rear camera 102 itself has a function for adjusting the shooting direction (lens direction), the adjustable range of the shooting direction can be further expanded by using the mounting fixture 1101. Furthermore, even when the rear camera 102 for photographing inside the vehicle interior is moved from an interior photographing position to an exterior photographing position such as the rear or left or right, the mounting state of the photographing device 101R and the rear camera 102 for photographing inside the vehicle interior can be stably maintained.

[0277] FIG. 15 shows an example of a variation of the mounting fixture. It is preferable to use the mounting fixtures 1201-1208 shown in Fig. 15 to provide a configuration in which multiple cylindrical cameras can be simply and easily attached and combined. In particular, by standardizing part of the outer shape of the cylindrical cameras and the shapes of the openings 1210 and 1220 of the mounting fixtures 1201-1208, it is possible to provide a wide range of expandability to the camera system at low cost. Some of the pipe-shaped mounting fixtures 1201-1208 have circular openings 1210 where the cylindrical cameras are attached, while others have openings on the sides of the pipe, such as opening 1220 (for example, with a C-shaped cross section).

[0278] The inner diameter of the openings 1210, 1220 of the pipe-shaped mounting fixtures 1201 to 1208 should be the same as the outer diameter of the cylindrical camera, or slightly larger. Furthermore, the circular openings 1210 of the fittings 1201 to 1208 may be shaped like a cutout made in a part of a pipe. In the structure using pipe-shaped fixtures 1201 to 1208, the angle and position of the attached camera can be adjusted and can be fixed after adjustment.

[0279] By using pipe-shaped mounting fixtures 1201 to 1208 as so-called pipe joints, an easy and simple configuration and good expandability can be applied to mounting an in-vehicle camera. Furthermore, it is preferable to have a cylindrical camera or drive recorder attached to (or integrated into) the pipe cap for the pipe joint. By standardizing the pipe joint standards in this way, it becomes possible to easily install a camera in a factory, exhibition, etc. by simply removing the cylindrical drive recorder or rear camera 102 for recording inside the vehicle and inserting it into a shelf, fence (enclosure), cart, or other pipe that is made up of pipes. In addition, there is no need to attach a pipe cap to the location where the camera is installed.

[0280] As described above, a configuration in which the photographing device 101R and the rear camera 102 for photographing inside the vehicle interior both have cylindrical housings is advantageous because it makes it easier and simpler to attach the rear camera 102 for photographing inside the vehicle interior to the photographing device 101R. The mounting fixture has multiple joint parts (openings) depending on its type, and the outer diameter of at least one end of each rear camera, which has an arc portion, is set to match or be slightly larger than the inner diameter of the mounting fixture so that different rear cameras can be inserted into each of the multiple joint parts (openings).

[0281] FIG. 16 shows an example of another variation of the mounting fixture. FIG. 16(a) illustrates an alternative fixture 1301 for connecting two cameras. FIG. 16(b) illustrates yet another fixture 1311 for connecting two cameras. A cylindrical camera can be attached to each of the openings of the other fixture 1301 and the further fixture 1311.

[0282] Another mounting fixture 1301 in FIG. 16(a) is made up of a first fixing part 1302, a second fixing part 1303, and a connection joint 1310 that rotatably connects them. FIG. 16(a-1) shows a state in which the openings of the first fixing portion 1302 and the second fixing portion 1303 face in the same direction. FIG. 16( a - 2 ) shows another mounting fixture 1301 of FIG. 16( a - 1 ) rotated 90° around a rotation axis 1307 .

[0283] The connection joint 1310 has a screw or the like attached in the direction of the rotation axis 1307, and by tightening the screw, the first fixed part 1302 and the second fixed part 1303 are fixed so that they cannot rotate relative to each other. First fixing portion 1302 and second fixing portion 1303 are each composed of two fixing members 1305a, 1305b, 1306a, and 1306b, and a pivotal connector 1304 that pivotally connects them to each other. The camera can be attached to first fixing portion 1302 and second fixing portion 1303 by loosening the screws of pivotal connector 1304, opening fixing members 1305a, 1305b, 1306a, and 1306b, attaching the camera, and then closing fixing members 1305a, 1305b or 1306a and 1306b again.

[0284] Yet another mounting fixture 1311 in FIG. 16(b) is made up of a first fixing part 1312, a second fixing part 1313, and a rotational connection part 1314 that rotatably connects them. FIG. 16(b-1) shows a state in which the openings of the first fixing portion 1312 and the second fixing portion 1313 face in the same direction. Fig. 16(b-2) shows a state in which the first fixing portion 1312 of the mounting fixture 1311 in Fig. 16(b-1) has been rotated around the rotating connector 1314. The user can fix the first fixing portion 1312 and the second fixing portion 1313 so that they cannot rotate relative to each other by screwing the rotating connector 1314 at a desired angle.

[0285] When the pivoting connection part 1314 is opened, one of the cameras that was facing in the same direction will face the opposite direction by 180 degrees. Therefore, for example, at one time, one camera can be facing the rear of the vehicle and one camera can be facing the interior of the vehicle, and at other times, two cameras can be installed facing the rear of the vehicle. The cameras can be attached and fixed to the openings of the first fixing part 1312 and the second fixing part 1313 in the manner described in connection with FIG.

[0286] FIG. 17 shows an example of a rear camera 102 for photographing the interior of a vehicle, which has a pin plug. In the embodiment of FIG. 17, the rear camera 102 for photographing the interior of the vehicle has a pin plug 1401, which can be inserted into a plug jack of a photographing device 101R such as a drive recorder or other additional camera.

[0287] An O-ring 1403 is attached to the pin plug 1401, and when the pin plug 1401 is inserted into the pin jack, the frictional force of the O-ring 1403 ensures that the rear camera 102 for photographing the interior of the vehicle is securely fixed in place. It is more preferable that the rear camera 102 for taking pictures inside the vehicle interior has a triangular protrusion on the inner circumference as the anti-rotation structure 1402 so that it can engage with, for example, the mating surface of the photographing device 101R. If the anti-rotation structure 1402 is configured to engage with the photographing device 101R at a predetermined angle such as 0°, 30°, 45°, 90°, or 180°, the angle of the rear camera 102 for taking pictures inside the vehicle interior can be easily changed, which is convenient.

[0288] Furthermore, the rear camera 102 for photographing the interior of the vehicle may be configured to have a triangular protrusion on the outer circumference as a rotation prevention structure 1405, for example, on the joint surface with another additional camera. These configurations are advantageous in preventing unwanted rotation due to vibrations.

[0289] In addition, when another rear camera 102 for interior photography of the same type is attached to the rear camera 102 for interior photography of the vehicle, it is advantageous if the anti-rotation structure 1402 of the rear camera 102 for interior photography and the anti-rotation structure 1405 of the other rear camera 102 for interior photography of the vehicle engage with each other.

[0290] FIG. 18 is an example of an enlarged view of another anti-rotation structure for the camera of FIG. FIG. 18 shows an enlarged view of the periphery of the rotation prevention structure when the rear camera 102 for photographing the interior of the vehicle shown in FIG. 17 is attached to the photographing device 101R. In FIG. 18, the camera rotation prevention structure 1402 in FIG. 17 is illustrated as a rotation prevention structure 1501 on the side of the rear camera for photographing inside the vehicle, and further illustrated as a rotation prevention structure 1502 on the side of the photographing device 101R that engages with it.

[0291] The rotation prevention structures 1501, 1502 on the rear camera side for photographing the interior of the vehicle, which prevent rotation, consist of a sawtooth-shaped photographing equipment side uneven portion 1504 formed on the photographing equipment 101R side and a camera side claw portion 1503 made of an elastic material formed on the rear camera 102 side for photographing the interior of the vehicle and engaged with the photographing equipment side uneven portion 1504.The camera side claw portion 1503 engaged with the recess of the photographing equipment side uneven portion 1504 is held in the recess unless a rotational torque greater than a predetermined value is applied to the rear camera 102 for photographing the interior of the vehicle, and when a rotational torque greater than the predetermined value is applied, the camera side claw portion 1503 can move from the recess of the photographing equipment side uneven portion 1504 over the protrusion to the adjacent recess. This structure is suitable for easily holding the rear camera 102 for photographing the interior of the vehicle at any rotational position.

[0292] In this way, it is possible to securely fix the interior-of-vehicle rear camera 102 and easily change the shooting direction by rotating the interior-of-vehicle rear camera 102. In this case, the interior-of-vehicle rear camera 102 is held within a predetermined range of 360°, and the same subject captured by the photographing device 101R that photographs the rear of the vehicle interior can be photographed at different angles.

[0293] FIG. 19 shows an embodiment in which the camera of FIG. 17 is attached to a cylindrical photographing device. When the rear camera 102 for photographing the interior of the vehicle is rotated with a force equal to or greater than a certain level from the attached state shown in FIG. 19(a), it becomes the state shown in FIG. 19(b).

[0294] In the embodiment of Fig. 19, the interior rear camera 102 is connected to the photographing device 101R that photographs the area behind the vehicle in the manner described in relation to Figs. 17 and 18. In Fig. 19(b), the interior rear camera 102 is fixed with the lens 1601 facing rearward, similar to the photographing device 101R. Conversely, in Fig. 19(a), the lens 1601 (not shown) is facing the interior of the vehicle, and photographs the interior of the vehicle from the rear toward the front of the vehicle.

[0295] The rear camera 102 for photographing inside the vehicle interior has the anti-rotation structure 1602 described in relation to Figures 17 and 18. This anti-rotation structure 1602 is preferably configured to allow rotation between the rear camera 102 for photographing inside the vehicle interior and the photographing device 101R and to achieve secure fixation.

[0296] The photographing device 101R in FIG. 19 includes a first housing 1603, a second housing 1604, a lens 1605, a bracket 1606, a non-slip surface 1607, support legs 1608, a mounting surface 1609, a mounting plate 1610, and the like. If the photographing device 101R and the rear camera 102 for photographing the interior of the vehicle have cylindrical housings with the same outer diameter, the sense of unity in appearance is improved, which is preferable in terms of appearance. Although the configuration has been described in which the rear camera 102 for photographing inside the vehicle interior is attached externally to the photographing device 101R, the camera shown in FIG. 19 can also be configured as a single device that is integrally formed in advance and has a changeable lens direction.

[0297] A similar rear camera 102 for photographing inside the vehicle can also be attached to the front photographing device 101F. Also, the rear camera 102 for photographing inside the vehicle according to Figures 17 to 19 may be configured so that it can be attached to various types of cameras, such as digital cameras and cylindrical cameras, that have a pin jack into which a pin plug can be inserted. As mentioned above, the in-car camera can be connected to either the front or rear camera, and it is preferable that the connected camera is detected so that image adjustments can be set to match the in-car image from the front or the in-car image from the rear. In particular, it is advantageous if the rear camera 102 for capturing images inside the vehicle outputs data that has been subjected to IPS processing or compression processing, but it may also output RAW data as is.

[0298] It is advantageous if the pin shape and pin arrangement of the pin jack for the in-vehicle camera are compatible with the pin shape and pin arrangement of the pin jack for an existing rear camera. In this configuration, a user can easily install the in-vehicle rear camera 102 via the pin jack of the existing photographing device 101R already installed in the vehicle. In addition, it is preferable to select a narrower angle of view for the rear camera 102 for capturing images of the interior of the vehicle than the angle of view for the rear-facing camera of the image capturing device 101R. In this configuration, the state inside the vehicle interior can be captured more clearly.

[0299] In the above configuration, it is even more advantageous if the drive recorder is provided with plug jacks into which the compatible connectors can be inserted on different surfaces of the drive recorder, particularly different sides or bottom surfaces, and is capable of recording all of the images from cameras inserted into the compatible plug jacks on the different surfaces in a time-synchronized manner so that they can be played back. With such a configuration, users can easily achieve the desired expansion of the shooting range, shooting purpose, and control functions depending on the total number of cameras to be added, the shooting direction, angle of view, shooting method, and other camera specifications.

[0300] 17 to 19, the rear camera 102 for taking photographs inside the vehicle interior has a male round pin plug protruding from the main body, but this may be a normal female and a male-male round pin plug may be used to connect the rear camera 102 for taking photographs inside the vehicle interior to other cameras such as the main camera, the photographing device 101R. In this configuration, it is preferable that the male-male round pin plug serves the roles of mechanical connection and fixation in addition to electrical connection between the rear camera 102 for taking photographs inside the vehicle interior and the photographing device 101R.

[0301] The use of the above-described Z-shaped male-male round pin plug allows the interior camera 102 to be rotated and positioned on a larger rotational radius track relative to the image capture device 101R. Because the interior camera 102 is lightweight, additional mounting fixtures other than the plug are not required to mount, position, and secure the interior camera 102 to the image capture device 101R.

[0302] Furthermore, the configuration using a male-male round pin plug prevents the pin plug from protruding from the interior camera 102, preventing damage during transportation and storage due to electrical shocks such as static electricity at the terminals and physical shocks such as bending of the terminals. It also prevents the end of the connecting cable from becoming a female connector and becoming larger when the interior camera 102 is placed separately without being directly connected to another camera. It also makes it easy to use the interior camera 102 as an exterior camera.

[0303] Another possible connection and fastening device is a fastener that allows the terminals of the image capture device 101R and the in-car camera to be connected in positions that are not coaxial (collinear), but that can be rotated and (semi-)fixed. This offsets the axes of the I-shaped ends of the cable, allowing the I-plugs to be positioned parallel to each other and reducing the distance between them. In other words, it is desirable to offset the axes of the male parts, which are usually designed to be aligned with each other. For example, the male round pin plugs could be configured in a Z-shape, handle shape, or crank shape. These connection and fastening devices can reduce the distance between the in-car camera and the main camera. A similar configuration can also be used for the I-portion of the L-shaped terminal.

[0304] FIG. 20 is an explanatory diagram illustrating an example of a procedure for attaching the camera of FIG. 17 to the digital camera type photographing device 101R. 20(a) shows a state in which imaging device 101R, which is a digital camera type drive recorder that captures images of the rear of a vehicle, is installed inside the vehicle and connected to a power source and a front camera by cable 1703. It is preferable to configure cable 1703 as a single cable that serves the roles of supplying and receiving power and transmitting and receiving data, and such a configuration is advantageous in that it allows for simplified wiring.

[0305] Figure 20(b) shows the state just before the connection between the photographing device 101R, cable 1703, power supply and front camera in Figure 20(a) is disconnected and the rear camera 102 for photographing inside the vehicle according to Figure 17 is installed between the photographing device 101R and cable 1703.

[0306] The pin plug 1705 and pin jack 1706 of the rear camera 102 for photographing inside the vehicle correspond to the standards of the pin jack 1704 of the photographing device 1701 and the pin plug 1707 of the cable 1703, respectively. In FIG. 20, the pin plug 1707 of the cable 1703 is shown with a larger outer diameter than the cable 1703, but this is simply for ease of viewing, and the outer diameter of the pin plug 1707 may be smaller than the outer diameter of the cable 1703.

[0307] FIG. 20(c) shows a state in which an additional rear camera 102 for capturing images inside the vehicle interior according to FIG. 17 is attached between the image capturing device 1701 and the cable 1703. The image capturing device 101R and the additional rear camera 102 for capturing images inside the vehicle interior are powered by a power source via the cable 1703, and are capable of transmitting and receiving captured image data with the front camera. In other words, the cable 1703 not only transmits digital data but also supplies power to each camera. Therefore, adding the rear camera 102 for capturing images inside the vehicle interior does not require new wiring, and simple routing is possible using a single cable. It is particularly preferable to use a coaxial cable for this cable.

[0308] 20(c), the rear camera 102 for taking pictures inside the vehicle can be easily attached to an existing photographing device 101R. Since the existing photographing device 101R generally has a standardized pin jack, by providing a standardized pin plug corresponding to the rear camera 102 for taking pictures inside the vehicle, various photographing devices 101R can be adopted in this information processing system.

[0309] 20(c), only one additional rear camera 102 for photographing the interior of the vehicle is attached between the image capturing device 101R and the cable 1703, but it is also possible to attach multiple additional rear cameras 102 for photographing the interior of the vehicle in series. Furthermore, a user may attach multiple additional rear cameras 102 for photographing the interior of the vehicle using a branch cable, and such a configuration will be described in FIG.

[0310] In the configuration of FIG. 20(c), the photographing direction of the photographing device 101R is set so as to photograph the outside rear of the vehicle, and the rear camera 102 for photographing the inside of the vehicle is set so as to photograph the inside of the vehicle. The rear camera 102 for photographing inside the vehicle interior is preferably configured to receive video data from the front camera via the cable 1703, and send both the received video data from the front camera and video data photographed by the rear camera 102 for photographing inside the vehicle interior to the photographing device 101R. As described above, it is even more preferable that the cable 1703 can also be used to supply power to the photographing device 101R and the rear camera 102 for photographing inside the vehicle interior.

[0311] The data sent from the front camera and the rear camera 102 for taking photos inside the vehicle interior to the photographing device 101R may be data that has been processed or compressed by an ISP (Image Signal Processor) in each of the front camera and the rear camera 102 for taking photos inside the vehicle interior, or it may be RAW data. In a configuration in which the front camera and the rear camera 102 for capturing images inside the vehicle transmit RAW data as is, the photographing device 101R may further be configured to perform ISP processing and compression processing on the RAW data.

[0312] The rear camera 102 for photographing inside the vehicle interior may be configured to receive video data from the photographing device 101R and send to the front camera both the received video data from the photographing device 101R and the video data photographed by the rear camera 102 for photographing inside the vehicle interior. In such a configuration, the data sent from the photographing device 101R and the rear camera 102 for photographing inside the vehicle interior to the front camera is preferably data that has been processed or compressed by an ISP in the photographing device 101R and the rear camera 102 for photographing inside the vehicle interior, respectively, but may also be RAW data.

[0313] In a configuration in which the photographing device 101R and the rear camera 102 for photographing inside the vehicle transmit RAW data as is, the front camera can also perform ISP processing and compression processing on the RAW data.

[0314] In a configuration in which the image capturing device 101R, the in-vehicle rear camera 102, and the front camera each perform IPS processing and compression processing, ISP processing and the like are not concentrated on one camera, so the control unit 303 of the image capturing device 101R, for example, can allocate more processing power to the image analysis and situation determination described in relation to Fig. 4. This allows image analysis and situation determination to be performed faster or with higher accuracy.

[0315] In a configuration in which the rear camera 102 and front camera for taking pictures inside the vehicle send RAW data to the photographing device 101R, and the photographing device 101R performs all IPS processing and compression processing of the video data from the individual cameras that make up the in-vehicle camera system, it is possible to make the photographing device 101R, which is the main unit, high-performance and configure the other units with inexpensive units, allowing the entire system to be manufactured inexpensively.

[0316] In the above configuration, it is preferable that the video recording medium be selectable from three options: front only, front and rear, and rear only. In a configuration in which recording media are provided on both the front and rear, since the recording media are separated into both, even if one of the cameras is damaged in the event of a major accident, it is possible to prevent any data from being left behind.

[0317] When there are two recording media, one for the front and one for the rear, it is preferable that the frames are synchronized with each other by communication, electrical signals, etc. Note that the synchronization in this configuration may be off to some extent. If there are two recording media, front and rear, it is a good idea to have a function to synchronize the time of the images on each side. If there are two recording media, a front and a rear, it is advisable to detect the event trigger on both.Furthermore, in this configuration, it is advisable to design the event trigger so that it is determined by conditions such as OR and AND.

[0318] FIG. 21 shows an embodiment in which two cameras shown in FIG. 17 are attached to a digital camera type photographing device using a branching cable 1804. In the embodiment of Fig. 21, it is possible to transmit images from two or more cameras to another device. In the embodiment of Fig. 21, the other device is, for example, an image capturing device 101R, which is a digital camera type drive recorder. Two rear cameras 102 for capturing images inside the vehicle are connected to the image capturing device 101R via a bifurcated branch cable 1804. The rear cameras 102 for photographing the interior of the vehicle are attached to desired positions of the vehicle, particularly to the rear area of ​​the vehicle, by position fixing means (not shown).

[0319] 21 is provided with two rear cameras 102 for photographing the interior of the vehicle, i.e., a first additional camera 1802a and a second additional camera 1802b. For example, it is advantageous to configure the first additional camera 1802a to photograph the interior of the vehicle from the rear toward the front, and the second additional camera 1802b to photograph the side of the vehicle from the rear of the vehicle. The imaging device 1801 may have an additional pin jack, and in such a configuration, two additional cameras can be connected using two branching cables. The branch cable is not limited to a two-branched cable, and may be branched into three or more branches.

[0320] In this embodiment, the additional rear camera 102 for photographing inside the vehicle interior may be configured to pass through to the outside, for example, the video data photographed by the rear camera 102 for photographing inside the vehicle interior itself and the video data from the photographing device 101R. The first additional camera 1802a may be configured to pass through the power supply from the cable 1803 and provide power to the second additional camera 1802b and the photographing device 101R via a branch cable 1804.

[0321] The rear camera 102 for shooting inside the vehicle interior can send video data captured by the rear camera 102 for shooting inside the vehicle interior itself, video data captured by the photographing device 101R, and power to the outside via a cable, and can also send video data sent from the outside via a cable, video data captured by the rear camera 102 for shooting inside the vehicle interior itself, and power to the photographing device 101R.

[0322] If the rear camera 102 for photographing the interior of the vehicle is an IR camera that lights up the IR LED when photographing the interior of the vehicle, it can clearly photograph the interior of the vehicle even at night, etc. If the camera for photographing the interior of the vehicle is an IR camera, it can also be used effectively for photographing the exterior of the vehicle / at night. In the above configuration, it is more preferable to use a lens that automatically inserts / removes an IR filter.

[0323] The camera for capturing images inside the vehicle can be an RGB-IR camera that can be configured to switch between RGB and IR modes as needed. This allows the camera for capturing images in IR mode in low-light conditions, such as at night, and to switch to RGB mode in conditions with sufficient light.

[0324] The rear camera 102 for photographing inside the vehicle is an in-vehicle photographing camera equipped with a camera that can also be directed outside the vehicle and a floodlight whose light projection direction is linked to adjustment of the camera direction, and is preferably configured so that when the camera is directed outside the vehicle, the light from the floodlight can be electrically or mechanically blocked so that the light cannot be seen outside the vehicle. For example, it is preferable that the projector is mechanically or electrically blocked when the camera turns backward in conjunction with the rotation of the camera. This configuration, on the one hand, makes it possible to secure the amount of light necessary for photographing the interior of the vehicle, and, on the other hand, achieves a configuration in which the light from the floodlight cannot be seen from outside the vehicle when photographing the exterior of the vehicle.

[0325] Furthermore, it is preferable to have a configuration in which the angle of view of the captured image can be changed, for example, by changing the lens used in conjunction with the rotation of the camera. For example, it is possible to change from standard to medium telephoto (e.g., a focal length of 50mm to 80mm in 35mm film camera terms) when shooting inside the vehicle and ultra-wide angle (e.g., a focal length of 10mm to 20mm in 35mm film camera terms) when shooting outside the vehicle, or to change from wide angle (e.g., a focal length of 24mm to 28mm in 35mm film camera terms) when shooting inside the vehicle and ultra-wide angle (e.g., a focal length of 10mm to 20mm in 35mm film camera terms) when shooting outside the vehicle.

[0326] The control unit 303 can execute these measures as a rearward capture mode. The control unit 303 executes switching to the rearward capture mode, for example, in conjunction with the rotation of the camera, thereby electrically or mechanically blocking light from a projector such as an infrared LED, or changing a lens. The control unit 303 may also be configured to output, for example, information notifying the user that the mode has been switched to the rearward capture mode or that the mode is the rearward capture mode, to a user terminal.

[0327] The in-car camera may be configured with a filter in front of the lens to improve the image through the glass when facing the rear. It is more advantageous to use an ND filter, PL filter, or the included positive film as the filter.

[0328] The filter can be applied to the rear portion of the hemispherical camera mounted on the rear of the vehicle, or even just the rear half. This configuration can ensure the amount of light necessary for capturing images inside the vehicle, while preventing the light from the floodlight from being visible from outside the vehicle.

[0329] Furthermore, the following may be done: There is a demand for further strengthening of safety measures against accidents that occur when a vehicle is backing up. In this regard, the information processing system that captures images of the area behind the vehicle 10 from the rear side of the vehicle 10 with two cameras and obtains distances may be used as follows. The following configuration can be applied to an information processing system that captures images of the area behind the vehicle with the above-mentioned two cameras, the image capturing device 101R and the rear camera 102 for capturing images inside the vehicle (hereinafter, sometimes referred to as "cameras that capture images of the rear side"), and measures the distance to an object behind the vehicle, such as a vehicle behind.

[0330] 1. The information processing system may be a system in which a camera installed near the rear of the vehicle is connected directly or indirectly via wire or wireless to a device having a display installed in a position visible to the driver, and the image from the camera capturing the rear view is acquired and displayed so that it can be seen by the driver.

[0331] The installation location of the device having a display may be on the dashboard, the rearview mirror, the A-pillar, or the sun visor, and the device may have mounting portions that allow at least three of these mounting fixtures to be attached to the same device having the display. It is also preferable that the at least three attachments that can be attached to the device and the device are shipped packed together in a packaging material. It is advisable to package the camera, the device, and the at least three mounting fixtures.

[0332] It would be good to have a function to input images from multiple different cameras and select the images to be displayed on the display or combine them into one screen. It is desirable to have a function that, even when the image from any camera or a composite image is being displayed, can transition, by voice recognition or one touch, to a state where the image from the camera capturing the rear view is displayed on one screen, and the images from the other cameras are not displayed on that single screen. It is preferable to provide a function of storing the state before the transition and returning to the stored selection state of the camera before the transition after a certain period of time has elapsed.

[0333] The vehicle may be provided with a function of detecting that the vehicle has been switched to a reverse state, and when the vehicle has been switched to a reverse state, the display may be provided with a function of displaying an image of at least the area behind the vehicle from the rear side of the vehicle as a shooting range. However, at this timing, in many cases the image behind the vehicle is displayed on the screen only after the start of backing up is detected, so there are cases where it cannot be used to check the situation when attempting to back up.

[0334] Therefore, it is preferable to provide a function that causes the display to display an image covering at least the area behind the vehicle from the rear side of the vehicle when the driver inputs an instruction to reverse the vehicle. The driver's command to reverse the vehicle may be input by voice recognition or by detecting the pressing of a button. The lighting of the hazard lamps may be detected, or the gear operation by the driver may be detected. In particular, when either of these is detected and the vehicle is "stopped," it is desirable to have a function that causes the display to display an image that covers at least the area behind the vehicle from the rear side of the vehicle.

[0335] The detection of whether the hazard lights are on can be performed by detecting the driver's operation of the hazard light on button in the image captured by a camera facing the interior of the vehicle (from the rear), or by detecting the lighting status of the lamp indicating the lighting status of the hazard lights, for example, by detecting the lighting of the turn signal lamps on both the left and right sides. The gear operation by the driver may be detected by detecting the gear operation by the driver in an image captured by a camera facing the interior of the vehicle (from the rear). In many vehicles, the hazard lamp operation buttons, gear operation levers, etc. are located in the center of the left and right sides of the vehicle, so their status can be easily obtained using a camera that photographs the interior of the vehicle from the rear.

[0336] 2. The information processing system may have a function to detect when the vehicle is switched to reverse, and a function to record video of the interior of the vehicle including the time before the trigger corresponding to the switching to reverse, and video of the rear view taken from the rear including the time before the trigger corresponding to the switching to reverse. This video may be recorded in association with information that can be distinguished from other events when played back.

[0337] 3. The function for detecting that the vehicle has switched to a reverse state, such as in 1.2. above, may use one of the following methods: a method using a change in the direction of the acceleration in the forward and backward directions of the G-sensor; a method using a change in the direction of the vehicle's travel obtained from a GPS or the like; or a method using a change in the image from a camera; and it is particularly preferable to use at least two of these methods for detection. The camera image may be one that captures images in front, behind, or to the side, but it is particularly preferable to use a camera that includes images to the side in its capture area. In particular, it is preferable to use changes in the direction of image flow around the periphery of a hemispherical camera installed facing downward, and it is particularly preferable to use a configuration that detects changes in the direction of image flowing in the circumferential direction.

[0338] 4. The information processing system may have a function to determine the distance to an object behind the vehicle from the image captured by a camera installed on the rear side of the vehicle, and to issue an alarm based on that distance. The information processing system may be provided with a switching function that switches between estimating the distance to an obstacle behind the vehicle by distance estimation using a monocular camera when video footage of the rear of the vehicle is acquired from one camera, and estimating the distance to an obstacle behind the vehicle by distance estimation using a stereo camera when video footage of the rear of the vehicle is acquired from two cameras. This switching can be set in advance using a setting screen or a switch on the device, and the device can switch depending on this setting, but it is better to have a function that automatically switches depending on the state of the input video and the installation state of the device. The information processing system may be provided with a function for selecting whether to switch automatically or manually.

[0339] 5. The information processing system may have a function to discriminate and determine the next area from the image captured by a camera capturing the interior of the vehicle (for example, from the image area of ​​one frame, or from the area of ​​one still image). (1) The area of ​​the car's structure (seats, pillars, etc.), (2) The area outside the car (e.g., the area visible through the window), and (3) The area of ​​movable objects inside the car, such as people and luggage. The information processing system should have the function of detecting people inside the vehicle, retain the video when it is determined that there are no people inside, and determine the areas (1) and (2) from the difference between the retained videos. In particular, it is advisable to provide a function for detecting people inside the vehicle from sources other than camera images and to perform this processing based on this.

[0340] It is preferable that a thermal image sensor, a pyroelectric sensor, or the like is installed inside the vehicle, and the information processing system has the function of detecting whether or not there is a person inside the vehicle by utilizing the property that such sensors do not detect people through glass. The information processing system may have the function of using a pre-trained learning model to infer images from a camera capturing the interior of the vehicle, obtain segmentation results, and determine the areas (1) to (3) above. The information processing system should have an object detection function that is not segmentation, a function to distinguish whether a person or object is outside or inside the vehicle, and a function to decide whether to issue an alarm or change the nature of the alarm (e.g., the content or level of the alarm) depending on whether the person or object is inside or outside the vehicle.

[0341] 6. The information processing system may have a function to automatically estimate the shooting range of the rear camera, and in particular, a function to determine the rear range within a single image area and display the image of that range on the display.

[0342] 7. The information processing system should have a function to calculate the distance to the vehicle behind from video footage taken from the rear of the vehicle, and a function to issue an alarm using the distance to the vehicle behind when backing up (reversing), and a function to detect aggressive driving by the vehicle behind based on the distance to the vehicle behind when moving forward.

[0343] 8. The information processing system should have a function to calculate the distance to an object behind the vehicle from an image taken from the rear of the vehicle, and a function to change the calculation process for the distance to an object behind the vehicle when moving forward and when backing up (reversing). The information processing system may be provided with a function for changing the range of the detection range, such as setting the range of the detection range so that it is calculated accurately within a range of 0 to several meters when backing up, and setting the range of the detection range so that it is calculated accurately within a range of several meters to several tens of meters when moving forward.

[0344] The above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0345] Other vehicles and other locations where the present invention may be applied include: This system is used for shuttle buses for kindergartens and special needs schools. Teachers are on board to pick up children, but in the event of an accident or other problem, the driver, teachers, and children's movements can be monitored. This system is expected to help prevent accidents in which kindergarten children are left behind on the bus. Installed on vehicles that transport people, such as tourist buses, hotel shuttle buses, and trains. Inside an airplane. It's possible that the state of the passenger cabin cannot be seen from the cockpit. In such cases, if the images from cameras installed in the cabin and cargo hold could be viewed in the cockpit or galley (such as the cabin attendant area), it would be useful in preventing trouble or incidents before they occur. These applications are similar to those of surveillance cameras.

[0346] While many rear hatches are of the flip-up type that open upward, some vehicle models have rear hatches that swing open to the side or double doors. In the case of such rear hatches, the interior rear camera 102 photographs the rear of the vehicle 500 from the side when the hatch is open. The rear hatch may be opened and closed manually or electrically. Furthermore, in the above-described configurations, the description of the rear hatch as an example of a rear opening and closing part that opens and closes an opening at the rear of the vehicle may be appropriately interpreted as an opening and closing part that opens and closes an opening in the vehicle. For example, a similar configuration may be adopted for an opening and closing part provided in a location other than the rear of the vehicle.

[0347] Incidentally, as already described, "processing video data inside the vehicle captured from the rear of the vehicle toward the front of the vehicle, for example, processing video data inside the vehicle captured from the rear of the vehicle toward the front of the vehicle and video data outside the vehicle captured from the rear of the vehicle to the outside of the rear of the vehicle." Also, as can be understood from the description of the embodiments so far, the imaging device provided on the rear side of the vehicle can be grasped as an imaging device having at least a function of imaging from the rear of the vehicle toward the front of the vehicle. For example, it can be grasped as an imaging device that does not have a function of imaging from the rear of the vehicle to the outside of the rear of the vehicle. For example, the imaging device 101F may be configured to process video data inside the vehicle captured using the rear camera 102 for interior imaging without having the camera 402.

[0348] [a. Configuration example of the imaging device 101F] The imaging device 101F may have, for example, the following configuration. [a-1. External configuration of the imaging device 101F] FIG. 24 is a diagram showing an example of the external configuration of the imaging device 101F. FIG. 24(a) is a view of the imaging device 101F seen from the upper right diagonal direction on the front side. FIG. 24(b) is a view of the imaging device 101F seen from the upper right diagonal direction on the rear side. The imaging device 101F has a housing 1010. The housing 1010 is a rectangular parallelepiped shape that is longer in the left-right direction than in the up-down direction and has a relatively small thickness. The housing 1010 has an upper surface 1011 that faces upward when attached to the vehicle, a second side surface 1012, a first side surface 1013, a third side surface 1014 located on the opposite side of the first side surface 1013, and a fourth side surface 1015 located on the opposite side of the second side surface 1012.

[0349] The upper surface 1011 is provided with a joint rail 552 and a camera jack 5591. The joint rail 552 is detachably mountable to a bracket (for example, a bracket 600, described later) for mounting the photographing device 101F at a predetermined mounting position on a vehicle. The mounting position may be, for example, the vehicle windshield (for example, near the top edge of the windshield), the vehicle's rearview mirror, or the ceiling of the vehicle cabin. When the photographing device 101F is mounted on a vehicle, the second side surface 1012 faces the front of the vehicle. In this case, the first side surface 1013 faces the right side when viewed from the rear of the vehicle. The third side surface 1014 faces the left side when viewed from the rear of the vehicle. The fourth side surface 1015 faces the rear of the vehicle.

[0350] The camera jack 5591 is a terminal to which one end of a cable for electrical connection with the photographing device 101R is connected. The cable has, for example, a signal line for transmitting and receiving various signals (e.g., video signals) between the photographing device 101R and the camera jack 5591, and a power line for supplying power from the photographing device 101F to the photographing device 101R. For example, the photographing device 101F may obtain a video signal specifying an image captured by the photographing device 101R from the photographing device 101R via the cable, or may supply power to the photographing device 101R via the cable. The camera jack 5591 may be compatible with, for example, the USB Type-C standard, and may be a terminal for the photographing device 101F to communicate with the photographing device 101R according to the Ethernet standard.

[0351] The second side surface 1012 is provided with an imaging lens 5551, a sound output hole 553, and a microphone hole 554. The imaging lens 5551 is a lens for collecting light included in an imaging unit (a camera 5515 described later; an example of a third camera) included in the photographing device 101F. The sound output hole 553 is provided above the imaging lens 5551 and is a hole that transmits sound output from an audio output unit (an audio output unit 5514 described later) included in the photographing device 101F from the inside to the outside of the housing 1010. The microphone hole 554 is provided below the imaging lens 5551 and is a hole that transmits external sound from the outside to the inside of the housing 1010. The sound that has transmitted into the inside of the housing 1010 is input to a microphone (a microphone 5521 described later) included in the photographing device 101F.

[0352] An event recording button 5522 is provided on the first side surface 1013. The event recording button 5522 is provided on the first side surface 1013 facing the driver's seat so that it can be easily operated by a driver of a right-hand drive vehicle. The event recording button 5522 is an operating means for instructing the start or end of recording of images captured by the camera 5515 (i.e., recording). When the event recording button 5522 is operated by a user while event recording (also referred to as event recording) is not being performed, the image capturing device 101F starts event recording. Event recording will be described in detail below.

[0353] While an image stored in the storage medium 5500 (see FIG. 26) is being played back (in other words, an image is being displayed), the event recording button 5522 functions as a button for switching the image displayed on the display surface 5531. In this case, when the event recording button 5522 is operated, the images displayed on the display surface 5531 are switched in the following order (hereinafter, loop): an image captured by the camera 5515 of the photographing device 101F (image indicated as "front" in FIG. 27(a)), an image captured by the camera 402 of the photographing device 101R (image indicated as "rear rear" in FIG. 27(a)), an image captured by the rear camera 102 for capturing images inside the vehicle of the photographing device 101R (image indicated as "rear interior" in FIG. 27(a)), an image captured by the camera 5515 of the photographing device 101F (image indicated as "front" in FIG. 27(a)), and so on (hereinafter, loop).

[0354] A terminal 5592 and a storage medium insertion port 5581 are provided on third side surface 1014. Terminal 5592 is a terminal for receiving power from an external device. Terminal 5592 is, for example, a DC jack. A connector on one end of a power cord (for example, a cigarette lighter plug cord) is connected to terminal 5592. The connector on the other end of the power cord is connected to, for example, a power supply terminal (for example, a cigarette lighter socket) provided on the vehicle side.

[0355] The terminal 5592 may be connected to an OBDII adapter that can be connected to the OBDII ("II" is the Roman numeral for "2") connector of a vehicle. The OBDII connector is an example of a diagnostic connector, and is connected to the vehicle's ECU (Engine Control Unit) and outputs various types of vehicle information at predetermined intervals (e.g., every 0.5 seconds). When the terminal 5592 is connected to the OBDII connector using an OBDII adapter (not shown), the image capturing device 101F receives a supply of operating power from the vehicle and acquires vehicle information from the vehicle.

[0356] The vehicle information is information relating to the state of the vehicle, and may be at least one of the following: vehicle speed (vehicle speed), engine RPM, engine load factor, throttle degree, ignition timing, remaining fuel percentage, intake manifold pressure, intake air flow rate (MAF), injection open time, engine coolant temperature (coolant temperature), air temperature taken into the engine (intake air temperature), air temperature outside the vehicle (outside air temperature), amount of remaining fuel in the fuel tank (remaining fuel amount), fuel flow rate, instantaneous fuel consumption, accelerator opening, turn signal information (operation of left and right turn signals (ON / OFF)), brake opening, steering wheel rotation angle, gear position, and door open / close status.

[0357] The storage medium insertion slot 5581 is an insertion slot for inserting a storage medium 5500 as an external storage means into the photographing device 101F. The storage medium 5500 is, for example, a storage medium, such as an SD card, on which images captured by the photographing device 101F or the photographing device 101R are recorded. The SD card may take any form, such as an SD memory card, a miniSD card, or a microSD card. The storage medium 5500 may further store a viewer (for example, a dedicated viewer) program for playing back the stored images on an information processing device such as a personal computer.

[0358] Fourth side surface 1015 is provided with operation unit 5523, display surface 5531, and light-emitting unit 5525. Operation unit 5523 has first button 5523A, second button 5523B, third button 5523C, and fourth button 5523D. First button 5523A, second button 5523B, third button 5523C, and fourth button 5523D are arranged vertically along one side of the right side of display surface 5531. Functions that can be assigned to each of these buttons include, for example, the following functions.

[0359] First button 5523A functions as a button for switching images when pressed for a long time, and functions as a button for instructing formatting of storage medium 5500 when pressed for a short time.

[0360] When pressed and held, the image displayed on display surface 5531 is, for example, the image currently being captured by photographing device 101F and part or all of the image currently being captured by photographing device 101R (for example, camera 402 and rear camera 102 for capturing images inside the vehicle interior). When pressed and held, as shown in Figure 27(b), the image switches in the following order: an image taken by the camera 5515 of the photographing device 101F (image shown as "front" in Figure 27(b)), an image taken by the camera 402 of the photographing device 101R (image shown as "rear rear" in Figure 27(b)), an image taken by the rear camera 102 of the photographing device 101R for photographing the interior of the vehicle (image shown as "rear interior" in Figure 27(b)), images taken by these three cameras (image shown as "front + rear" in Figure 27(b)), a screen with the display turned off and no images taken by these three cameras (screen shown as "screen off" in Figure 27(b)), and the camera 5515 of the photographing device 101F (image shown as "front" in Figure 27)... (hereinafter referred to as a loop). 27(b), the display area of ​​the image captured by camera 5515 is larger than the display areas of the images captured by camera 402 and interior-use rear camera 102, and is specified by an L-shaped area. The display areas of the images captured by camera 402 and interior-use rear camera 102 are each rectangular, and may be, for example, of the same size.

[0361] Formatting the storage medium 5500 means initializing the storage medium 5500, and can be understood as at least one of erasing data such as images stored in the storage medium 5500, writing setting information indicating the contents of the operation settings to the storage medium 5500 so that the photographing equipment 101F can use the storage medium 5500 (for example, so that images can be recorded and read), and putting the storage medium 5500 into a specific file state.

[0362] The second button 5523B is a button for displaying a selection screen for selecting an image to be played back by the imaging device 101F. The third button 5523C is a button for displaying a menu regarding the settings of the imaging device 101F and the imaging device 101R. The fourth button 5523D is a button for instructing the start and stop of image recording. For example, when the fourth button 5523D is short-pressed during recording by the continuous recording function described later, the recording is temporarily stopped. When the fourth button 5523D is short-pressed during the temporary stop, the image recording by the continuous recording function resumes. When the fourth button 5523D is long-pressed, the frame rate at the time of image recording can be changed.

[0363] The display surface 5531 is an area where an image displayed by a display unit (display unit 5513 described later) of the imaging device 101F is displayed. The display surface 5531 is, for example, a rectangular or square area. A touch sensor 5524 for detecting a touch operation of the user is provided so as to overlap the display surface 5531.

[0364] The light emitting unit 5525 is provided above the first button 5523A and emits light in a predetermined color.

[0365] <a-2. Configuration of the bracket> FIG. 25 is a diagram showing the configuration of the bracket 600. FIG. 25(a) is a view of the bracket 600 as seen from the upper right obliquely forward side. FIG. 25(b) is a view of the bracket from the side. FIG. 25(c) is a view of the bracket 600 as seen from the lower right obliquely forward side. The bracket 600 is an example of a mounting member for attaching the imaging device 101F to a vehicle.

[0366] The bracket 600 is a mounting member using a ball joint mechanism. In the bracket 600, the flat base portion 610 has an attachment surface 611 to be attached to the windshield of the vehicle. The base portion 610 is inclined by a predetermined angle with respect to the column of the ball stud 620. An adhesive member such as double-sided tape is attached to the attachment surface 611, and through the adhesive member, it is attached to the windshield of the vehicle or the like.

[0367] The ball stud 620 is a part that stands up from the surface of the base part 610 opposite to the mounting surface 611. The socket part 630 is where the ball part 621 of the ball stud 620 is mounted. The nut 640 is detachably attached around the socket part 630. A thread groove is formed on the outer periphery of the socket part 630. The nut 640 that fits into the thread groove is attached to the outer periphery of the socket part 630. Before the nut 640 is tightened with the ball part 621 of the ball stud 620 mounted in the socket part 630, the socket part 630 can rotate in a desired direction along the peripheral surface of the ball part 621, and the posture and position of the base part 650 can be displaced. When the nut 640 is tightened, the posture and position of the base part 650 are fixed.

[0368] The base part 650 is formed in a pair with the socket part 630 and is a part for mounting on the imaging device 101F. The base part 650 has a pair of guide rails 651 on its lower surface. The pair of guide rails 651 are configured to be slidable along the joint rail 5502 of the imaging device 101F. A claw-shaped tip part 652 is provided at the tip of the base part 650. By hooking the tip part 652 near the front end of the imaging device 101F on the joint rail 5502, the bracket 600 is attached to the imaging device 101F.

[0369] <a-3. Electrical Configuration of Imaging Device 101F> FIG. 26 is a block diagram showing the electrical configuration of the photographing device 101F. The control unit 5511 controls each unit of the photographing device 101F. The control unit 5511 is a computer including, for example, a processor 5511A and a memory 5511B. The processor 5511A has, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA). The memory 5511B is a main storage device having, for example, a random access memory (RAM) and a read-only memory (ROM). The processor 5511A temporarily stores a program read from the ROM of the memory 5511B in the RAM. The RAM of the memory 5511B provides a working area for the processor 55511A. Processor 5511A performs various controls by performing arithmetic processing while temporarily storing data generated during program execution in RAM, and control unit 5511 further includes clock unit 5511C that measures time. Clock unit 5511C is, for example, a real-time clock. Clock unit 5511C may be mounted on the motherboard of processor 5511A, or may be external to processor 5511A.

[0370] The input unit 5512 accepts input of information from the user. The input unit 5512 has, for example, the above-mentioned microphone 5521, event recording button 5522, operation unit 5523, and touch sensor 5524. The microphone 5521 converts sound incident via the microphone hole 554 or the like into an electric signal. The microphone 5521 is, for example, a condenser microphone. The touch sensor 5524 detects a position touched by the user on the display surface 5531. The touch sensor 5524 is, for example, of a capacitance type.

[0371] Display unit 5513 displays an image on display surface 5531. Display unit 5513 is, for example, a liquid crystal display (LCD).

[0372] The audio output unit 5514 outputs audio. Examples of the audio include notification sounds, background music, audio messages, etc. The audio output unit 5514 includes, for example, an audio processing circuit and a speaker.

[0373] The camera 5515 captures images and generates video data obtained by capturing the images. The camera 5515 includes, for example, an imaging lens 5551 and an imaging element that captures light collected by the imaging lens 5551. The imaging element is, for example, a CMOS (Complementary MOS) or a CCD (Charge Coupled Device). The camera 5515 generates a color (multi-color) image consisting of, for example, red (R), green (G), and blue (B) color components.

[0374] The communication unit 5516 communicates with an external device. The communication unit 5516 has a communication circuit for wirelessly communicating with the external device, for example, by Wi-Fi (registered trademark), Bluetooth (registered trademark), or other wireless LAN (Local Area Network) communication or short-range wireless communication. The communication unit 5516 may have a communication circuit for performing communication in compliance with standards for mobile communication systems such as LTE (Long Term Evolution), 4G, and 5G.

[0375] The sensor unit 5517 has various sensors. The sensor unit 5517 has, for example, at least one of an acceleration sensor, a gyro sensor, a barometric pressure sensor, and an illuminance sensor. The acceleration sensor is, for example, a three-axis acceleration sensor that detects the acceleration of the vehicle in the forward / backward, left / right, and up / down directions. The gyro sensor is a sensor that detects the tilt of the image capturing device 101F. The acceleration sensor and the gyro sensor may be used, for example, to estimate the position of the vehicle using autonomous navigation when signals from GNSS satellites cannot be received. The barometric pressure sensor measures barometric pressure. The barometric pressure sensor is used, for example, to detect differences in elevation and determine whether the vehicle is on a highway or an ordinary road. The illuminance sensor is a sensor that detects illuminance, which indicates the brightness inside the vehicle cabin, which is the area around the image capturing device 101F. The illuminance sensor is used, for example, to adjust the display brightness of the display unit 5513.

[0376] The reader / writer 5518 functions as a medium holding unit that holds the storage medium 5500 inserted into the photographing device 101F through the storage medium insertion slot 5581. The reader / writer 5518 writes data to the storage medium 5500 and reads data from the storage medium 5500. The reader / writer 5518 may hold only one storage medium 5500, or may be configured to be able to hold two or more storage media 5500 simultaneously.

[0377] The terminal unit 5519 has a terminal for electrically connecting to an external device. The terminal unit 5519 has the above-mentioned camera jack 5591 and terminal 5592. An external battery may be used as a device connected to the terminal unit 5519 so that the image capturing device 101F and the image capturing device 101R can operate without power supply from the vehicle side. The device connected to the terminal unit 5519 may be, for example, a device having a function to support safe driving by a user. Examples of such devices include a device having a function to capture an image of a driver (e.g., a face) and detect and notify the driver's state, such as distracted driving and drowsy driving, and a device having a function to detect and notify obstacles around the vehicle (e.g., a device used for vehicle detection for forward vehicle collision warning systems (FCWS)). The device connected to the terminal unit 5519 may also be an in-vehicle device such as a radar detector, a laser detector, a car navigation device, or a display device.

[0378] The location information acquisition unit 5520 acquires location information indicating the location (more specifically, the current location) of the photographing device 101F. The location of the photographing device 101F can be considered to be the same as the location of the vehicle in which the photographing device 101F is located, the location of the photographing device 101R located in the vehicle, and the location of the driver or other people in the vehicle. The location information acquisition unit 5520 acquires location information (latitude information and longitude information) of the photographing device 101F based on, for example, a signal from GPS, which is one of the GNSS. The location information acquisition unit 5520 may also use Michibiki as a QZSS (Quasi-Zenith Satellite System).

[0379] The light emitting unit 5525 emits light in a predetermined color and includes, for example, a light emitting diode.

[0380] The power control unit 5526 controls the power supply to each part of the imaging device 101F and the imaging device 101R. The power control unit 5526 has, for example, a power switch and a power control circuit. The power control unit 5526 supplies the power supplied from the vehicle side via the terminal unit 5519 to each part of the imaging device 101F and the imaging device 101R. The power control unit 5526 may further have, as power storage means, a secondary battery, a button battery, or an electric double layer capacitor (also called a supercapacitor).

[0381] The imaging device 101F may further have, as internal storage means, an auxiliary storage device exemplified by a flash memory (e.g., eMMC, SSD). As the auxiliary storage device, various storage media exemplified by an optical storage medium, a magnetic storage medium, and a semiconductor storage medium can be used.

[0382] <a-4. Recording function of the imaging device 101F> The imaging device 101F may have, as a recording function, one or more of the following functions. The recording function is a function of recording (i.e., storing) the images captured by the imaging device 101F as video data in a predetermined file format. The video data is preferably in a video format, for example, in the MPEG (Moving Picture Experts Group) format (e.g., MPEG2, MPEG4), but it can also be in AVI, MOV, WMV, etc.

[0383] (a-4-1. Continuous recording function) The continuous recording function (also called the continuous storage function) is a function of continuously (i.e., constantly) recording the images captured by the camera 5515 of the imaging device 101F and a part or all of the cameras 402 of the imaging device 101R and the rear camera 102 for interior vehicle imaging during the operation of the imaging device 101F. When the control unit 5511 is executing the continuous recording function, it stores the video data captured from the start to the stop of the vehicle engine.

[0384] (a-4-2. Event recording function) The event recording function (also referred to as the event recording function) is a function for recording images captured by the camera 5515 of the photographing device 101F and some or all of the camera 402 and the interior rear camera 102 of the photographing device 101R in response to the occurrence of a specific event. The event is an occurrence that serves as a condition (i.e., a trigger) for recording an image, and may be, for example, when the user suddenly steers or brakes while the vehicle is traveling, or when the vehicle collides with another object. The control unit 5511 determines that an event has occurred, for example, based on a measurement value obtained by the acceleration sensor of the sensor unit 5517. Specifically, the control unit 5511 determines that an event has occurred when the measurement value obtained by the acceleration sensor is equal to or greater than a predetermined threshold value or when a predetermined change occurs over time. The conditions for determining the occurrence of an event are not limited to these. The control unit 5511 may also determine that an event has occurred based on vehicle information, for example, when the vehicle state, such as the vehicle speed or steering state, satisfies a predetermined condition. The control unit 5511 may determine the occurrence of an event by analyzing an image captured by at least one of the camera 5515, the camera 402, and the interior rear camera 102. The control unit 5511 may determine that an event has occurred, for example, when detecting dangerous driving of the vehicle or another vehicle (for example, tailgating, approaching, or abnormally approaching). The control unit 5511 also determines that an event has occurred when the event recording button 5522 is operated.

[0385] When the control unit 5511 determines that an event has occurred, the control unit 5511 records images captured during a predetermined period before and after the occurrence of the event (hereinafter referred to as the "event recording period") in the storage medium 5500. For example, the control unit 5511 temporarily records images captured by the camera 5515, the camera 402, and the interior-use rear camera 102 in memory 5511B (e.g., RAM). When the control unit 5511 determines that an event has occurred, the control unit 5511 may record images during the event recording period read from the memory 5511B in the storage medium 5500. For example, the control unit 5511 may generate, as one file, images for a total of 40 seconds, including 20 seconds before and 20 seconds after the occurrence of the event, for each of the camera 5515, the camera 402, and the interior-use rear camera 102. This event recording period is an example and may vary depending on the type of event. The image capturing devices 101F and 101R may be configured so that the user can change the event recording period. The control unit 5511 may set different event recording periods for some or all of the camera 5515, the camera 402, and the rear camera 102 for photographing inside the vehicle interior, or may set the same length of event recording period for all of them. The control unit 5511 may further record, in the storage medium 5500, information specifying the type of event, values ​​measured by the sensor unit 5517 during the event recording period (for example, acceleration in each of the three axes), and position information acquired by the position information acquisition unit 5520, in association with the image.

[0386] The control unit 5511 may change the conditions for performing event recording for some or all of the camera 5515, the camera 402, and the rear camera 102 for photographing inside the vehicle interior. For example, when a value measured by the sensor unit 5517 satisfies a predetermined condition (for example, when the value is equal to or greater than a threshold), the control unit 5511 may change the conditions for performing event recording for some or all of the camera 5515, the camera 402, and the rear camera 102 for photographing inside the vehicle interior. The condition may be that any one of the accelerations along three axes or the absolute value of the acceleration along three axes is equal to or greater than a threshold. The condition for performing event recording may be configured so that the user can set it for each of the camera 5515, the camera 402, and the rear camera 102 for photographing inside the vehicle interior. The acceleration occurring on the photographing device 101F side (specifically, camera 5515) is measured using the sensor unit 5517, and the acceleration occurring on the photographing device 101R side (specifically, camera 402 and rear camera 102 for photographing inside the vehicle) is measured using the acceleration sensor 305, so that the acceleration occurring at the position of each camera can be measured more accurately and event recording can be performed using the measurement results as a trigger.

[0387] (a-4-3. Parking monitoring function) The parking monitoring function is a function for recording images captured by the camera 5515 of the photographing device 101F, and some or all of the camera 402 of the photographing device 101R and the rear camera 102 for capturing images inside the vehicle while the vehicle is parked. The parking monitoring function is a function for monitoring the interior of the vehicle or the exterior surrounding the vehicle while the vehicle is parked. When not receiving power from the vehicle, the control unit 5511 may receive power from an external battery and record images in the storage medium 5500. The control unit 5511 may determine whether the vehicle is parked based on one or more of the following: the accessory power supply is turned off; the engine is turned off; power supply from the external battery has started; the vehicle speed is 0 km / h or a predetermined speed or less; and the location information acquired by the location information acquisition unit 5520 is predetermined location information (e.g., location information of the home, workplace, or parking lot). However, the present invention is not limited to this, and the control unit 5511 may determine whether or not the vehicle is parked by analyzing an image captured by the rear camera 102 for capturing images inside the vehicle, as already explained.

[0388] The parking monitoring function may have a time lapse mode and a motion detection mode. In this case, when the user selects the time lapse mode, the control unit 5511 may record at a frame rate lower than that of other image recording functions such as the continuous recording function and the event recording function. For example, the frame rate of the time lapse mode may be 1 frame / second, while the frame rate of the other image recording functions is 20 to 30 frames / second.

[0389] The moving object detection mode is a mode in which images are recorded in response to the detection of a moving object (i.e., the movement of an object). Examples of moving objects include people and other vehicles. Specifically, when the moving object detection mode is selected by the user, the control unit 5511 may record images captured within a predetermined period before and after the detection when the moving object is detected based on a change in images captured by at least one of the camera 5515, the camera 402, and the interior-photography rear camera 102. When the control unit 5511 detects a moving object based on images captured by at least one camera, recording images captured by all of the cameras, i.e., the camera 5515, the camera 402, and the interior-photography rear camera 102, allows a wide range of the situation at that time, including the front, rear, and interior of the vehicle, to be later confirmed. When the control unit 5511 records images captured by only the camera that detected the moving object, it is possible to conserve storage capacity of the storage medium 5500, etc., while recording images of the moving object. The frame rate should preferably be the same as that of the continuous recording function, the event recording function, and the like.

[0390] In the motion detection mode, the conditions for recording captured images (i.e., trigger conditions) may be different for some or all of the cameras 5515, 402, and 102 for interior photography. The control unit 5511 may, for example, vary the motion detection sensitivity for some or all of the cameras 5515, 402, and 102 for interior photography. The conditions for recording images may be configured so that the user can set them for each of the cameras 5515, 402, and 102 for interior photography. For example, the motion detection mode may be configured so that the user can select from "OFF" (i.e., no image is recorded in response to motion detection), "low," "normal," and "high." The latter setting makes it easier for even the slightest movement of a moving object to be recorded. By default, the sensitivity for each of the cameras 5515, 402, and 102 for interior photography may be set to "normal."

[0391] Note that the photographing device 101F may be a photographing device that does not have a display unit 5513. The housing of the photographing device 101F does not have to be rectangular parallelepiped, and may be, for example, a cylindrical photographing device. The process of recording data in the storage medium 5500 can be replaced with or combined with, for example, a process of recording data in a storage device other than the storage medium 5500 or a process of transmitting data to an external device.

[0392] [b. Viewer and related functions] FIG. 28 shows an example of a viewer screen for playing back images captured by the camera 5515 of the photographing device 101F, the camera 402 of the photographing device 101R, and the in-vehicle rear camera 102. This viewer may be implemented by a viewer program for a personal computer, smartphone, tablet, or other information processing device, as described above, or may be implemented by a viewer program used to display images captured by the photographing device 101F. FIGS. 29 to 31 are diagrams illustrating the display elements included on the viewer screen. The circled numbers in FIG. 29 correspond to the numbers in the "No." column in the tables of FIGS. 30 and 31. The display elements included on the viewer screen have the display names shown in the tables of FIGS. 30 and 31 and perform the functions described in FIGS. 30 and 31. 30 and 31, the "front camera" corresponds to the camera 5515, the "rear rear camera" corresponds to the camera 402, and the "rear interior camera" corresponds to the rear camera 102 for photographing the interior of the vehicle.

[0393] As shown in FIG. 28, the image captured by the camera 5515 capturing the view ahead of the vehicle is displayed in the largest size in a rectangular area T1. To the right of area T1, smaller images captured by the camera 402 of the photographing device 101R are displayed in a rectangular area T2, and images captured by the interior-use rear camera 102 are displayed in a rectangular area T3. The viewer may have a function for synchronously playing back images captured by the camera 5515, the camera 402, and the interior-use rear camera 102. This allows the user to simultaneously check the view ahead, behind, and inside the vehicle at each point in time. As shown in FIG. 32(a), a playback list L is displayed on the viewer screen by default. However, in response to a user operation, a map M showing the shooting locations may be displayed instead of the playback list L, as shown in FIG. 32(b).

[0394] Furthermore, the viewer program may have two modes for displaying images captured by camera 402: a mode for displaying the captured image as is (i.e., a mode for displaying a normal image in which the image is viewed as is from top to bottom and left to right), and a mode for displaying an image that is flipped left to right (i.e., a mode for displaying a mirror image). Camera 402 and the interior-use rear camera 102 may capture images in opposite directions. In this case, the left and right relationships between the images captured by these cameras are reversed. Therefore, the viewer program may be configured to allow the user to select whether to display the captured image as is (i.e., display a normal image) or display a mirror image to align the left and right.

[0395] It is preferable that the images from each camera recorded in storage medium 5500 by photographing device 101F are fixed as normal images, but for example, a configuration may be adopted in which it is possible to record a mirror image taken by camera 5515 or rear camera 102 for photographing the interior of the vehicle. Also, while the image taken by rear camera 102 for photographing the interior of the vehicle is fixed as a normal image, a mode for displaying a mirror image with left and right determined may be provided in the viewer program. In this way, it is expected that the degree of freedom in recording and displaying images will be improved.

[0396] [c. Another example of an image captured by the rear camera 102 for capturing images inside the vehicle] Fig. 33 is a diagram showing another example of an image captured by the interior rear camera 102, different from Fig. 11. As shown in Fig. 33, the interior rear camera 102 can capture and record a wide-angle image of the left and right situations when viewed from the rear of the vehicle toward the front of the vehicle. Furthermore, as shown in areas T4 and T5, it can capture and record images of the left and right situations of the vehicle, more specifically, the surroundings such as side roads and adjacent lanes located on the outside in the vehicle width direction. Furthermore, as shown in area T6, it is possible to capture images that respect the privacy of passengers, with their faces not or hardly visible in the image.

[0397] [d. Relationship between the forward vehicle approach warning function and the recording of captured images] The photographing device 101F may have a forward vehicle approach warning function. The forward vehicle approach warning function is a function that warns the vehicle when it approaches a forward vehicle (i.e., a leading vehicle) while the vehicle is traveling. The forward vehicle approach warning function is a function for avoiding, for example, unintentional tailgating by the driver against the forward vehicle or a collision with the forward vehicle. The detection of the approach of the forward vehicle may be performed, for example, by the control unit 5511 analyzing an image captured by the camera 5515 or by using a sensor such as a laser ranging sensor. The warning may be interpreted as an alarm or a notification, and may be performed by a sound, voice, display, light emission, or other method that can be perceived by a human (e.g., the driver).

[0398] With this configuration, when issuing a warning using the forward vehicle approach warning function, the control unit 5511 of the photographing device 101F may record images captured by the camera 5515, the camera 402, and the rear camera 102 for capturing images inside the vehicle interior. This allows the driver to later check a wide range of conditions, including the areas in front of the vehicle, behind the vehicle, and inside the vehicle interior, when the vehicle approaches the forward vehicle. The period for recording images may coincide with the period during which the forward vehicle approach warning function issues a warning, but may also include a predetermined period before, after, or both before and after that period. This allows the driver to later check the conditions in front of, behind, and inside the vehicle interior around the period when the vehicle approaches the forward vehicle. The length of the recording period for images captured by camera 5515, camera 402, and rear camera 102 for capturing images inside the vehicle interior may be the same, or at least some of them may be different from the others. For example, if the length of the recording period for camera 5515 is made longer than the length of the recording period for camera 402 and rear camera 102 for capturing images inside the vehicle interior, this will save on the recording capacity of storage medium 5500, etc., while being useful for later understanding the situation in front of the vehicle in detail.

[0399] Here, the relationship between the forward vehicle approach warning function and the recording of images captured by the camera 5515, the camera 402, and the interior rear camera 102 has been described. However, the forward vehicle approach warning function can be replaced with or combined with the lane departure warning function or the forward vehicle departure warning function. The lane departure warning function is a function that issues a warning when the host vehicle deviates from its lane while the vehicle is traveling. The forward vehicle departure warning function is a function that issues a warning when the host vehicle does not start moving even though the forward vehicle has started moving while the host vehicle is stopped. The method of detecting lane departure in the lane departure warning function and the method of detecting that the forward vehicle has started moving while the host vehicle is stopped in the forward vehicle departure warning function may be a method in which the control unit 5511 analyzes and detects images captured by the camera 5515, or another well-known method.

[0400] [e. Relationship between rear vehicle approach recording / warning function and recording of captured images] The image capturing device 101F may have a rear vehicle approach recording and warning function. The rear vehicle approach recording and warning function is a function that issues a warning when the vehicle approaches a rear vehicle (i.e., a following vehicle) while traveling. The rear vehicle approach recording and warning function is a function that detects dangerous driving, such as tailgating (failure to maintain a safe distance) by a rear vehicle, and records at least an image captured by the camera 402 and issues a warning. The detection of the approach of a rear vehicle may be performed, for example, by the control unit 5511 analyzing an image captured by the camera 402 or by using a sensor such as a laser ranging sensor. The warning may be interpreted as an alarm or a notification, and may be performed by a sound, voice (for example, outputting a message such as "Dangerous driving behind has been detected!"), display, light emission, or other method that can be perceived by a human (for example, the driver).

[0401] With this configuration, when issuing a warning using the rear vehicle approach recording and warning function, the control unit 5511 of the photographing device 101F may record images captured by the camera 5515, the camera 402, and the rear camera 102 for capturing images inside the vehicle interior. This allows the driver to later check a wide range of conditions, including the area in front of the vehicle, the area behind the vehicle, and the interior of the vehicle, when the rear vehicle approaches. The period for recording images may coincide with the period during which the warning using the rear vehicle approach recording and warning function is issued, but it may also be set to include a predetermined period before, after, or both before and after that period. This allows the driver to later check the conditions in front of, behind, and the interior of the vehicle around the time the rear vehicle approached. The time lengths of the recording periods for images captured by camera 5515, camera 402, and interior-photography rear camera 102 may be the same, or at least some of them may be different from the others; for example, if the time lengths of the recording periods for camera 402 and interior-photography rear camera 102 are made longer than the time length of the recording period for camera 5515, this is useful for later understanding the situation behind the vehicle in detail while saving the recording capacity of storage medium 5500, etc. Note that although the "rear vehicle approaching recording and warning function" has been described here, it may be replaced with a "rear vehicle approaching recording function" that does not provide a warning.

[0402] In the above-described embodiment, the terms “engine” and “accessory power source” may be replaced with “drive source” and “vehicle power source.” The vehicle may be, for example, an internal combustion engine vehicle having an engine as a drive source, a hybrid vehicle having an engine and a traction motor as drive sources, or an electric vehicle having a traction motor as a drive source.

[0403] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. The above-described embodiments disclose at least the configurations described in the claims.

[0404] The scope of the present invention is not limited to the structures explicitly described in the specification, but also includes combinations of various aspects of the present invention disclosed herein. The structures of the present invention that are sought to be patented are specified in the appended claims, but it is the intention of the present inventors to claim structures disclosed in this specification in the future, even if they are not currently specified in the claims.

[0405] The present invention is not limited to the configurations described in the above-described embodiments. The components of each of the above-described embodiments and variations may be arbitrarily selected and combined. Furthermore, any component of each embodiment or variation may be arbitrarily combined with any component described in the Summary of the Invention or any component embodying any component described in the Summary of the Invention. The present invention also intends to obtain rights to these configurations through amendments or divisional applications of the present application. Even if a description is made of "in the case of..." or "when...," it is not intended to describe a configuration limited to that case or time. Configurations that are not limited to those cases or times are also disclosed, and the present invention intends to obtain rights to them. Furthermore, any descriptions that specify an order are not limited to this order. Configurations in which some parts are deleted or the order is changed are also disclosed, and the present invention intends to obtain rights to them.

[0406] Furthermore, we intend to obtain rights to the overall design or partial design by converting the application to a design registration. The drawings depict the entire device using solid lines, but they also include partial designs claimed for parts of the device. For example, a partial design can be a partial design for a part of the device, or a part of that part. A partial design can be a part of the device, or a part of that part. We intend to obtain rights not only for the overall design, but also for partial designs in which any part of the solid line portion of the drawing is drawn with a broken line. Furthermore, the modules, components, parts, images, etc. inside the device's casing, all of which are shown in the drawings, are subject to independent commerce, and we intend to similarly obtain rights by converting the application to a design registration.

[0407] The above-described embodiment discloses at least the following configurations. [Configuration 1] This information processing system It is preferable to process video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle. In this way, it is possible to process video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle. For example, in conventional video data of the interior of the vehicle captured from the front of the vehicle toward the rear of the vehicle, objects, elements, events, etc. that are located in blind spots and are impossible or difficult to process can be detected, identified, recognized, judged, etc. by capturing the interior of the vehicle from a direction different from the conventional one, i.e., by capturing the image from the rear of the vehicle toward the front of the vehicle, and thus, the user can utilize newly provided functions based on this processing. [Configuration 2] This information processing system It is preferable to process video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle, and video data of the exterior of the vehicle captured from the rear of the vehicle toward the outside of the vehicle. In this way, it is possible to process video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle, and video data of the exterior of the vehicle captured from the rear of the vehicle. For example, in conventional video data of the interior of the vehicle captured from the front of the vehicle toward the rear of the vehicle, objects, elements, events, etc. that are located in blind spots and are impossible or difficult to process can be detected, identified, recognized, judged, etc. by capturing the interior of the vehicle from a direction different from the conventional one, i.e., by capturing the image from the rear of the vehicle toward the front of the vehicle, and therefore, the user can use newly provided functions based on this processing. [Configuration 3] The information processing system according to the present invention comprises: It is preferable that the vehicle control device has a control unit that analyzes the video data of the inside of the vehicle cabin and determines the situation based on the analysis results. In this way, the control unit can analyze video data of the interior of the vehicle captured from a different direction than before, making it possible to perform new or more accurate judgment processing for various situations than before, allowing the user to take advantage of new functions based on new, more accurate judgments. [Configuration 4] The information processing system according to the present invention comprises: The control unit may determine the presence or absence of a person in the vehicle cabin based on the video data of the vehicle cabin, and create and output video data that has been filtered to remove the person. In this way, by filming the interior of the vehicle from the rear toward the front of the vehicle, the chances of people's faces being photographed are reduced, and furthermore, by applying a filter to the people photographed, the actions of people inside the vehicle can be photographed and recorded with greater consideration given to their privacy than before. This reduces the resistance felt by users, for example, by passengers of buses and taxis, about having their images recorded. [Configuration 5] The information processing system according to the present invention comprises: The control unit may determine the state of an object outside the vehicle that is captured in the video data based on the video data of the interior of the vehicle, and output information relating to the state of the object. In this way, the control unit can determine, for example, when a turn signal is turned on, a situation where there is a car, person, or obstacle on the side, a situation where someone is approaching the vehicle unnaturally when the vehicle is parked with no one inside, a situation where such a person is looking into the vehicle, a situation where people, particularly children, are present, approaching, or playing around the vehicle when the engine is started, when the vehicle is slowing down, when the vehicle is traveling, etc., a situation where the vehicle is being tailgated from the side from behind, etc. Furthermore, the control unit can output appropriate notifications and warnings to the driver, etc. depending on the results of the determination of these situations, so that the user can quickly notice and respond to abnormalities or dangers in or around the vehicle. [Configuration 6] The information processing system according to the present invention comprises: The control unit may determine an abnormality in an object installed on the ceiling of the vehicle based on the video data of the interior of the vehicle compartment, and output information related to the abnormality. In this way, the control unit can determine, for example, the lighting status of the room lamp and the presence or absence of a person in the vehicle, whether the net is sagging due to excessive luggage, whether the left and right fasteners have come loose, the orientation and status of the projector, the position and status of the projector's image, whether the relative positions of the projector and rear seat passengers are appropriate, whether the flip-down monitor is open or closed, whether the activation status (whether an image is displayed or not) is appropriate, etc. Furthermore, the control unit can output appropriate notifications and warnings to the driver or the like according to the results of the determination of these conditions, so that the user can easily maintain the vehicle in an appropriate state and can quickly notice any abnormalities in the vehicle. [Configuration 7] The information processing system according to the present invention comprises: The control unit may determine a state of luggage in the vehicle based on the video data of the interior of the vehicle, and output information relating to the state of the luggage. In this way, the control unit can determine, for example, the state of the luggage placed behind the driver's seat, the positional relationship between the luggage and the luggage bed or seat, the estimated position of the center of gravity of the luggage, the positional relationship between the centers of gravity of multiple luggage, the vehicle speed and acceleration and changes therein, the risk of luggage falling from the luggage bed or seat, the risk of stacked luggage collapsing, the degree of slippage of luggage on the luggage bed or seat, and the relationship between the collapse of luggage and tailgating or contact by another vehicle photographed from behind through a vehicle window. Furthermore, the control unit can output appropriate notifications and warnings to the driver, etc., depending on the results of the determination of the luggage state. This allows the driver to concentrate on driving without worrying about the luggage collapsing. [Configuration 8] The information processing system according to the present invention comprises: The control unit may determine the state of a display unit disposed in the vehicle based on the video data of the interior of the vehicle, and output information relating to the state of the display unit. In this way, the display unit can determine the status of, for example, a monitor installed on the back of a seat, the status of a monitor installed on the ceiling, the display content and presence or absence of an operation on a center monitor installed near the center of the front between the driver's seat and the passenger seat, the specific operation performed on the display unit, whether the vehicle was in autonomous driving or manual driving at the time of video recording, etc. Furthermore, the control unit can output appropriate notifications and warnings to the driver, etc., depending on the results of determining these conditions. Furthermore, if the user is involved in an accident, etc. while the driver is operating the vehicle appropriately, the user can provide objective evidence that the accident did not occur while operating the terminal, that the accident was not due to a terminal misoperation, and whether the accident occurred during autonomous driving or manual driving, etc. [Configuration 9] The information processing system according to the present invention comprises: The control unit may determine the presence of a person's head based on the video data, and output information relating to the proportion of the video data that is occupied by the head. In this way, the display unit can determine, for example, whether a head is visible on the screen, what proportion of the screen the visible head occupies, whether the installation state is such that such an event is anticipated, etc. Furthermore, the control unit can output appropriate notifications and warnings to the driver, etc., depending on the results of the determination of these situations. This allows the user to avoid recording only video data in which the back of a person's head is constantly visible on the entire screen when sitting in the back seat. [Configuration 10] The information processing system according to the present invention comprises: The control unit may determine whether the vehicle is in a parked state based on the video data, and if the vehicle is in the parked state, may output a command to switch to a parking monitoring mode. In this way, the control unit can determine whether a person is inside the vehicle, whether a person has left the vehicle, whether the vehicle is parked, etc. If the control unit determines that the vehicle is parked, it outputs a command to switch the shooting mode to parking monitoring mode. As a result, when the vehicle is parked, the control unit can preferentially allocate computing power to processing for identifying a suspicious person approaching from within the area of ​​the vehicle's windshield, rear window, or side window, for example, allowing the user to use a more efficient information processing system. [Configuration 11] The information processing system according to the present invention comprises: The control unit may determine the state of the person inside the vehicle based on the video data, and output information relating to the state of the person inside the vehicle. In this way, the control unit can determine the status of people inside the vehicle, especially those behind the vehicle, with higher accuracy than before, and the user can receive appropriate notifications and warnings depending on the results of the determination of the person's status. [Configuration 12] The information processing system according to the present invention comprises: The control unit may determine whether a person in the vehicle is abandoned based on the video data, and output information relating to the abandoned state. In this way, the control unit determines whether there is a person in the video, whether the person in the video is a child or a disabled person, whether there is a person other than a child or a disabled person in the vehicle, whether the vehicle is parked, whether a child, a disabled person, or the like has been left behind, etc. Furthermore, the control unit can output appropriate notifications and warnings to the driver, the surroundings of the vehicle, etc., depending on the determination result of the abandonment situation. This allows the user to use the highly accurate rear-side abandonment prevention function. [Configuration 13] The information processing system according to the present invention comprises: The control unit may determine whether a person in the vehicle is dozing or sleeping based on the video data, and output information relating to the dozing or sleeping state. In this way, the control unit determines whether there is a person in the video, whether the person in the video is the driver or a passenger, whether the driver's head is visible, whether the state of the driver's head indicates a warning pattern for drowsiness, whether the face or head of a passenger in the rear seat is visible, and whether the state of the face or head indicates a sleeping pattern, etc. Furthermore, the control unit can output appropriate notifications or warnings to the driver or the surroundings of the vehicle, etc., depending on the determination result of the abandoned vehicle situation. This makes it easier for the driver to check for signs of drowsiness himself or herself and to take into consideration the sleeping state of the passenger when driving. [Configuration 14] The information processing system according to the present invention comprises: It is preferable that a human detection sensor is provided, and a detection signal from the human detection sensor is at least used as a trigger for starting the determination or for the determination. In this way, for example, by using a pyroelectric infrared sensor or a thermosensor, it is possible to identify children, persons with disabilities, etc. with a lower load than AI processing of video data. Since constantly executing AI processing requires a large computational load, it is advantageous to detect the presence of people using a separate human-detection sensor and use the result of the human-detection sensor's detection of a person inside the vehicle as a trigger to start human detection processing based on video data, in terms of reducing the computational load and achieving high-accuracy human detection. Furthermore, for example, by limiting the detection range of the human-detection sensor to the rear area of ​​the vehicle or near the rear seats, it is possible to distinguish between the driver and passengers with high accuracy. This allows users to reduce the risk to passengers. [Configuration 15] The information processing system according to the present invention comprises: It is preferable to photograph the interior of the vehicle from the rear of the vehicle toward the front of the vehicle at a plurality of different positions. In this way, for example, by combining video data captured at multiple positions using a wide-angle camera, the user can get a full view of the interior and exterior of the vehicle. [Configuration 16] The information processing system according to the present invention comprises: It is preferable that the opening and closing operation of the opening / closing part of the vehicle is used as a trigger to start taking or recording images, or to change settings for taking or recording images. In this way, by triggering the opening or closing of a hatch or the like to start shooting or recording video or change the video shooting or recording settings, the user can reliably obtain video data that has been shot with appropriate settings even when there are major changes in the shooting environment or shooting conditions. [Configuration 17] The information processing system according to the present invention comprises: In addition to the opening and closing operation of the opening / closing part of the vehicle, the vehicle running state, the state of the drive source of the vehicle, and the state of the power source of the vehicle are used as triggers, Begin filming or recording footage; or It is advisable to change the settings for taking or recording video. In this way, in addition to the opening and closing operation of the vehicle's opening and closing part, it is possible to start taking or recording images or change settings in a manner that corresponds to at least one of the following conditions: the vehicle's running state, the state of the vehicle's drive source, and the state of the vehicle's power supply. [Configuration 18] The information processing system according to the present invention comprises: Recording the sounds collected by at least one of the sound collection means provided at a plurality of locations in the vehicle together with the captured video; It is preferable that the settings of the audio to be recorded be changed using the opening and closing operation of the opening and closing part of the vehicle as a trigger. In this way, the settings for the audio to be recorded can be changed using the opening and closing operation of the opening and closing part of the vehicle as a trigger. [Configuration 19] The information processing system according to the present invention comprises: The opening / closing unit of the vehicle includes a rear opening / closing unit that opens and closes an opening provided at the rear of the vehicle, The camera of the photographing device provided in the rear opening / closing section photographs the outside behind the vehicle when the opening / closing section is in the open state, and photographs from the rear of the vehicle toward the front of the vehicle when the rear opening / closing section is in the closed state. In this way, the shooting direction changes depending on the open / closed state of the opening provided at the rear of the vehicle, and when the rear opening / closing part is in the open state, the outside behind the vehicle is photographed, and when it is in the closed state, the front of the vehicle is photographed from the rear of the vehicle, so that the appropriate direction can be photographed depending on the open / closed state. [Configuration 20] The information processing system according to the present invention comprises: The rear opening / closing portion opens upward, The camera of the photographing device may take photographs of the exterior of the rear of the vehicle facing downward when the rear opening / closing section is in an open state, and may take photographs from the rear of the vehicle facing forward when the rear opening / closing section is in a closed state. In this way, the photographing direction changes depending on the open / closed state of the opening that opens upward, and when the rear opening / closing part is in the open state, the outside behind the vehicle is photographed facing downward, and when it is in the closed state, the front of the vehicle is photographed from the rear of the vehicle, so that the appropriate direction can be photographed depending on the open / closed state. [Configuration 21] The information processing system according to the present invention comprises: When the rear opening / closing section changes from a closed state to an open state, it is preferable to start recording images captured by the camera of the photographing device in a predetermined manner. In this way, when the rear opening / closing section changes from the closed state to the open state, the predetermined recording of the image captured by the photographing device can be started. When the rear opening / closing section changes from the closed state to the open state, it is conceivable that a person is getting on or off, so it is possible to photograph and confirm an event that occurs when a person is present. [Configuration 22] The information processing system according to the present invention comprises: the photographing device is operable by receiving power from an external source or by an internal power source when a drive source or a power source of the vehicle is turned off; When the rear opening / closing section is opened while the drive source or power source of the vehicle is turned off, it is preferable to start recording the images captured by the photographing device in a predetermined manner. In this way, when the rear opening / closing section changes from the closed state to the open state, the image captured by the photographing device can be recorded in a predetermined manner. When the rear opening / closing section changes from the closed state to the open state, it is thought that an event such as people getting on or off or putting something in or taking something out may occur, and it is possible to photograph such an event. [Configuration 23] The information processing system according to the present invention comprises: The photographing device is in standby mode with its power turned on during the designated time period, If the rear opening / closing section changes from a closed state to an open state within the specified time period, the predetermined recording may be started. In this w...

Claims

1. a control unit that processes video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle, analyzes the video data of the interior of the vehicle, and determines the situation based on the analysis result; the control unit has a function of determining a state of an object installed on a ceiling of the vehicle that is displayed in the video data of the vehicle interior and a state of a person inside the vehicle interior, and outputting result information based on a determination result of the state of the object and the state of the person. Information processing system.

2. The control unit determines the state of the installation and the state of the person by: The person is in contact with the installation, or There is a risk of contact between the installation and the person; To determine The information processing system according to claim 1 .

3. The control unit determines the state of the installed object and the state of the person by: The startup state of the installation determining whether or not a person is present; changing the result information based on the determination result based on the presence or absence of the person for the same state of the installation; The information processing system according to claim 1 .

4. The control unit determines the state of the installed object and the state of the person by: The posture or direction of the installation object; determining a positional relationship between the installation and the person; changing the result information based on the determination result based on the positional relationship for the same state of the installation object; The information processing system according to claim 1 .

5. a control unit that processes video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle, analyzes the video data of the interior of the vehicle, and determines the situation based on the analysis result; the control unit has a function of determining a state of luggage in the vehicle and a state of a person in the vehicle cabin that are shown in the video data of the vehicle cabin, and outputting result information based on the determination results of the state of the luggage and the state of the person. Information processing system.

6. The control unit determines the state of the luggage and the state of the person by: At least one of the risk of the luggage falling, the risk of the stacked luggage collapsing, and the slipperiness of the luggage; and determining the riding state of the person; outputting, as the result information based on the determination result, information regarding a warning or notification to the person or the person getting on the vehicle regarding at least one of a risk of the luggage falling, a risk of the stacked luggage collapsing, and a degree of slipperiness of the luggage; The information processing system according to claim 5 .

7. The control unit determines the state of the luggage and the state of the person by: At least one of the following: a drop of the luggage during driving, a collapse of the stacked luggage, and a slip of the luggage; The driving state of the driver in response to tailgating by another vehicle is determined, As the result information based on the determination result, when at least one of the luggage falling, the stacked luggage collapsing, and the luggage sliding occurs, the driving state of the driver responding to the tailgating by the other vehicle, the driving trajectory of the other vehicle, and the timing at which at least one of the luggage falling, the stacked luggage collapsing, and the luggage sliding occurs are recorded in one video data. The information processing system according to claim 5 .

8. The control unit determines the state of the luggage and the state of the person by: The position of at least one of the dropped luggage, the collapsed luggage, and the slipped luggage; determining the location of the person; outputting information regarding a warning or notification to the person according to a positional relationship between the person and at least one of the dropped luggage, the collapsed luggage, and the slipped luggage as the result information based on the determination result; The information processing system according to claim 5 .

9. a control unit that processes video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle, analyzes the video data of the interior of the vehicle, and determines the situation based on the analysis result; The control unit has a function of determining a state of a display unit disposed in the vehicle and a state of a person in the vehicle based on the video data of the vehicle interior, and outputting result information based on a determination result of the state of the display unit and the state of the person in the vehicle interior. Information processing system.

10. The control unit determines the state of the display unit and the state of the person by: a playback state of the display unit; determining whether or not the person faces the display unit or whether or not the person is sleeping; When the person facing the display unit is not present or when the person facing the display unit is asleep, the display unit outputs information to temporarily suspend or hide the display as the result information based on the determination result. The information processing system according to claim 9 .

11. The control unit determines the state of the display unit and the state of the person by: The display state of the display unit, and determining an operation state of the driver, who is the person, on the display unit; When the person is in the operating state with respect to the display unit, video images before and after the operating state are recorded as the result information based on the determination result. The information processing system according to claim 9 .

12. a control unit that processes video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle, analyzes the video data of the interior of the vehicle, and determines the situation based on the analysis result; The control unit has a function of determining the state of a person inside the vehicle cabin and the state of a person outside the vehicle cabin shown in the video data, and when it determines that the state of the person inside the vehicle cabin is a state in which no person is inside the vehicle cabin, performing a process of discriminating the person outside the vehicle cabin from within an area of ​​the window glass of the vehicle with priority. Information processing system.

13. a control unit that processes video data of the interior of the vehicle captured from the rear of the vehicle toward the front of the vehicle, analyzes the video data of the interior of the vehicle, and determines the situation based on the analysis result; the control unit has a function of determining a state of a first person and a state of a second person shown in the video data, and when it is determined that the first person has reached out his / her hand toward the second person as a result of the determination of the state of the first person and the state of the second person, outputting image information of an image taken inside the vehicle cabin to a predetermined output destination. Information processing system.

14. The control unit When it is determined that the first person is a passenger in the rear seat of the vehicle and the second person is a driver of the vehicle, the image information is output to the predetermined output destination; When it is determined that the first person is the driver of the vehicle and the second person is a passenger in the rear seat of the vehicle, the image information is not output to the predetermined output destination. The information processing system according to claim 13.

15. A program for causing a computer to realize the functions of the control unit of the system according to any one of claims 1 to 14.

Images