Image correction system

Abstract

To provide a technique that is preferable for business operators, and users using vehicles when processing related to abnormalities such as dirt inside the vehicles is performed by communication between the vehicles and the cloud.SOLUTION: An image correction system 1 is configured to be able to transmit data of an image taken by a camera 19 mounted on a vehicle 3 to a display device 9 via an on-vehicle device 5 and a cloud 7. The image correction system 1 includes a tilt detection unit 41 and a correction processing unit 43. The tilt detection unit 41 is configured to detect the tilt of the photographic image taken by the camera 19 with respect to a predetermined direction. The correction processing unit 43 is configured to correct the tilt of the photographic image so that it has the desired orientation based on the tilt detected by the tilt detection unit 41, and to create a display image for displaying on the display device 9.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a technique for correcting an image captured by a camera.

Background Art

[0002] Conventionally, various techniques have been proposed for capturing the surroundings of a vehicle with an in-vehicle camera and recording the data of the captured image (i.e., image data) (see, for example, Patent Document 1). Further, as a camera, a configuration in which a lens is disposed on the front side of the camera (i.e., the side facing the imaging target) and a display is disposed on the rear side of the camera is known.

[0003] When mounting the above-described camera on a vehicle, for example, the camera may be attached to the windshield or placed on the dashboard. However, since the up and down directions of the camera are defined, when the camera is attached to the windshield and when it is placed on the dashboard, the image displayed on the display (i.e., the display image) is upside down.

[0004] For example, when the mounting portion of the camera is attached to the windshield and the camera is mounted so as to be suspended, in a device in which the up and down directions of the display on the rear side of the camera are correctly displayed, if the camera is mounted upside down and the mounting portion is fixed on the dashboard so that the camera is located on the mounting portion, the up and down directions of the display image are reversed on the display.

[0005] In addition, even when the camera is installed at an inclination, due to the above-described reasons, the display image displayed on the display is also inclined. Regarding the problem that the display image is upside down due to this camera mounting method, a device is known in which a switch for reversing the up and down directions of the display image is provided in the camera and the up and down directions of the display image are manually corrected. Further, a device is also known in which a G-sensor is disposed in the camera to detect the inclination of the camera (for example, the mounting state regarding the up and down directions of the camera) and the up and down directions of the display image on the display are adjusted (for example, the display image is reversed) according to the inclination of the camera. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2008-271417 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, after detailed examination by the inventor, the following problems were identified with the conventional technology. Specifically, if a camera is equipped with a switch to invert the displayed image vertically, it complicates the camera's structure, and since the switch must be operated manually, it becomes cumbersome and inconvenient to use.

[0008] Furthermore, in configurations that use a G-sensor to switch the top and bottom of the displayed image, a G-sensor is required, which complicates the camera's structure. In addition, this increased complexity leads to higher manufacturing costs.

[0009] Furthermore, a similar problem arises when a separate display is provided from the camera, and the display itself is used to switch between the top and bottom of the screen. One aspect of this disclosure is to suppress the complexity of the camera's structure while providing a highly convenient technology for users. [Means for solving the problem]

[0010] An image correction system (1) in one aspect of the present disclosure is an image correction system configured to transmit image data captured by a camera (19) mounted on a vehicle (3) to a display device (9) via an in-vehicle device (5) and an external device (e.g., a server such as a cloud) (7).

[0011] This image correction system comprises a tilt detection unit and a correction processing unit. The tilt detection unit is configured to detect the tilt of an image captured by a camera relative to a predetermined orientation.

[0012] The correction processing unit is configured to correct the tilt of the captured image to the desired orientation based on the tilt detection unit, and to create a display image for display on the display device.

[0013] With this configuration, an image correction system according to one aspect of the present disclosure can suppress the complexity of the camera structure and provide a highly convenient technology for users.

[0014] In this disclosure, for example, even if the image captured by the camera is tilted relative to the vehicle, causing it to be tilted in a predetermined direction (e.g., the upward direction on the vehicle), the tilt can be corrected to match the intended direction (e.g., the upward direction on the vehicle or display device). Therefore, by displaying the image obtained after such correction on a display device, the sense of incongruity in the image for the viewer of the display device can be suppressed. For example, the display device can display an image with the top of the vehicle facing upwards, thus reducing the likelihood of incongruity.

[0015] Furthermore, as described above, this disclosure uses the captured image itself to correct the tilt of the captured image, thus eliminating the need for switches or G-sensors to correct image tilt, as is done in conventional systems. This results in a simplification of the camera structure and a significant improvement in user convenience.

[0016] The tilt detection unit can be installed in an on-board device or in an external device. The correction processing unit can also be installed in an on-board device or in an external device or display device.

[0017] Also, the reference signs within parentheses described in this column and in the claims indicate the correspondence with the specific means described in the embodiments described below as one aspect, and do not limit the technical scope of the present disclosure.

Brief Description of the Drawings

[0018] [Figure 1] It is a block diagram showing the overall configuration of the image correction system according to the first embodiment. [Figure 2] It is a block diagram functionally showing the second control unit of the cloud according to the first embodiment. [Figure 3] It is an explanatory diagram showing the coordinates of a camera, a vehicle, etc. [Figure 4] It is an explanatory diagram schematically showing a captured image of the interior of a vehicle taken by a camera. [Figure 5] FIG. 5A is an explanatory diagram schematically showing a captured image when the camera is not tilted, and FIG. 5B is an explanatory diagram schematically showing a displayed image when the camera is not tilted. [Figure 6] FIG. 6A is an explanatory diagram showing a state where the camera is tilted with respect to the vehicle, FIG. 6B is an explanatory diagram schematically showing a captured image when the camera is tilted, FIG. 5C is an explanatory diagram schematically showing an uncorrected displayed image when the camera is tilted, and FIG. 5D is an explanatory diagram schematically showing a corrected displayed image when the camera is tilted. [Figure 7] It is an explanatory diagram schematically showing a captured image of the interior of a vehicle with a marker taken by a camera. [Figure 8] FIG. 8A is an explanatory diagram schematically showing a captured image and a displayed image of a building when the camera is not tilted, and FIG. 8B is an explanatory diagram schematically showing a captured image of a building when the camera is tilted. [Figure 9] FIG. 9A is an explanatory diagram schematically showing a captured image of a road when the camera is tilted, FIG. 9B is an explanatory diagram schematically showing a captured image of a road sign when the camera is tilted, and FIG. 9C is an explanatory diagram schematically showing a captured image of a scenery when the camera is tilted. [Figure 10] It is an explanatory diagram showing the procedure of the processing of the image correction system according to the first embodiment. [Figure 11]This is a flowchart showing the control process of the image correction system according to the first embodiment. [Figure 12] This is an explanatory diagram showing the processing procedure of the image correction system according to the second embodiment. [Figure 13] This is a flowchart showing the control process of the image correction system according to the second embodiment. [Figure 14] This is an explanatory diagram showing the processing procedure of the image correction system according to the third embodiment. [Figure 15] This is an explanatory diagram showing a display method in the display device of the third embodiment. [Figure 16] This is a flowchart showing the control process of the image correction system according to the third embodiment. [Figure 17] This is an explanatory diagram showing the processing procedure of the image correction system according to the fourth embodiment. [Figure 18] This is a flowchart showing the control process of the image correction system according to the fourth embodiment. [Modes for carrying out the invention]

[0019] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. [1. First Embodiment] In this first embodiment, as an example of a mobility IoT system, an image correction system that corrects the tilt of images captured by a camera mounted on a vehicle (for example, an automobile) will be described. IoT stands for Internet of Things.

[0020] [1-1. Overall Structure] First, the overall configuration of the image correction system 1 of this first embodiment will be described based on Figure 1. As shown in Figure 1, the image correction system 1 comprises an in-vehicle device 5 mounted on the vehicle 3, a cloud (i.e., a cloud server) 7, and a display device 9 capable of displaying images.

[0021] Although Figure 1 shows only one in-vehicle device 5 for convenience, the image correction system 1 may have multiple in-vehicle devices 5, and each of the multiple in-vehicle devices 5 may be mounted on a different vehicle 3.

[0022] The in-vehicle device 5 can communicate wirelessly with the cloud 7 via the first communication unit 11 on the vehicle 3 side. This in-vehicle device 5 can transmit data such as images to the cloud 7. The detailed configuration of the vehicle 3 and the in-vehicle device 5 will be described later.

[0023] Cloud 7 can communicate wirelessly with the in-vehicle device 5 and the display device 9 via the second communication unit 13 on the Cloud 7 side. This Cloud 7 can collect data such as images from the in-vehicle device 5 and transmit image data and other data to the display device 9. The detailed configuration of Cloud 7 will be described later.

[0024] The display device 9 is a device capable of displaying images (i.e., display images). This display device 9 can communicate wirelessly with the cloud 7 via the third communication unit 15 on the display device 9 side. In other words, the display device 9 can display an image (for example, an image with tilt correction) on its own display 17 based on data such as images transmitted from the cloud 7.

[0025] Examples of the display device 9 include a notebook computer equipped with a display 17, a desktop computer, and the display 17. Other examples include mobile devices (i.e., information terminals) such as smartphones and tablet devices. While this first embodiment primarily uses a notebook computer as an example, it is not limited to that. The detailed configuration of the display device 9 will be described later.

[0026] The following provides a detailed explanation of each component. [1-2. Vehicle-side configuration] First, the configuration of vehicle 3 will be explained based on Figure 1.

[0027] As shown in Figure 1, the vehicle 3 is equipped with an on-board device 5 and a camera 19. Furthermore, as will be described later, the vehicle 3 may be equipped with a display device (not shown) having a display capable of showing images inside the vehicle (i.e., inside the vehicle).

[0028] Camera 19 is used to photograph the interior or exterior of the vehicle (for example, the scenery around vehicle 3), and one or more cameras are installed inside the vehicle. Camera 19 can be a digital camera, such as a CCD camera. The captured images can be color images.

[0029] Possible mounting locations for camera 19 include the top of the windshield, near the rearview mirror, on the ceiling, and on the dashboard. The camera's field of view, for example, when photographing the interior of the vehicle, would include the seats 61, windows 63, and doors (see Figure 4).

[0030] <In-vehicle equipment> Next, we will explain the in-vehicle device 5 in detail. The in-vehicle device 5 comprises a first communication unit 11, a first storage unit 21, and a first control unit 23.

[0031] As described above, the first communication unit 11 can wirelessly transmit image data and the like to the cloud 7 via the second communication unit 13. The first storage unit 21 is a storage device capable of storing information. This first storage unit 21 can store, for example, information about images (i.e., captured images) taken by the camera 19. Examples of the first storage unit 21 include a hard disk drive (i.e., HDD) and a solid disk drive (i.e., SSD).

[0032] The first control unit 23 comprises a CPU 25 and a semiconductor memory such as RAM or ROM (hereinafter referred to as memory 27). The first control unit 23 is configured, for example, by a microcomputer.

[0033] The functions of the first control unit 23 are realized by the CPU 25 executing a program stored in a non-transitional physical recording medium (i.e., memory 27). Furthermore, when this program is executed, the method corresponding to the program is executed.

[0034] Furthermore, the method for realizing the various functions of the first control unit 23 is not limited to software; some or all of its elements may be realized using one or more hardware components. For example, if the above functions are realized by an electronic circuit, which is hardware, that electronic circuit may be a digital circuit containing a large number of logic circuits, an analog circuit, or a combination thereof.

[0035] [1-3. Cloud-side configuration] Next, we will explain the configuration on the Cloud 7 side based on Figures 1 and 2. Cloud 7 comprises a second communication unit 13, a second storage unit 31, and a second control unit 33.

[0036] The second communication unit 13 can communicate wirelessly with the first communication unit 11 and the third communication unit 15. For example, Cloud 7 can receive image data etc. transmitted from the in-vehicle device 5 via the first communication unit 11 and the second communication unit 13, and can transmit image data etc. to the display device 9 via the second communication unit 13 and the third communication unit 15.

[0037] The second storage unit 31 is a storage device that stores information similar to that of the first storage unit 21 of the in-vehicle device 5, and can store image data of images captured by the camera 19, etc. The second control unit 33 comprises a CPU 35 and a semiconductor memory such as RAM or ROM (hereinafter referred to as memory 37, which is a non-transitional physical recording medium). The configuration and functions of the second control unit 33 are basically the same as those of the first control unit 23 of the in-vehicle device 5, and are realized by the CPU 35 executing a program stored in memory 37. Furthermore, when this program is executed, a method corresponding to the program is executed.

[0038] Furthermore, the Cloud 7 with the above configuration can collect vehicle 3 data transmitted from each of the multiple in-vehicle devices 5 via the first communication unit 11. In addition, the Cloud 7 can store the collected data for each vehicle 3 in the second storage unit 31.

[0039] Furthermore, Cloud 7 creates a digital twin based on the vehicle 3 data stored in the second memory unit 31. The digital twin is normalized index data. Furthermore, if Cloud 7 is connected to a service provision server (not shown), it can retrieve data for a predetermined vehicle 3 stored in the second storage unit 31 using index data acquired from the digital twin. The service provision server can determine the control content for vehicle 3 and send instructions corresponding to the control content to Cloud 7. Based on these instructions, Cloud 7 can send the control content to vehicle 3.

[0040] <Functional configuration of the second control unit in the cloud> Here, we will describe the functional configuration of the second control unit 33 of Cloud 7. As shown in Figure 2, the second control unit 33 functionally comprises a tilt detection unit 41 and a correction processing unit 43.

[0041] The tilt detection unit 41 is configured to detect the tilt (for example, the angle of rotation from the upward direction) of an image captured by the camera 19 with respect to a predetermined orientation (for example, the upward direction in the vertical direction of the vehicle 3).

[0042] Here, "direction" refers to orientation in a direction such as the up-and-down direction or the Y-axis direction (for example, the direction of a vector or the direction of a clock hand). The "upward direction" is sometimes simply referred to as "upward."

[0043] The correction processing unit 43 is configured to correct the tilt of the captured image to the desired orientation (for example, the original orientation of the captured image when the camera 19 is not tilted) based on the tilt detected by the tilt detection unit 41, and to create a display image to be displayed on the display device 9.

[0044] For example, the captured image is corrected so that the top of the captured image is the top of the vehicle 3, as if the camera 19 were not tilted, and a display image is created. Therefore, as will be described later, the display 17 will display the image so that the upper part of the vehicle 3 is at the top of the display 17.

[0045] [1-4. Display Device Configuration] Next, the configuration of the display device 9 will be explained based on Figure 1. The display device 9 is, for example, a laptop computer and comprises a third communication unit 15, a third storage unit 51, a third control unit 53, and a display 17.

[0046] As described above, the third communication unit 15 can wirelessly receive image data and the like from the cloud 7 via the second communication unit 13. The third storage unit 51 is a storage device that stores information similar to that of the first storage unit 21 of the in-vehicle device 5, and can store image data and the like transmitted from the cloud 7.

[0047] The third control unit 53 comprises a CPU 57 and a semiconductor memory such as RAM or ROM (hereinafter referred to as memory 59, which is a non-transitional physical recording medium). The configuration and functions of the third control unit 53 are basically the same as those of the first control unit 23 of the in-vehicle device 5, and are realized by the CPU 57 executing a program stored in memory 59. Furthermore, when this program is executed, a method corresponding to the program is executed.

[0048] The display 17 is a part of the display device 9 that displays various images, etc. (for example, a liquid crystal display). The display 17 can, for example, display an image with the tilt of the captured image corrected based on image data transmitted from the cloud 7.

[0049] If the display 17 has a rectangular shape when viewed from the front, and the display device 9 is placed on a horizontal surface, then one long side of the display 17 becomes the upper end and the other long side becomes the lower end.

[0050] [1-5. Method for detecting and correcting image tilt] Next, we will explain a method for correcting the captured image taken by camera 19 to create a display image that is in the desired orientation after correction (i.e., a correction method).

[0051] In this first embodiment, as will be described later, when photographing the interior of a vehicle, the tilt of the captured image is corrected so that the ceiling side of the vehicle 3 is at the top to create the display image. Then, the display 17 displays the top of the display image (i.e., the ceiling side of the vehicle 3) so that it aligns with the top of the display 17.

[0052] Furthermore, when photographing the outside of the vehicle, the tilt of the captured image is corrected so that the sky side of the scenery is at the top when creating the display image. Then, on the display 17, the top of the display image (i.e., the sky side) is displayed so that it aligns with the top of the display 17.

[0053] In the following examples, we will use orthogonal coordinate systems where the X, Y, and Z axes are orthogonal, as shown in Figure 3. For example, in the coordinate system SZ of vehicle 3, the Y axis represents the vertical direction of vehicle 3, the X axis represents the left-right direction when viewed from the front of vehicle 3, and the Z axis represents the front-to-back direction. The origin of this coordinate system SZ could be, for example, the center of gravity of vehicle 3.

[0054] Furthermore, in the coordinate system CZ of camera 19, the Z axis represents the optical axis direction of camera 19, the Y axis represents the vertical direction of camera 19, and the X axis represents the horizontal direction when viewed from the opposite side of camera 19 from the lens side (for example, the indoor side).

[0055] In the coordinate system DZ of display 17, the Y axis represents the vertical direction (short side) of display 17, the X axis represents the horizontal direction (long side) of display 17, and the Z axis represents the direction perpendicular to the display surface of display 17.

[0056] Furthermore, the positive side in the X-axis direction is denoted as X+ and the negative side as X-, the positive side in the Y-axis direction as Y+ and the negative side as Y-, and the positive side in the Z-axis direction as Z+ and the negative side as Z-. Z+ is the direction of the back side of the paper in Figure 3, and Z- is the direction of the front side of the same paper.

[0057] In this first embodiment, the display 17 displays a display image in which the tilt of the captured image has been corrected, that is, a display image in which the upward orientation of the vehicle 3, scenery, etc., matches the upward orientation of the display 17, as described below.

[0058] In other words, since the vertical direction of vehicle 3 and the vertical direction of scenery etc. are the same as the vertical direction of display 17, we will explain using the direction indicating upwards (i.e., upward) as an example of the desired orientation.

[0059] Several correction methods are possible, so the following sections will explain various correction methods. <Method of correction based on images taken inside the vehicle> Here, we will consider an example where camera 19 is mounted on the windshield, for example, to film the inside of the car.

[0060] The optical axis of camera 19 (the Y-axis direction in coordinate system CZ) is parallel to the front-to-back direction of vehicle 3 (the Y-axis direction in coordinate system SZ). Even if the optical axis of camera 19 and the front-to-back direction of vehicle 3 are not parallel, the tilt of the captured image (e.g., vertical) can be corrected using the correction principle described later. A simple example will be given here to illustrate this point.

[0061] Note that the vertical orientation of camera 19 and the vertical orientation of the captured image are the same, and the vertical orientation of camera 19 is predetermined by the camera 19 itself, for example, by the arrangement of the image sensor that receives light and the coordinates of the image sensor.

[0062] First, we will explain the case where the upward direction (Y+) of camera 19 in the vertical direction coincides with the upward direction (Y+) of vehicle 3 in the vertical direction, that is, the case where camera 19 is not rotated and tilted around the optical axis.

[0063] If camera 19 is not tilted, when camera 19 photographs the interior of the vehicle, as shown in Figure 4, the top and bottom of structures inside the vehicle, such as seats 61 and windows 63, will also be aligned in the captured image (i.e., the captured image). For example, in the captured image, the upper end of the backrest 61a of seat 61 is above, and the seat surface 61b is below.

[0064] Therefore, the captured image is displayed directly on the display 17. In other words, the image is displayed so that the top and bottom of the image match the top and bottom of the display 17. Specifically, for a captured image of the interior of vehicle 3, the top of vehicle 3 is displayed on the top of the display 17, and the bottom of vehicle 3 is displayed on the bottom of the display 17.

[0065] This will be explained in detail using Figures 5A and 5B. In Figures 5A and 5B, only the rear window 63 in the captured image is schematically shown as a trapezoid, with the lower base of the trapezoid representing the linear end (lower end) extending in the left-right direction of the vehicle 3 below the window 63, and the upper base, which is shorter than the lower base, representing the linear end (upper end) extending in the left-right direction of the vehicle 3 above the window 63.

[0066] Furthermore, when vehicle 3 is on a level surface, the upper and lower edges of the window 63 are horizontal and perpendicular to the vertical direction indicating the up-and-down direction of vehicle 3. As shown in Figure 3, when viewed from the front of the vehicle 3, if the camera 19 is not tilted relative to the vehicle 3, then the Y-axis direction and orientation of the camera 19 (e.g., upward: Y+) and the Y-axis direction and orientation of the vehicle 3 (e.g., upward: Y+) are in the same direction and orientation.

[0067] Therefore, when the image captured by camera 19 (see Figure 5A) is displayed on display 17, as shown in Figure 5B, the top and bottom of camera 19, the top and bottom of vehicle 3, and the top and bottom of display 17 are aligned, so the image captured by camera 19 is displayed as is. In other words, the top and bottom of the captured image and the top and bottom of the displayed image are aligned. More specifically, the top and bottom of the trapezoidal window 63 in the captured image and the top and bottom of the trapezoidal window 63 in the displayed image are aligned.

[0068] In contrast, when viewed from the front of vehicle 3, if camera 19 is rotated around its optical axis and tilted relative to vehicle 3 (see Figure 6A), the orientation of camera 19 (e.g., upward) and the orientation of vehicle 3 (e.g., upward) will be different. Note that if camera 19 is rotated 180 degrees around its optical axis, the Y-axis directions of camera 19 and vehicle 3 will coincide, but their orientations (i.e., the upper side of camera 19 and the upper side of vehicle 3) will be reversed because their tops and bottoms are inverted.

[0069] For example, as shown in Figure 6A, if the Y-axis of camera 19 is rotated α degrees (e.g., 45 degrees) to the right when viewed from the front of vehicle 3, then the orientation of camera 19 (e.g., upward: Y+) is also rotated to the right by, for example, 45 degrees.

[0070] In this state, when the interior of the vehicle is photographed with camera 19, the interior structures will appear in an image rotated in the opposite direction to the rotation of camera 19. In other words, as shown in Figure 6B, the image captured by camera 19 will be rotated and tilted 45 degrees to the left (i.e., 315 degrees to the right). Note that the upward orientation of camera 19 (upward) and the upward orientation of the captured image (upward) are the same.

[0071] Therefore, when this captured image is displayed on the display 17 as is (i.e., without correction), the upward orientation of the camera 19 (i.e., upward) and the upward orientation of the display 17 are displayed to match. In other words, the upward orientation of the captured image and the upward orientation of the displayed image are displayed to match. That is, as shown in Figure 6C, the interior structures of the vehicle are displayed rotated 45 degrees to the left and tilted, just as in the captured image.

[0072] When displayed in this way, the image is in an unusual orientation, which naturally causes discomfort to the viewer. Therefore, in this first embodiment, the captured image is corrected to prevent this discomfort.

[0073] In other words, as shown in Figure 6D, the tilt of the captured image caused by the tilt of the camera 19 is corrected so that the upper side of the structure inside the vehicle becomes the upper side of the display 17 and the lower side of the structure becomes the lower side of the display 17. Specifically, the captured image is corrected by rotating it 45 degrees to the right to create the display screen, and the display image obtained through this correction is displayed on the display 17.

[0074] Thus, in this first embodiment, the display 17 displays a display image in which the tilt of the captured image has been corrected, that is, a display image in which the upward orientation (upward) of the vehicle 3 matches the upward orientation (upward) of the display 17.

[0075] Therefore, even if the camera 19 is tilted, the display 17 can display an image that looks as if the camera 19 were not tilted. This reduces the unnatural appearance of the image displayed on the display 17, resulting in a very easy-to-view image.

[0076] Here, the in-vehicle structures used to correct the captured images can include, for example, various structures that can recognize the left-right and up-down directions of vehicle 3. For example, as mentioned above, the linear upper and lower ends of the rear window 63 and the upper end of the seat backrest 61a of the seat 61 coincide with the left-right direction of the vehicle 3, so they can be used to recognize the left-right direction of the vehicle 3. Furthermore, the difference in length between the upper and lower ends of the trapezoidal window 63, and the inclination of the left and right ends of the trapezoid, can be used to determine the top and bottom of the window 63. In addition, if there is a lamp in the center of the ceiling, the direction of the lamp in the captured image can be determined to be upward in the vertical direction.

[0077] As shown in Figure 7, a marker 71 indicating the position inside the vehicle is placed inside the vehicle, and the tilt of the captured image can be detected from the image of the marker 71.

[0078] Specifically, as shown in Figure 7, for example, markers 71, such as two-dimensional codes, are placed on the upper left and right and lower left and right sides of the vehicle interior to indicate their respective positions. The line connecting the upper left and right markers 71 and the line connecting the lower left and right markers 71 are parallel. Furthermore, the left and right markers 71 are positioned symmetrically with respect to the Y-axis of the vehicle's coordinate system SZ.

[0079] Therefore, when the camera 19 photographs the interior of the vehicle, each marker 71 is captured in the image, and the position of each marker 71 (i.e., its position in the vehicle, both vertically and horizontally) can be recognized from the image of each marker 71. In other words, from each marker 71, it is possible to determine the orientation of the vehicle 3 in each direction, in addition to its vertical and horizontal directions.

[0080] Furthermore, for example, the upper side of vehicle 3 can be determined from the upper left and right markers 71, and the line connecting these left and right markers 71 indicates the left and right sides of vehicle 3. Therefore, from the captured image of the markers 71, it is possible to determine how much the camera 19 (and thus the captured image) is tilted relative to vehicle 3.

[0081] Therefore, since the tilt of the captured image can be detected, as described above, the correct orientation of the image can be displayed by correcting the tilt of the captured image. The tilt of camera 19 (and therefore the captured image) can be detected from the captured image of the scenery outside the vehicle, as described below. Furthermore, as will be described later, when using an image of the scenery outside the vehicle, the image is displayed on the display 17 such that the vertically upward direction of the scenery (i.e., the surroundings outside the vehicle) is above the display 17.

[0082] For the sake of simplicity, we will use the example of vehicle 3 being on level ground. a) Use a photograph taken of building 65 in front of vehicle 3. If the camera 19 is not tilted with respect to the vertical direction (and therefore the vertical direction) of the vehicle 3 with its optical axis as the center of rotation, then, as in Figures 5A and 5B, the top and bottom of the captured image coincide with the top and bottom of the display 17 (i.e., the top and bottom of the displayed image).

[0083] Therefore, when the building 65 is photographed, as shown in Figure 8A, the roof 67 of the building 65 is displayed above the side wall (i.e., the exterior wall) 69 in both the photographed image and the displayed image, so there is no sense of incongruity in the image displayed on the display 17.

[0084] On the other hand, if camera 19 is tilted, the building 65 will be photographed at an angle, similar to when photographing the interior of a car, so displaying it as is on display 17 would look unnatural. For example, as shown in Figure 8B, if the camera 19 is rotated α degrees (e.g., 45 degrees) to the right with respect to the upward direction around the optical axis, the captured image will be rotated α degrees to the left (e.g., an image where the building 65 is tilted). Therefore, it is necessary to correct the tilt of the captured image.

[0085] This section describes the case where image recognition or other processing is performed on the captured image, and it is detected that the captured object is a building 65. For example, if the outline of an object is enclosed by lines that include parallel lines and lines that extend perpendicular to those parallel lines, that is, if one of the outlines is convex, the object may be considered a building 65.

[0086] Therefore, the parallel lines constituting the building 65 can be considered as lines constituting the exterior wall 69, and the direction of these lines can be considered as the vertical direction. In this case, by considering the line extending perpendicularly to the line of the exterior wall 69 as the upper end of the roof 67, the upward direction in the vertical direction can be detected. Accordingly, by detecting the inclination of the line of the exterior wall 69 with respect to the upward direction in the vertical direction of the captured image, the inclination of the captured image can be detected.

[0087] Therefore, the tilt of this captured image is corrected so that the straight line of the exterior wall 69 in the captured image matches the original vertical direction. More specifically, the straight line of the exterior wall 69 is corrected so that it matches the vertical direction of the original camera 19 in the correct position (i.e., the vertical direction of the original captured image), and the roof 67 is on the upper side in the vertical direction (i.e., the upper side of the display screen).

[0088] Specifically, for example, by rotating the captured image 45 degrees to the right and correcting it, it is possible to create a display image that does not have any vertical inconsistencies (for example, with the roof 67 side facing upwards).

[0089] b) Use images taken of the road in front of vehicle 3. As shown in Figure 9A, if there are straight lines on the road, the tilt of the captured image can be detected using the white lines on the side of the road or the center line.

[0090] For example, in the case of left-hand traffic, a virtual line is determined at the midpoint between the left white line and the center line, and this virtual line is considered to represent the front-to-back direction of the vehicle 3 (and therefore the up-and-down direction in the original captured image) when the camera 19 is not tilted and is photographed from its original position.

[0091] Therefore, the tilt of the virtual line relative to the vertical direction of the actual captured image (for example, upward) is detected as the tilt of the captured image, and the captured image is rotated and corrected so that the virtual line is in the vertical direction. At this time, the correction is made so that the upper end of the virtual line (i.e., the side furthest from vehicle 3) is at the top of the captured image.

[0092] Furthermore, as shown in Figure 9B, if a road sign is present in the captured image, the vertical direction and upward direction from the road sign can be recognized using known image recognition techniques. In other words, when recognizing text or other elements from road signs, it is possible to recognize the upward direction in the vertical direction, thus enabling the detection of the tilt of the captured image. Furthermore, the vertical direction can be recognized from the support posts of road signs.

[0093] Furthermore, even when guardrails or guardrail posts are present in the captured image, known image recognition techniques can recognize the vertical direction and the upward direction (i.e., the sky side, the upper side). In other words, since the posts extend vertically and the guardrails are positioned on top of the posts, the vertical direction and the upward direction can be recognized. The same applies to utility poles, etc.

[0094] Therefore, based on the detected tilt, the image can be corrected to eliminate the tilt of the captured image and then displayed. c) Use images taken of the terrain in front of vehicle 3. As shown in Figure 9C, known image recognition techniques can identify, for example, a straight line crossing an image during the daytime. If one region separated by the line is brighter and the other region (indicated by the diagonal lines) is darker than the first region, then one region can be considered the sky and the other region the land or sea.

[0095] Furthermore, if there are roads or buildings 65 in the other area, it can be considered land, and if there is a curved line that can be considered a coastline, it can be considered sea. In such cases, the aforementioned straight line can be considered as the horizon or horizontal line, and the tilt of the captured image can be detected using this horizon or horizontal line.

[0096] For example, by considering a line perpendicular to the horizon as a vertical line, it is possible to detect how much this vertical line is tilted relative to the vertical direction of the captured image. In this case, the image is corrected so that the sky is at the top of the vertical direction.

[0097] Therefore, based on the detected tilt, the image can be corrected to eliminate the tilt of the captured image and then displayed. Furthermore, when using images taken from outside the vehicle, even if the ground is sloped relative to the horizontal plane (for example, if the left and right sides of vehicle 3 are tilted), the vertical orientation of the captured image is corrected, so that the display 17 can display an image with the correct vertical orientation (i.e., an image in which the sky is displayed at the top of the display 17).

[0098] [1-6. Overall Processing Procedure] Next, the overall processing procedure of the image correction system 1 of this first embodiment will be described based on Figure 10. Note that Figure 10 shows only the essential parts of the configuration shown in Figure 1.

[0099] In this first embodiment, the process of detecting the tilt of the captured image and the process of correcting the tilt of the captured image are performed in Cloud 7. As shown in Figure 10, in this first embodiment, the data of the image captured by the camera 19 (i.e., the captured image) is transmitted from the in-vehicle device 5 to the cloud 7. Next, the cloud 7 corrects the captured image (i.e., corrects the tilt) to create a display image. Then, the data of the display image is transmitted to the display device 9, and the display image is displayed on the display 17 of the display device 9.

[0100] The details are explained below. As shown in Figure 10, camera 19 takes pictures of, for example, the outside of the vehicle (for example, the front of vehicle 3) or the inside of the vehicle. Here, we will explain using an example image of the scenery outside the vehicle. In this example image, we will use an image that shows the land in the shaded area, the white sky, and the horizon.

[0101] The image shown in Figure 10 may also be an image of the rear of the vehicle interior. In this case, the image may show the rear door indicated by the diagonal lines, the white window of that door, and the line at the bottom edge of the window.

[0102] This section describes the case where vehicle 3 is on a horizontal plane (the same applies to other embodiments below). Note that if the ground is sloped, the image captured from outside the vehicle will be tilted even if camera 19 is not tilted relative to vehicle 3. In other words, if camera 19 is tilted relative to the scenery outside the vehicle, the image will be up and down incorrectly (the same applies to other embodiments below). However, even in that case, processing is performed to recognize the up and down of the captured image, so a display image with correct orientation and minimal discrepancy can be obtained from a captured image of the scenery outside the vehicle that is tilted, as an image of the scenery outside the vehicle.

[0103] On the other hand, when using images taken inside the vehicle, if the camera 19 is tilted relative to the vehicle 3, the camera performs the same process as described above to recognize the top and bottom of the captured image, regardless of the tilt of the ground. Therefore, even if the captured image inside the vehicle is tilted, a display image with the correct top and bottom orientation and minimal discrepancy can be obtained, as an image taken inside the vehicle.

[0104] First, as shown in Figure 10, if the camera 19 is mounted at an angle to the vehicle 3, the image captured by the camera 19 (i.e., the leftmost image) will also be an image where the top and bottom of the sky and land are incorrect. In other words, an image with incorrect vertical orientation is obtained. In the following, the vertical orientation refers to the direction in which it is possible to determine which is up and which is down, and the correct vertical orientation refers to the direction that is both correctly upward and correctly downward.

[0105] Note that the leftmost image in Figure 10 shows an image that has been rotated 135 degrees to the right compared to the original image with the correct vertical orientation (i.e., the image taken when camera 19 is not tilted).

[0106] Next, the image data of the captured image taken by camera 19 is sent to the first control unit 23 of the in-vehicle device 5 and transmitted to the cloud 7 from the antenna 11a of the first communication unit 11. Next, the image data transmitted to Cloud 7 is sent to the second control unit 33, where the display orientation is corrected. That is, the tilt of the captured image, which is incorrect in the vertical direction, is corrected to create a display image with the correct vertical direction. For example, the captured image that is rotated 135 degrees to the right is rotated 135 degrees to the left. This results in an image where the sky is at the top, the land is at the bottom, and the horizon is horizontal (i.e., it matches the left-right direction of the display image).

[0107] Next, the cloud 7 transmits the image data corrected for the display orientation (i.e., image data of the display image) to the display device 9. This image data is sent to the third control unit 53, and is displayed on the display 17 after processing by the third control unit 53.

[0108] As a result, the display 17 displays a corrected, vertically oriented image of the captured image. In other words, the display 17 displays an image with the sky at the top, the land at the bottom, and the horizon horizontal (i.e., an image where the horizon aligns with the left-right direction of the displayed image), as shown in the image on the far right of Figure 10.

[0109] [1-7. Control Processing] Next, the control processing performed by the image correction system 1 will be explained based on Figure 11. Note that this control processing may be performed continuously after the vehicle 3 is started, for example, when photographing the exterior of the vehicle. Alternatively, when photographing the interior of the vehicle, it may be performed at least once after the vehicle 3 is started (for example, when the ignition switch is turned on).

[0110] In S100 of Figure 11, the in-vehicle device 5 takes a photograph of the outside (or inside) of the vehicle using the camera 19. In the subsequent step S110, the in-vehicle device 5 receives the image captured by the camera 19 (i.e., the captured image) into the first control unit 23.

[0111] In the following step S120, the in-vehicle device 5 sends the image data of the captured image from the first communication unit 11 to the cloud 7. In the subsequent S130, the second control unit 33 of Cloud 7 recognizes the tilt (i.e., the vertical direction) of the image data and corrects the image data so that it is in the correct vertical direction.

[0112] Here, the correct vertical direction includes, as mentioned above, which direction is up and which is down. For example, when using an image from outside the vehicle, it indicates the vertical direction of up and down, and when using an image from inside the vehicle, it indicates the vertical direction of up and down of the vehicle 3. Therefore, on the display 17, the vertical direction of the corrected image is displayed to match the vertical direction of the display 17.

[0113] In the following step S140, Cloud 7 sends the corrected image data (i.e., the image data of the display image) to the display device 9. In the subsequent S150, the image (i.e., the displayed image) made from the corrected image data is displayed on the display 17, and this process is temporarily terminated.

[0114] [1-8. Effects] According to this first embodiment, the following effects can be obtained. (1a) In this first embodiment, the camera 19 detects the tilt of the vehicle 3 in the vertical direction (for example, upward) of the captured image, corrects the captured image to the correct vertical direction, and displays the corrected image on the display 17 of the display device 9.

[0115] This allows for correction of the tilt so that, for example, if the camera 19 is tilted relative to the vehicle 3, the image captured by the camera 19 is tilted relative to the vertical direction of the vehicle 3 (for example, upward), so that the tilt matches the desired orientation (for example, the upward orientation of the vehicle 3 or the display 17).

[0116] Therefore, by displaying the corrected image on the display 17, the sense of incongruity in the image for the viewer can be suppressed. In other words, the display 17 can display the correct vertical image with the top of the vehicle 3 at the top, making it less likely to cause discomfort.

[0117] Furthermore, if camera 19 is tilted relative to the scenery outside the vehicle, the same correction can be made. In other words, the image can be corrected so that the sky is at the top and the land or sea is at the bottom, thus reducing the unnatural appearance of the displayed image.

[0118] (1b) Moreover, in this first embodiment, the tilt of the captured image is corrected as described above, so a switch or G-sensor for correcting the tilt of the image can be omitted as in the conventional method. Therefore, the structure of the camera 19 can be simplified, and the convenience for the user can be improved, which is a significant benefit.

[0119] (1c) Examples of images captured by camera 19 include structures inside the vehicle, markers 71, or scenery outside the vehicle (i.e., around the vehicle). In other words, the tilt of the captured image can be corrected based on images of seats 61, windows 63, markers 71 inside the vehicle, or buildings 65, roads, signs, scenery, etc. outside the vehicle.

[0120] (1d) The detection and correction of the tilt of the captured image can be performed by Cloud 7, but as will be described later, it may also be performed by the in-vehicle device 5. [1-9. Correspondence] Next, the relationship between this first embodiment and this disclosure will be described.

[0121] Image correction system 1 corresponds to the image correction system, vehicle 3 corresponds to the vehicle, in-vehicle device 5 corresponds to the in-vehicle device, cloud 7 corresponds to the cloud, display device 9 corresponds to the display device, camera 19 corresponds to the camera, tilt detection unit 41 corresponds to the tilt detection unit, and correction processing unit 43 corresponds to the correction processing unit.

[0122] [2. Second Embodiment] Next, a second embodiment will be described. Since the basic configuration of the second embodiment is the same as that of the first embodiment, the following description will mainly focus on the differences from the first embodiment. Note that the same reference numerals as in the first embodiment indicate the same components, and refer to the preceding description.

[0123] The hardware configuration of the image correction system 1 in this second embodiment is the same as that of the first embodiment, so its description will be omitted. In this second embodiment, the in-vehicle device 5 performs the process of detecting the tilt of the captured image and the process of correcting the tilt of the captured image.

[0124] [2-1. Processing Procedure] As shown in Figure 12, camera 19 takes pictures of the outside or inside of the vehicle. Here, we will explain using an image taken of the scenery outside the vehicle as an example.

[0125] As shown in the leftmost image of Figure 12, if the camera 19 is mounted at an angle to a vehicle 3 on a horizontal surface, for example, the image captured by the camera 19 will also be an image where the top and bottom of the sky and land are not correctly represented.

[0126] Furthermore, if camera 19 is tilted relative to the scenery outside the vehicle, the resulting image will be upside down, as described above. Next, the image data of the captured image taken by camera 19 is sent to the first control unit 23 of the in-vehicle device 5, where the display orientation is corrected. That is, the tilt of the captured image, which is incorrect in the vertical direction, is corrected to create a display image with the correct vertical direction.

[0127] Next, the first communication unit 11 of the in-vehicle device 5 transmits the image data with the corrected display orientation (i.e., the image data of the display image) to the cloud 7. Next, the cloud 7 transmits the image data with the corrected display orientation to the display device 9. This image data is then displayed on the display 17 by processing by the third control unit 53.

[0128] As a result, the display 17 shows a corrected, vertically oriented table image of the captured image, with the tilt corrected. [2-2. Control Processing] Next, the control process of this second embodiment will be explained with reference to Figure 13.

[0129] In S200 of Figure 13, the in-vehicle device 5 takes a photograph of the outside (or inside) of the vehicle using the camera 19. In the following step S210, the in-vehicle device 5 takes in the image (captured image) captured by the camera 19 into the first control unit 23.

[0130] In the subsequent S220, the first control unit 23 recognizes the tilt (i.e., the vertical direction) of the image data and corrects the image data so that it is in the correct vertical direction. In the following step S230, the in-vehicle device 5 sends the corrected image data from the first communication unit 11 to the cloud 7.

[0131] In the subsequent S240, Cloud 7 sends the corrected image data to the display device 9. In the subsequent S250, the image (i.e., the displayed image) made from the corrected image data is displayed on the display 17, and this process is temporarily terminated.

[0132] This second embodiment provides the same effects as the first embodiment. [3. Third Embodiment] Next, a third embodiment will be described. Since the basic configuration of the third embodiment is the same as that of the first embodiment, the following description will mainly focus on the differences from the first embodiment. Note that the same reference numerals as in the first embodiment indicate the same components, and refer to the preceding description.

[0133] The hardware configuration of the image correction system 1 in this third embodiment is the same as that of the first embodiment, so its description will be omitted. In this third embodiment, the cloud 7 performs a process to detect the tilt of the captured image and assign a vector, and the display device 9 performs a process to correct the tilt of the captured image.

[0134] [3-1. Processing Procedure] As shown in Figure 14, camera 19 takes pictures of the outside or inside of the vehicle. Here, we will explain using an image taken of the scenery outside the vehicle as an example.

[0135] Image data of the captured image by camera 19 is sent to the first control unit 23 of the in-vehicle device 5 and transmitted to the cloud 7 from the antenna 11a of the first communication unit 11. Next, in Cloud 7, the second control unit 33 assigns a vertical vector V1 to the image data. This vertical vector V1 indicates, for example, how much the image is tilted with respect to the vertical direction of the vehicle 3 (for example, upward), and a unit vector represented by coordinates (x, y) is an example. The origin of these coordinates can be, for example, the center of the captured image in terms of top, bottom, left, and right.

[0136] In addition to the vertical vector V1, rotation angle data indicating the degree to which the image is rotated in the vertical direction (for example, upward) may also be added. Next, the cloud 7 sends image data with the vertical vector V1 data attached to it to the display device 9.

[0137] Next, in the display device 9, the third control unit 53 corrects the image data of the captured image based on the vertical vector V1 to create a display image with the correct vertical orientation, and displays it on the display 17.

[0138] As a result, as shown in Figure 15, the display 17 of the display device 9 displays a corrected, vertically oriented image of the captured image. In other words, a display image is displayed that has been processed so that the vertical vector V1 of the image data matches the vertical vector V2 of the display device 9 (and therefore the display 17), which indicates the orientation of the display device 9.

[0139] Note that the upper and middle diagrams of Figure 15 show the case where the vertical direction of the display 17 coincides with the vertical direction. The lower diagram of Figure 15 shows the case where the vertical direction of the display 17 is tilted relative to the vertical direction, as is the case with the display 17 of a mobile device. The tilt of a mobile device can be detected by a G-sensor or similar device.

[0140] [3-2. Control Processing] Next, the control process of this third embodiment will be described with reference to Figure 16. In S300 of Figure 16, the in-vehicle device 5 takes a photograph of the outside (or inside) of the vehicle using the camera 19.

[0141] In the following step S310, the in-vehicle device 5 receives the image (captured image) taken by the camera 19 into the first control unit 23. In the following step S320, the in-vehicle device 5 sends the image data of the captured image from the first communication unit 11 to the cloud 7.

[0142] In the subsequent S330, Cloud 7, using the second control unit 33, recognizes the tilt (i.e., the vertical direction) of the image data and assigns a vertical vector V1 to the image data. In the subsequent S340, image data with the vertical vector V1 data attached is transmitted from Cloud 7 to Display Device 9.

[0143] In the subsequent step S350, the third control unit 53 of the display device 9 corrects the tilt of the captured image so that the vertical vector V1 of the image data matches the vertical vector V2 of the display device 9 (and therefore the display 17).

[0144] This creates a correctly oriented display image, which is then displayed on display 17, and the process is temporarily terminated. This third embodiment provides the same effects as the first embodiment.

[0145] [4. Fourth Embodiment] Next, the fourth embodiment will be described. Since the basic configuration of the fourth embodiment is the same as that of the third embodiment, the following description will mainly focus on the differences from the first embodiment. Note that the same reference numerals as in the third embodiment indicate the same components, and refer to the preceding description.

[0146] The hardware configuration of the image correction system 1 in this fourth embodiment is the same as that of the first embodiment, so its description will be omitted. In this fourth embodiment, the in-vehicle device 5 detects the tilt of the captured image and assigns a vector to it, and the display device 9 performs a process to correct the tilt of the captured image.

[0147] [4-1. Processing Procedure] As shown in Figure 17, camera 19 takes pictures of the outside or inside of the vehicle. Here, we will explain using an image taken of the scenery outside the vehicle as an example.

[0148] Image data of the captured image by camera 19 is sent to the first control unit 23 of the in-vehicle device 5. The first control unit 23 assigns a vertical vector V1 to the image data. Next, the antenna 11a of the first communication unit 11 transmits image data with the vertical vector V1 data attached to it to the cloud 7.

[0149] Next, the cloud 7 sends image data with the vertical vector V1 data attached to it to the display device 9. The third control unit 53 corrects the image data of the captured image based on the vertical vector V1 to create a display image with the correct vertical orientation and displays it on the display 17.

[0150] [4-2. Control Processing] Next, the control process of this fourth embodiment will be described with reference to Figure 18. In S400 of Figure 18, the in-vehicle device 5 takes a photograph of the outside (or inside) of the vehicle using the camera 19.

[0151] In the following step S410, the in-vehicle device 5 takes in the image (captured image) captured by the camera 19 into the first control unit 23. In the following step S42, the first control unit 23 of the in-vehicle device 5 recognizes the tilt (i.e., vertical direction) of the image data and assigns a vertical vector V1 to the image data.

[0152] In the subsequent S430, the in-vehicle device 5 transmits image data with vertical vector V1 data attached to it to the cloud 7. In the subsequent S440, Cloud 7 transmits image data with vertical vector V1 data attached to it to the display device 9.

[0153] In the subsequent step S450, the third control unit 53 of the display device 9 corrects the tilt of the captured image so that the vertical vector V1 of the image data matches the vertical vector V2 of the display device 9 (and therefore the display 17).

[0154] This creates a correctly oriented display image, which is then displayed on display 17, and the process is temporarily terminated. This fourth embodiment provides the same effects as the third embodiment.

[0155] [5. Other Embodiments] While embodiments of this disclosure have been described above, it goes without saying that this disclosure is not limited to the embodiments described above and can take various forms.

[0156] (5a) The control processing in each embodiment may be performed continuously after the vehicle has started. In addition, if correction is performed based on captured images taken inside the vehicle, it may be performed at least once after the vehicle has started.

[0157] (5b) If there are multiple cameras (for example, cameras that photograph the interior of a vehicle), the tilt described above may be detected based on the image captured by one camera, and the correction described above may be applied to the image captured by the other cameras.

[0158] For example, if there is a camera A that photographs the second row of seats inside the vehicle and a camera B that photographs the third row, and it is known that cameras A and B are mounted in the same orientation relative to the vehicle's vertical ceiling, then the tilt correction of camera B will be corrected based on the correction information of camera A, and no correction will be detected from the image of camera B itself. This reduces the processing load on the control unit compared to correcting the tilt of each camera separately.

[0159] (5c) Examples of display devices equipped with a display include, as mentioned above, personal computers and mobile terminals used outside the vehicle by businesses such as car sharing companies or by individuals, but also display devices placed on the dashboard inside the vehicle, for example. Such display devices are also suitable because they can display images in the correct vertical orientation, causing less discomfort to the driver and others.

[0160] (5d) The cloud, or at least a part of the cloud, may be on-premises. For example, cloud servers or on-premises servers can be used as devices outside the vehicle (i.e., outside the vehicle). The cloud may also be a hybrid cloud combining a public cloud, a private cloud, and on-premises systems.

[0161] For example, at least a portion of the cloud in each embodiment may be managed and operated on external servers or other equipment provided by the company that provides the vehicle or in-vehicle device. (5e) The image correction system described herein may be implemented by a dedicated computer provided by configuring a processor and memory to perform one or more functions embodied by a computer program.

[0162] Alternatively, the image correction system described herein may be implemented by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits.

[0163] Alternatively, the image correction system described herein may be implemented by one or more dedicated computers comprising a combination of a processor and memory programmed to perform one or more functions and a processor comprising one or more hardware logic circuits.

[0164] Furthermore, computer programs may be stored on a computer-readable, non-transitional tangible recording medium as instructions executed by a computer. The methods for realizing the functions of each part included in the image correction system do not necessarily need to include software; all of these functions may be realized using one or more hardware components.

[0165] (5f) In addition to the image correction system described above, this disclosure can also be implemented in various forms, such as a program for operating the computer of the image correction system, a non-transition tangible recording medium such as a semiconductor memory on which this program is recorded, and a control method (i.e., the image correction direction).

[0166] (5g) Multiple functions of one component in each of the above embodiments may be realized by multiple components, or one function of one component may be realized by multiple components. Also, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Furthermore, some of the configurations of each of the above embodiments may be omitted. Furthermore, at least some of the configurations of each of the above embodiments may be added to or replaced with the configurations of other embodiments. [Explanation of Symbols]

[0167] 1…Image correction system, 3…Vehicle, 5…In-vehicle device, 7…Cloud, 9…Display device, 17…Display, 19…Camera, 41…Tilt detection unit, 43…Correction processing unit, 71…Marker

Claims

1. An image correction system (1) is configured such that image data captured by a camera (19) mounted on a vehicle (3) can be transmitted to a display device (9) via an in-vehicle device (5) and an external device (7), A tilt detection unit (41) is configured to detect tilt relative to a predetermined direction in the captured image of the interior of the vehicle taken by the camera, A correction processing unit (43) is configured to correct the tilt of the captured image to the desired orientation based on the tilt detected by the tilt detection unit, and to create a display image for display on the display device. In addition to being equipped, The display image is configured to be displayed on the display device, Furthermore, a marker (71) indicating a position within the interior of the vehicle is provided, and the system is configured to perform the correction based on the captured image including the marker. Image correction system.

2. An image correction system (1) configured such that image data captured by a camera (19) mounted on a vehicle (3) can be transmitted to a display device (9) via an in-vehicle device (5) and an external device (7), A tilt detection unit (41) is configured to detect tilt relative to a predetermined direction in an image taken by the camera inside the vehicle or an image taken around the vehicle, A correction processing unit (43) is configured to correct the tilt of the captured image to the desired orientation based on the tilt detected by the tilt detection unit, and to create a display image for display on the display device. In addition to being equipped, The display image is configured to be displayed on the display device, Furthermore, if there are multiple cameras, the tilt detection unit determines the tilt based on the image captured by the first camera, and the correction is performed on the image captured by a second camera different from the first camera based on the tilt. Image correction system.

3. An image correction system according to claim 1 or claim 2, The in-vehicle device mounted on the vehicle is configured to perform the processing of the tilt detection unit and the correction processing unit. Image correction system.

4. An image correction system according to claim 1 or claim 2, The device located outside the vehicle is configured to perform the processing of the tilt detection unit and the correction processing unit. Image correction system.

5. The image correction system according to claim 3, The display device is configured to display the display image based on the data of the display image transmitted from the device outside the vehicle. Image correction system.

6. The image correction system according to claim 4, The display device is configured to display the display image based on the data of the display image transmitted from the device outside the vehicle. Image correction system.

7. An image correction system (1) configured such that image data captured by a camera (19) mounted on a vehicle (3) can be transmitted to a display device (9) via an in-vehicle device (5) and an external device (7), A tilt detection unit (41) is configured to detect tilt relative to a predetermined direction in an image taken by the camera inside the vehicle or an image taken around the vehicle, A correction processing unit (43) is configured to correct the tilt of the captured image to the desired orientation based on the tilt detected by the tilt detection unit, and to create a display image for display on the display device. In addition to being equipped, The display image is configured to be displayed on the display device, Furthermore, the in-vehicle device mounted on the vehicle is configured to process the tilt detection unit and to add the tilt data to the captured image data. Image correction system.

8. An image correction system (1) configured such that image data captured by a camera (19) mounted on a vehicle (3) can be transmitted to a display device (9) via an in-vehicle device (5) and an external device (7), A tilt detection unit (41) is configured to detect tilt relative to a predetermined direction in an image taken by the camera inside the vehicle or an image taken around the vehicle, A correction processing unit (43) is configured to correct the tilt of the captured image to the desired orientation based on the tilt detected by the tilt detection unit, and to create a display image for display on the display device. In addition to being equipped, The display image is configured to be displayed on the display device, Furthermore, the device located outside the vehicle is configured to process the tilt detection unit and to add the tilt data to the captured image data. Image correction system.

9. An image correction system according to claim 7 or claim 8, The system is configured to create a display image based on the data of the captured image and the tilt data transmitted from the external device, and to display the said display image. Image correction system.

10. An image correction system according to claim 1, claim 2, claim 7, or claim 8, The tilt detection unit and the correction processing unit are configured to perform processing when the vehicle is started. Image correction system.

11. An image correction system according to claim 1, claim 2, claim 7, or claim 8, After the vehicle is started, the tilt detection unit and the correction processing unit are configured to perform processing at all times. Image correction system.

12. The image correction system according to claim 1, After the vehicle is started, the system is configured to perform the correction based on the captured image taken at least once of the interior of the vehicle. Image correction system.

Images