Image Processing Apparatus, Vehicle, and Image Processing Method

Abstract

The image processing apparatus according to the present invention processes an image of a vehicle. In the image processing apparatus, there are provided: a first acquisition unit that acquires an image representing the appearance around the vehicle as a vehicle surrounding image; a second acquisition unit that acquires information representing the state of the vehicle; a third acquisition unit that acquires an image of the driver of the vehicle as a driver image; and an image generation unit that generates an image of the vehicle as a vehicle image based on the information acquired by the second acquisition unit, and overlaps the vehicle image and the driver image with the vehicle surrounding image to generate a composite image.

Description

Technical Field

[0001] The present invention mainly relates to an image processing apparatus. Background Art

[0002] An image processing technique is described in Patent Document 1. In this image processing technique, a virtual viewpoint is set in an image showing a vehicle and its surroundings, and the image can be observed and confirmed while changing the virtual viewpoint. In Patent Document 1, an example of applying such a technique to an anti-theft use is shown.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2015-76062 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] When further diversifying the uses of the image obtained by the above-described image processing technique, further technical improvements are desired.

[0008] An exemplary object of the present invention is to diversify the uses of the image obtained by the above-described image processing technique in a relatively simple manner.

[0009] Solutions to the Problems

[0010] A first aspect of the present invention relates to an image processing apparatus that processes an image of a vehicle. The image processing apparatus is characterized by including: a first acquisition unit that acquires an image showing the surroundings of the vehicle as a vehicle surrounding image; a second acquisition unit that acquires information indicating the state of the vehicle; a third acquisition unit that acquires an image of the driver of the vehicle as a driver image; and an image generation unit that generates an image of the vehicle as a vehicle image based on the information acquired by the second acquisition unit, and generates a composite image by overlapping the vehicle image and the driver image with the vehicle surrounding image.

[0011] Effects of the Invention

[0012] According to the present invention, it is beneficial to diversify the uses of the image obtained by the above-described image processing technique. Brief Description of the Drawings

[0013] Figure 1 It is a schematic diagram showing a structural example of an image display system.

[0014] Figure 2 It is a schematic diagram showing a structural example of a vehicle.

[0015] Figure 3 It is a flowchart showing an example of an image processing method.

[0016] Figure 4A It is a schematic diagram showing an image around a vehicle.

[0017] Figure 4B It is a schematic diagram showing an image of a driver.

[0018] Figure 4C It is a schematic diagram showing an image of a vehicle.

[0019] Figure 4D It is a schematic diagram showing a composite image.

[0020] Figure 5A It is a diagram showing an example of a composite image of a certain virtual viewpoint.

[0021] Figure 5B It is a diagram showing an example of a composite image of other virtual viewpoints.

[0022] Figure 6A It is a schematic diagram showing another structural example of an image display system.

[0023] Figure 6B It is a schematic diagram showing another structural example of an image display system. Detailed implementation mode

[0024] Hereinafter, the implementation mode will be described in detail with reference to the drawings. In addition, the following implementation mode does not limit the invention related to the claims, and the invention does not require the combination of all the features described in the implementation mode. It is also possible that any two or more of the multiple features described in the implementation mode are arbitrarily combined. In addition, the same reference numerals are added to the same or similar structures, and repeated descriptions are omitted.

[0025] (An example of an image display system)

[0026] Figure 1 It is a schematic diagram showing a structural example of an image display system SY according to the implementation mode. The image display system SY includes a vehicle 1, an image processing device 2, and a terminal 3, and in this implementation mode, they can communicate with each other via a network N.

[0027] In the present embodiment, the vehicle 1 is a riding type vehicle. Here, the riding type vehicle refers to a riding type vehicle in which the driver straddles the vehicle body. In its concept, in addition to including ordinary two-wheel vehicles (including step type vehicles), it also includes three-wheel vehicles (a vehicle with one front wheel and two rear wheels, or a vehicle with two front wheels and one rear wheel), all-terrain vehicles (ATV) such as four-wheel buggies, and the like. Additionally, as another embodiment, the vehicle 1 may be a passenger type vehicle. The vehicle 1 includes a camera device 11A, a camera device 11B, a detection device 12, and a communication device 13.

[0028] Figure 2 It is a schematic diagram showing a structural example of the vehicle 1.

[0029] A plurality of camera devices 11A are provided at the peripheral portion of the vehicle body in such a manner that an image representing the appearance around the vehicle 1 can be captured, and these plurality of camera devices 11A are arranged in such a manner that these imaging regions include the entire region around the vehicle 1. That is, the plurality of camera devices 11A are arranged in such a manner that the imaging regions of two adjacent camera devices 11A partially overlap each other. In the figure, the pointing directions of the camera devices 11A are schematically shown by dashed lines, and the actual detection range of the camera device 11A is wider than the illustrated range.

[0030] Camera devices 11B are respectively provided in front of and behind the driver's seat in such a manner that the driver can be captured from the front and the rear, respectively. In the figure, similar to the camera device 11A, the pointing directions of the camera devices 11B are schematically shown by dashed lines, but the actual detection range of the camera device 11B is wider than the illustrated range. Details will be described later, but thereby, the appearance, posture, motion state, etc. of the driver can be captured.

[0031] Regarding the camera devices 11A and 11B, a known camera composed of a CCD / CMOS image sensor or the like can be used. Additionally, in the present embodiment, as an example, in order to reduce the cost required for the camera devices 11A and 11B, a monocular camera is used.

[0032] The detection device 12 is provided at each part of the vehicle body in such a manner that the state of the vehicle 1 can be detected. Here, in the present embodiment, the state of the vehicle 1 includes the vehicle speed, the steering angle (or the wheel angle), the posture of the vehicle body, and the state of the lamp body (headlamp, taillight, turn indicator, etc.).

[0033] For example, the vehicle speed can be detected based on the rotational speed of the wheels per unit time, which can be achieved by using a well-known speed sensor. For example, the steering angle can be detected based on the direction of the steering wheel relative to the vehicle body (or the direction of the handlebar relative to the vehicle body), which can be achieved by using a well-known rudder angle sensor. For example, the attitude of the vehicle body can be detected based on the direction of the vehicle body relative to the direction of gravity, which can be achieved by a well-known acceleration sensor. Additionally, for example, the state of the lamp body can be detected based on the conduction state of the light source, which can be achieved by using a well-known ammeter.

[0034] The communication device 13 sends the imaging results of the imaging devices 11A and 11B and the detection results of the detection device 12 to the image processing device 2 via the network N. The communication device 13 can be described as a sending and receiving device or the like, and in this embodiment, it can also be described only as a sending device. Details will be described later, but the imaging result of the imaging device 11A shows an image representing the appearance around the vehicle 1 (hereinafter referred to as the vehicle surrounding image 9A). The imaging result of the imaging device 11B shows an image of the driver (hereinafter referred to as the driver image 9B). The detection result of the detection device 12 shows information representing the state of the vehicle 1 (hereinafter referred to as the vehicle information 9i).

[0035] Referring again to Figure 1 , the image processing device 2 includes a communication unit 21 and an arithmetic unit 22. The communication unit 21 can communicate the image processing device 2 with the vehicle 1 and the terminal 3 via the network N respectively. Details will be described later, but the arithmetic unit 22 performs predetermined arithmetic processing including image processing. In this embodiment, the arithmetic unit 22, as a processor including a CPU and a memory, executes a predetermined program to implement the functions of the arithmetic unit 22. That is, it can be that the program is read via a network or a storage medium and executed on a computer.

[0036] In addition, as another embodiment, it can also be that the arithmetic unit 22 is composed of semiconductor devices such as a PLD (programmable logic device) and an ASIC (application-specific semiconductor integrated circuit). That is, the functions of the arithmetic unit 22 can be implemented by either hardware or software.

[0037] In this embodiment, the terminal 3, as a portable terminal (such as a smartphone), includes a communication unit 31, an operation unit 32, and a display unit 33. The user of the terminal 3 can be the driver of the vehicle 1 or a third party different from the driver. The communication unit 31 can communicate the terminal 3 with the image processing device 2 via the network N. Details will be described later, but the operation unit 32 can accept operation inputs from the user, and in addition, the display unit 33 can display images. The operation unit 32 and the display unit 33 can be set integrally (for example, it can be a touch panel type display), or can be set individually.

[0038] Details will be described later. In such an image display system SY, the vehicle 1 can communicate with the image processing device 2 and send the imaging results of the imaging devices 11A and 11B and the detection results of the detection device 12 to the image processing device 2. Based on the above imaging results and detection results, the image processing device 2 performs predetermined image processing by the arithmetic unit 22 to generate a composite image (hereinafter referred to as composite image 9X), and sends the composite image 9X to the terminal 3. The user can use the terminal 3 to observe and confirm the composite image 9X on the display unit 33 while operating the operation unit 32.

[0039] (An example of an image processing method)

[0040] Figure 3 It is a flowchart showing an example of an image processing method for generating the composite image 9X. The content of this flowchart is mainly executed by the arithmetic unit 22. The outline is to generate an image of the vehicle 1 (hereinafter referred to as vehicle image 9C) based on the vehicle information 9i, and use the images 9A to 9C to generate the composite image 9X. In addition, this flowchart can be executed when the vehicle 1 is in use (during driving) or after the vehicle 1 is used (when not driving).

[0041] In step S1000 (hereinafter simply referred to as "S1000". The same applies to other steps described later), a vehicle surrounding image 9A is obtained from the vehicle 1. As described above, the vehicle surrounding image 9A is obtained by the plurality of imaging devices 11A, and these plurality of imaging devices 11A are arranged at the vehicle body peripheral part so that these imaging areas include the entire area around the vehicle 1. Thus, the vehicle surrounding image 9A shows the appearance of the entire area around the vehicle 1 and is obtained in a so-called panoramic (360-degree panoramic) view. Thus, by using the spherical coordinate system, the vehicle surrounding image 9A can be processed in a relatively simple manner.

[0042] In S1010, a driver image 9B is obtained from the vehicle 1. As described above, the driver image 9B is obtained by a pair of imaging devices 11B arranged in front of and behind the driver's seat, and this pair of imaging devices 11B is arranged so as to be able to photograph the driver from the front and the rear. Thus, the driver image 9B shows the driver's driving appearance such as the driver's appearance (for example, skeleton, clothing (including helmet in addition to clothes)), posture (in the case of a still image), and motion state (in the case of a moving image). Thus, by using a three-dimensional coordinate system based on a predetermined human model, the driver image 9B can be processed in a relatively simple manner.

[0043] In S1020, vehicle information 9i is acquired from vehicle 1. As described above, the vehicle information 9i is obtained by the detection device 12, which is arranged at various parts of the vehicle body in a manner capable of detecting the state of vehicle 1. The state of vehicle 1 includes vehicle speed, steering angle, attitude of the vehicle body, and state of the lamp body.

[0044] In S1030, a vehicle image 9C is generated based on the vehicle information 9i. Since the vehicle information 9i represents vehicle speed, steering angle, attitude of the vehicle body, and state of the lamp body, an image of vehicle 1 corresponding to these states is generated as the vehicle image 9C. Thus, by using a three-dimensional coordinate system based on the corresponding vehicle model, image processing of the vehicle image 9C can be performed in a relatively simple manner.

[0045] In S1040, images 9B to 9C are superimposed on the vehicle surrounding image 9A to generate a composite image 9X. As described above, in the present embodiment, the vehicle surrounding image 9A is processed in a spherical coordinate system, and the driver image 9B and the vehicle image 9C are processed in a three-dimensional coordinate system.

[0046] Here, the three-dimensional coordinate system generally uses the distance x in the longitudinal direction of the vehicle body from the coordinate center to the object, the distance y in the lateral direction of the vehicle body from the coordinate center to the object, and the distance z in the vertical direction of the vehicle body from the coordinate center to the object, and can be represented by the coordinates (x, y, z). In addition, the spherical coordinate system generally uses the distance r from the coordinate center to the object, the angle θ formed by the line connecting the coordinate center and the object and the vertical direction of the vehicle body, and the angle formed by the line connecting the coordinate center and the object and the longitudinal direction of the vehicle body can be represented by the coordinates shown.

[0047] Figure 4A is a schematic diagram showing the vehicle surrounding image 9A. The vehicle surrounding image 9A is processed in a spherical coordinate system and is depicted at a position with a distance r from the coordinate center. In other words, the panoramic vehicle surrounding image 9A is formed in the shape depicted on the inner wall of a sphere with a radius r. The above r can be set to a position outside vehicle 1.

[0048] Figure 4B is a schematic diagram showing the driver image 9B. The driver image 9B is processed in a three-dimensional coordinate system. For example, the head, shoulders, torso (chest and abdomen), waist, arm parts (upper arm and forearm), hands, leg parts (thigh and calf), feet, etc. can be depicted based on a predetermined human model. Clothing can also be depicted incidentally.

[0049] Figure 4CIt is a schematic diagram showing a vehicle image 9C. The vehicle image 9C is processed using a three-dimensional coordinate system. For example, an image of the vehicle 1 in a state based on vehicle information 9i (information indicating vehicle speed, steering angle, attitude of the vehicle body, and state of the lamp body) can be depicted. For example, the vehicle image 9C can depict the turning vehicle 1 in an attitude with the vehicle body tilted.

[0050] Here, since the vehicle information 9i includes information indicating the attitude of the vehicle body, the imaging results of the imaging devices 11A and 11B can be corrected according to the degree of tilt of the vehicle body. For example, when the vehicle surrounding image 9A is acquired by the imaging device 11A while the vehicle body is in an attitude tilted at an inclination angle λ1, the image 9A can be processed by rotating only an amount corresponding to the angle λ1. The correction process for the imaging results of the imaging devices 11A and 11B can be performed in the image processing device 2, but can also be performed in the vehicle 1.

[0051] Figure 4D It is a schematic diagram showing a composite image 9X. It can be that the images 9A to 9C are synthesized in such a way that the coordinate center, distance, and direction are the same. In addition, the coordinate center is set to the position directly above the seat in the present embodiment, but in other embodiments, it can also be other positions (for example, any position in the vehicle body).

[0052] Refer again to Figure 3 , in step S1050, the composite image 9X is sent to the terminal 3. The user of the terminal 3 operates the operation unit 32, whereby the composite image 9X can be displayed on the display unit 33 from a viewpoint at an arbitrary position (hereinafter referred to as a virtual viewpoint). In addition, the user operates the operation unit 32, whereby the composite image 9X can also be enlarged or reduced.

[0053] Figure 5A Shows an example of the composite image 9X in the case of a touch panel type display in which the operation unit 32 and the display unit 3 are integrated. Figure 5B Shows other examples of the composite image 9X (examples of the composite image 9X in a virtual viewpoint different from Figure 5A . Icons 8a and 8b for changing the virtual viewpoint, an icon 8c for magnification, and an icon 8d for reduction are displayed on the display unit 33 as part of the operation unit 32. The user performs a predetermined operation input (such as a tap operation, a slide operation, a flick operation, etc.) on these icons 8a, etc., whereby the appearance of the vehicle 1 and its surroundings can be observed and confirmed from a desired virtual viewpoint.

[0054] In the following situation, when the virtual viewpoint is changed, the size of the vehicle image 9C in the composite image 9X is changed, and the size of the driver image 9B is also changed accordingly, thereby reducing the discomfort caused by the change in the visibility of the composite image 9X due to this change. On the other hand, it may also be that the vehicle surrounding image 9A is processed in a spherical coordinate system, so when the virtual viewpoint is changed, the size of the vehicle surrounding image 9A in the composite image 9X can be maintained.

[0055] In addition, when performing image processing on the vehicle surrounding image 9A, the distance r in the spherical coordinate system is set to a relatively large value, thereby reducing the discomfort (such as distortion) caused by the change of the vehicle surrounding image 9A when the virtual viewpoint is changed. In this case, a camera device 11A with a relatively large number of pixels is used, thereby enabling the vehicle surrounding image 9A to be clearly displayed.

[0056] When performing the above S1040 (generating the composite image 9X), the desired images 9A to 9C are expected to show approximately the same time appearance. Thus, the attribute information indicating the time (the image captured at which timing, or the image generated based on the information obtained at which timing) can be associated with the images 9A to 9C. Accordingly, the attribute information indicating the place (the image captured at which place, or the image generated based on the information obtained at which place) can also be associated with the images 9A to 9C.

[0057] In addition, as described above, the three-dimensional coordinate system can generally be represented by coordinates (x, y, z), and the spherical coordinate system can generally be represented by coordinates representation. Thus, as another embodiment, it may also be that, using a known coordinate transformation, the driver image 9B and the vehicle image 9C, like the vehicle surrounding image 9A, are processed in a spherical coordinate system. Or, it may also be that the vehicle surrounding image 9A, like the driver image 9B and the vehicle image 9C, is processed in a three-dimensional coordinate system.

[0058] In addition, in the present embodiment, a monocular camera is used as the camera device 111, but a compound eye camera can also be used instead. Thus, the imaging object and the distance information can be captured together, so the vehicle surrounding image 9A can be processed in a three-dimensional coordinate system in a relatively simple manner.

[0059] In addition, in the present embodiment, since the vehicle 1 is a riding vehicle, the driver image 9B is used when generating the composite image 9X, but it may also be that when the vehicle 1 is a passenger vehicle (for example, when it is difficult to observe and confirm the driver from outside the vehicle), the driver image 9B is not used. In this case, the camera device 11B can also be omitted.

[0060] As described above, according to the present embodiment, an image representing the appearance around the vehicle 1 is obtained as the vehicle surrounding image 9A, and vehicle information 9i representing the state of the vehicle 1 is obtained. Then, an image of the vehicle 1 is generated based on the vehicle information 9i as the vehicle image 9C, and the vehicle image 9C is superimposed on the vehicle surrounding image 9A to generate a composite image 9X. At this time, it is also possible to superimpose the driver image 9B on the vehicle surrounding image 9A incidentally. A user (such as a driver) can use the composite image 9X to, for example, observe and confirm the appearance of the vehicle 1 being driven and its surroundings from a desired virtual viewpoint, or can also allow a third party to observe the appearance. Thus, according to the present embodiment, the composite image 9X can be utilized for various purposes. As an example, it can be utilized as a driving recorder representing a more detailed driving condition.

[0061] (Other examples of the image display system)

[0062] According to the image display system SY (refer to Figure 1 ), the function of the image processing device 2 is implemented at a location different from the vehicle 1 (such as a server), and the display of the composite image 9 and the change of the virtual viewpoint are performed at the terminal 3, but it is not limited to this method.

[0063] Figure 6A A structural example of the image display system SYa is shown. In this system SYa, the image processing device 2 is mounted on the vehicle 1. In this case, the composite image 9X can be sent from the vehicle 1 to the terminal 3 via the network N or by a known communication unit (such as bluetooth (registered trademark)).

[0064] Figure 6B A structural example of the image display system SYb is shown. In this system SYb, the image processing device 2 is provided in the terminal 3. That is, it is possible that the terminal 3 receives the images 9A, 9B and the vehicle information 9i from the vehicle 1 via the network N or the like, generates the composite image 9X based on them, and displays it on the display unit 33.

[0065] As another example, the terminal 3 can also be an in-vehicle monitor (for example, a car navigation system). In this case, the driver can observe and confirm the appearance of the surroundings from a desired virtual viewpoint while driving the vehicle 1.

[0066] (Others)

[0067] In the above description, for ease of understanding, each element is represented by a name functionally related to the function of each element. However, each element is not limited to having the content described in the embodiment as the main function, and can also have those functions subsidiarily.

[0068] In addition, in this specification, as a typical example, the vehicle 1 is illustrated. However, the content of the embodiment can also be applied to facilities (such as ships) that do not have wheels, that is, it can be applied to various moving bodies.

[0069] (Summary of the Embodiment)

[0070] The first aspect relates to an image processing apparatus (such as 2) that processes an image of a vehicle (such as 1). The image processing apparatus is characterized by including: a first acquisition unit (such as S1000) that acquires an image representing the appearance around the vehicle as a vehicle surrounding image (such as 9A); a second acquisition unit (such as S1020) that acquires information representing the state of the vehicle (such as 9i); and an image generation unit (such as S1030, S1040) that generates an image of the vehicle as a vehicle image (such as 9C) based on the information acquired by the second acquisition unit, and overlaps the vehicle image with the vehicle surrounding image to generate a composite image (such as 9X).

[0071] According to the first aspect, the composite image obtained in this way can be utilized for various purposes.

[0072] Incidentally, in the first aspect, it is characterized by further including a third acquisition unit (such as S1010) that acquires an image of the driver of the vehicle as a driver image (such as 9B), and the image generation unit further overlaps the driver image with the vehicle surrounding image to generate the composite image.

[0073] Thereby, a more detailed appearance can be depicted in the composite image.

[0074] In the second aspect, it is characterized in that the image generation unit processes the vehicle surrounding image in a spherical coordinate system.

[0075] According to the second aspect, the processing of the vehicle surrounding image can be achieved in a relatively simple manner.

[0076] In the third aspect, it is characterized in that the image generation unit processes the vehicle image in a three-dimensional coordinate system.

[0077] According to the third aspect, the processing of the vehicle image can be achieved in a relatively simple manner.

[0078] In the fourth aspect, it is characterized by further including a transmission unit (such as S1050) that transmits the composite image to a predetermined terminal (such as 3), and a user of the terminal performs an operation input so that the composite image can be displayed from a viewpoint at an arbitrary position.

[0079] According to the fourth aspect, the user can observe and confirm the appearance of the vehicle and its surroundings from any viewpoint.

[0080] In the fifth aspect, it is characterized in that it further includes a display unit (such as 33, Figure 5A , Figure 5B ), and the display unit displays the synthesized image from the viewpoint at any position.

[0081] According to the fifth aspect, the user can observe and confirm the appearance of the vehicle and its surroundings from any viewpoint.

[0082] In the sixth aspect, it is characterized in that the display unit changes the size of the vehicle image along with the change of the viewpoint.

[0083] According to the sixth aspect, it is possible to reduce the discomfort caused by the change of the synthesized image accompanying the change of the viewpoint.

[0084] In the seventh aspect, it is characterized in that the state of the vehicle includes vehicle speed, steering angle, attitude of the vehicle body, and / or state of the lamp body.

[0085] According to the seventh aspect, in the synthesized image, a more detailed appearance can be depicted.

[0086] The eighth aspect relates to a vehicle (such as 1), which is a vehicle capable of communicating with the above-mentioned image processing device, and the vehicle is characterized in that it includes: a first imaging device (such as 11A) that captures an image representing the appearance of the surroundings of the vehicle; a detection device (such as 12) that detects the state of the vehicle; and a communication device (such as 13) that sends the imaging result of the first imaging device and the detection result of the detection device to the image processing device.

[0087] That is, the above-mentioned image processing device can be applied to a known vehicle.

[0088] The ninth aspect relates to a vehicle (such as 1), which is a vehicle capable of communicating with the above-mentioned image processing device, and the vehicle is characterized in that it includes: a first imaging device (such as 11A) that captures an image representing the appearance of the surroundings of the vehicle; a second imaging device (such as 11B) that captures an image of the driver; a detection device (such as 12) that detects the state of the vehicle; and a communication device (such as 13) that sends the imaging result of the first imaging device, the imaging result of the second imaging device, and the detection result of the detection device to the image processing device.

[0089] That is, the above-mentioned image processing device can be applied to a known vehicle.

[0090] The tenth aspect relates to a vehicle (e.g., 1), characterized in that it includes: the above-described image processing device; a first imaging device (e.g., 11A) that captures an image representing the surroundings of the vehicle; and a detection device (e.g., 12) that detects the state of the vehicle.

[0091] That is, the above-described image processing device can be mounted on a known vehicle.

[0092] The eleventh aspect relates to a vehicle (e.g., 1), characterized in that it includes: the above-described image processing device; a first imaging device (e.g., 11A) that captures an image representing the surroundings of the vehicle; a second imaging device (e.g., 11B) that captures an image of the driver; and a detection device (e.g., 12) that detects the state of the vehicle.

[0093] That is, the above-described image processing device can be mounted on a known vehicle.

[0094] In the twelfth aspect, it is characterized in that the vehicle is a riding vehicle (e.g., 1), and the second imaging device is respectively provided in front of and behind the driver's seat.

[0095] According to the twelfth aspect, an image of the driver can be appropriately captured.

[0096] In the thirteenth aspect, it is characterized in that a plurality of the first imaging devices are provided at the peripheral part of the vehicle body.

[0097] In the thirteenth aspect, an image representing the surroundings of the vehicle can be appropriately captured.

[0098] The fourteenth aspect relates to an image processing method that processes an image of a vehicle (e.g., 1). The image processing method is characterized by including: a step (e.g., S1000) of obtaining an image representing the surroundings of the vehicle as a vehicle surrounding image (e.g., 9A); a step (e.g., S1020) of obtaining information representing the state of the vehicle (e.g., 9i); a step (e.g., S1010) of obtaining an image of the driver of the vehicle as a driver image (e.g., 9B); steps (e.g., S1030, S1040) of generating an image of the vehicle as a vehicle image (e.g., 9C) based on the information representing the state of the vehicle, and overlapping the vehicle image and the driver image with the vehicle surrounding image to generate a composite image (e.g., 9X); and a step (e.g., 33, Figure 5A 、 Figure 5B ) of displaying the composite image from a viewpoint at an arbitrary position.

[0099] According to the fourteenth aspect, the composite image obtained in this way can be utilized for various purposes.

[0100] The invention is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the gist of the invention.

Claims

1. An image processing apparatus processes an image of a vehicle, and is characterized in that it includes: A first acquisition unit that acquires an image representing the appearance around the vehicle as a vehicle surrounding image; A second acquisition unit that acquires information representing the state of the vehicle; A third acquisition unit that acquires an image of the driver of the vehicle as a driver image; And An image generation unit that generates an image of the vehicle as a vehicle image based on the information acquired by the second acquisition unit, and overlaps the vehicle image and the driver image with the vehicle surrounding image to generate a composite image, The second acquisition unit includes an acceleration sensor that detects the attitude of the vehicle body of the vehicle, When the vehicle is a riding vehicle and the vehicle surrounding image is acquired by the first acquisition unit during a period in which the vehicle body is in an inclined attitude, the image generation unit depicts the vehicle image in the attitude of the inclined vehicle body according to the degree of inclination of the vehicle body detected by the acceleration sensor, and corrects the vehicle surrounding image by performing a rotation process at an angle corresponding to the degree of inclination of the vehicle body.

2. The image processing apparatus according to claim 1, characterized in that The image generation unit processes the vehicle surrounding image in a spherical coordinate system.

3. The image processing apparatus according to claim 1, characterized in that The image generation unit processes the vehicle image in a three-dimensional coordinate system.

4. The image processing apparatus according to claim 1, characterized in that It further includes a transmission unit that transmits the composite image to a predetermined terminal, and a user of the terminal can perform an operation input on the terminal so as to display the composite image from an arbitrary viewpoint.

5. The image processing apparatus according to claim 1, characterized in that It further includes a display unit that displays the composite image from an arbitrary viewpoint.

6. The image processing apparatus according to claim 5, characterized in that The display unit changes the size of the vehicle image as the viewpoint changes.

7. The image processing apparatus according to claim 1, characterized in that The state of the vehicle includes vehicle speed, steering angle, attitude of the vehicle body, and / or state of the lamp body.

8. A vehicle capable of communicating with the image processing apparatus according to any one of claims 1 to 7, characterized in that it includes: A first imaging device that captures an image representing the appearance around the vehicle; A detection device that detects the state of the vehicle; and A communication device that transmits the imaging result of the first imaging device and the detection result of the detection device to the image processing apparatus.

9. The vehicle according to claim 8, characterized in that A plurality of the first imaging devices are provided at the peripheral part of the vehicle body.

10. A vehicle capable of communicating with the image processing apparatus according to claim 1, characterized in that it includes: A first imaging device that captures an image representing the appearance around the vehicle; A second imaging device that captures an image of the driver; A detection device that detects the state of the vehicle; and A communication device that transmits the imaging result of the first imaging device, the imaging result of the second imaging device, and the detection result of the detection device to the image processing device.

11. The vehicle according to claim 10, wherein A plurality of the first imaging devices are provided at the peripheral portion of the vehicle body.

12. The vehicle according to claim 10, wherein The vehicle is a riding-type vehicle, The second imaging devices are respectively provided in front of and behind the driver's seat.

13. A vehicle, characterized in that it comprises: The image processing device according to any one of claims 1 to 7; A first imaging device that captures an image representing the appearance around the vehicle; and A detection device that detects the state of the vehicle.

14. The vehicle according to claim 13, wherein A plurality of the first imaging devices are provided at the peripheral portion of the vehicle body.

15. A vehicle, characterized in that it comprises: The image processing device according to claim 1; A first imaging device that captures an image representing the appearance around the vehicle; A second imaging device that captures an image of the driver; and A detection device that detects the state of the vehicle.

16. The vehicle according to claim 15, wherein A plurality of the first imaging devices are provided at the peripheral portion of the vehicle body.

17. The vehicle according to claim 15, wherein The vehicle is a riding-type vehicle, The second imaging devices are respectively provided in front of and behind the driver's seat.

18. An image processing method for processing an image of a vehicle, characterized by including: A step of acquiring an image representing the appearance around the vehicle as a vehicle peripheral image; A step of using an acceleration sensor that detects the attitude of the vehicle body of the vehicle to acquire information representing the state of the vehicle; A step of acquiring an image of the driver of the vehicle as a driver image; A step of generating an image of the vehicle as a vehicle image based on the information representing the state of the vehicle, and overlapping the vehicle image and the driver image with the vehicle peripheral image to generate a composite image; And A step of displaying the composite image from a viewpoint at an arbitrary position, The vehicle is a riding-type vehicle. When the vehicle peripheral image is acquired during a period in which the vehicle body is in an inclined attitude, in the step of generating the composite image, according to the degree of inclination of the vehicle body detected by the acceleration sensor, the vehicle image is depicted in the attitude of the inclined vehicle body, and the vehicle peripheral image is corrected by performing a rotation process at an angle corresponding to the degree of inclination of the vehicle body.

Images