Image generation device and image generation method

Abstract

Image generation device for a vehicle, with: an image reference unit (F1) configured to receive multiple camera images obtained from multiple cameras configured to photograph the vehicle's surroundings; an image synthesis unit (F7) configured to project data from the majority of camera images onto a virtual projection surface corresponding to the vehicle's environment and to generate a synthetic image showing the vehicle's environment as viewed from a virtual viewpoint, using the data projected onto the projection surface; and a driving environment determination unit (F62) configured to determine, based on a signal from another on-board device of the vehicle, whether a driving environment of the vehicle is terrain or on a road, wherein the image synthesis unit (F7) is configured to change the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment determination unit (F62) determines that the driving environment is terrain, the image synthesis unit (F7) is configured to generate the synthetic image by using a road projection surface with a predefined shape as the projection surface when the driving environment determination unit (F62) determines that the driving environment is on a road, and The image synthesis unit (F7) is configured to produce the synthetic image by using a terrain projection surface with a shape different from the shape of the road projection surface when the driving environment determination unit (F62) determines that the driving environment is terrain.

Description

Technical field

[0001] The present disclosure relates to a technique for generating a synthetic image showing the environment of a vehicle by using images taken by a plurality of vehicle on-board cameras. State of the art

[0002] JP 5 911 775 B2 and JP 6 257 978 B2 disclose the following image display systems. That is, the image display systems synthesize images captured by cameras installed at multiple positions within a vehicle to create a synthetic image depicting the vehicle's surroundings as viewed from a virtual viewpoint. Furthermore, the image display systems display this synthetic image. Examples of synthetic images include overview images showing the vehicle's surroundings as if viewed from above, and driver viewpoint images showing the vehicle's surroundings as viewed from the driver's perspective, in such a way that one can see through an instrument panel, and the like.

[0003] A synthetic image showing the environment of a vehicle viewed from a virtual viewpoint, as previously described, is generated by projecting data from a plurality of captured images onto a virtual projection surface corresponding to the vehicle's environment and by using the data on the projection surface.

[0004] A horizontal surface, a shell shape, and similar shapes are proposed for the projection surface. In this case, the shell shape is designed to be essentially horizontal near the vehicle and to have a slope that gradually increases with distance from the vehicle's position. The curved surface of the shell shape corresponds to a parabolic surface formed by rotating a parabola. Such shell shapes consist of a flat surface area and a downwardly convexly curved surface area coupled to the perimeter of the flat surface area.

[0005] JP 5 911 775 B2 further discloses a structure for switching or changing the projection surface from a normal projection surface with a shell shape, which has a flat vehicle surrounding area, to a predetermined specific projection surface if an obstacle is detected or based on user operations to instruct a magnified display. The specific projection surface in JP 5 911 775 B2 is a projection surface with a shape consisting of a rectangular flat surface corresponding to a vehicle surrounding area and an upwardly convexly curved surface area surrounding the flat surface.

[0006] By using the aforementioned image display system, the user (a representative example being a driver) can perceive the condition of the vehicle's surroundings essentially in real time.

[0007] The previously mentioned imaging devices examined so far are structured based on the premise that the vehicle is located on a road surface paved with asphalt, concrete, bricks, stones, and the like. In other words, the aforementioned imaging devices are based on the premise that there is a flat ground surface in the vicinity of the vehicle. Therefore, the projection surface corresponding to the vehicle's surroundings is essentially flat. In this case, "the vehicle's surroundings" refers, for example, to an area at a distance of 2 to 3 meters from the vehicle.

[0008] However, the developer of the present disclosure discovered the following by examining structures configured to display synthetic images, such as driver viewpoint images, even while driving off-road. Specifically, it was found that with such synthetic images, which utilize a projection surface that is essentially flat in the vehicle's surroundings, images of rocks and stones present in the vehicle's environment are compressed or squashed to flatten them, which can cause the user to perceive a sense of dissonance or incongruity in these images.

[0009] Reference is also made to DE 10 2017 117 243 A1, JP 2018 - 16 250 A, and DE 10 2017 112 359 A1, which were identified as state of the art. Summary of the invention

[0010] It is an objective of the present disclosure to provide an image-generating device and an image-generating method that can produce a synthetic image that is less prone to causing the user to feel a sense of incongruity or dissonance about it than a synthetic image displayed while driving off-road.

[0011] The problem is solved by the subject matter or procedures of the independent claims. Advantageous further developments can be found in the dependent claims.

[0012] According to one aspect, an image generation device for a vehicle includes an image reference unit configured to acquire multiple camera images obtained from multiple cameras configured to photograph the vehicle's surroundings. The image generation device further includes an image synthesis unit configured to project data from the multiple camera images onto a virtual projection surface corresponding to the vehicle's surroundings and to generate a synthetic image depicting the vehicle's surroundings as viewed from a virtual viewpoint, using the data projected onto the projection surface.The image generation device further includes a driving environment determination unit, which is configured to determine, based on a signal from another onboard device of the vehicle, whether the vehicle's driving environment is terrain or a road. The image synthesis unit is configured to modify the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment determination unit determines that the driving environment is terrain.

[0013] With the aforementioned structure, if the driving environment is determined to be terrain, a synthetic image is generated by using a projection surface with a different shape than the one used when the vehicle is determined to be on a road. Specifically, it is possible to use a projection surface with a shape based on the premise that the vehicle's driving environment is terrain. Therefore, it is possible to generate a synthetic image that is less likely to elicit a sense of incongruity from the user than a synthetic image intended to be displayed while driving off-road.

[0014] According to another aspect, an image generation method serves to produce an image to assist in the driving of a vehicle. The method involves acquiring multiple camera images obtained from multiple cameras configured to photograph the vehicle's surroundings. The method further involves projecting data from these multiple camera images onto a virtual projection surface corresponding to the vehicle's surroundings and generating a synthetic image depicting the vehicle's surroundings as viewed from a virtual viewpoint, using the data projected onto the projection surface. The method also includes determining whether the vehicle's driving environment is terrain or a road, based on a signal from another onboard device of the vehicle.The procedure also includes changing the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment is a road.

[0015] The aforementioned structure is a method corresponding to the previously described image-generating device. Using this method, when the vehicle's driving environment is terrain, it is possible to use a projection surface based on the premise that the driving environment is terrain, and this surface has a different shape than the projection surface used when the vehicle is on a road. Therefore, it is possible to generate a synthetic image that is less likely to elicit a sense of incongruity or dissonance from the user than a synthetic image intended to be displayed while driving off-road. Brief description of the drawings

[0016] The previously described and other tasks, features, and advantages of the present disclosure will become clearer from the following detailed description with reference to the accompanying drawings. In the drawings: Fig. 1 is a block diagram that provides an overview of an environment display system (Sys). Fig. Figure 2 is a view that shows an example of the positions where each camera 2 is installed and the areas of photography by each camera 2. Fig. Figure 3 is a view that shows an example of the positions where the respective Sonare 8 are installed, and the areas of photography through the respective Sonare 8. Fig. Figure 4 is a block diagram to explain the functions of an image generation ECU 1. Fig. Figure 5 is a view that shows an example of camera images obtained from an image reference unit F1. Fig. Figure 6 is a view to explain a projection surface TS. Fig. Figure 7 is a view to illustrate a driver's viewpoint VPd. Fig. Figure 8 is a view to explain the shape of a normal projection surface TS1. Fig. Figure 9 is a view to explain the shape of the normal projection surface TS1. Fig. Figure 10 is a view to explain the shape of a terrain projection surface TS2. Fig. Figure 11 is a view to explain the shape of the terrain projection surface TS2. Fig. 12 is a view that represents an example of a driver viewpoint image CPd. Fig. Figure 13 is a view that represents an example of a bird's-eye view CPb image. Fig. Figure 14 is a flowchart of synthetic image display processing. Fig. Figure 15 is a view that shows an example of the layout of a displayed DP image. Fig. Figure 16 is a view to explain the way in which a rock Rk is viewed when the normal projection surface TS1 is used. Fig. Figure 17 is a view to explain the way in which a rock Rk is viewed when the terrain projection surface TS2 is used. Fig. Figure 18 is a line diagram of a driver viewpoint image CPd, which is generated by using the normal projection surface TS1. Fig. Figure 19 is a line graph of a driver viewpoint image CPd, which is generated by using the terrain projection surface TS2. Fig. 20 is a view that represents a modification of the terrain projection surface TS2. Fig. 21 is a view that represents a modification of the terrain projection surface TS2. Fig. 22 is a view that represents a modification of the terrain projection surface TS2. Fig. 23 is a view that shows an example of a notification image NP display indicating that a terrain display mode is set. Fig. Figure 24 is a view that provides an example of the position where a virtual viewpoint VP is set instead of the driver's viewpoint VPd within a vehicle space. Detailed description

[0017] The following describes exemplary embodiments of an environmental display system Sys according to the present disclosure, with reference to the drawings. The environmental display system Sys displays images of the surroundings or periphery of a vehicle containing this system on a display. Hereinafter, the vehicle V containing the environmental display system Sys is also referred to as the vehicle itself. <einleitung>

[0018] The vehicle in the present embodiment is an all-wheel-drive car, which is expected to drive not only on paved roads (i.e., roads) but also off-road. The vehicle has a normal mode suitable for driving on roads and off-road modes. The off-road modes can be a first off-road mode suitable for driving on slippery surfaces such as muddy or sandy roads, and a second off-road mode suitable for driving on surfaces with greater concavity and convexity, such as bumpy or rocky roads. In each of these driving modes, the distribution of drive power to the respective wheels is controlled differently. In this case, the term "off-road" primarily refers to a surface with greater concavity and convexity, such as a rocky road.Of course, the term "terrain" can refer to ground surfaces other than road surfaces, namely unpaved ground surfaces in general.

[0019] In the following description, the forward, reverse, left, right, upward, and downward directions are defined in relation to the vehicle itself. Specifically, the forward and reverse directions correspond to the longitudinal directions of the vehicle. The left and right directions correspond to the width directions of the vehicle. The upward and downward directions correspond to the vehicle's vertical directions. From another perspective, the upward and downward directions correspond to directions perpendicular to a plane parallel to the forward and reverse directions and the left and right directions. In this disclosure, a plane perpendicular to the vehicle's vertical direction is also referred to as the horizontal vehicle plane. Directions perpendicular to the vehicle's vertical direction that include the forward, reverse, left, and right directions are also referred to as horizontal vehicle directions.The horizontal vehicle directions correspond to directions that move further away from the vehicle itself.

[0020] Furthermore, in the present disclosure, "parallel" is not limited to a completely parallel state. "Parallel" can refer to a state inclined by several degrees up to about 20 degrees. In general, "parallel" can encompass parallel states (so-called essentially parallel states). Similarly, in the present disclosure, the term "perpendicular" is not limited to a completely perpendicular state and can include states inclined by several degrees up to about 20 degrees. <Beschreibung der gesamten Struktur>

[0021] Fig. Figure 1 is a view that presents an example of a schematic structure of an environment display system (Sys) according to the present disclosure. As in Fig. As shown in Figure 1, the environmental display system Sys includes an image generation ECU 1, a plurality of cameras 2, a display 3, a touch panel 4, control buttons 5, vehicle condition sensors 6, a sonar ECU 7, a plurality of sonar units 8, and an integration ECU 9. Among the element designations, the term "ECU," short for Electronic Control Unit, refers to an electronic control device.

[0022] The image generation ECU 1 is connected to the majority of cameras 2, the display 3, the touch panel 4, the control buttons 5, the vehicle condition sensors 6, the sonar ECU 7, and the integration ECU 9 in a manner that allows communication with each of them. The majority of sonar 8 are connected to the sonar ECU 7. The image generation ECU 1 can be connected to the aforementioned various devices or sensors either individually via dedicated lines or via an in-vehicle network, which is a structured communication network within the vehicle. For example, the image generation ECU 1 can be directly connected to the cameras 2 via dedicated image signal lines.

[0023] The image generation ECU 1 is an ECU as follows. That is, the ECU generates a synthetic image CP of the vehicle's surroundings, viewed from any point of view, based on image data input from the respective cameras 2. This image serves to support operations and manipulations for driving a vehicle V. Furthermore, the ECU displays the synthetic image on the display 3. The image generation ECU 1 is implemented using a computer. Specifically, the image generation ECU 1 contains a processing unit 11, RAM 12, memory 13, I / O 14, and bus lines for connecting these components.

[0024] Processing Unit 11 is hardware for arithmetic processing (in other words, an arithmetic core) that is connected to RAM 12. Processing Unit 11 is, for example, a CPU. Processing Unit 11 performs various processing operations to implement the functions of the respective functional units, which will be described later, by accessing RAM 12. RAM 12 is a volatile memory medium.

[0025] Memory 13 is structured to contain a non-volatile storage medium, such as flash memory. Memory 13 stores an image generation program Pg as firmware and various data to enable an image synthesis unit F7 to generate synthetic images. This data for generating synthetic images includes multiple projection surface data Dt with different parameters that define surface shapes and the like. The execution of the image generation program Pg by the processing unit 11 corresponds to the execution of an image generation process, which is a process that corresponds to this image generation program Pg. The I / O 14 is a circuit module for communication with other devices. The I / O 14 is implemented using analog circuit elements, ICs, and the like. The image generation ECU 1 will be described separately in detail later.

[0026] The cameras 2 are in-vehicle cameras configured to photograph the vehicle's surroundings and output data from the resulting images to the image generation ECU 1. Each camera 2 contains at least one lens and an image capture element and electronically acquires an image showing the vehicle's surroundings. The majority of cameras 2 are mounted at predetermined positions on the vehicle in predetermined orientations to photograph different areas. The environmental display system Sys according to the present embodiment includes a front camera 2F, a rear camera 2B, a left-side camera 2L, and a right-side camera 2R as the cameras 2, as shown in Fig. Figure 2 shows that these four cameras are positioned at different locations within the vehicle and configured to photograph the vehicle's surroundings from different directions. This is achieved as follows.

[0027] The front camera 2F is a camera for photographing the front of the vehicle from a predetermined angle. The front camera 2F is mounted in such a position that its optical axis 2Fa points forward from the vehicle, at a leading edge of the vehicle, such as the front grille. The rear camera 2B is a camera for photographing the rear of the vehicle from a predetermined angle. The rear camera 2B is mounted in such a position that its optical axis 2Ba points backward from the vehicle, at a predetermined position on the rear surface of the body, such as near the rear license plate or the rear window. The left-side camera 2L is a camera for photographing the left side of the vehicle.The left-side camera 2L is mounted on a left-side mirror in such a position that its optical axis 2La is aligned to the left of the vehicle. The right-side camera 2R is a camera for photographing the right side of the vehicle. The right-side camera 2R is mounted on a right-side mirror in such a position that its optical axis 2Ra is aligned to the right of the vehicle.

[0028] The lenses used in these cameras 2 are wide-angle lenses, such as fisheye lenses, and each camera 2 has a field of view θ equal to or greater than 180 degrees. Therefore, by using the four cameras 2, it is possible to photograph the entire area around the vehicle. Furthermore, it is possible to adjust the mounting positions of the respective cameras 2. The front camera 2F can also be mounted on the rearview mirror or the upper edge of the windshield. The right-side camera 2R and the left-side camera 2L can also be placed near the base of an A-pillar and a B-pillar, respectively. The surround-view system Sys can also include a roof-mounted camera 2. Some or all of the cameras 2 can also be relocated, for example, to the roof, the dashboard, near the window frames, and so on.

[0029] Memory 13 stores data specifying the positions and orientations of the respective cameras 2 mounted in the vehicle V (hereinafter referred to as "mounting position data"). The positions of the respective cameras 2 can be expressed, for example, as points in 3D coordinates centered at any location within the vehicle V (hereinafter referred to as a vehicle 3D coordinate system). An X-axis forming the vehicle's 3D coordinate system can, for example, be an axis parallel to the vehicle's left-right orientation. Furthermore, a Y-axis can be an axis parallel to the forward and reverse directions. A Z-axis can be an axis parallel to the vehicle's vertical direction.The vehicle's clockwise direction corresponds to the positive X-axis direction, its forward direction to the positive Y-axis direction, and its upward direction to the positive Z-axis direction. The center point of the vehicle's 3D coordinate system could, for example, be the center point of the rear axle.

[0030] Display 3 is a device that includes, for example, a thin-type display panel, such as a liquid crystal display, and shows various types of information and images. Display 3 is a display device. Display 3 is located in the instrument panel or similar location in a vehicle to allow a user to view the screen. Display 3 can also be located within the same housing as the image-generating ECU 1 and can thus be integrated into the image-generating ECU 1. Of course, display 3 can also be a separate device from the image-generating ECU 1. Display 3 includes the touch panel 4 laminated onto the display panel and is therefore capable of receiving user input.Touch field 4, for example, is formed by a capacitive touch field and is configured to output signals indicating positions touched by the user. Here, the term "user" primarily refers to an occupant in the driver's seat (the so-called driver). The term "user" can also include an occupant in the passenger seat next to the driver.

[0031] The control buttons 5 are operating elements that receive user input. The Sys ambient display system can have a display toggle switch 51 and a driving mode switch 52 as control buttons 5. The display toggle switch 51 is a switch for displaying a synthetic image CP generated by the image generation ECU 1 on the display 3 and for changing the virtual viewpoint and the like of a synthetic image CP displayed on it. The display toggle switch 51 is, for example, located on the steering wheel of the vehicle and primarily receives input from the driver. The user can perform various operations on the Sys ambient display system using the display toggle switch 51 and the touch panel 4 on the display 3.If the user operates the display selector switch 51 or the touch panel 4, an operating signal indicating the content of this operation is input into the image generation ECU 1. The display selector switch 51 can also be referred to as a multi-information switch. The display selector switch 51 can also be located on the instrument panel.

[0032] The driving mode switch 52 is a button that allows the driver to switch the vehicle's driving mode. The driving mode switch 52 can also be located in a center console, instrument panel, or similar location. Buttons corresponding to the respective driving modes can be positioned as part of the driving mode switch 52. The driving mode switch 52 can be designed as a rotary switch. The control element for switching the driving mode can be of the selector type. The gearshift lever can also be designed to function as the driving mode switch 52. The driving mode switch 52 and the gearshift lever together constitute an input device for switching the driving mode to an off-road mode.

[0033] The drive mode switch 52 outputs a signal to the image generation ECU 1 indicating the drive mode selected by the user. The output signal from the drive mode switch 52 is also input to the integration ECU 9, which integrally controls multiple ECUs that comprise a powertrain system within the vehicle. The integration ECU 9 corresponds, for example, to a domain ECU in a powertrain system. The integration ECU 9 switches the vehicle's drive mode V and also integrally controls the behavior of a powertrain ECU and a chassis ECU in response to the input signal from the drive mode switch 52. This integration ECU 9 can, in a sense, be considered an ECU for managing the vehicle's drive mode. The powertrain unit controlled by the powertrain ECU is not limited to an internal combustion engine and can also be an electric motor.The powertrain ECU can also control the torque distribution to the front and rear wheels, and furthermore, independently control the torque to the respective left and right rear wheels. For example, the powertrain ECU can also function as a TVD (torque vectoring differential) ECU.

[0034] The vehicle condition sensors 6 are sensors for detecting state variables related to the vehicle's driving control. The vehicle condition sensors 6 include a gear position sensor, a vehicle speed sensor, a steering angle sensor, an acceleration sensor, and similar components. The gear position sensor detects the position of the gearshift lever. The vehicle speed sensor detects the vehicle's speed. The steering angle sensor detects the angle of rotation of the steering wheel (the so-called steering angle). The acceleration sensor detects the acceleration acting on the vehicle in at least one of the following directions: forward and reverse, lateral, and upward and downward. In this case, a triaxial acceleration sensor is assumed to be used.Values ​​acquired by the accelerometer can be used as data to determine the vehicle's position relative to a horizontal plane. The types of sensors used in the environmental display system Sys as the vehicle condition sensors 6 can be appropriately selected, and the environmental display system Sys need not necessarily include all of the aforementioned sensors. The vehicle condition sensors 6 can also include a vehicle height sensor and the like. The respective sensors output data to the image generation ECU 1, which indicates the current values ​​of the quantities of physical conditions to be detected (namely, the results of the detections).

[0035] The Sonar ECU 7 is an ECU that controls the operation of the Sonars 8. The Sonars 8 generate an ultrasound as a sounding wave and receive reflected waves resulting from the reflection of the ultrasound by an object, thus detecting the object present in the vicinity of the vehicle. The Sonars 8 are also capable of determining the distance to the object based on the time from the generation of the ultrasound to its return. Information about the distance to the object, acquired by each Sonar 8, is, for example, fed into the Image Generation ECU 1 via the Sonar ECU 7.

[0036] In the present embodiment, as in Fig. Figure 3 shows, as an example, the eight sonars 8. Specifically, four sonars 8A to 8D are arranged from the right corner of the front bumper to its left corner. Furthermore, four sonars 8E to 8H are arranged from the right corner of the rear bumper of the vehicle to its left corner. The sonars 8, positioned as described above, enable the sonar ECU 7 to detect an object located in front of or behind the vehicle. The aforementioned aspect of the sonar 8 placement is merely an example, and the placement of the sonars 8 is not limited to the one shown in Figure 3. Fig. The example shown is limited. Furthermore, the sonars 8 can be mounted on a side sill, a fender, or a door panel to generate ultrasound towards an area near one side of the vehicle.

[0037] Furthermore, the sonar ECU 7 identifies the relative position of an object in the vicinity of the vehicle by combining the results of the detections by the respective sonar units 8. For example, regarding the direction of a detected object, the direction of that object relative to the vehicle is derived based on the respective distances to the same object detected by two or more sonar units 8. If the sonar units 8 detect an object, the sonar ECU 7 inputs data to the image generation ECU 1 indicating the direction and distance to that object as a result of the detection. The sonar ECU 7 can also be configured to output the height of a detected object, the intensity of the reflection it causes, and the waveforms received from it to the image generation ECU 1 as information that indirectly indicates whether the vehicle's surroundings are in a terrain environment.The sonar ECU 7 can also be structured to determine whether the peripheral environment is terrain, such as a rocky area, and to output the result of this determination to the image generation ECU 1. The sonar ECU 7 can also be integrated into the image generation ECU 1. <Struktur der Bilderzeugungs-ECU>

[0038] The image generation ECU 1 contains an image reference unit F1, an operating receiver unit F2, a vehicle state reference unit F3, an obstacle information reference unit F4, an image recognition unit F5, a display control unit F6, the image synthesis unit F7, a display image generation unit F8 and an image output unit F9 as functional units, as shown in Fig. 4 shown.

[0039] The image reference unit F1 receives camera images, which are images produced by the respective four cameras 2. If the majority of cameras 2 perform a photograph, as in Fig. As shown in Figure 5, four camera images, SF, SB, SL, and SR, are acquired, showing the front, rear, left, and right sides of the vehicle. These four camera images contain data of the entire area around the vehicle. Camera image SF is generated by the front camera 2F, and camera image SB is generated by the rear camera 2B. Camera image SL is generated by the left-side camera 2L, and camera image SR is generated by the right-side camera 2R. The image reference unit F1 converts the image signals input from cameras 2 into digital image data of a predefined data format. The image reference unit F1 performs predefined image processing on the acquired camera images and inputs the processed camera images to the image synthesis unit F7 and the display image generation unit F8.

[0040] The operating receiver unit F2 receives control signals issued by the control buttons 5 and the touch panel 4 when the user performs operations on them. Thus, the operating receiver unit F2 receives the user's instructions regarding the display of synthetic images CP and camera images. The operating receiver unit F2 then inputs data into the display control unit F6 according to the received control signals.

[0041] The vehicle state reference unit F3 is structured to obtain information indicating the vehicle's state from other devices located within the vehicle, separate from the image generation ECU 1, such as the vehicle state sensors 6. For example, the vehicle state reference unit F3 obtains the currently set gear position from the gear position sensor. The vehicle state reference unit F3 can obtain the driving mode from the integration ECU 9. Furthermore, the vehicle state reference unit F3 detects accelerations in the respective detection axes, vehicle speed, steering angle, and similar parameters.

[0042] The obstacle information reference unit F4 receives information about a stereoscopic object present in the vehicle's vicinity from the sonar ECU 7. Specifically, the obstacle information reference unit F4 receives the size, height, and relative position of the detected stereoscopic object, among other parameters. If the sonar ECU 7 has an identifier for identifying the type of detected object by analyzing characteristics of signal waveforms from reflected waves received by it, the obstacle information reference unit F4 also receives the result of the sonar ECU 7's identification of the detected object.

[0043] The image recognition unit F5 is structured to determine the position, type, and other characteristics of a predefined object by analyzing images input from cameras 2. The image recognition unit F5 functions as an identifier for identifying object types, for example, based on characteristic vectors in images. The image recognition unit F5 identifies objects using convolutional neural network (CNN) and deep neural network (DNN) techniques, which employ, for example, deep learning. These detected objects include pedestrians and other vehicles, as well as traffic signs such as road markings, which may be present on paved roads, road edges, and the like.Such captured objects may also include stereoscopic structures intended for use on roads, such as traffic signs, destination signs, guardrails, telegraph poles and the like.

[0044] Additionally, the image recognition unit F5 can be structured to recognize stones and rocks. The result of the recognition by the image recognition unit F5 is output to the display control unit F6 and the like. Furthermore, the image recognition unit F5 can be structured to output the recognition result, indicating whether the driving environment is on a road or terrain, to the display control unit F6 based on the pattern of an image area that is estimated to be a ground surface (for example, the result of edge detection).

[0045] The display control unit F6 is structured to comprehensively control the entire image generation ECU 1. For example, the display control unit F6 controls the image synthesis unit F7 and the display image generation unit F8 based on information input from the operating receiver unit F2 and the vehicle state reference unit F3, causing them to generate a synthetic image CP and a displayed image DP according to the vehicle's driving conditions and the user's settings.

[0046] The display control unit F6 contains a driving direction reference unit F61, a driving environment determination unit F62, a projection surface control unit F63, and a gaze point control unit F64 as sub-function blocks. The driving direction reference unit F61 determines whether the vehicle's direction of travel is forward or reverse, for example, based on the direction of tire rotation or signals from the shift position sensor.

[0047] The driving environment determination unit F62 is designed to determine whether the vehicle's location (in other words, the driving environment) is on a road or off-road. The driving environment determination unit F62 can also be referred to as a road surface type determination unit for determining road surface types. For example, the driving environment determination unit F62 can determine whether the driving environment is off-road based on input signals from the driving mode switch 52. Specifically, the driving environment determination unit F62 determines that the driving environment is off-road based on the fact that a signal indicating a specific off-road mode is input from the driving mode switch 52.In contrast, the driving environment determination unit F62 can determine that the driving environment is a normal mode based on the fact that a signal indicating a defined normal mode is input from the driving mode switch 52. The driving environment determination unit F62 can also determine whether the driving environment is off-road based on driving mode information obtained by the vehicle state reference unit F3 from the integration ECU 9.

[0048] The driving environment determination unit F62 can also determine whether the driving environment is terrain based on the detection result from the image recognition unit F5. For example, the driving environment determination unit F62 can determine that the driving environment is terrain based on the fact that equal to or more than a predefined number of objects with properties indicative of rocks have been detected, or that rocks have been detected forward, backward, left, and right of the vehicle through image recognition processing. The driving environment determination unit F62 can also determine that the driving environment is terrain based on the fact that multiple obstacles have been continuously detected.In this case, "obstacles" refer to natural stereoscopic objects such as rocks, differences in ground level, and the like, rather than artificial stereoscopic objects such as guardrails or other vehicles. Natural stereoscopic objects, such as rocks, have various shapes, making it difficult to determine through image recognition that the detected object is a natural stereoscopic object. In contrast, the type of an artificial object can be identified relatively easily. Given these circumstances, the F62 driving environment determination unit can also determine whether the driving environment is off-road or not by treating detected stereoscopic objects, whose types could not be identified through image recognition processing, as natural stereoscopic objects.The driving environment determination unit F62 can also determine that the driving environment is terrain, provided that the image recognition unit F5 has not detected any element indicating a paved road at the front, rear, left, or right sides of the vehicle. Such an element indicating a paved road would be, for example, a lane marking, a road edge, or the like.

[0049] The F62 driving environment determination unit can also determine that the driving environment is terrain based on the fact that the sonar has continuously detected obstacles as described above, either in front of, behind, to the left, and to the right of the vehicle. The F62 driving environment determination unit can also determine whether the driving environment is terrain through sensor fusion between the image recognition processing and the sonar.

[0050] Furthermore, the F62 driving environment detection unit can also determine whether the driving environment is terrain or not based on patterns detected by the accelerometer. For example, if the acceleration in the upward and downward directions has continuously changed at an amplitude of a predefined threshold for a constant distance (e.g., 3 m), the F62 driving environment detection unit can determine that the driving environment is terrain. Similarly, the F62 driving environment detection unit can determine that the driving environment is terrain if a vibration sensor continuously detects vibrations equal to or greater than a predefined threshold for a constant distance while the vehicle is in motion. The F62 driving environment detection unit can also determine that the driving environment is terrain based on the fact that one of a plurality of tires was in neutral.The F62 driving environment determination unit can also determine whether the driving environment is terrain or not by using map data and position information about the vehicle, which has been identified by a GNSS (Global Navigation Satellite System). Various types of information and determination criteria can be used as materials to determine that the driving environment is terrain.

[0051] The F62 driving environment determination unit can determine that the driving environment is a road based on the fact that a condition for determining that the driving environment is terrain is no longer met. For example, the F62 driving environment determination unit can determine that the driving environment is a road based on the fact that a spatial line, a road edge, a guardrail, or a road sign has been detected by image recognition processing. The F62 driving environment determination unit can also determine that the driving environment is a road based on the fact that the amplitude of the acceleration in the upward and downward directions has fallen below a predefined value during travel. Various conditions can be used as conditions required to determine that the driving environment is a road.

[0052] The projection surface control unit F63 is structured to switch a projection surface TS, which is used for image synthesis processing. The projection surface TS is a virtual stereoscopic surface that corresponds to the area around the vehicle, as conceptually described in Fig. Figure 6 illustrates this. A central region of the projection surface TS is defined as a vehicle area R0, representing the position of the vehicle. The image generation ECU 1, according to the present embodiment, is structured to selectively use a normal projection surface and a terrain projection surface. The normal projection surface TS1 is used when the vehicle is on a road, and the terrain projection surface TS2 is used when the vehicle is off-road. The respective shapes of the plurality of projection surfaces TS are defined by a plurality of parts of projection surface data Dt, which are provisionally stored in the driving environment determination unit F62. The projection surfaces TS are described in more detail separately later.

[0053] Switching the projection surface to generate a synthetic image CP is, in one respect, equivalent to switching the display mode for the synthetic image CP. For simplicity, a mode for displaying a synthetic image CP using the terrain projection surface is referred to as a terrain display mode. A display mode for displaying a synthetic image CP using the normal projection surface is referred to as a normal display mode. Since the normal projection surface TS1 is a projection surface TS used when the vehicle is on a road, the normal projection surface TS1 can be referred to as a road projection surface. The normal display mode can be referred to as a road display mode.

[0054] The gaze point control unit F64 defines the position and viewing direction of a virtual gaze point VP for generating a synthetic image CP, described later, based on at least one of the following factors: the result of the determination by the driving direction reference unit F61, the signal from the touch field 4, and the signal from the display switch 51. Configurable patterns for the virtual gaze point VP include a bird's-eye view view point VPb, a driver's gaze point VPd, and the like. The bird's-eye view view point VPb is a configurable pattern of the virtual gaze point VP where the gaze point position is directly above the vehicle and its viewing direction is directed precisely downwards. The bird's-eye view view point VPb can be used in cases where bird's-eye view images CPb are generated, which are images of the vehicle and its surroundings viewed from just above the vehicle.The viewpoint position of the bird's-eye viewpoint VPb is not limited to the position directly above the vehicle and can be a position that deviates from the position directly above the vehicle to the rear, front, or side. The bird's-eye viewpoint VPb corresponds to an example of a vehicle interior exterior viewpoint, where the virtual viewpoint VP is placed outside the vehicle interior.

[0055] The driver's gaze point VPd is a definable pattern of the virtual gaze point VP, where the gaze point position is defined at an estimated position of the driver's eyes within the vehicle space, as shown in Fig. Figure 7 illustrates this. The direction of view of the driver's gaze point VPd can be set to be a forward direction and a downward angle, for example, to include the area around the front wheels. The downward angle can be defined, for example, as a downward angle of approximately 20 to 30 degrees relative to a horizontal plane of the vehicle. The direction of view of the driver's gaze point VPd can be made changeable in any direction based on user actions (for example, swiping) on ​​the touch panel 4, while the forward and downward angle is defined as the default direction.

[0056] An eye ellipse, defined for each vehicle type, can be used as the estimated position of the driver's eyes. Such an eye ellipse is a virtual space defined for each vehicle type and is set to have a virtual elliptical shape based on an eye area that statically expresses a specific distribution of occupant eye points (see JISD0021: 1998). For example, the estimated position of the driver's eyes is set near the headrest of the driver's seat. The driver's eye point VPd corresponds to an example of an interior vehicle eye point, with the virtual eye point VP placed within the vehicle space. The position of the driver's eye point VPd can also be placed at a location that differs from the estimated position of the driver's eyes.For example, the driver's gaze point VPd can also be placed in a position that differs from the estimated position of the driver's eyes by a predetermined amount in the direction of the passenger seat, such as a position between the driver's seat and the passenger seat.

[0057] Switching the viewpoint to generate a synthetic image CP is equivalent to switching the display mode for the synthetic image CP. For simplicity, states in which the bird's-eye viewpoint VPb is used are referred to as bird's-eye viewpoint mode. States in which the driver's viewpoint VPd is used are referred to as driver's viewpoint mode.

[0058] The image synthesis unit F7 is structured to perform image processing to generate a synthetic image CP. The image synthesis unit F7 projects data from multiple camera images onto a virtual projection surface that corresponds to the vehicle's surroundings. Furthermore, the image synthesis unit F7 generates a synthetic image CP, depicting the vehicle's surroundings as viewed from a virtual viewpoint VP, by using the data on this projection surface. The display control unit F6 manages the operation of the image synthesis unit F7. For example, the display control unit F6 controls the projection surface and the virtual viewpoint VP to be used for generating the synthetic image CP.In particular, the image synthesis unit F7 generates a synthetic image CP by selectively using one of the multiple projection surfaces, each with a different shape, controlled by the display control unit F6. The procedure for generating a synthetic image CP will be described in more detail later.

[0059] The display image generation unit F8 generates a displayed image DP, which is to be shown on the display 3. The display image generation unit F8 generates a displayed image DP that contains a synthetic image CP and a camera image, using the synthetic image CP generated by the image synthesis unit F7 and the camera image obtained from the image reference unit F1. The display control unit F6 determines the combination of images contained in the displayed image DP according to the vehicle's direction of travel and the user's operations on the touch panel 4, and the like. Specifically, the display control unit F6 controls the operation of the display image generation unit F8. The operations of the display image generation unit F8 and the display control unit F6 for generating a displayed image DP are described separately later.

[0060] The image output unit F9 converts the displayed image DP generated by the display image generation unit F8 into image signals of a predefined signal format and then outputs the image signals to the display 3, causing the display 3 to show the displayed image DP. Thus, a synthetic image CP is displayed on the display 3, showing the environment of the vehicle as viewed from the virtual viewpoint VP 3. <Normale Projektionsfläche>

[0061] The following describes the normal projection surface TS1 for use in generating a synthetic image CP. The normal projection surface TS1 has a shell shape with a downwardly convexly curved surface. Fig. Figure 8 is a view showing a cross-section of the normal projection surface TS1 along the left-right direction of the own vehicle. Fig. Figure 9 is a view showing a cross-section of the normal projection surface TS1 along the forward and reverse directions of the vehicle. As shown in the figures, the normal projection surface TS1 generally has a shape that is a flat surface formed along a horizontal vehicle direction near the vehicle area R0, and also has greater slopes (gradients) at greater distances from the vehicle area R0. In this description, the term "slope" of the projection surface TS refers to its length in the vertical direction relative to a unit length in the horizontal vehicle direction. The "slope" at any position on the projection surface TS can also be described as the slope of a tangent line at that position.

[0062] The normal projection surface TS1 can be subdivided into the vehicle area R0, which has a flat surface shape corresponding to the road surface area in which the vehicle is located; a flat surface area R1, which forms a flat surface that is continuous or connected to the vehicle area R0; and a curved surface area R2, which forms a downwardly convex curved surface that is separated from the vehicle area R0. The flat surface area R1 corresponds to a road flat surface area, and the curved surface area R2 corresponds to a road curved surface area.

[0063] It can be said that the normal projection surface TS1, as previously described, has the flat surface area R1 with relatively smaller slopes and is located at a relatively greater distance from the vehicle area R0, and the curved surface area R2 with relatively larger slopes and is located at a relatively greater distance from the vehicle area R0. The vehicle area R0 corresponds to an area that overlaps with the vehicle itself in a top view.

[0064] The flat surface area R1 borders the vehicle area R0 and is positioned to surround the area of ​​vehicle area R0. The curved surface area R2 is positioned outside the flat surface area R1. This structure of the normal projection surface TS1 corresponds to a structure that provides the flat surface area R1 between the vehicle area R0 and the curved surface area R2 from a different perspective. The flat surface area R1 extends from the edge segment of the vehicle area R0 at least to a point that is separated from it by a predetermined minimum flat surface distance Dmin or more in the horizontal vehicle direction. In other words, an area from the vehicle area R0 at least to the position separated from it by the minimum flat surface distance Dmin or less is designated as the flat surface area R1.The minimum flat surface distance Dmin can be, for example, 0.3 m, 0.5 m, or 1 m. A flat surface training distance D1, which is the distance from the edge section of the vehicle area R0 to the boundary between the flat surface area R1 and the curved surface area R2, is defined as the minimum flat surface distance Dmin or greater. The flat surface training distance D1 can, for example, be set to 1.5 m. The flat surface training distance D1 can differ in dimension between the forward and reverse directions and between the left and right directions. For example, the flat surface training distance D1 can be set approximately 0.25 to 0.5 m greater in the forward and reverse directions than the flat surface training distance D1 in the left and right directions.

[0065] The curved surface region R2 is formed outside the flat surface region R1, specifically at the circumferential edge segment of the normal projection surface TS1. Furthermore, the curved surface region R2 has a shape with a gradually increasing slope. For example, the curved surface region R2 can be designed to have a shape similar to a quadratic curve, in other words, a parabolic shape. Therefore, the shape of the curved surface region R2 can be defined by a coefficient parameter of a quadratic curve.For example, the cross-sectional shape of the curved surface region R2 contained within the normal projection surface TS1 can be expressed as in the following formula (2) by using a coefficient “a1”, assuming that an origin point is located at an outer edge section of the flat surface region R1, the x-axis lies in the horizontal direction of the vehicle, and the z-axis lies in the vertical direction of the vehicle. The positive direction of the x-axis is assumed to be the direction that is farther away from the vehicle region R0. z=a1⋅x2

[0066] The coefficient "a1" is a parameter that defines the degree of inclination of the curved surface region R2 and can also be called a slope coefficient. "a1" can be appropriately sized. Increasing the coefficient "a1" increases the slope of the curved surface region R2. Of course, the model of the curved surface region R2 is not limited to a quadratic curve and can also be an arc shape, a logarithmic function, or an exponential function.

[0067] In this case, for example, data from camera images is not projected onto the vehicle area R0, and data from camera images is projected onto the areas outside the vehicle area R0. The vehicle area R0 corresponds to a non-projection area onto which no camera image is projected. In the following, the areas onto which the data from camera images is projected (the areas outside the vehicle area R0) in the various projection surfaces TS are also referred to as "projection object areas." In the normal projection surface TS1, the flat surface area R1 and the curved surface area R2 correspond to the projection object areas.

[0068] The standard projection area TS1 contains a front area PF, corresponding to the front of the vehicle; a rear area PB, corresponding to its rear; a left area PL, corresponding to its left side; and a right area PR, corresponding to its right side. The front area PF is the area onto which an image SF from the front camera 2F is projected. The rear area PB is the area onto which an image SB from the rear camera 2B is projected. The left area PL is the area onto which an image SL from the left-side camera 2L is projected. The right area PR is the area onto which an image SR from the right-side camera 2R is projected. The flat surface area R1 is contained within the respective areas PF, PB, PL, and PR in the standard projection area TS1.Each position in the projection object areas in the normal projection surface TS is assigned to one of the four camera images SF, SB, SL and SR by correspondence information, such as table data. <Geländeprojektionsfläche>

[0069] Next, with reference to the Fig. 10 and Fig. Section 11 describes the terrain projection surface TS2. The terrain projection surface TS2 is a projection surface TS for generating a synthetic image CP, similar to the normal projection surface TS1, and corresponds to a 3D model surface onto which the respective camera images are projected (in other words, performing a texture mapping).

[0070] Fig. Figure 10 is a view showing a cross-section of the terrain projection surface TS2 along the left-right direction of the own vehicle. Fig. Figure 11 is a view showing a cross-section of the terrain projection surface TS2 along the forward and reverse directions of the vehicle. As shown in the Fig. 10 and Fig. As shown in Figure 11, the terrain projection surface TS2 generally has a shell shape with a downwardly convexly curved surface. This terrain projection surface TS2 can be subdivided into a vehicle area R0, in which the vehicle is located, and a curved surface area R2a, which surrounds the vehicle area R0 and has a downwardly convexly curved surface. Specifically, the terrain projection surface TS2 according to the present embodiment does not contain a flat surface area between the vehicle area R0 and the curved surface area R2a. The curved surface area R2a corresponds to a terrain-curved surface area.

[0071] The curved surface area R2a is designed to be coupled to the edge section of the vehicle area R0. In the terrain projection surface TS2 according to the present embodiment, only the curved surface area R2a corresponds to the projection object area. The curved surface area R2a includes a front area PF opposite the front surface of the vehicle, a rear area PB opposite the rear surface section of the vehicle, a left-side area PL opposite its left side surface, and a right-side area PR opposite its right side surface.

[0072] The curved surface area R2a is designed to be a curved surface with smaller slopes at positions closer to the vehicle area R0 and larger slopes at positions further away from the vehicle area R0. Specifically, the curved surface area R2a has a shape with a slope that gradually increases in the horizontal direction of the vehicle. For example, the shape of a cross-section of the curved surface area R2a can be expressed as in the following formula (2), assuming that an origin point is located at the boundary section of the vehicle area R0, the x-axis lies in the horizontal direction of the vehicle, and the z-axis lies in the vertical direction of the vehicle. z=a2⋅x2

[0073] "a2" is a coefficient that defines the magnitude of the slope of the curved surface area R2a. The coefficient "a2" can also be called a slope coefficient. Increasing the coefficient "a2" increases the slope of the curved surface area R2a. The coefficient "a2" is set to have a value greater than that of the previously mentioned coefficient "a1." Specifically, the curved surface area R2a, which is contained within the terrain projection surface TS2, is designed to have a steeper slope than that of the curved surface area R2, which forms the normal projection surface TS1. For example, the slope of the curved surface area R2a contained within the terrain projection surface TS2 is set to be equal to or greater than 1.5 times the slope of the curved surface area R2, which forms the normal projection surface TS1.For example, the inclination of the curved surface area R2a is set to satisfy the relationship a2 ≥ 1.5·a1.

[0074] The curved surface area R2a in the terrain projection surface TS2 can have a different slope between areas PL and PR relative to the side surfaces of the vehicle and between areas PF and PB relative to the front and rear surfaces of the vehicle. For example, the front area PF and the rear area PB can be designed with a greater slope than the left and right areas. The coefficient "a2" in the direct forward and reverse directions of the vehicle can be set equal to or greater than 1.2 times the coefficient "a2" in the right and left directions of the vehicle. The slope changes gradually between sections with different slopes in the curved surface area R2a.

[0075] Although the present embodiment uses an aspect in which the curved surface region R2a has a downwardly convex parabolic shape, the present disclosure is of course not limited to this. The model of the curved surface region R2a is not limited to a quadratic curve and can also be an arc shape, a logarithmic function, or an exponential function. <Verfahren zur Erzeugung eines synthetischen Bildes>

[0076] The following describes a methodology or procedure by which the image synthesis unit F7 is instructed to generate a synthetic image CP that shows the state of the environment of the vehicle as viewed from the virtual viewpoint VP. The procedure for generating a synthetic image CP is described below by illustrating a case where the normal projection surface TS1 is used as the projection surface TS. The same can be applied to cases where the terrain projection surface TS2 is used as the projection surface TS.

[0077] When generating a synthetic image CP, the image synthesis unit F7 first projects data (the values ​​of the respective pixels) contained in four camera images SF, SB, SL, and SR, input by the image reference unit F1, onto the normal projection surface TS1 in a virtual 3D space. The respective camera images and the positions at which they are projected onto the normal projection surface TS1 were previously linked with correspondence information, such as tabular data. The image synthesis unit F7 projects the data from the four camera images SF, SB, SL, and SR onto the corresponding areas on the normal projection surface TS1.

[0078] Specifically, the image synthesis unit F7 projects the data from the front camera 2F's camera image SF onto the front area PF in the normal projection surface TS1. Furthermore, the image synthesis unit F7 projects the data from the rear camera 2B's camera image SB onto the rear area PB in the normal projection surface TS1. Additionally, the image synthesis unit F7 projects the data from the left camera 2L's camera image SL onto the left area PL in the normal projection surface TS1, and furthermore, it projects the data from the right camera 2R's camera image SR onto the right area PR.

[0079] Sections of the camera images that extend beyond the projection object areas can be eliminated. For overlapping areas, which are areas redundantly photographed by two cameras, it is possible to use a methodology or procedure for blending images taken by two cameras in a predefined ratio, or a methodology or procedure for combining images taken by two cameras at a predefined boundary line.

[0080] After projecting the data from the corresponding camera images onto the respective sections of the normal projection surface TS1, the image synthesis unit F7 next generates a vehicle image Pv, which is a 3D model of the vehicle. Memory 13 can temporarily store data for drawing the vehicle image Pv as image drawing data. The vehicle image Pv is then placed on the vehicle area R0.

[0081] Next, the image synthesis unit F7 defines a virtual viewpoint VP in a 3D space containing the normal projection surface TS1, controlled by the display control unit F6. The image synthesis unit F7 can position the virtual viewpoint VP, oriented in an arbitrary viewing direction, at any viewpoint position in the 3D space. Furthermore, the image synthesis unit F7 extracts the data projected onto an area contained within a predefined viewing angle, as viewed from the defined virtual viewpoint VP, as an image from the normal projection surface TS1. Finally, the image synthesis unit F7 performs rendering, or playback, on the vehicle image Pv according to the defined virtual viewpoint VP.The image synthesis unit F7 then overlays a two-dimensional image of the vehicle Pv, resulting from the rendering process, onto the cropped image. Thus, the image synthesis unit F7 generates a synthetic image CP, which shows the vehicle and an area of ​​its surroundings as viewed from the virtual viewpoint VP.

[0082] For example, if the position and such of the virtual viewpoint VP are set to be those of the driver's viewpoint VPd, as in Fig. As shown in Figure 12, the image synthesis unit F7 generates a driver viewpoint image CPd, which shows the front of the vehicle, so that it passes transparently through the instrument panel and the like. The driver viewpoint image CPd corresponds to an image generated by superimposing component element images Pve, which show parts of the vehicle as viewed from the driver's viewpoint VPd, onto an image of a photographed object that is in front of the vehicle and seen by the driver. By recognizing this driver viewpoint image CPd, the user can perceive the state of the vehicle's surroundings from the viewpoint inside the vehicle and can furthermore intuitively understand the state of the vehicle's surroundings.Furthermore, the user can intuitively understand in which direction the driver's viewpoint image CPd points from the surroundings of the vehicle, based on the component element images Pve contained in the driver's viewpoint image CPd.

[0083] The component element images Pve contain, for example, a ground image Pvf, a tire image Pvt, and interior images Pvi. The ground image Pvf, which is an image showing the area where the vehicle's underbody is located, is defined, for example, to be opaque. The tire image Pvt, which is an image drawn using a 3D model of the tires, is defined as opaque or semi-transparent. The tire image Pvt can also be an image of just an opaque or semi-transparent contour line. Within the tire image Pvt, the interior of the contour line can be made colorless or transparent. Additionally, the synthetic image CP can contain a frame line image that indicates the road surface areas where the tires are located, either instead of or parallel to the tire image Pvt. The interior images Pvi are images showing an instrument panel, a handle, an A-pillar, and the like.The interior images (Pvi) are also made transparent or semi-transparent in areas other than the contour sections. Additionally, the contour lines in the interior images (Pvi) can also be made semi-transparent. Furthermore, it is not necessary to display the interior images (Pvi). Although... Fig. Since image 12 represents an aspect where the handle, A-pillar, and headlight are shown, it is not necessary to display images indicating the positions of these elements. It is possible to modify the elements contained in the component element images Pve appropriately.

[0084] As previously described, in the synthetic image CP, component elements whose display is less necessary to allow the driver to recognize the positional relationship between the vehicle V and surrounding objects are either overlaid or shown only by contour lines. In other words, the component element images Pve contained in the synthetic image CP are defined in such a way as to minimize their number. This structure makes it possible to prevent the deterioration of visibility of the vehicle's surroundings.

[0085] Furthermore, if the position and the like of the virtual viewpoint VP is set to be that of the bird's-eye viewpoint VPb, the image synthesis unit F7 generates a bird's-eye view image CPb, which is a synthetic image CP of the vehicle and an area of ​​its surroundings viewed from above, as shown in Fig. 13 shown.

[0086] The initial state of the position and viewing direction of the virtual viewpoint VP in the synthetic image CP is determined based on at least one of the following parameters: the vehicle's direction of travel, a preliminary setting made by the user, and the steering angle. Furthermore, the position and viewing direction of the virtual viewpoint VP in the synthetic image CP can be changed based on user input via the touchscreen and similar actions. The image synthesis unit F7, according to the present embodiment, is structured to primarily utilize the driver's viewpoint VPd.The image synthesis unit F7 is structured in such a way as to generate a driver viewpoint image CPd, which is a synthetic image CP that was captured along the driver's line of sight in such a way that it shines transparently through the structure of the vehicle, as a synthetic image CP that is first generated when a display start condition, which will be described later, has been met. <Betriebsabläufe>

[0087] Regarding a Fig. The flowchart shown in Figure 14 describes the process of synthetic image display processing, which is a series of processes performed by the image generation ECU 1 to display a synthetic image CP. The Fig. The flowchart shown in Figure 14 is started when a predefined display start condition is met. The display start condition can be understood as a condition required to display a synthetic image CP; in other words, a condition required to start the current sequence. For example, the image generation ECU 1 determines that the display start condition has been met when the user presses the display toggle switch 51. Furthermore, the image generation ECU 1 can determine that the display start condition has been met when it detects that the user has performed a predefined operation to display a synthetic image CP via the touch panel 4. The current sequence can be started based on the fact that the operator receiver F2 has received a signal indicating that an operation to display a synthetic image CP has been performed.

[0088] The display start condition can be formed by elements relating to the shift position, vehicle speed, and the like. For example, the display start condition can include at least one of the following: that the shift position is set within a predefined range, and that the vehicle speed is lower than a predefined threshold. Additionally, the current sequence can be started when a viewpoint switching operation is received, in a state where a front camera image (SF) or a rear camera image (SB) is displayed.

[0089] The in Fig. The processing sequence shown in Figure 14 can be executed repeatedly in a predefined cycle (for example, a cycle of 1 / 30 of a second) until a predefined display cancellation condition is met. The display cancellation condition could be, for example, that the display toggle switch 51 has been pressed again, that the switch position has been set outside a predefined range, or that the vehicle speed has become equal to or greater than a predefined threshold, and so on. In addition, the image generation ECU 1 can determine that the display cancellation condition has been met when it detects that an operation to end the display of a synthetic image CP has been performed via the touch panel 4.

[0090] In this case, the synthetic image display processing includes steps S1 to S10 as an example. Of course, the number of steps comprising the synthetic image display processing and the order of those steps can be changed as needed.

[0091] First, in step S1, the image reference unit F1 retrieves four camera images SF, SB, SL, and SR, captured by the four cameras 2, and processing proceeds to step S2. Step S1 can be referred to as the image reference step. In step S2, the vehicle state reference unit F3 retrieves information indicating the state of the vehicle, such as the gear position, vehicle speed, and the like, and processing proceeds to step S3. Step S2 can be referred to as the vehicle state reference step.

[0092] In step S3, the driving environment determination unit F62 determines whether the driving environment corresponds to terrain or not, for example, based on signals from the driving mode switch 52. Of course, it is possible to use different methods to determine that the driving environment is terrain. Step S3 can be referred to as the driving environment determination step. If, in this case, the driving environment determination unit F62 determines that the driving environment is terrain, step S4 results in a positive determination, and processing proceeds to step S5. Conversely, if it is determined that the driving environment is on a road, step S4 results in a negative determination, and step S6 is executed.

[0093] In step S5, the display control unit F6, acting as the projection surface control unit F63, sends a signal to the image synthesis unit F7, instructing it to use the terrain projection surface TS2 as the projection surface TS used in generating a synthetic image CP. The image synthesis unit F7 obtains the shape of the terrain projection surface TS2 by reading the projection surface data Dt corresponding to the terrain projection surface TS2 from memory 13, based on the instruction from the display control unit F6. Thus, the terrain projection surface TS2 is used for image synthesis processing. This step S5 can be described as a terrain projection surface application step, in which the image synthesis unit F7 is instructed to select the terrain projection surface TS2 as the projection surface TS by being controlled by the display control unit F6.After processing is completed in step S5, processing proceeds to step S7.

[0094] In step S6, the display control unit F6, acting as the projection surface control unit F63, sends a signal to the image synthesis unit F7, instructing it to use the normal projection surface TS1 as the projection surface TS used in generating a synthetic image CP. Thus, the normal projection surface TS1 is used for image synthesis processing. This step S6 can be described as a normal projection surface application step, in which the image synthesis unit F7 is instructed to select the normal projection surface TS1 as the projection surface TS by being controlled by the display control unit F6. After processing in step S6 is complete, processing proceeds to step S7.This series of processing steps S4 to S6 can be described as a projection surface selection step for switching the projection surface TS for use in generating a synthetic image CP based on the result of the determination by the driving environment determination unit F62.

[0095] In step S7, the gaze control unit F64 determines the virtual gaze point VP to generate a synthetic image CP. If a synthetic image CP is initially displayed after a driving power source is activated, the virtual gaze point VP can be set to a position and direction previously defined as a default by the user or designer. For example, the default might be the driver's gaze point VPd, which has a forward and slightly downward direction. Alternatively, the virtual gaze point VP read in step S7 can be the position and direction of the virtual gaze point VP displayed by the driver at the previous time.In this case, the display control unit F6 is structured to store data about the setting of the virtual viewpoint VP when a synthetic image CP is displayed at the previous time in memory 13 and the like as a preparatory processing.

[0096] The position and viewing direction of the virtual viewpoint VP can also be determined depending on the direction of travel of the vehicle. For example, if the vehicle's direction of travel, as referenced by the direction reference unit F61, is forward, the driver's viewpoint VPd, which has a viewing direction oriented forward and diagonally downward as described previously, is used as the virtual viewpoint VP. Conversely, if the vehicle's direction of travel, as referenced by the direction reference unit F61, is reverse, a driver's viewpoint VPd, which has a viewing direction oriented backward and diagonally downward, can be used as the virtual viewpoint VP to generate a synthetic image CP. Furthermore, the viewing direction of the virtual viewpoint VP can also be set according to the steering angle.The gaze point control unit F64 can also obtain the position and direction of the virtual gaze point VP specified by the user via the touch field 4 and the like, and can furthermore set the virtual gaze point VP at the specified position and with the specified direction.

[0097] Information about the virtual viewpoint VP, determined in step S7, is output to the image synthesis unit F7. In this case, it is assumed, as an example, that the driver's viewpoint VPd, with a forward and slightly downward gaze direction, is used as the virtual viewpoint VP. This step S7 can be referred to as a virtual viewpoint determination step. After completion of step S7, processing proceeds to step S8.

[0098] In step S8, the image synthesis unit F7 generates a synthetic image CP, captured from the virtual viewpoint VP defined in step S7, using the projection surface TS according to the determination result by the driving environment determination unit F62. Specifically, the image synthesis unit F7 generates a driver viewpoint image CPd, showing the ground surface in front of the vehicle, which in this case is captured from the driver viewpoint VPd.

[0099] For example, when the normal projection surface TS1 is used, the image synthesis unit F7 projects data from the respective camera images onto the projection object areas, including the flat surface area R1, within the normal projection surface TS1. Furthermore, the image synthesis unit F7 generates a driver viewpoint image CPd by using the data on the normal projection surface TS1 onto which the various camera images were projected. When the terrain projection surface TS2 is used, the image synthesis unit F7 also projects data from the respective camera images onto the curved surface area R2a as the projection object area within the terrain projection surface TS2. Furthermore, the image synthesis unit F7 generates a driver viewpoint image CPd by using the data on the terrain projection surface TS2, where the respective camera images were textured.The data from the synthetic image CP generated by the image synthesis unit F7 is output to the display image generation unit F8. This step S8 can be referred to as a synthetic image generation step. After processing in step S8 is complete, processing proceeds to step S9.

[0100] In step S9, the display image generation unit F8 creates a displayed image DP, to be shown on display 3, using the synthetic image CP generated by the image synthesis unit F7. For example, the display image generation unit F8 creates an image that includes a driver viewpoint image CPd as a synthetic image CP, a right-side camera image SR, a left-side camera image SL, and a viewpoint switch image SW as a displayed image DP, as shown in Fig. Figure 15 illustrates this. Specifically, the driver's viewpoint image CPd is positioned at the top, in a central area of ​​the displayed image DP, and the viewpoint switch image SW is positioned below it. The right-side camera image SR is positioned to the right of the driver's viewpoint image CPd, and the left-side camera image SL is positioned to the left of the driver's viewpoint image CPd. This layout of the displayed image DP allows the driver to be visually notified of the left and right sides while maintaining a primary view of the driver's viewpoint image CPd. The driver can thus assess the condition of the road surface near the front wheels and the front of the vehicle, and simultaneously assess the condition of areas around the sides of the vehicle by viewing the aforementioned displayed image DP.

[0101] The viewpoint switch image SW contained within the displayed image DP is an image that acts as a switch for toggling the display content from the displayed image DP by the user's touch. Based on touch position signals output by touch field 4, it can be determined whether the user has touched the viewpoint switch image SW or not. For example, if a user touch operation is detected on the viewpoint switch image SW, the display image generation unit F8, upon instruction from the display control unit F6, switches the image displayed in the center of the screen from the driver viewpoint image CPd to the front camera image SF. The viewpoint switch image SW can also be configured to switch the virtual viewpoint VP from the driver viewpoint VPd to the bird's-eye view viewpoint VPb.In this case, the display image generation unit F8 creates an image that, for example, contains a bird's-eye view image CPb as a displayed image DP. Data from the displayed image DP, generated by the display image generation unit F8, is output to the image output unit F9. Step S9 can be described as a display image generation step.

[0102] In step S10, the image output unit F9 converts the digital data of the displayed image DP, generated by the display image generation unit F8, into signals of a predefined signal format and then outputs the signals to the display 3. Thus, the displayed image DP, including the driver's viewpoint image CPd, is shown on the display 3. Step S10 can be described as an image output step.

[0103] The previously mentioned aspect of the control is merely an example, and the display image generation unit F8 can select a camera image according to the vehicle's direction of travel and use it to generate a displayed image DP, controlled by the display control unit F6. For example, if the direction of travel is reverse, a rear camera image SB can be placed in the center of the displayed image DP. The display control unit F6 can also modify the combination of images displayed in the displayed image DP, its layout, the areas where camera images are displayed, and the like, based on control signals received from the control receiver unit F2. <Effekte des Umschaltens der Projektionsfläche>

[0104] The following describes the effects of switching the projection surface TS for use in generating a synthetic image from the normal projection surface TS1 to the terrain projection surface TS2 when the driving environment is terrain. The premise is that terrain contains a greater number of stereoscopic objects, such as rocks and geographical elevation changes, in the vicinity of the vehicle, compared to situations where the vehicle is on a road. Furthermore, the ground surface itself is often not flat or level in terrain. Therefore, when the driving environment is terrain, it is important to display such rocks, stones, and geographical features without compromising the stereoscopic effect and its realism.

[0105] Fig. Figure 16 is a view to explain the position at which an image of a rock Rk located near the own vehicle is projected, in a case where the normal projection surface TS1 is used to generate a synthetic image CP. The rock Rk, which is a rock located at a position relatively close to the own vehicle, represents a rock that is present within the flat surface area R1 in the normal projection surface TS1, namely at the flat surface formation distance D1 or less from the own vehicle. The term "rock" in this case refers, for example, to a rock with a size of 20 cm or greater.

[0106] An image of rock Rk, obtained by photographing rock Rk with camera 2 (for example, the front camera 2F), is projected onto the position where a straight line connecting the position of this camera 2 and the position of rock Rk is intersected by the normal projection surface TS1. Since rock Rk is located near the vehicle, the image of rock Rk is projected onto the flat surface area R1 in the normal projection surface TS1. In the figure, “X1” indicates the area where the image of rock Rk is projected onto the normal projection surface TS1. The dashed lines drawn on rock Rk indicate the surface section photographed by the front camera 2F. As shown in Fig. As shown in Figure 16, the section photographed by the front camera 2F is located outside the surface of rock Rk at a position higher than the horizontal plane where the flat surface area R1 is located. Therefore, the image of rock Rk is projected onto the flat surface area R1 in a shape that is stretched deeper (on other sides, further outwards) than its actual position.

[0107] Therefore, in a synthetic image CP of the normal projection surface TS1, viewed from the driver's viewpoint VPd, the image of the rock Rk is such that the rock near the vehicle appears larger than its actual size, with a flat surface shape and a degraded stereoscopic effect. This phenomenon is also induced in stereoscopic objects such as other rocks and stones located within the flat surface area R1. Furthermore, in an off-road environment, the driver tends to view a section closer to the vehicle V from the driver's viewpoint image CPd in ​​order to make adjustments such as positioning the tires on a particular rocky area or avoiding a larger rock that could damage the vehicle body.Consequently, when viewing the driver's viewpoint image CPd using the normal projection surface TS1, the user may perceive the vehicle's surroundings as flat. Furthermore, the user may tend to perceive stereoscopic objects as appearing squashed or distorted.

[0108] Naturally, using the normal projection surface TS1 provides the user with a visual impression that corresponds to the actual ground surface shape when the vehicle's driving environment is a road, i.e., when the vehicle is on a flat road. Therefore, it can be said that the normal projection surface TS1 has a preferred shape than the projection surface TS for generating a synthetic image when the vehicle is on a road. However, as previously described, in an off-road environment, the ground surface is rarely flat, which increases the likelihood that the aforementioned visual impression will not correspond to the actual ground surface shape. Consequently, a structure configured to use the normal projection surface TS1 even in off-road environments carries a relatively higher risk of causing the user to experience a sense of inconsistency.It has an incongruity.

[0109] Fig. Figure 17 is a view to explain the position at which an image of the aforementioned rock Rk is projected in a case where the terrain projection surface TS2 is used to generate a synthetic image CP. The image of rock Rk captured by camera 2 is projected onto the position where a straight line connecting the position of camera 2 and the position of rock Rk is intersected by the terrain projection surface TS2. In this case, there is no flat surface area R1 between the vehicle area R0 and the curved surface area R2a in the terrain projection surface TS2. The image of rock Rk is projected onto the curved surface area R2a. Specifically, the image of rock Rk is projected onto a projection position X2 on the curved surface area R2a.

[0110] As previously described, when using the terrain projection surface TS2, the image of rock Rk is projected onto projection position X2, which is closer to the vehicle area R0 than projection position X1, where the image of rock Rk is projected when using the normal projection surface TS1. Furthermore, the photographed surface of rock Rk and the projection surface are positioned relatively close to each other, which can reduce the risk of image distortion and its degree of distortion. In other words, it is possible to reduce the risk of the image of rock Rk appearing distorted in the driver's viewpoint image CPd. Consequently, when the driver views the synthetic image CPd, the impression is closer to that of directly seeing rock Rk.Furthermore, it prevents the driver from having the impression that the surroundings of the vehicle are flat, and the impression that stereoscopic objects are crushed or squashed.

[0111] Fig. 18 and Fig. 19 represent two driver viewpoint images CPd that were generated in the same terrain environment. Fig. Figure 18 is a line drawing of a driver viewpoint image CPd, which is generated by using the normal projection surface TS1. Fig. 19 is a line drawing of a driver viewpoint image CPd, generated using the terrain projection surface TS2. In the Fig. 18 and Fig. 19 These images are represented by eliminating the display of other component element images Pve besides the ground image Pvf, such as the tire image, to ensure the visibility of the figures.

[0112] As can be seen from the comparison between the driver viewpoint images CPd, which are in the Fig. 18 and Fig. As can be seen from Figure 19, the driver's viewpoint image CPd, by using the terrain projection surface TS2, represents rocks, stones and the like in the vicinity of the vehicle more stereoscopically than the driver's viewpoint image CPd, by using the normal projection surface TS1.

[0113] As previously described, if the driving environment has been determined to be terrain, it is possible to reduce the risk of the user experiencing a sense of incongruity or dissonance by generating a synthetic image CP using the terrain projection surface TS2, which does not contain a flat surface area R1 outside the vehicle area R0. Furthermore, this prevents the user from misinterpreting the road surface conditions in the vicinity of the vehicle, thus enabling the user to drive the vehicle safely.

[0114] Furthermore, if it has been determined that the driving environment is on a road, it is possible to give the user a visual impression of the road surface shape that corresponds to the actual ground surface shape by using the normal projection surface TS1. By switching the projection surface TS depending on whether the driving environment is terrain or not, it is possible to reduce the risk of the user experiencing a sense of incongruity or dissonance between different scenes.

[0115] Although the effects of the present embodiment were described in the preceding description using the virtual viewpoint VP of the synthetic image CP as an example, specifically the case of using the driver's viewpoint VPd with a forward-facing gaze, the same effects can be achieved in cases where the gaze is directed in other directions, such as backwards and oblique lateral directions. Furthermore, the same effects can be achieved when the virtual viewpoint VP is placed at any position within the vehicle compartment instead of the driver's viewpoint VPd. Additionally, the same effects can be expected when the virtual viewpoint VP is located on the exterior surface of the vehicle V or in a vehicle environment area outside the vehicle compartment.In this case, "the vehicle's surrounding area" refers to an area at a distance of, for example, 0.2 m or less from the vehicle's exterior surface area. "The exterior surface area" can include the left and right side surface areas, the rear surface area, the front edge area, and the roof. "Side surface areas" can include the door panels, fender sections, pillars, and the like.

[0116] Although embodiments of the present disclosure have been described previously, the present disclosure is not limited to these embodiments. Various modifications, which will be described later, are also included within the technical scope of the present disclosure. Furthermore, the present disclosure can be implemented by making various other modifications than those described later without departing from its spirit. For example, the various modifications described later can be implemented by properly combining them within a scope that does not cause technical inconsistency. Elements with the same functions as those described in the preceding embodiments are designated by the same reference numerals and are not described redundantly.If only one section of a structure is described, the structures can be applied to the other section according to the previously mentioned examples. <Anwendungen eines Unterbodenerkennungsbildes>

[0117] Although the foregoing description discloses one aspect in which a synthetic image CP containing an opaque ground image Pvf is displayed as the driver's viewpoint image CPd, the present disclosure is not limited thereto. For example, while the vehicle is moving forward, a front camera image SF, resulting from photographing the ground surface from just below the vehicle's front edge to 3 m in front of it, can be stored in RAM 12 and the like, and a synthetic image CP, showing the vehicle body-ground section transparently, can be generated using this image data stored while the vehicle is moving. The image data to be projected onto the underbody section can be updated at any time while the vehicle is moving V. This display control can also be implemented while the vehicle is moving in reverse as well as while it is moving forward.During reversing, it is possible to create an image of an area under the floor using rear camera images (SB). <Ergänzende Anmerkungen zur Geländeprojektionsfläche TS2>

[0118] Although the terrain projection surface TS2 in the preceding description does not include the flat surface area, the present disclosure is not limited thereto. The terrain projection surface TS2 may also include a flat surface area R1a with a width of less than 0.3 m between the vehicle area R0 and the curved surface area R2a, as shown in the Fig. 20 and Fig. Figure 21 shows the flat surface area R1a, which corresponds to a terrain flat surface area. Because the flat surface area R1a is less than 0.3 m wide, its display area in a driver's viewpoint image and the like is small. Furthermore, the area located 0.3 m or less from the vehicle body is photographed essentially directly above it, as camera 2 is positioned essentially directly above this area. As a result, the image is less susceptible to distortion, even when projected onto a flat projection surface. Therefore, because the flat surface area R1a is small, the driver is prevented from having the impression that stereoscopic objects near the vehicle are being squashed or crushed.The same effects as in the previously mentioned embodiments can be achieved even if the terrain projection surface TS2 contains the small flat surface area R1a on the side closer to the vehicle than the curved surface area R2a.

[0119] The structures described above, including the exemplary embodiments, correspond to a structure configured to place a curved surface region in the projection surface TS closer to the vehicle when the driving environment is defined as terrain than when it is defined as a road. Specifically, if the driving environment is defined as a road, the starting position of the curved surface is placed at a distance of 0.3 m or more from the vehicle. Conversely, if the driving environment is defined as terrain, the starting position of the curved surface is placed at a distance of less than 0.3 m from the vehicle. The "starting position of the curved surface" refers to the ground point where the curved surface region R2, R2a begins and can be considered the position where the projection surface rises.The starting position of the curved surface can be defined, for example, as the ground point with an inclination of 3 to 5 degrees or more relative to the flat plane to which the vehicle area R0 belongs. The structure configured to place the starting position of the curved surface at a location less than 0.3 m from the vehicle includes a structure configured to place it at a location 0.0 m from the vehicle, and a structure configured to place it within the vehicle area. <Nutzung von Hindernisinformationen>

[0120] In situations where the driving environment determination unit F62 has determined that the driving environment is terrain, the slope and shape of the terrain projection surface TS2 can be dynamically adjusted according to the results of the sonar 8 detection. For example, even if the environment has been determined to be terrain, the slope of the curved surface area R2a can be made smaller than a standard value if no obstacle has been detected in the environment. Furthermore, if the environment has been determined to be terrain when it is detected that multiple stereoscopic objects with heights equal to or greater than a predefined threshold are present within an area at a predefined distance or less from the vehicle, the slope (actually the coefficient a2) of the curved surface area R2a can be made larger than the standard value.If the ground surface in the vicinity of the vehicle has a degree of convexity and concavity equal to or greater than a predetermined value—for example, if the number of detected stereoscopic objects is equal to or greater than a predetermined value—then the slope (actually the coefficient a2) of the curved surface area R2a can similarly be made larger than the standard value. The slope of the curved surface area R2a can, in fact, be adjusted using the coefficient a2.

[0121] In Fig. In section 22, “R2aH” represents the curved surface area when the slope is greater than the default value, and “R2aL” represents the curved surface area when the slope is less than the default value. For simplicity, the projection surface TS with the curved surface area R2aL is also referred to as a low-slope projection surface TS3. The projection surface TS with the curved surface area R2aH is also referred to as a high-slope projection surface TS4. Given the slope of the curved surface area R2a, a terrain projection surface TS2 positioned between the low-slope projection surface TS3 and the high-slope projection surface TS4 is also referred to as an intermediate projection surface.Instead of setting the inclination of the curved surface area R2a in the terrain projection area TS2 relative to the standard value, it is also possible to dynamically determine the inclination of the curved surface area R2a within a predefined range, according to the obstacle detection situation by the sonar 8. Furthermore, the shape of the curved surface area R2a in the terrain projection area TS2 is not limited to a downward-convex curved surface shape. The curved surface area R2a can have an upward-convex curved surface shape. <Ergänzende Anmerkungen zu den Hindernissensoren>

[0122] Although the preceding description uses sonar 8 as an example sensor for detecting objects in the vehicle's environment (so-called obstacle sensors), these obstacle sensors can also be millimeter-wave radar or LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging). The image generation ECU 1 can be used by connecting it to various obstacle sensors. <Ergänzende Anmerkungen zu der Fahrumgebungsbestimmungseinheit>

[0123] The driving environment determination unit F62 can also be structured to determine terrain types by combining the various determination materials mentioned above. Terrain can be broadly divided into stereoscopic roads with many convexities and concavities, such as stony roads and bumpy roads, and slippery roads, such as muddy roads, sandy roads, glass fields, and snowfields. Such slippery roads refer to ground surfaces with relatively moderate gradient changes and relatively moderate concavities and convexities. For example, the driving environment determination unit F62 can also be structured to differentiate stereoscopic roads and slippery roads from each other as terrain types, based on at least one of the following parameters: the image acquisition result, the sonar acquisition result, and the defined driving mode.

[0124] The projection surface control unit F63 can be configured to change the shape pattern of the projection surface used according to the type of terrain determined by the driving environment determination unit F62. For example, if the driving environment determination unit F62 determines that the driving environment is a stereoscopic road, a terrain projection surface TS2 for stereoscopic roads is used. Conversely, if the driving environment determination unit F62 determines that the driving environment is a slippery road, a terrain projection surface TS2 for slippery roads is used. For example, the low-slope projection surface TS3 can be used as the terrain projection surface TS2 for slippery roads. For example, the intermediate projection surface or the high-slope projection surface TS4 can be used as the terrain projection surface TS2 for stereoscopic roads.The terrain projection surface TS2 for stereoscopic roads only needs to have a greater slope in the curved surface area R2a than that of the curved surface area R2a in the low-slope projection surface TS3.

[0125] With the aforementioned structure, it is possible to use the projection surface TS from different terrains, depending on the terrain type. This allows for the creation of a driver's viewpoint image CPd that is less prone to conveying a sense of inconsistency or incongruity. <Ergänzende Anmerkungen zu dem Geländeanzeigemodus>

[0126] The F8 display image generation unit can place a notification image NP, indicating that a terrain display mode is set, at a corner section and the like of a displayed image DP, as shown in Fig. Figure 23 shows the terrain display mode when the terrain projection area TS2 is set, specifically when the terrain projection area TS2 is used as the projection area TS. This structure reduces the risk of driver confusion, as the ground surface is displayed differently than usual.

[0127] It is also possible to toggle between displaying and hiding the NP notification image, depending on the information used to determine the terrain. For example, if the driving environment is determined to be terrain, the NP notification image will not be displayed based on user input via controls such as the control buttons 5, the touch panel 4, and the gearshift lever. Conversely, if the driving environment is automatically determined to be terrain based on information other than user input, such as the image acquisition or sonar readings, the NP notification image can be displayed.If the projection surface is switched in response to user input on the controls, the user recognizes that the vehicle is operating in off-road mode, thus eliminating unnecessary notifications and reducing annoyance. However, if the projection surface switches automatically, there is a possibility of malfunction, and therefore providing a notification that the off-road display mode is active reduces the risk of driver confusion. <Variationen des virtuellen Blickpunkts>

[0128] Although the bird's-eye viewpoint VPb and the driver's viewpoint VPd were presented as examples of the virtual viewpoint VP in the preceding description, the combination of position and direction that can be defined for the virtual viewpoint VP is not limited to the aforementioned example. For the virtual viewpoint VP, it is possible to use a pattern where the viewpoint position is set to the left and behind the vehicle, and the viewing direction is set to the forward direction of the vehicle. The virtual viewpoint VP can be placed at various positions outside the vehicle. Furthermore, the virtual viewpoint VP can be made placeable at any position inside the vehicle.

[0129] As a virtual gaze point VP, it is also possible to use a pattern where the gaze point position is placed near a side mirror, a fixed pattern where the gaze point position is placed in the middle of the ceiling section within the vehicle compartment, and the like. Furthermore, the gaze point control unit F64 can be structured to define an interior rear gaze point VPr, which is located a predetermined distance behind the eye ellipse at a gaze point position and also has a rearward or backward and downward viewing direction, as shown in Fig. Figure 24 illustrates this. By using this interior rear viewpoint VPr, it is possible to generate and display an interior rear viewpoint image, which is a synthetic image CP containing the area around the rear wheels and the rear side of the vehicle, as a synthetic image CP that is displayed while the vehicle is reversing. Consequently, the driver is able to easily perceive the states of the rear wheels and the area around the rear bumper during reversing, similar to how they can perceive them during forward driving. Furthermore, it is preferred that the interior rear viewpoint image includes a tire image corresponding to the rear wheels, a body boundary line, and the like, similar to the driver viewpoint image CPd. With this structure, it is possible to easily perceive the positional relationship between the respective elements that make up the vehicle and objects outside the vehicle space. <anmerkungen>

[0130] The device, system, and method or procedure disclosed herein can be implemented by dedicated computers comprising processors programmed to execute one or more functions specified by computer programs. The device and method or procedure disclosed herein can also be implemented by using dedicated hardware logic circuits. The device and method or procedure disclosed herein can also be implemented by one or more dedicated computers comprising a combination of a processor for executing computer programs and one or more hardware logic circuits.Computer programs, as instructions to be executed by computers, can be stored on computer-readable, non-temporary physical storage media. The means and / or functions provided by the processing unit 11 and the like can be provided by software stored on physical storage devices and computers for their execution, by software alone, by hardware alone, or a combination thereof. For example, some or all of the functions contained in the processing unit 11 can be implemented as hardware. Aspects where a particular function is implemented by hardware include aspects where the function is implemented using one or more integrated circuits (ICs) and the like. The processing unit 11 can also be implemented using an MPU, a GPU, or a DFP (data flow processor) instead of a CPU.The processing unit 11 can also be implemented by combining several types of arithmetic processing devices, such as a CPU, an MPU, and a GPU. The processing unit 11 can also be implemented as a system-on-a-chip (SoC). Furthermore, the various types of processing units can also be implemented using FPGAs (field-programmable gate arrays) or ASICs (application-specific integrated circuits). The different program types can be stored on non-temporary physical storage media. Various storage media can be used for the programs, such as HDDs (hard disk drives), SSDs (solid-state drives), flash memory, SD (secure digital) cards, and the like.

[0131] The scope of this disclosure includes not only the aforementioned image-generating device, but also various aspects, such as systems that incorporate this image-generating device as a component. For example, the scope of this disclosure also includes programs that cause a computer to function as an image-generating device, non-temporary physical storage media such as semiconductor memory that store these programs, and other aspects.< / anmerkungen> < / einleitung>

Claims

[1] Image generation device for a vehicle, comprising: an image reference unit (F1) configured to receive multiple camera images obtained from multiple cameras configured to photograph the vehicle's surroundings; an image synthesis unit (F7) configured to project data from the majority of camera images onto a virtual projection surface corresponding to the vehicle's environment and to generate a synthetic image showing the vehicle's environment as viewed from a virtual viewpoint, using the data projected onto the projection surface; and a driving environment determination unit (F62) configured to determine, based on a signal from another on-board device of the vehicle, whether a driving environment of the vehicle is terrain or on a road, wherein the image synthesis unit (F7) is configured to change the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment determination unit (F62) determines that the driving environment is terrain, the image synthesis unit (F7) is configured to generate the synthetic image by using a road projection surface with a predefined shape as the projection surface when the driving environment determination unit (F62) determines that the driving environment is on a road, and The image synthesis unit (F7) is configured to produce the synthetic image by using a terrain projection surface with a shape different from the shape of the road projection surface when the driving environment determination unit (F62) determines that the driving environment is terrain. [2] Image generation device for a vehicle, comprising: an image reference unit (F1) configured to receive multiple camera images obtained from multiple cameras configured to photograph the vehicle's surroundings; an image synthesis unit (F7) configured to project data from the majority of camera images onto a virtual projection surface corresponding to the vehicle's environment and to generate a synthetic image showing the vehicle's environment as viewed from a virtual viewpoint, using the data projected onto the projection surface; a driving environment determination unit (F62) configured to determine, based on a signal from another on-board device of the vehicle, whether a driving environment of the vehicle is terrain or on a road, wherein the image synthesis unit (F7) is configured to change the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment determination unit (F62) determines that the driving environment is terrain; and an image recognition unit (F5) configured to analyze the camera images in order to detect a predefined captured object, wherein the driving environment determination unit (F62) is configured to determine that the driving environment is terrain based on at least one of the following detections: detection of multiple stones by the image recognition unit (F5), detection of a missing lane marking by the image recognition unit (F5), and detection of a missing road edge by the image recognition unit (F5). [3] Image generation device for a vehicle, comprising: an image reference unit (F1) configured to receive multiple camera images obtained from multiple cameras configured to photograph the vehicle's surroundings; an image synthesis unit (F7) configured to project data from the majority of camera images onto a virtual projection surface corresponding to the vehicle's environment and to generate a synthetic image showing the vehicle's environment as viewed from a virtual viewpoint, using the data projected onto the projection surface; and a driving environment determination unit (F62) configured to determine, based on a signal from another on-board device of the vehicle, whether a driving environment of the vehicle is terrain or on a road, wherein the image synthesis unit (F7) is configured to change the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment determination unit (F62) determines that the driving environment is terrain, the image synthesis unit (F7) is configured to selectively use a plurality of projection surfaces, each with different shapes, as the projection surface for a terrain, the driving environment determination unit (F62) is configured to determine a terrain type based on the signal from the on-board device when it is determined that the driving environment is terrain, and the image synthesis unit (F7) is configured to change the projection surface used to generate the synthetic image according to the terrain type determined by the driving environment determination unit (F62). [4] Image generation device for a vehicle, comprising: an image reference unit (F1) configured to receive multiple camera images obtained from multiple cameras configured to photograph the vehicle's surroundings; an image synthesis unit (F7) configured to project data from the majority of camera images onto a virtual projection surface corresponding to the vehicle's environment and to generate a synthetic image showing the vehicle's environment as viewed from a virtual viewpoint, using the data projected onto the projection surface; and a driving environment determination unit (F62) configured to determine, based on a signal from another on-board device of the vehicle, whether a driving environment of the vehicle is terrain or on a road, wherein the image synthesis unit (F7) is configured to change the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment determination unit (F62) determines that the driving environment is terrain, and the image generation device is configured to cause a display device to display a notification image (NP) indicating that an image for a terrain is being displayed when the image synthesis unit (F7) generates the synthetic image by using the projection surface for a terrain. [5] Image generating device according to claim 1, wherein the road projection surface has a road flat surface area (R1) with a flat surface shape around a vehicle area (R0), which is an area in which the vehicle is located, and the terrain projection surface has a terrain-curved surface area (R2a) with a curved surface shape, which is connected to an edge section of the vehicle area (R0). [6] Image generating device according to claim 1, wherein the road projection surface has a road flat surface area (R1) with a flat surface shape around a vehicle area (R0), which is an area where the vehicle is located, and a road curved surface area (R2) with a curved surface shape outside the road flat surface area (R1), the terrain projection surface has a terrain flat surface area (R1a) with a flat surface shape around the vehicle area (R0) and a terrain curved surface area (R2a) with a curved surface shape outside the terrain flat surface area (R1a), and The terrain flat surface area (R1a) is smaller than the road flat surface area (R1). [7] Image generating device according to claim 6, wherein the road flat surface area (R1) extends from the vehicle area (R0) to a position with a specified minimum flat surface distance (Dmin) or more in forward, backward, left and right directions, and In the terrain projection area, the terrain flat surface area (R1a), which is formed between the vehicle area (R0) and the terrain curved surface area (R2a), has a width that is smaller than the minimum flat surface distance (Dmin). [8] Image generating device according to claim 1, wherein the road projection surface has a road-curved surface area (R2) with a downward convex curved surface shape outside a vehicle area (R0), which is an area in which the vehicle is located, the terrain projection surface has a terrain-curved surface area (R2a) with a downwardly convex curved surface shape outside the vehicle area (R0), and The terrain curved surface area (R2a) is closer to the vehicle area (R0) than the road curved surface area (R2). [9] Image generating device according to any one of claims 6 to 8, wherein The terrain-curved surface area (R2a) and the road-curved surface area (R2) each have downwardly convex curved surface shapes, and The terrain curved surface area (R2a) has a slope that is greater than the slope of the road curved surface area (R2). [10] Image generating device according to one of claims 1, 5, 6, 7, 8 or 9, wherein the image generation device is configured to be connected to an obstacle sensor (8) which is configured to detect a stereoscopic object present in the vicinity of the vehicle, the image generation device further comprises: an obstacle information reference unit (F4) configured to receive a signal indicating a result of the detection by the obstacle sensor (8), wherein the driving environment determination unit (F62) is configured to determine that the driving environment is terrain, based on the detection of a plurality of stereoscopic objects in the vicinity of the vehicle by the obstacle sensor (8). [11] Image generating device according to one of claims 1, 2, 5, 6, 7, 8, 9 or 10, wherein the vehicle has an off-road mode as a driving mode, which is a mode for driving off-road, whereby The driving environment determination unit (F62) is configured to determine whether the driving environment is off-road, based on a signal from an on-board input device of the vehicle that is configured to switch the driving mode to off-road mode. [12] Image generation method for producing an image to assist in driving a vehicle, comprising: Obtaining a plurality of camera images obtained from a plurality of cameras configured to photograph an environment of the vehicle (S1); Projecting data from the majority of camera images onto a virtual projection surface corresponding to the vehicle's environment and generating a synthetic image showing the vehicle's environment as viewed from a virtual viewpoint by using the data projected onto the projection surface (S8); Determine whether the vehicle's driving environment is terrain or a road, based on a signal from another on-board device of the vehicle (S3); Changing the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment is a road (S5, S6); Generating the synthetic image by using a road projection surface with a predetermined shape as the projection surface, when it is determined that the driving environment is on a road; and Generating the synthetic image by using a terrain projection surface with a shape different from the shape of the road projection surface, when it is determined that the driving environment is terrain. [13] Image generation method for producing an image to assist in driving a vehicle, comprising: Obtaining a plurality of camera images obtained from a plurality of cameras configured to photograph an environment of the vehicle (S1); Projecting data from the majority of camera images onto a virtual projection surface corresponding to the vehicle's environment and generating a synthetic image showing the vehicle's environment as viewed from a virtual viewpoint by using the data projected onto the projection surface (S8); Determine whether the vehicle's driving environment is terrain or a road, based on a signal from another on-board device of the vehicle (S3); Changing the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment is a road (S5, S6); Analyzing camera images to detect a predefined object; and Determine that the driving environment is terrain based on at least one of the detections, detection of a plurality of stones, detection of a missing road marking, and detection of a missing road edge. [14] Image generation method for producing an image to assist in driving a vehicle, comprising: Obtaining a plurality of camera images obtained from a plurality of cameras configured to photograph an environment of the vehicle (S1); Projecting data from the majority of camera images onto a virtual projection surface corresponding to the vehicle's environment and generating a synthetic image showing the vehicle's environment as viewed from a virtual viewpoint by using the data projected onto the projection surface (S8); Determine whether the vehicle's driving environment is terrain or a road, based on a signal from another on-board device of the vehicle (S3); Changing the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment is a road (S5, S6); Use as the projection surface for a terrain selectively a plurality of projection surfaces, each having different shapes, Determining a terrain type based on the signal from the on-board device when it is determined that the driving environment is terrain, and Changing the projection surface used to generate the synthetic image according to the terrain type that is determined. [15] Image generation method for producing an image to assist in driving a vehicle, comprising: Obtaining a plurality of camera images obtained from a plurality of cameras configured to photograph an environment of the vehicle (S1); Projecting data from the majority of camera images onto a virtual projection surface corresponding to the vehicle's environment and generating a synthetic image showing the vehicle's environment as viewed from a virtual viewpoint by using the data projected onto the projection surface (S8); Determine whether the vehicle's driving environment is terrain or a road, based on a signal from another on-board device of the vehicle (S3); Changing the shape of the projection surface used to generate the synthetic image, depending on whether the driving environment is a road (S5, S6); and Causing a display device to display a notification image (NP) indicating that an image for a terrain is being displayed when the synthetic image is generated by using the projection surface for a terrain.

Images