Image processing system, image processing device, image processing method, and image processing program

Abstract

A processor of this image processing system is configured to execute acquisition of a plurality of captured images which are captured by exterior cameras and have at least partially overlapping imaging range portions. The processor is configured to execute generation of a composite image in which a plurality of viewpoint images, from specific virtual viewpoints, that resulted from conversion from the captured images are combined. Generation of the composite image includes generation of a composite image in which: a region of a road surface is curved such that the horizon has an upwardly convex shape; viewpoint images are combined with each other such that image-captured road surfaces overlap each other at the same actual road surface in at least a part of overlapping imaging range portions of the viewpoint images; and a three-dimensional object rendered across the horizon in the viewpoint images is curved smoothly across the horizon.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Patent Application No. 2022-98442 filed in Japan on June 17, 2022, and the contents of the original application are incorporated by reference in their entirety. [Technical Field]

[0002] The present disclosure relates to an image processing technique for processing images captured by multiple external cameras mounted on a host mobile body. [Background technology]

[0003] Patent Document 1 discloses a technology for generating a panoramic image from a virtual viewpoint by combining camera images acquired by multiple cameras installed around a vehicle. This technology identifies the position of each pixel in the image below the horizon, determined from the camera installation position and angle, on the bottom surface of a virtual projection plane, which is a hemisphere with a flat bottom. It also identifies the position of each pixel in the image above the horizon on the hemispherical surface of the virtual projection plane. Furthermore, it identifies the positions of each camera's pixels identified on the virtual projection plane on a stereoscopic projection plane with a three-dimensional shape centered on the vehicle's position. Furthermore, it identifies positions on a display image frame corresponding to the positions of each camera's pixels identified on the stereoscopic projection plane based on a predetermined viewpoint, and plots the pixel values ​​of the corresponding camera at each identified position. In this way, the technology of Patent Document 1 reduces the effect of parallax between camera images by using the virtual projection plane. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5500255 Summary of the Invention

[0005] In the technology of Patent Document 1, the virtual projection plane is defined by a flat bottom surface and a hemispherical surface. In this case, the image appears to bend at the horizon position where the hemispherical surface rises from the bottom surface.

[0006] An object of the present disclosure is to provide an image processing system capable of avoiding image distortion while reducing the influence of parallax between images. Another object of the present disclosure is to provide an image processing device capable of avoiding image distortion while reducing the influence of parallax between images. Yet another object of the present disclosure is to provide an image processing method capable of avoiding image distortion while reducing the influence of parallax between images. Yet another object of the present disclosure is to provide an image processing program capable of avoiding image distortion while reducing the influence of parallax between images.

[0007] The technical means of the present disclosure for solving the problems will be described below. Note that the reference numerals in parentheses in the claims indicate the correspondence with the specific means described in the embodiments described later in detail, and do not limit the technical scope of the present disclosure.

[0008] A first aspect of the present disclosure is an image processing system having a processor and processing images captured by a plurality of external cameras mounted on a host mobile object, The processor Acquiring a plurality of captured images captured by external cameras whose imaging ranges at least partially overlap; An outer projection surface onto which the captured image is projected is defined, the outer projection surface being defined to include a plane and a rising surface that rises smoothly from the plane, and an inner projection surface onto which the captured image projected onto the outer projection surface is further projected, the inner projection surface being defined as a three-dimensional curved surface shape that rises inward from the outer projection surface and that increases in gradient as it moves away from the host moving body; Projected onto the inner projection surface generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image; configured to run Generating a composite image includes: The road surface area is curved so that the horizon has an upward convex shape, and the viewpoint images are synthesized so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging ranges of the viewpoint images, and a composite image is generated in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they also cross the horizon.

[0009] A second aspect of the present disclosure is an image processing device having a processor and processing images captured by a plurality of external cameras mounted on a host mobile body, The processor Acquiring a plurality of captured images captured by external cameras whose imaging ranges at least partially overlap; An outer projection surface onto which the captured image is projected is defined, the outer projection surface being defined to include a plane and a rising surface that rises smoothly from the plane, and an inner projection surface onto which the captured image projected onto the outer projection surface is further projected, the inner projection surface being defined as a three-dimensional curved surface shape that rises inward from the outer projection surface and that increases in gradient as it moves away from the host moving body; Projected onto the inner projection surface generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image; configured to run Generating a composite image includes: The road surface area is curved so that the horizon has an upward convex shape, and the viewpoint images are synthesized so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging ranges of the viewpoint images, and a composite image is generated in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they also cross the horizon.

[0010] A third aspect of the present disclosure is an image processing method executed by a processor to process images captured by a plurality of external cameras mounted on a host mobile object, the method comprising: Acquiring a plurality of captured images captured by an external camera; Acquiring a plurality of captured images captured by external cameras whose imaging ranges at least partially overlap; An outer projection surface onto which the captured image is projected is defined, the outer projection surface being defined to include a plane and a rising surface that rises smoothly from the plane, and an inner projection surface onto which the captured image projected onto the outer projection surface is further projected, the inner projection surface being defined as a three-dimensional curved surface shape that rises inward from the outer projection surface and that increases in gradient as it moves away from the host moving body; Projected onto the inner projection surface generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image; Including, Generating a composite image includes: The road surface area is curved so that the horizon has an upward convex shape, and the viewpoint images are synthesized so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging ranges of the viewpoint images, and a composite image is generated in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they also cross the horizon.

[0011] A fourth aspect of the present disclosure is an image processing program stored in a storage medium for processing images captured by a plurality of external cameras mounted on a host mobile body, the image processing program including instructions to be executed by a processor, the image processing program including: The command is, Acquiring a plurality of captured images captured by external cameras whose imaging ranges at least partially overlap; An outer projection surface onto which the captured image is projected is defined, the outer projection surface being defined to include a plane and a rising surface that rises smoothly from the plane, and an inner projection surface onto which the captured image projected onto the outer projection surface is further projected, the inner projection surface being defined as a three-dimensional curved surface shape that rises inward from the outer projection surface and that increases in gradient as it moves away from the host moving body; Projected onto the inner projection surface generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image; configured to run Generating a composite image includes: The road surface area is curved so that the horizon has an upward convex shape, and the viewpoint images are synthesized so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging ranges of the viewpoint images, and a composite image is generated in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they also cross the horizon.

[0012] According to these first to fourth aspects, by combining the outer projection surface and the inner projection surface to synthesize an image, the road surface is curved, three-dimensional objects are smoothly curved at the horizon position, and a synthetic image is generated in which the road surface is represented as a single image inside the outer projection surface. That is, the influence of parallax between images is reduced by projection onto the outer projection surface, so the road surface is represented as a single image inside the outer projection surface. And, because the outer projection surface rises smoothly, bending of the image can also be avoided. Therefore, bending of the image can be avoided while reducing the influence of parallax between images. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing the overall configuration of a first embodiment. [Figure 2] FIG. 1 is a block diagram showing a functional configuration of an image processing system according to a first embodiment. [Figure 3] FIG. 2 is a diagram showing the positional relationship between an outer projection surface and an inner projection surface. [Figure 4] 4 is a flowchart showing an image processing flow according to the first embodiment. [Figure 5] 10A and 10B are diagrams comparing a composite image according to the first embodiment with a composite image according to a reference example. [Figure 6] 10A and 10B are diagrams comparing a composite image according to the first embodiment with a composite image according to a reference example. [Figure 7] 10A and 10B are diagrams for explaining differences in composite images depending on the position of the outer projection surface. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. Note that corresponding components in each embodiment are designated by the same reference numerals, and redundant description may be omitted. Furthermore, when only a portion of the configuration is described in each embodiment, the configuration of another previously described embodiment may be applied to the remaining portions of the configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments may be partially combined together even if not explicitly stated, provided that there is no particular problem with the combination.

[0015] (First embodiment) The image processing system 100 of the first embodiment shown in Fig. 1 processes images captured by an external camera 11 of a host vehicle A shown in Fig. 3 as a host moving body. From a viewpoint centered on the host vehicle A, the host vehicle A can also be said to be an ego-vehicle. The host vehicle A is a moving body, such as an automobile, that can travel on a road with an occupant on board.

[0016] The host vehicle A is provided with an autonomous driving mode that is classified into levels according to the degree of manual intervention by the occupant in the dynamic driving task. The autonomous driving mode may be realized by autonomous driving control, such as conditional driving automation, high driving automation, or full driving automation, in which the system performs all dynamic driving tasks when activated. The autonomous driving mode may also be realized by advanced driving assistance control, such as driving assistance or partial driving automation, in which the occupant performs some or all of the dynamic driving tasks. The autonomous driving mode may be realized by either autonomous driving control or advanced driving assistance control, or by a combination of these, or by switching between them.

[0017] The host vehicle A is equipped with a camera system 10, an internal sensor system 20, and a display system 30, as shown in Figures 1 and 2. The camera system 10 acquires camera information about the external and internal worlds of the host vehicle A that can be used by an image processing system 100. The camera system 10 is configured to include multiple external cameras 11.

[0018] The external camera 11 captures an image of a predetermined range of the external world of the host vehicle A to acquire image data of the external world. The external camera 11 includes, for example, a light receiving unit and a control unit. The light receiving unit includes a light receiving lens and a light receiving element. The light receiving unit collects incident light from the imaging range using, for example, the light receiving lens and directs the light to a light receiving element such as a CCD sensor or a CMOS sensor. The light receiving element has a plurality of light receiving pixels arranged in a two-dimensional array. The control unit controls the light receiving unit. The control unit is mainly composed of a broad processor such as a microcomputer or FPGA. The control unit realizes an imaging function. The imaging function is a function of capturing the above-mentioned color image. The control unit reads out a voltage value based on the incident light received by each light receiving pixel using, for example, a global shutter system, at a timing based on an operating clock of a clock oscillator provided in the external camera 11, and senses and measures the intensity of the incident light. The control unit can generate image data in which the intensity of incident light is associated with two-dimensional coordinates on an image plane corresponding to the imaging range. Such image data is sequentially output to the image processing system 100.

[0019] The above-described external cameras 11 are installed so that the imaging ranges of at least adjacent external cameras 11 at least partially overlap each other.

[0020] The host vehicle A may be equipped with an external sensor other than the external camera 11 that detects targets present in the external world of the host vehicle A. The external sensor other than the external camera 11 is at least one of, for example, LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), radar, sonar, etc.

[0021] The internal sensor system 20 acquires internal information as sensor information from the internal world, which is the internal environment of the host vehicle A. The internal sensor system 20 detects specific physical quantities of motion in the internal world of the host vehicle A. The internal sensor system 20 is at least one type of sensor, such as a driving speed sensor, an acceleration sensor, or a gyro sensor.

[0022] The display system 30 presents visual information to the occupants of the host vehicle A. The display system 30 is at least one of a head-up display (HUD), a multi-function display (MFD), a combination meter, a navigation unit, a light-emitting unit, and the like.

[0023] The image processing system 100 is connected to a camera system 10, an internal sensor system 20, and a display system 30 via at least one of, for example, a LAN (Local Area Network) line, a wire harness, an internal bus, or a wireless communication line. The image processing system 100 is configured to include at least one dedicated computer.

[0024] The dedicated computer constituting the image processing system 100 may be an HCU (Human Machine Interface Control Unit (HMI)) that controls the presentation of information on the display system 30 of the host vehicle A. The dedicated computer constituting the image processing system 100 may be a driving control ECU (Electronic Control Unit) that controls the driving of the host vehicle A. The dedicated computer constituting the image processing system 100 may be a navigation ECU that navigates the driving route of the host vehicle A. The dedicated computer constituting the image processing system 100 may be a locator ECU that estimates the self-state quantity of the host vehicle A. The dedicated computer constituting the image processing system 100 may be an actuator ECU that controls the driving actuator of the host vehicle A. The dedicated computer constituting the image processing system 100 may be a computer other than the host vehicle A that constitutes, for example, an external center or mobile terminal capable of communicating with the host vehicle A.

[0025] The dedicated computer constituting the image processing system 100 may be an integrated ECU (Electronic Control Unit) that integrates the driving control of the host vehicle A. The dedicated computer constituting the image processing system 100 may be a determination ECU that determines a driving task in the driving control of the host vehicle A. The dedicated computer constituting the image processing system 100 may be a monitoring ECU that monitors the driving control of the host vehicle A. The dedicated computer constituting the image processing system 100 may be an evaluation ECU that evaluates the driving control of the host vehicle A.

[0026] The dedicated computer constituting the image processing system 100 has at least one memory 101 and one processor 102. The memory 101 is at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores computer-readable programs, data, and the like. Here, "storage" may refer to accumulation in which data is retained even when the host vehicle A is turned off, or may refer to temporary storage in which data is erased when the host vehicle A is turned off. The processor 102 includes at least one type of core, such as a central processing unit (CPU), a graphics processing unit (GPU), a reduced instruction set computer (RISC)-CPU, a data flow processor (DFP), or a graph streaming processor (GSP).

[0027] In the image processing system 100, the processor 102 executes a plurality of instructions included in an image processing program stored in the memory 101 in order to process images captured by the external camera 11 in the host vehicle A. As a result, the image processing system 100 constructs a plurality of functional blocks for processing images captured by the external camera 11 in the host vehicle A. The plurality of functional blocks constructed in the image processing system 100 include an image acquisition block 110, a projection block 120, a generation block 130, and a display block 140, as shown in FIG.

[0028] The image processing method in which the image processing system 100 processes the image captured by the external camera 11 in the host vehicle A by cooperation of these blocks 110, 120, 130, and 140 is executed according to the image processing flow shown in FIG.

[0029] This image processing flow is repeatedly executed during startup of the host vehicle A in an external display scene in which image data captured by the external camera 11 is displayed to the occupant. The external display scene includes, for example, a parking scene in which the host vehicle A is parked in a parking space. The external display scene may be determined based on sensor information or the like in the host vehicle A or an external center. Alternatively, the external display scene may be determined as a trigger when the occupant operates an in-vehicle device. Note that each "S" in this image processing flow represents multiple steps executed by multiple commands included in the image processing program.

[0030] First, in S10, the image acquisition block 110 acquires the images captured by the external cameras 11 in the camera system 10.

[0031] Next, in S20, the projection block 120 defines a virtual viewpoint Pv for creating a composite image, which will be described later (see FIG. 3). Pv is the viewpoint that determines the appearance of the composite image. Pv The composite image is a composite of a plurality of viewpoint images converted into images seen from different angles.

[0032] In the next step S30, the projection block 120 defines an inner projection surface Pi and an outer projection surface Po, as shown in Fig. 3. The projection block 120 defines the inner projection surface Pi as a three-dimensional curved surface that is centered on the position of the host vehicle A, can approximate the road plane around the host vehicle A, and whose slope increases with increasing distance from the host vehicle A. The projection block 120 defines the shape of the inner projection surface Pi in the form of a mathematical formula or a polyhedron shape.

[0033] The projection block 120 defines the outer projection plane Po as a shape having a plane Po1 that is centered on the position of the host vehicle A and can approximate the road plane around the host vehicle A, and a rising plane Po2 that rises from the center of the outer projection plane Po at a predetermined rising start position R.

[0034] The plane Po1 on the outer projection surface Po is defined as a plane that extends further outward than the inner projection surface Pi. In other words, the rising start position R of the outer projection surface Po is set outside the inner projection surface Pi. The distance from the center position of the outer projection surface Po to the rising start position R is set to a large value that allows the distance between the external cameras 11 to be considered an error, for example.

[0035] The rising surface Po2 on the outer projection plane Po is defined as a curved surface that rises smoothly from the plane Po1. The rising surface is defined as a three-dimensional curved surface whose slope increases with increasing distance from the host vehicle A. For example, the cross section of the rising surface has a shape defined by a downwardly convex quadratic function with the rising start position R as the vertex. This cross-sectional shape may be a parabolic shape or a circular arc shape.

[0036] Then, in S40, the projection block 120 uses the outer projection surface Po to execute a projection process of the acquired image data onto the inner projection surface Pi. Here, the projection process corresponds to a process of determining the coordinate position on the inner projection surface Pi corresponding to each pixel in the image data. In this way, the projection block 120 calculates the correspondence relationship between each pixel and the position on the inner projection surface Pi when viewed from the virtual viewpoint Pv.

[0037] In projecting image data onto the inner projection surface Pi, the projection block 120 first projects the image data onto the outer projection surface Po. For example, the projection block 120 calculates a coordinate position in the vehicle coordinate system on the outer projection surface Po that corresponds to the coordinate position of each pixel in the image coordinate system based on the installation position and installation attitude of the external camera 11 and the position and shape information of the outer projection surface Po that are stored in advance. Then, the projection block 120 further projects the image data projected onto the outer projection surface Po onto the inner projection surface Pi. For example, the projection block 120 calculates a coordinate position on the inner projection surface Pi that corresponds to the outer corresponding position when viewed from the center point Pr of the inner projection surface Pi (inner corresponding position) based on the position of the center point Pr and the position and shape information of the inner projection surface Pi. The projection block 120 converts this inner corresponding position into a coordinate position as viewed from the virtual viewpoint Pv, thereby acquiring position information of each pixel as viewed from the virtual viewpoint Pv. In this way, the projection block 120 defines the correspondence between the image and the inner projection plane Pi.

[0038] After the above processing of S40, the flow proceeds to S50. In S50, the generation block 130 synthesizes the image data from each external camera 11 based on the defined correspondence relationship to generate a composite image viewed from the virtual viewpoint Pv. For image data that overlaps on the inner projection plane Pi, the generation block 130 sets a predetermined transmittance for the overlapping area and synthesizes them.

[0039] Then, in S60, the display block 140 displays the generated composite image on the display system 30. The display block 140 may further superimpose other display objects on the composite image. For example, the display block 140 may superimpose display objects representing parking spaces, a route to a target position, guide lines, obstacle positions, etc.

[0040] The composite image Ic generated by the first embodiment described above will be described below with reference to FIGS. 5 and 6, while being compared with a reference image Ir, which is a composite image generated by a reference example.

[0041] 5, the image at the top of the page is a reference image Ir generated by a reference example in which an image is projected only onto the inner projection surface without using the outer projection surface, and the image at the bottom of the page is a composite image Ic generated by the first embodiment.

[0042] Each of the images Ir and Ic is a composite image of viewpoint images Iv1, Iv2, and Iv3 generated from images captured by three external cameras 11 installed in the host vehicle A. The installation positions of the three external cameras 11 are assumed to be the same for the reference image Ir and the composite image Ic.

[0043] The viewpoint image Iv1 is the portion from the left edge of the paper to the dotted line. The captured ground surface S1 and the captured three-dimensional object O1 are captured in the viewpoint image Iv1. The viewpoint image Iv2 is the portion between the dashed lines. The captured ground surface S2 and the captured three-dimensional object O2 are captured in the viewpoint image Iv2. The viewpoint image Iv3 is the portion from the long dashed line to the right edge of the paper. The captured ground surface S3 is captured in the viewpoint image Iv3. The viewpoint images Iv1 and Iv2 are superimposed and combined at the superimposition portion SA1. The viewpoint images Iv2 and Iv3 are superimposed and combined at the superimposition portion SA2. In the example of FIG. 5, the captured three-dimensional objects O1 and O2 are captured as three-dimensional objects located closer to the position of the inner projection plane Pi. Note that this three-dimensional object is a columnar three-dimensional object extending from the ground.

[0044] As shown in Figure 5, compared to a reference image Ir synthesized from viewpoint images generated using only the inner projection plane Pi, a composite image Ic synthesized from viewpoint images generated using the inner projection plane Pi and the outer projection plane Po shows a greater degree of curvature of the captured ground surfaces S1, S2, and S3. Also, the overlapping portions SA of the viewpoint images show a smaller degree of misalignment between the captured ground surfaces. For example, the bases of the captured three-dimensional objects O1 and O2 on the captured ground surface are misaligned in the reference image Ir, but are substantially aligned in the composite image Ic.

[0045] 6, the image at the top of the page is a reference image Ir generated by a reference example in which an image is projected using a hemispherical outer projection surface and an inner projection surface. The composite image at the bottom of the page is a composite image Ic generated by the first embodiment. The external camera 11 that captured the captured images that served as the basis for each image is assumed to be the same as that shown in FIG.

[0046] 6, the reference image Ir shows the captured three-dimensional object straddling the horizon line, curved across the horizon line, while the composite image Ic shows the outer projection surface Po rising smoothly from a flat surface, so the composite image Ic shows the object rising smoothly across the horizon line, curved across the horizon line.

[0047] As described above, in the composite image Ic of this embodiment, the road surface area is curved so that the horizon has an upward convex shape, the viewpoint images are combined so that the imaged road surface overlaps at the same position on the actual road surface in at least part of the overlapping area of ​​the imaging ranges between the viewpoint images, and three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they cross the horizon.

[0048] Fig. 7 is a diagram showing changes in an image when the position of the outer projection surface Po is changed in the first embodiment. In Fig. 7, the image at the top of the page has the rising start position R set at a point 5000 mm from the center position, and the image at the bottom of the page has the rising start position R set at a point 15000 mm from the center position. Also, in Fig. 7, when the quadratic function representing the cross section of the rising surface of the outer projection surface Po is expressed as y = αx^2, the coefficient α is set to 10^-6. According to Fig. 7, the field of view in the composite image Ic becomes wider as the outer projection surface Po becomes farther away.

[0049] According to the first embodiment described above, by combining the outer projection surface and the inner projection surface to synthesize an image, the road surface is curved, three-dimensional objects are smoothly curved at the horizon position, and a synthetic image is generated in which the road surface is represented as a single image inside the outer projection surface. That is, projection onto the outer projection surface reduces the influence of parallax between images, so the road surface is represented as a single image inside the outer projection surface. Furthermore, because the outer projection surface rises smoothly, image bending can also be avoided. Therefore, it is possible to avoid image bending while reducing the influence of parallax between images.

[0050] Second Embodiment The second embodiment is a modification of the first embodiment.

[0051] In the second embodiment, the projection block 120 changes at least one of the virtual viewpoint Pv and the rising start position R of the outer projection plane Po based on a specific condition.

[0052] For example, the projection block 120 performs the change process using the speed of the host vehicle A as a condition. Specifically, the projection block 120 moves the virtual viewpoint Pv upward as the speed of the host vehicle A increases. In addition, the projection block 120 moves the rising start position of the outer projection surface farther away from the host vehicle A as the speed of the host vehicle A increases.

[0053] Alternatively, the projection block 120 performs the change process using the position of an obstacle around the host vehicle A as a condition. Specifically, the projection block 120 sets the rising start position R to be farther away than the position of the obstacle. For example, the projection block 120 may compare a preset initial rising start position R with the position of the obstacle, and change the rising start position R if the position of the obstacle is farther away than the initial rising start position. Alternatively, the projection block 120 may determine the initial rising start position R based on the position of the obstacle.

[0054] In this case, the projection block 120 maintains the virtual viewpoint Pv at a preset position, or the projection block 120 may set the virtual viewpoint Pv at a position where it is determined that the occupant can easily visually recognize the distance between the obstacle and the host vehicle A.

[0055] Alternatively, the projection block 120 performs the change process using the display position of the display object superimposed on the composite image as a condition. Specifically, the projection block 120 sets the rise start position R farther away than the superimposition position of the display object superimposed on the road surface of the composite image. For example, the projection block 120 may compare a preset initial rise start position R with the superimposition position, and change the rise start position R if the superimposition position is farther away than the initial rise start position. Alternatively, the projection block 120 may determine the initial rise start position R based on the superimposition position.

[0056] In this case, the projection block 120 maintains the virtual viewpoint Pv at a preset position, or the projection block 120 may set the virtual viewpoint Pv at a position where it is determined that the display object is easily visible.

[0057] (Other embodiments) Although multiple embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0058] In a modified example, the dedicated computer constituting image processing system 100 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of the following: an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SOC), a programmable gate array (PGA), and a complex programmable logic device (CPLD). Such a digital circuit may also have a memory that stores a program.

[0059] In a modified example, the host mobile body to which image processing system 100 is applied may be, for example, an autonomous robot capable of autonomously or remotely traveling to transport luggage or collect information, etc. In addition to the forms described so far, the above-described embodiments and modified examples may be implemented in the form of a processing circuit (e.g., a processing ECU, etc.) or a semiconductor device (e.g., a semiconductor chip, etc.) as a control device that is configured to be mountable on a host mobile body and has at least one processor 102 and one memory 101.

[0060] (Disclosed technical idea) This specification discloses multiple technical ideas described in the following paragraphs, and also discloses multiple combined technical ideas indicated by alternatively citing the preceding technical ideas in the subsequent technical ideas.

[0061] (Technical thought 1) An image processing system having a processor (102) for processing images captured by a plurality of external cameras (11) mounted on a host moving body (A), The processor: acquiring a plurality of captured images captured by the external cameras whose imaging ranges at least partially overlap; generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image; configured to run generating the composite image includes: An image processing system that includes: curving a road surface area so that the horizon has an upward convex shape; synthesizing each of the viewpoint images so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging range in the viewpoint images; and generating a synthesized image in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they cross the horizon.

[0062] (Technical thought 2) The image processing system described in Technical Idea 1 is configured to further define an outer projection surface (Po) onto which the captured image is projected, which is defined to include a plane and a rising surface that rises smoothly from the plane, and an inner projection surface (Pi) onto which the captured image projected onto the outer projection surface is further projected, which is defined as a shape that rises further inward than the outer projection surface.

[0063] (Technical Thought 3) Defining the outer projection surface and the inner projection surface includes: defining the virtual viewpoint, which is a viewpoint in the synthetic image; defining a rising start position (R) of the rising surface from the plane, which is correlated with the position of the virtual viewpoint; The image processing system according to Technical Idea 2 includes:

[0064] (Technical Thought 4) Defining the outer projection surface and the inner projection surface includes: The image processing system according to Technical Idea 2 or Technical Idea 3 includes defining the virtual viewpoint correlated with the moving speed of the host moving object.

[0065] (Technical Thought 5) Defining the outer projection surface and the inner projection surface includes: An image processing system according to any one of technical ideas 2 to 4, which includes defining a rising start position (R) of the rising surface from the plane, which correlates with the position of an obstacle shown in the captured image.

[0066] (Technical Thought 6) generating the composite image includes: further generating an object to be superimposed on the composite image; Defining the outer projection surface and the inner projection surface includes: An image processing system according to any one of technical ideas 2 to 5, which includes defining a rising start position (R) of the rising surface from the plane, which is correlated with the superimposition position of the object.

[0067] (Technical Thought 7) An image processing device having a processor (102) for processing images captured by a plurality of external cameras (11) mounted on a host moving body (A), The processor: acquiring a plurality of captured images captured by the external cameras whose imaging ranges at least partially overlap; generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image; configured to run generating the composite image includes: An image processing device that includes: curving a road surface area so that the horizon has an upward convex shape; synthesizing each of the viewpoint images so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging range in the viewpoint images; and generating a synthesized image in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they cross the horizon.

[0068] (Technical Thought 8) An image processing method executed by a processor (102) to process images captured by a plurality of external cameras (11) mounted on a host moving body (A), comprising: acquiring a plurality of the captured images captured by the external camera; acquiring a plurality of captured images captured by the external cameras whose imaging ranges at least partially overlap; generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image; Including, generating the composite image includes: An image processing method that includes curving a road surface area so that the horizon has an upward convex shape, synthesizing each of the viewpoint images so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging range in the viewpoint images, and smoothly curving three-dimensional objects depicted across the horizon in the viewpoint images to generate a synthesized image that is curved so that they straddle the horizon.

[0069] (Technical Thought 9) An image processing program stored in a storage medium (101) for processing images captured by a plurality of external cameras (11) mounted on a host moving body (A), the image processing program including instructions to be executed by a processor (102), The instruction: The instruction: acquiring a plurality of captured images captured by the external cameras whose imaging ranges at least partially overlap; generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image; configured to run generating the composite image includes: An image processing program that includes: curving a road surface area so that the horizon has an upward convex shape; synthesizing each of the viewpoint images so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging range in the viewpoint images; and generating a composite image in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they cross the horizon.

Claims

1. An image processing system having a processor (102) for processing images captured by a plurality of external cameras (11) mounted on a host mobile body (A), The processor: acquiring a plurality of captured images captured by the external cameras whose imaging ranges at least partially overlap; defining an outer projection surface (Po) onto which the captured image is projected, the outer projection surface (Po) being defined to include a plane and a rising surface that rises smoothly from the plane, and an inner projection surface (Pi) being defined as a three-dimensional curved surface shape that rises inward from the outer projection surface and that increases in gradient as it moves away from the host moving body, onto which the captured image projected onto the outer projection surface is further projected; generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image projected on the inner projection surface; configured to run generating the composite image includes: An image processing system that includes: curving a road surface area so that the horizon has an upward convex shape; synthesizing each of the viewpoint images so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging range in the viewpoint images; and generating a synthesized image in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they cross the horizon.

2. Defining the outer projection surface and the inner projection surface includes: defining the virtual viewpoint, which is a viewpoint in the synthetic image; defining a rising start position (R) of the rising surface from the plane, which is correlated to the position of the virtual viewpoint; The image processing system of claim 1 , comprising:

3. Defining the outer projection surface and the inner projection surface includes: The image processing system according to claim 2 , further comprising: defining the virtual viewpoint higher as the moving speed of the host mobile body acquired from a traveling speed sensor of the host mobile body increases.

4. Defining the outer projection surface and the inner projection surface includes: The image processing system according to claim 2 , further comprising: defining a rising start position (R) of the rising surface from the flat surface, which is correlated with a position of an obstacle shown in the captured image.

5. generating the composite image includes: further generating an object to be superimposed on the composite image; Defining the outer projection surface and the inner projection surface includes:

4. The image processing system according to claim 3, further comprising: defining a rising start position (R) of the rising surface from the plane, which is correlated with the superimposed position of the object.

6. An image processing device having a processor (102) and processing images captured by a plurality of external cameras (11) mounted on a host moving body (A), The processor: acquiring a plurality of captured images captured by the external cameras whose imaging ranges at least partially overlap; defining an outer projection surface (Po) onto which the captured image is projected, the outer projection surface (Po) being defined to include a plane and a rising surface that rises smoothly from the plane, and an inner projection surface (Pi) being defined as a three-dimensional curved surface shape that rises inward from the outer projection surface and that increases in gradient as it moves away from the host moving body, onto which the captured image projected onto the outer projection surface is further projected; generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image projected on the inner projection surface; configured to run generating the composite image includes: An image processing device that includes: curving a road surface area so that the horizon has an upward convex shape; synthesizing each of the viewpoint images so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging range in the viewpoint images; and generating a synthesized image in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they cross the horizon.

7. An image processing method executed by a processor (102) to process images captured by a plurality of external cameras (11) mounted on a host moving body (A), comprising: acquiring a plurality of the captured images captured by the external camera; acquiring a plurality of captured images captured by the external cameras whose imaging ranges at least partially overlap; defining an outer projection surface (Po) onto which the captured image is projected, the outer projection surface (Po) being defined to include a plane and a rising surface that rises smoothly from the plane, and an inner projection surface (Pi) being defined as a three-dimensional curved surface shape that rises inward from the outer projection surface and that increases in gradient as it moves away from the host moving body, onto which the captured image projected onto the outer projection surface is further projected; generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image projected on the inner projection surface; Including, generating the composite image includes: An image processing method including: curving a road surface area so that the horizon has an upward convex shape; synthesizing the viewpoint images so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging range in the viewpoint images; and generating a synthesized image in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they cross the horizon.

8. An image processing program stored in a storage medium (101) for processing images captured by a plurality of external cameras (11) mounted on a host mobile body (A), the image processing program including instructions to be executed by a processor (102), The instruction: acquiring a plurality of captured images captured by the external cameras whose imaging ranges at least partially overlap; defining an outer projection surface (Po) onto which the captured image is projected, the outer projection surface (Po) being defined to include a plane and a rising surface that rises smoothly from the plane, and an inner projection surface (Pi) being defined as a three-dimensional curved surface shape that rises inward from the outer projection surface and that increases in gradient as it moves away from the host moving body, onto which the captured image projected onto the outer projection surface is further projected; generating a composite image by combining a plurality of viewpoint images from a specific virtual viewpoint converted from the captured image projected on the inner projection surface; configured to run generating the composite image includes: An image processing program that includes: curving a road surface area so that the horizon has an upward convex shape; synthesizing each of the viewpoint images so that the imaged road surface overlaps at the same position on the actual road surface in at least a portion of the overlapping portion of the imaging range in the viewpoint images; and generating a composite image in which three-dimensional objects depicted across the horizon in the viewpoint images are smoothly curved so that they cross the horizon.

Images