Image processing system, image processing device, image processing method, and image processing program

Abstract

A processor of an image processing system is configured to acquire a plurality of captured images captured by external cameras whose imaging ranges at least partially overlap. The processor is configured to generate a composite image in which a plurality of viewpoint images, which are converted from the captured image and observed from a specific virtual viewpoint, are synthesized. The generation of the composite image includes the generation of a composite image in which the road surface region is bent so that the horizon line becomes an upward convex shape, and the viewpoint images are synthesized so that at least a portion of the overlapping portion of the imaging range of the viewpoint images overlaps each other at the same position on the actual road surface. A three-dimensional object drawn across a horizon in a viewpoint image is smoothly curved so as to cross the horizon.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on Japanese Patent Application No. 2022-98442 filed in Japan on June 17, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to an image processing technology for processing images captured by a plurality of external cameras mounted on a main moving body. Background Art

[0004] Patent document 1 discloses a technique for generating a whole-circle image observed from a virtual viewpoint by synthesizing camera images acquired by a plurality of cameras arranged around a vehicle. In this technique, on the bottom surface of a virtual projection surface of a hemisphere having a flat bottom surface, the position of each pixel of an image below the horizon obtained from each acquired camera image is determined, and on the hemispherical surface of the virtual projection surface, the position of each pixel of an image above the horizon is determined. Furthermore, on a stereoscopic projection surface having a three-dimensional shape centered on the position of the vehicle, the position of the pixel of each camera determined on the virtual projection surface is determined. Furthermore, based on a predetermined viewpoint position, the position on the image frame for display corresponding to the position of the pixel of each camera determined on the stereoscopic projection surface is determined, and the value of the pixel of the corresponding camera is depicted at each determined position. In this way, in the technique of patent document 1, the influence of parallax between camera images is reduced by the virtual projection surface.

[0005] Patent Document 1: Japanese Patent No. 5500255

[0006] In the technology of Patent Document 1, the virtual projection plane is defined by the bottom surface of the plane and the hemispherical surface. In this case, the image appears to be bent at the horizon position where the hemispherical surface rises from the bottom surface. Summary of the invention

[0007] The subject of the present disclosure is to provide an image processing system capable of reducing the influence of parallax between images and avoiding image bending. Another subject of the present disclosure is to provide an image processing device capable of reducing the influence of parallax between images and avoiding image bending. Another subject of the present disclosure is to provide an image processing method capable of reducing the influence of parallax between images and avoiding image bending. Yet another subject of the present disclosure is to provide an image processing program capable of reducing the influence of parallax between images and avoiding image bending.

[0008] The technical means of the present disclosure for solving the problems will be described below. In addition, the reference numerals in parentheses in the claims indicate the correspondence with the specific means described in the embodiments described in detail later, and do not limit the technical scope of the present disclosure.

[0009] A first aspect of the present disclosure is an image processing system having a processor and processing images captured by a plurality of external cameras mounted on a main moving body.

[0010] The processor is configured to perform the following processing:

[0011] acquiring a plurality of captured images captured by respective external cameras whose capturing ranges at least partially overlap; and

[0012] generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint,

[0013] Generating a composite image includes generating the following composite image:

[0014] The road surface area is bent so that the horizon becomes convex upward, and the viewpoint images are synthesized into at least a part of the overlapping part of the shooting range in the viewpoint images. The photographed road surfaces overlap with each other at the same position on the actual road surface, and the three-dimensional objects depicted across the horizon in the viewpoint images are smoothly bent and bent to cross the horizon.

[0015] A second aspect of the present disclosure is an image processing device having a processor and processing images captured by a plurality of external cameras mounted on a main moving body.

[0016] The processor is configured to perform the following processing:

[0017] acquiring a plurality of captured images captured by respective external cameras whose capturing ranges at least partially overlap; and

[0018] generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint,

[0019] Generating a composite image includes generating the following composite image:

[0020] The road surface area is bent so that the horizon becomes convex upward, and the viewpoint images are synthesized into at least a part of the overlapping part of the shooting range in the viewpoint images. The photographed road surfaces overlap with each other at the same position on the actual road surface, and the three-dimensional objects depicted across the horizon in the viewpoint images are smoothly bent and bent to cross the horizon.

[0021] A third aspect of the present disclosure is an image processing method executed by a processor for processing images captured by a plurality of external cameras mounted on a main mobile body, comprising:

[0022] Acquire a plurality of captured images captured by an external camera;

[0023] acquiring a plurality of captured images captured by respective external cameras whose capturing ranges at least partially overlap; and

[0024] generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint,

[0025] Generating a composite image includes generating the following composite image:

[0026] The road surface area is bent so that the horizon becomes convex upward, and the viewpoint images are synthesized into at least a part of the overlapping part of the shooting range in the viewpoint images. The photographed road surfaces overlap with each other at the same position on the actual road surface, and the three-dimensional objects depicted across the horizon in the viewpoint images are smoothly bent and bent to cross the horizon.

[0027] A fourth aspect of the present disclosure is an image processing program including instructions stored in a storage medium and executed by a processor for processing images captured by a plurality of external cameras mounted on a main moving body.

[0028] The command is structured to perform the following processing:

[0029] acquiring a plurality of captured images captured by respective external cameras whose capturing ranges at least partially overlap; and

[0030] generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint,

[0031] Generating a composite image includes generating the following composite image:

[0032] The road surface area is bent so that the horizon becomes convex upward, and the viewpoint images are synthesized into at least a part of the overlapping part of the shooting range in the viewpoint images. The photographed road surfaces overlap with each other at the same position on the actual road surface, and the three-dimensional objects depicted across the horizon in the viewpoint images are smoothly bent and bent to cross the horizon.

[0033] According to the first to fourth methods, an image is synthesized by combining the outer projection plane and the inner projection plane, thereby generating a synthesized image in which the road surface is curved, the three-dimensional object is smoothly curved at the horizon position, and the road surface is represented in a single manner on the inner side of the outer projection plane. That is, since the influence of parallax between images is reduced by projection onto the outer projection plane, the road surface is represented in a single manner on the inner side of the outer projection plane. Moreover, since the outer projection plane rises smoothly, it is also possible to avoid bending of the image. Therefore, it is possible to reduce the influence of parallax between images and avoid bending of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a block diagram showing the overall structure of the first embodiment.

[0035] Figure 2 It is a block diagram showing the functional structure of the image processing system according to the first embodiment.

[0036] Figure 3 It is a diagram showing the positional relationship between the outer projection plane and the inner projection plane.

[0037] Figure 4 This is a flowchart showing the flow of image processing in the first embodiment.

[0038] Figure 5 This is a diagram comparing the synthesized image of the first embodiment and the synthesized image of the reference example.

[0039] Figure 6 This is a diagram comparing the synthesized image of the first embodiment and the synthesized image of the reference example.

[0040] Figure 7 It is a diagram for explaining the difference of the synthesized image according to the position of the outer projection surface. DETAILED DESCRIPTION

[0041] Hereinafter, multiple embodiments of the present disclosure are described based on the accompanying drawings. In addition, there are cases where repeated descriptions are omitted by marking the corresponding constituent elements in each embodiment with the same figure mark. In addition, when only a part of the structure is described in each embodiment, the structure of other embodiments described above can be applied to the other parts of the structure. Moreover, not only the combination of structures clearly shown in the description of each embodiment, but also the structures of multiple embodiments can be partially combined with each other even if it is not clearly shown, especially if the combination does not cause any hindrance.

[0042] (First Embodiment)

[0043] Figure 1 The image processing system 100 of the first embodiment shown in the figure is used for the main moving body. Figure 3 The image captured by the external camera 11 of the host vehicle A shown in the figure is processed. The host vehicle A can also be called an ego vehicle from a viewpoint centered on the host vehicle A. The host vehicle A is a mobile object such as a car that can travel on a travel road with passengers on board.

[0044] In the host vehicle A, an automatic driving mode graded according to the degree of manual intervention of the occupant in the dynamic driving task is given. The automatic driving mode can also be achieved by autonomous driving control in which the system performs all dynamic driving tasks when operating, such as conditional driving automation, high driving automation, or full driving automation. The automatic driving mode can also be achieved by driving assistance, or high driving assistance control in which the occupant performs part or all of the dynamic driving tasks, such as partial driving automation. The automatic driving mode can also be achieved by any one of these autonomous driving controls and high driving assistance controls, or a combination or switching.

[0045] In the main vehicle A, Figure 1 , Figure 2 The camera system 10 , the interior sensor system 20 , and the display system 30 are shown. The camera system 10 acquires camera information of the exterior and interior of the host vehicle A that can be utilized by the image processing system 100 . The camera system 10 includes a plurality of exterior cameras 11 .

[0046] The external camera 11 acquires external image data by photographing a predetermined range outside the main vehicle A. The external camera 11, for example, includes a light receiving unit and a control unit. The light receiving unit includes a light receiving lens and a light receiving element. The light receiving unit, for example, uses a light receiving lens to converge incident light incident from the photographing range and causes it to be incident on a light receiving element such as a CCD sensor or a CMOS sensor. The light receiving element is in an array shape so that a plurality of light receiving pixels are arranged in a two-dimensional direction. The control unit is a unit that controls the light receiving unit. The control unit is mainly composed of a broad processor such as a microcomputer or FPGA. The control unit implements a photographing function. The photographing function is a function of photographing the above-mentioned color image. The control unit reads out a voltage value based on the incident light received by each light receiving pixel, for example, using a global shutter method, based on the timing of an action clock of a clock oscillator provided in the external camera 11, and senses and measures the intensity of the incident light. The control unit can generate image data in which the intensity of the incident light is associated with the two-dimensional coordinates on the image plane corresponding to the photographing range. Such image data is sequentially output to the image processing system 100.

[0047] The above-described external cameras 11 are provided so that the imaging ranges of at least adjacent external cameras 11 at least partially overlap.

[0048] In addition, the host vehicle A may be equipped with an external sensor other than the external camera 11 for detecting an object existing outside the host vehicle A. The external sensor other than the external camera 11 is, for example, at least one of LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), radar, and sonar.

[0049] The interior sensor system 20 acquires interior information as sensor information from the interior of the host vehicle A, which is the interior environment of the host vehicle A. The interior sensor system 20 detects a specific motion physical quantity inside the host vehicle A. The interior sensor system 20 is, for example, at least one of a travel speed sensor, an acceleration sensor, and a gyro sensor.

[0050] The display system 30 presents visual information to the occupants of the host vehicle A. The display system 30 is, for example, at least one of a HUD (Head-Up Display), an MFD (Multi-Function Display), a combination meter, a navigation unit, and a lighting unit.

[0051] The image processing system 100 is connected to the camera system 10, the internal sensor system 20, and the display system 30 via at least one of a LAN (Local Area Network) line, a harness, an internal bus, and a wireless communication line. The image processing system 100 includes at least one dedicated computer.

[0052] The dedicated computer constituting the image processing system 100 may also be an HCU (HMI (Human Machine Interface) Control Unit) that controls the information prompt in the display system 30 of the main vehicle A. The dedicated computer constituting the image processing system 100 may also be a driving control ECU (Electronic Control Unit) that controls the driving of the main vehicle A. The dedicated computer constituting the image processing system 100 may also be a navigation ECU that navigates the driving path of the main vehicle A. The dedicated computer constituting the image processing system 100 may also be a locator ECU that infers the own state quantity of the main vehicle A. The dedicated computer constituting the image processing system 100 may also be an actuator ECU that controls the driving actuator of the main vehicle A. The dedicated computer constituting the image processing system 100 may also be a computer other than the main vehicle A, which is constructed, for example, as an external center or mobile terminal that can communicate with the main vehicle A.

[0053] The dedicated computer constituting the image processing system 100 may also be an integrated ECU (Electronic Control Unit) that integrates the driving control of the host vehicle A. The dedicated computer constituting the image processing system 100 may also be a determination ECU that determines the driving task in the driving control of the host vehicle A. The dedicated computer constituting the image processing system 100 may also be a monitoring ECU that monitors the driving control of the host vehicle A. The dedicated computer constituting the image processing system 100 may also be an evaluation ECU that evaluates the driving control of the host vehicle A.

[0054] The dedicated computer constituting the image processing system 100 has at least one memory 101 and one processor 102. The memory 101 is at least one non-transitory tangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium that stores programs and data readable by the computer in a non-temporary manner. The storage mentioned here can be a storage of data that is maintained even when the host vehicle A is turned on and off, or a temporary storage of data that is deleted when the host vehicle A is turned on and off. The processor 102 includes at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a RISC (Reduced Instruction Set Computer)-CPU, a DFP (Data Flow Processor), and a GSP (Graph Streaming Processor) as a core.

[0055] In the image processing system 100, the processor 102 executes a plurality of commands included in the image processing program stored in the memory 101 for processing the image captured by the external camera 11 in the host vehicle A. Thus, the image processing system 100 constructs a plurality of functional modules for processing the image captured by the external camera 11 in the host vehicle A. Figure 2 As shown, the multiple functional modules constructed in the image processing system 100 include an image acquisition module 110 , a projection module 120 , a generation module 130 and a display module 140 .

[0056] Through the cooperation of these modules 110, 120, 130, 140, the image processing system 100 processes the image captured by the external camera 11 in the host vehicle A according to the image processing method. Figure 4 The image processing flow shown is used for execution.

[0057] This image processing flow is repeatedly executed in the external display scene in which the image data of the external camera 11 is displayed to the occupants during the startup of the main vehicle A. The external display scene includes, for example, a parking scene in which the main vehicle A is parked in a parking space. The external display scene can also be determined based on sensor information, etc. in the main vehicle A or an external center. Alternatively, the external display scene can also be determined by the occupants' operation of the in-vehicle equipment as a trigger. In addition, each "S" in this image processing flow means a plurality of steps executed according to a plurality of commands included in the image processing program.

[0058] First, in S10 , the image acquisition module 110 acquires images captured by each external camera 11 in the camera system 10 .

[0059] Next, in S20, the projection module 120 defines a virtual viewpoint Pv (see Figure 3 ). The virtual viewpoint Vp is a viewpoint that determines the appearance of the synthesized image. That is, the synthesized image is an image synthesized from a plurality of viewpoint images that convert each captured image into an image observed from the virtual viewpoint Vp.

[0060] In the following S30, if Figure 3 As shown, the projection module 120 defines an inner projection surface Pi and an outer projection surface Po. For the inner projection surface Pi, the projection module 120 defines a three-dimensional curved surface that is centered on the position of the host vehicle A and can approximate the host vehicle A to the road plane at the periphery and increases in slope as it moves away from the host vehicle A as the shape of the inner projection surface Pi. The projection module 120 defines the shape of the inner projection surface Pi in the form of a mathematical expression or a polyhedron shape.

[0061] The projection module 120 defines the outer projection surface Po as a shape having a plane Po1 and a rising surface Po2, wherein the plane Po1 is centered on the position of the main vehicle A and can approximate the main vehicle A to the road plane in the periphery, and the rising surface Po2 rises from the center of the outer projection surface Po from a specified rising start position R.

[0062] The plane Po1 in the outer projection surface Po is defined as a plane extending to the outside of the inner projection surface Pi. In other words, the rising start position R of the outer projection surface Po is set to the outside of the inner projection surface Pi. The distance from the center position of the outer projection surface Po to the rising start position R is set to a large value such that the distance between the external cameras 11 is regarded as an error.

[0063] The rising surface Po2 in the outer projection surface Po is defined as a curved surface that rises smoothly from the plane Po1. The rising surface is defined as a three-dimensional curved surface whose slope increases as it moves away from the host vehicle A. For example, the cross section of the rising surface is defined by a quadratic function that is convex downward and has the rising start position R as the vertex. In addition, the cross-sectional shape may be a parabola or an arc.

[0064] Then, in S40, the projection module 120 performs a projection process of the acquired image data onto the inner projection surface Pi using the outer projection surface Po. Here, the projection process is equivalent to a process of determining the coordinate position on the inner projection surface Pi corresponding to each pixel in the image data. Thus, the projection module 120 calculates the correspondence relationship between each pixel and which position on the inner projection surface Pi corresponds when observed from the virtual viewpoint Pv.

[0065] In the projection of the image data to the inner projection surface Pi, the projection module 120 first projects the image data to the outer projection surface Po. For example, the projection module 120 calculates the coordinate position (outer corresponding position) in the vehicle coordinate system on the outer projection surface Po corresponding to the coordinate position of each pixel in the image coordinate system based on the pre-stored setting position and setting posture of the external camera 11 and the position and shape information of the outer projection surface Po. Then, the projection module 120 further projects the image data projected to the outer projection surface Po to the inner projection surface Pi. For example, the projection module 120 calculates the coordinate position (inner corresponding position) on the inner projection surface Pi corresponding to the outer corresponding position when observed from the center point Pr of the inner projection surface Pi based on the position of the center point Pr, the position of the inner projection surface Pi and the shape information. The projection module 120 obtains the position information of each pixel observed from the virtual viewpoint Pv by converting the inner corresponding position into the coordinate position observed from the virtual viewpoint Pv. Thus, the projection module 120 defines the corresponding relationship between the image and the inner projection surface Pi.

[0066] After the above processing of S40, the flow moves to S50. In S50, the generation module 130 synthesizes the image data from each external camera 11 based on the defined correspondence relationship to generate a synthesized image observed from the virtual viewpoint Pv. The generation module 130 sets a predetermined transmittance for the overlapping area of ​​the image data overlapping each other on the inner projection surface Pi and synthesizes them.

[0067] Then, in S60, the display module 140 displays the generated composite image on the display system 30. In addition, the display module 140 may further overlap and display other display objects on the composite image. For example, the display module 140 may overlap and display display objects such as a parking frame, a path to a target location, a guide line, and an obstacle location.

[0068] Below, refer to Figure 5-6 The synthetic image Ic generated by the above-described first embodiment and the reference image Ir, which is a synthetic image generated by the reference example, will be described by comparison.

[0069] exist Figure 5 In the example shown, the image on the upper side of the paper is a reference image Ir generated by a reference example in which the image is projected only onto the inner projection surface without using the outer projection surface. Also, the image on the lower side of the paper is a synthetic image Ic generated by the first embodiment.

[0070] Each image Ir, Ic is a composite image of viewpoint images Iv1, Iv2, and Iv3 generated from images captured by three external cameras 11 installed in the host vehicle A. The installation positions of the three external cameras 11 are common to the reference image Ir and the composite image Ic.

[0071] Viewpoint image Iv1 is the portion from the left end of the paper to the dotted line. In viewpoint image Iv1, the ground S1 and the three-dimensional object O1 are photographed. Viewpoint image Iv2 is the portion from the dotted line to the dotted line. In viewpoint image Iv2, the ground S2 and the three-dimensional object O2 are photographed. Viewpoint image Iv3 is the portion from the long dotted line to the right end of the paper. In viewpoint image Iv3, the ground S3 is photographed. Viewpoint images Iv1 and Iv2 overlap and are synthesized at the overlapping portion SA1. Viewpoint images Iv2 and Iv3 overlap and are synthesized at the overlapping portion SA2. Figure 5 In the example of , the photographed three-dimensional objects O1 and O2 are obtained by photographing a three-dimensional object located closer to the front side than the position of the inner projection plane Pi. In addition, the three-dimensional object is a columnar three-dimensional object extending from the ground.

[0072] like Figure 5 As shown, the curvature of the shooting ground S1, S2, and S3 of the synthetic image Ic obtained by synthesizing the viewpoint images generated by the inner projection surface Pi and the outer projection surface Po is increased compared to the reference image Ir obtained by synthesizing the viewpoint images generated only by the inner projection surface Pi. In addition, the magnitude of the offset between the shooting grounds of the overlapping parts SA of the viewpoint images is reduced. For example, the root parts of the shooting three-dimensional objects O1 and O2 on the shooting ground are offset in the reference image Ir, but are basically consistent in the synthetic image Ic.

[0073] In addition, Figure 6 In the example shown, the image on the upper side of the paper is a reference image Ir generated by projecting images on the hemispherical outer projection surface and inner projection surface. Moreover, the composite image on the lower side of the paper is a composite image Ic generated by the first embodiment. The external camera 11 and the image capture camera 11 that capture the captured images that are the basis of each image are Figure 5 same.

[0074] like Figure 6 As shown, the reference image Ir is displayed such that the solid object photographed to cross the horizon position curves across the horizon position. On the other hand, since the outer projection surface Po rises smoothly from the plane, the synthetic image Ic is displayed such that the horizon position curves and rises smoothly.

[0075] In summary, the composite image Ic of the present embodiment is a road surface area that is bent so that the horizon becomes an upward convex shape, and each viewpoint image is synthesized with each other into at least a part of the overlapping part of the shooting range in each viewpoint image, and the photographed road surfaces overlap with each other at the same position on the actual road surface, and the three-dimensional object depicted across the horizon in the viewpoint image is smoothly bent and bent into an image that crosses the horizon.

[0076] Figure 7 1 is a diagram showing changes in an image when the position of the outer projection surface Po is changed in the first embodiment. Figure 7 In the figure, the image on the upper side of the paper is an image in which the rising start position R is set at a location 5000 mm from the center position, and the image on the lower side of the paper is an image in which the rising start position R is set at a location 15000 mm from the center position. Figure 7 In the case where the quadratic function representing the cross section of the rising surface of the outer projection surface Po is represented by y=αx^2, the coefficient α is set to 10^-6. Figure 7 The farther the outer projection surface Po is, the wider the field of view in the composite image Ic is.

[0077] According to the first embodiment described above, an image is synthesized by combining the outer projection surface and the inner projection surface, thereby generating a synthetic image in which the road surface is curved, the three-dimensional object is smoothly curved at the horizon position, and the road surface is represented in a single manner on the inner side of the outer projection surface. That is, since the influence of parallax between images is reduced by projection onto the outer projection surface, the road surface is represented in a single manner on the inner side of the outer projection surface. Moreover, since the outer projection surface rises smoothly, it is also possible to avoid bending of the image. Therefore, it is possible to reduce the influence of parallax between images and avoid bending of the image.

[0078] (Second Embodiment)

[0079] The second embodiment is a modification of the first embodiment.

[0080] In the second embodiment, the projection module 120 changes at least one of the virtual viewpoint Pv and the ascending start position R of the outer projection surface Po based on a specific condition.

[0081] For example, the projection module 120 implements the change processing based on the speed of the host vehicle A. Specifically, the greater the speed of the host vehicle A, the more the projection module 120 moves the virtual viewpoint Pv upward. In addition, the greater the speed of the host vehicle A, the farther the projection module 120 moves the rising start position of the outer projection surface away from the host vehicle A.

[0082] Alternatively, the projection module 120 implements the change processing based on the position of the obstacle around the host vehicle A. Specifically, the projection module 120 sets the rise start position R farther than the position of the obstacle. For example, the projection module 120 compares the preset initial rise start position R with the position of the obstacle, and when the position of the obstacle is farther than the initial rise start position, the rise start position R may be changed. Alternatively, the projection module 120 may also determine the initial rise start position R based on the position of the obstacle.

[0083] In this case, the projection module 120 maintains the virtual viewpoint Pv at a preset position. Alternatively, the projection module 120 may set the virtual viewpoint Pv at a position where the occupant can easily visually recognize the distance between the obstacle and the host vehicle A.

[0084] Alternatively, the projection module 120 implements the change processing based on the display position of the display object overlapped on the synthetic image. Specifically, the projection module 120 sets the rising start position R farther than the overlapping position of the display object overlapped on the road surface of the synthetic image. For example, the projection module 120 may also compare the preset initial rising start position R with the overlapping position, and change the rising start position R when the overlapping position is farther than the initial rising start position. Alternatively, the projection module 120 may also determine the initial rising start position R based on the overlapping position.

[0085] In this case, the projection module 120 maintains the virtual viewpoint Pv at a preset position. Alternatively, the projection module 120 may set the virtual viewpoint Pv at a position where it is determined that the display object is easy to visually recognize.

[0086] (Other embodiments)

[0087] Although a plurality of embodiments have been described above, the present disclosure is not to be construed as being limited to these embodiments, and can be applied to various embodiments and combinations within a scope not departing from the gist of the present disclosure.

[0088] In a modified example, the dedicated computer constituting the image processing system 100 may have at least one of a digital circuit and an analog circuit as a processor. The so-called digital circuit here is, for example, at least one of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a SOC (System on a Chip), a PGA (Programmable Gate Array), and a CPLD (Complex Programmable Logic Device). In addition, such a digital circuit may also have a memory storing a program.

[0089] In a modified example, the main moving body to which the image processing system 100 is applied may also be, for example, an autonomous driving robot capable of carrying goods or collecting information by autonomous driving or remote driving. In addition to the above-described methods, the above-described embodiments and modified examples may also be implemented in the form of a processing circuit (e.g., a processing ECU, etc.) or a semiconductor device (e.g., a semiconductor chip, etc.) as a control device that can be mounted on the main moving body and has at least one processor 102 and one memory 101.

[0090] (Public technical ideas)

[0091] This specification discloses a plurality of technical concepts described in the following enumerated items, and discloses a plurality of combined technical concepts shown by selectively citing prior technical concepts in subsequent technical concepts.

[0092] (Technical Thought 1)

[0093] An image processing system is provided, which has a processor (102) and processes images captured by a plurality of external cameras (11) mounted on a main moving body (A).

[0094] The processor is configured to perform the following processing:

[0095] Acquire the plurality of images captured by the external cameras whose capturing ranges at least partially overlap; and

[0096] generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint,

[0097] Generating the above-mentioned composite image includes generating the following composite image:

[0098] The road surface area is bent so that the horizon becomes convex upward, and the above-mentioned viewpoint images are synthesized into at least a part of the overlapping part of the above-mentioned shooting range in the above-mentioned viewpoint images, and the photographed road surfaces overlap with each other at the same position in the actual road surface. The three-dimensional object depicted across the above-mentioned horizon in the above-mentioned viewpoint image is smoothly bent and bent to cross the above-mentioned horizon.

[0099] (Technical Thought 2)

[0100] The image processing system according to technical concept 1, wherein:

[0101] The structure is further configured to perform: defining the above-mentioned outer projection surface (Po) and the inner projection surface (Pi), wherein the above-mentioned outer projection surface (Po) is defined as a rising surface including a plane and a rising surface smoothly rising from the above-mentioned plane, the above-mentioned captured image is projected onto the above-mentioned outer projection surface, the above-mentioned inner projection surface (Pi) is defined as a shape rising innerward than the above-mentioned outer projection surface, and the above-mentioned captured image projected onto the above-mentioned outer projection surface is further projected onto the above-mentioned inner projection surface.

[0102] (Technical Thought 3)

[0103] The image processing system according to technical idea 2, wherein:

[0104] Defining the above outer projection surface and the above inner projection surface includes:

[0105] defining the virtual viewpoint, wherein the virtual viewpoint is a viewpoint in the synthetic image; and

[0106] A rising start position (R) of the rising surface from the plane is defined in relation to the position of the virtual viewpoint.

[0107] (Technical Thought 4)

[0108] The image processing system according to technical idea 2 or technical idea 3, wherein:

[0109] Defining the above outer projection surface and the above inner projection surface includes:

[0110] The virtual viewpoint is defined in relation to the moving speed of the main moving object.

[0111] (Technical Thought 5)

[0112] The image processing system according to any one of technical concepts 2 to 4, wherein:

[0113] Defining the above outer projection surface and the above inner projection surface includes:

[0114] A rising start position (R) from the plane in the rising surface is defined in relation to the position of the obstacle shown in the captured image.

[0115] (Technical Thought 6)

[0116] The image processing system according to any one of technical concepts 2 to 5, wherein:

[0117] Generating the above composite image involves:

[0118] Also generate an object that overlaps with the above composite image,

[0119] Defining the above outer projection surface and the above inner projection surface includes:

[0120] A rising start position (R) of the rising surface from the plane is defined in relation to the overlapping position of the object.

[0121] (Technical Thought 7)

[0122] An image processing device is provided, which has a processor (102) and processes images captured by a plurality of external cameras (11) mounted on a main moving body (A).

[0123] The processor is configured to perform the following processing:

[0124] Acquire the plurality of images captured by the external cameras whose capturing ranges at least partially overlap; and

[0125] generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint,

[0126] Generating the above-mentioned composite image includes generating the following composite image:

[0127] The road surface area is bent so that the horizon becomes convex upward, and the above-mentioned viewpoint images are synthesized into at least a part of the overlapping part of the above-mentioned shooting range in the above-mentioned viewpoint images, and the photographed road surfaces overlap with each other at the same position in the actual road surface. The three-dimensional object depicted across the above-mentioned horizon in the above-mentioned viewpoint image is smoothly bent and bent to cross the above-mentioned horizon.

[0128] (Technical Thought 8)

[0129] An image processing method is an image processing method executed by a processor (102) for processing images captured by a plurality of external cameras (11) mounted on a main moving body (A).

[0130] The above image processing method comprises:

[0131] Acquire the plurality of images captured by the external camera;

[0132] Acquire the plurality of images captured by the external cameras whose capturing ranges at least partially overlap; and

[0133] generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint,

[0134] Generating the above composite image involves:

[0135] The road surface area is bent so that the horizon becomes convex upward, and the above-mentioned viewpoint images are synthesized into at least a part of the overlapping part of the above-mentioned shooting range in the above-mentioned viewpoint images, and the photographed road surfaces overlap with each other at the same position in the actual road surface. The three-dimensional object depicted across the above-mentioned horizon in the above-mentioned viewpoint image is smoothly bent and bent to cross the above-mentioned horizon.

[0136] (Technical Thought 9)

[0137] An image processing program includes commands stored in a storage medium (101) and executed by a processor (102) for processing images captured by a plurality of external cameras (11) mounted on a main moving body (A).

[0138] The above command is structured to execute the following processing:

[0139] Acquire the plurality of images captured by the external cameras whose capturing ranges at least partially overlap; and

[0140] generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint,

[0141] Generating the above-mentioned composite image includes generating the following composite image:

[0142] The road surface area is bent so that the horizon becomes convex upward, and the above-mentioned viewpoint images are synthesized into at least a part of the overlapping part of the above-mentioned shooting range in the above-mentioned viewpoint images, and the photographed road surfaces overlap with each other at the same position in the actual road surface. The three-dimensional object depicted across the above-mentioned horizon in the above-mentioned viewpoint image is smoothly bent and bent to cross the above-mentioned horizon.

Claims

1. An image processing system comprising a processor (102) and processing images captured by a plurality of external cameras (11) mounted on a main moving body (A). The processor is configured to perform the following processing: Acquire the plurality of images captured by the external cameras whose capturing ranges at least partially overlap; and generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint, Generating the above-mentioned composite image includes generating the following composite image: The road surface area is bent so that the horizon becomes convex upward, and the above-mentioned viewpoint images are synthesized with each other so that at least a part of the photographed road surface in the overlapping part of the above-mentioned shooting range in the above-mentioned viewpoint images overlaps with each other at the same position on the actual road surface, and the three-dimensional object depicted across the above-mentioned horizon in the above-mentioned viewpoint image is smoothly bent and bent to cross the above-mentioned horizon.

2. The image processing system according to claim 1, wherein: The structure is further configured to perform: defining an outer projection surface (Po) and an inner projection surface (Pi), wherein the outer projection surface (Po) is defined as a rising surface including a plane and a rising surface smoothly rising from the plane, and the captured image is projected onto the outer projection surface, and the inner projection surface (Pi) is defined as a shape rising inwardly than the outer projection surface, and the captured image projected onto the outer projection surface is further projected onto the inner projection surface.

3. The image processing system according to claim 2, wherein: Defining the above outer projection surface and the above inner projection surface includes: defining the virtual viewpoint, wherein the virtual viewpoint is a viewpoint in the synthetic image; and A rising start position (R) of the rising surface from the plane is defined in relation to the position of the virtual viewpoint.

4. The image processing system according to claim 3, wherein: Defining the above outer projection surface and the above inner projection surface includes: The virtual viewpoint is defined in relation to the moving speed of the main moving object.

5. The image processing system according to claim 3, wherein: Defining the above outer projection surface and the above inner projection surface includes: A rising start position (R) from the plane in the rising surface is defined in relation to the position of the obstacle shown in the captured image.

6. The image processing system according to claim 2, wherein: Generating the above composite image involves: Also generate an object that overlaps with the above composite image, Defining the above outer projection surface and the above inner projection surface includes: A rising start position (R) of the rising surface from the plane is defined in relation to the overlapping position of the object.

7. An image processing device comprising a processor (102) and processing images captured by a plurality of external cameras (11) mounted on a main moving body (A), The processor is configured to perform the following processing: Acquire the plurality of images captured by the external cameras whose capturing ranges at least partially overlap; and generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint, Generating the above-mentioned composite image includes generating the following composite image: The road surface area is bent so that the horizon becomes convex upward, and the above-mentioned viewpoint images are synthesized with each other so that at least a part of the photographed road surface in the overlapping part of the above-mentioned shooting range in the above-mentioned viewpoint images overlaps with each other at the same position on the actual road surface, and the three-dimensional object depicted across the above-mentioned horizon in the above-mentioned viewpoint image is smoothly bent and bent to cross the above-mentioned horizon.

8. An image processing method, which is executed by a processor (102) for processing images captured by a plurality of external cameras (11) mounted on a main moving body (A). The above image processing method comprises: Acquire the plurality of images captured by the external camera; Acquire the plurality of images captured by the external cameras whose capturing ranges at least partially overlap; and generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint, Generating the above-mentioned composite image includes generating the following composite image: The road surface area is bent so that the horizon becomes convex upward, and the above-mentioned viewpoint images are synthesized with each other so that at least a part of the photographed road surface in the overlapping part of the above-mentioned shooting range in the above-mentioned viewpoint images overlaps with each other at the same position on the actual road surface, and the three-dimensional object depicted across the above-mentioned horizon in the above-mentioned viewpoint image is smoothly bent and bent to cross the above-mentioned horizon.

9. An image processing program, comprising instructions stored in a storage medium (101) for processing images captured by a plurality of external cameras (11) mounted on a main moving body (A) and for causing a processor (102) to execute, The above command is structured to execute the following processing: Acquire the plurality of images captured by the external cameras whose capturing ranges at least partially overlap; and generating a composite image, wherein the composite image is formed by synthesizing a plurality of viewpoint images converted from the captured images and observed from a specific virtual viewpoint, Generating the above-mentioned composite image includes generating the following composite image: The road surface area is bent so that the horizon becomes convex upward, and the above-mentioned viewpoint images are synthesized with each other so that at least a part of the photographed road surface in the overlapping part of the above-mentioned shooting range in the above-mentioned viewpoint images overlaps with each other at the same position on the actual road surface, and the three-dimensional object depicted across the above-mentioned horizon in the above-mentioned viewpoint image is smoothly bent and bent to cross the above-mentioned horizon.

Images