Image processing device, image processing method, and computer program

Abstract

To provide an image processing device capable of generating a virtual viewpoint image that does not misidentify images of structures protruding from a moving body as obstacles.SOLUTION: An image processing device has virtual viewpoint image generation means for generating a virtual viewpoint image on the basis of multiple captured images from multiple imaging means mounted on a moving body, and identification means for identifying an area in the captured images where a structure protruding from the moving body is captured as a protruding area. The virtual viewpoint image generation means generates the virtual viewpoint image so as to exclude the protruding area identified by the identification means.SELECTED DRAWING: Figure 6A

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a computer program. [Background technology]

[0002] In recent years, an image processing system has been proposed that displays an image of the surroundings of a vehicle seen from a virtual viewpoint based on images captured by multiple cameras mounted on the vehicle. Patent Document 1 discloses a system that generates a virtual viewpoint image showing the surroundings of a vehicle seen from a virtual viewpoint based on images captured by cameras installed on the front, rear, left and right sides of the vehicle, and displays the generated image on a display device.

[0003] Furthermore, in Patent Document 1, the area of ​​a mask that hides the vehicle reflected in the captured image is adjusted, and the mask is superimposed on the virtual viewpoint image, thereby preventing the body of the vehicle from being mistaken for an obstacle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-118435 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, in minivans and trucks, which are larger than passenger cars, auxiliary mirrors are installed to check blind spots and are used in conjunction with camera systems. The installation position of the mirror is determined by the relationship between the driver's seat and the blind spot, so it is often installed in a position that protrudes from the vehicle.

[0006] On the other hand, the installation position of the camera is determined by the range of the object to be photographed, and in order to capture a wider road surface, it is preferable to install the camera at a higher position, which means that protruding vehicle structures such as mirrors are captured in the camera's imaging range.

[0007] The conventional problems will be described with reference to Figures 9A to 9D. Figure 9A is a side view of a vehicle for explaining the conventional problems, showing a side view of a minivan 100. Cameras 101a to 101d (101b is not shown as it is located on the opposite side) are installed on the front, rear, left and right sides of minivan 100, and a protruding auxiliary mirror 102 is installed at the rear.

[0008] 9B is a top view of a vehicle for explaining the conventional problem, and shows a top view of a minivan 100 and examples of images 103a to 103d captured by cameras 101a to 101d. In image 103c captured by camera 101c as a rear camera, auxiliary mirror 102 is captured as image 105.

[0009] 9C is a diagram showing an example of a virtual viewpoint image for explaining the conventional problem, and shows virtual viewpoint image 106 obtained by combining images 103a to 103d from a top-view virtual viewpoint. When a general flat or bowl-shaped projection surface is used to generate the virtual viewpoint image, auxiliary mirror 102, which is a protruding object from minivan 100, is synthesized at a position away from the vehicle in the generated virtual viewpoint image.

[0010] Reference numeral 107 denotes the synthesized image of the auxiliary mirror 102. When the driver looks at the virtual viewpoint image 106, there is a risk that the driver will mistakenly recognize the image 107 as an obstacle.

[0011] 9D is a front view of a vehicle, for example, a truck 108, for explaining the conventional problem. On the left side of the truck 108, opposite the driver's seat, a camera 101d and mirrors 109a to 109c for checking blind spots are installed in positions that protrude from the truck. In this case as well, the synthesized virtual viewpoint image includes images of mirrors 109a to 109c at positions distant from the vehicle, which may be mistaken for obstacles by the driver.

[0012] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device capable of generating a virtual viewpoint image in which the image of a structure protruding from a moving body is not mistaken for an obstacle. [Means for solving the problem]

[0013] In the image processing device, a virtual viewpoint image generating means for generating a virtual viewpoint image based on a plurality of captured images from a plurality of imaging means mounted on the moving body; and specifying means for specifying, as a protruding region, a region in the captured image in which a structure protruding from the moving body is captured, The virtual viewpoint image generating means generates the virtual viewpoint image so as not to include the protruding region identified by the identifying means. [Effects of the Invention]

[0014] According to the present invention, it is possible to realize an image processing device capable of generating a virtual viewpoint image in which an image of a structure protruding from a moving body is not mistaken for an obstacle. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an example of the positional relationship between a vehicle 1 and camera units 11 to 14 according to the first embodiment. [Figure 2] 1 is a functional block diagram for explaining an example of the configuration of an image processing system 20 according to a first embodiment. [Figure 3] 4 is a flowchart illustrating the processing flow of an image processing method in the integration processing unit 21 according to the first embodiment. [Figure 4] 10 is a flowchart illustrating an example of processing in step S32 for identifying an image area in which a structure on the vehicle is reflected. [Figure 5A] FIG. 10 is a diagram showing an example of a camera selection screen 51. [Figure 5B] FIG. 10 is a diagram showing an example of an area editing screen 55a in step S43. [Figure 5C]FIG. 10 is a diagram showing an example of an area addition screen 56 in step S45. [Figure 5D] FIG. 10 is a diagram showing an example of an area addition screen 56b in a state where a partial area 58a in a camera image 52d is specified. [Figure 5E] FIG. 5D shows an example of region edit screen 55b when the user selects added region confirmation button 54e and returns to step S43. [Figure 5F] FIG. 10 is a diagram showing an example of an area deletion screen 59a. [Figure 5G] FIG. 10 is a diagram showing an example of an area deletion screen 59b in a state in which the user has operated the operation unit 28 to select a part of the designated area. [Figure 6A] 10A to 10C are diagrams illustrating an example of synthesis processing when a top-view image is generated as a virtual viewpoint image. [Figure 6B] FIG. 4 is a diagram illustrating an example of virtual viewpoint synthesis parameters according to the first embodiment. [Figure 6C] 6B is a diagram showing an example of generating virtual viewpoint synthesis parameters such that the host vehicle structure region 62 in FIG. 6A is not used as the blending region, and a surrounding region 64g that avoids the host vehicle structure region 62 is used as the blending region. [Figure 6D] FIG. 10 is a diagram showing an example of generating virtual viewpoint synthesis parameters that do not use the host vehicle structure region 62 while keeping the boundary line of the blend region as a straight line. [Figure 6E] 10 is a diagram showing an example of a virtual viewpoint image 66 generated by the OSD superimposing unit 24. FIG. [Figure 7] 10 is a flowchart illustrating an example of processing for identifying an image area in which a vehicle structure is captured, according to the second embodiment. [Figure 8A] FIG. 10 is a diagram showing an example of an automatic setting screen 81 according to the second embodiment. [Figure 8B] FIG. 10 is a diagram showing an example of virtual viewpoint transformation parameters for a top view corresponding to the front camera unit 11. [Figure 8C] 1 shows a top view of the vehicle 1 and images 85a to 85d obtained by processing the images corresponding to the camera units 11 to 14, respectively. [Figure 8D] FIG. 10 is a diagram showing examples of virtual viewpoint images 88a to 88d that are generated by the image synthesis unit 23 from images 85a to 85d after image processing based on the virtual viewpoint conversion parameters set in step S73 and stored in memory 25. [Figure 8E] FIG. 10 is a diagram showing an example of a vehicle structure area confirmation screen 90 displayed in step S76. [Figure 9A] FIG. 1 is a side view of a vehicle for explaining a conventional problem. [Figure 9B] FIG. 1 is a top view of a vehicle for explaining a conventional problem. [Figure 9C] FIG. 10 is a diagram showing an example of a virtual viewpoint image for explaining a problem in the related art. [Figure 9D] FIG. 1 is a front view of a vehicle for explaining a conventional problem. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In the drawings, the same members or elements are designated by the same reference numerals, and redundant descriptions thereof will be omitted or simplified.

[0017] <Embodiment 1> 1 is a diagram illustrating an example of the positional relationship between a vehicle 1 and camera units 11 to 14 according to embodiment 1. The vehicle 1 is a moving body such as an automobile, and the camera units 11 to 14 each function as an imaging device that outputs a captured image.

[0018] That is, in the first embodiment, a plurality of imaging units that output a plurality of captured images are mounted on a moving body, a vehicle 1. The moving body may be any moving body that can move, such as an automobile, a train, a ship, an airplane, a robot, or a drone.

[0019] 1, camera units 11, 12, 13, and 14 are installed at the front, right side, rear, and left side of a vehicle 1 serving as a moving body. Note that, although the image processing system has four camera units in the first embodiment, the number of camera units included in the image processing system is not limited to four.

[0020] The image processing system of this embodiment may be an image processing system that generates a virtual viewpoint image using images from two or more camera units.

[0021] The camera units 11 to 14 are installed so as to capture predetermined imaging ranges in front, to the right, to the left, and behind of the vehicle 1 as a moving body. The camera units 11 to 14 each have substantially the same components. For example, the camera units 11 to 14 each have an imaging element that captures an optical image and an optical system that generates the optical image on the light receiving surface of the imaging element.

[0022] The optical systems of camera units 11 to 14 are each configured to capture images with a wide angle of approximately 180 degrees in the horizontal direction. In Fig. 1, 11a to 14a indicate the imaging angles of view that can be captured by camera units 11 to 14. The imaging angles of view, number of pixels, and other characteristics of camera units 11 to 14 may differ.

[0023] Next, FIG. 2 is a functional block diagram for explaining an example of the configuration of the image processing system 20 according to the first embodiment, and the image processing system 20 according to the first embodiment will be explained with reference to FIG.

[0024] Note that some of the functional blocks shown in FIG. 2 are realized by causing a CPU or the like serving as a computer (not shown) included in the image processing system or camera unit to execute a computer program stored in a memory serving as a storage medium (not shown).

[0025] However, some or all of these functions may be implemented by hardware, which may be a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP).

[0026] 2 may not be built into the same housing, but may be configured as separate devices connected to each other via signal paths. Also, for example, the integrated processing unit 21, display unit 27, operation unit 28, etc. may not be mounted on a vehicle as a mobile object, but may be provided in, for example, an external terminal for remotely controlling the mobile object.

[0027] 2, image processing system 20 is mounted on vehicle 1. Camera unit 11 installed at the front has wide-angle lens 11b, image sensor 11c, and image processing unit 11d. Camera units 12 to 14 installed at the sides and rear have similar structures.

[0028] Each of the imaging elements 11c to 14c has, for example, a CMOS image sensor or a CCD image sensor. Each of the wide-angle lenses 11b to 14b as an optical system is made up of one or more optical lenses, and generates an optical image on the light-receiving surface of the corresponding imaging element 11c to 14c.

[0029] The imaging elements 11c to 14c each photoelectrically convert an optical image and output imaging data. On the light receiving surface of each of the imaging elements 11c to 14c, for example, RGB color filters are arranged for each pixel.

[0030] The RGB array is, for example, a Bayer array, and therefore, from each of the image sensors 11c to 14c, R, G, R, G pixel data is sequentially output from a given row in accordance with the Bayer array, and G, B, G, B pixel data is sequentially output from the adjacent row.

[0031] The imaging processing units 11d to 14d perform image processing on the imaging data output from the corresponding imaging elements 11c to 14c, respectively, and transmit the processed data to the integration processing unit 21. For example, the imaging processing units 11d to 14d process the image data input from the corresponding imaging elements 11c to 14c in accordance with the Bayer array, and convert the image data into an image data format for transmission to the integration processing unit 21.

[0032] The image capture processing units 11d to 14d may each perform de-Bayer processing, white balance adjustment, gain / offset adjustment, gamma processing, color matrix processing, lossless compression processing, etc. Note that some or all of the processing by the image capture processing units 11d to 14d may be performed by a signal processing unit provided in a semiconductor stack within the image capture elements 11c to 14c.

[0033] Furthermore, some or all of the processing of the imaging processing units 11d to 14d may not be performed within the camera units, but may be performed by an image processing unit within the integrated processing unit 21. Parameters for the imaging processing are stored in a ROM (not shown) provided within the camera units, and are set at startup. Note that the imaging processing parameters may be configured to be set from the integrated processing unit 21.

[0034] The integrated processing unit 21 has an image processing unit 22, an image synthesis unit 23, an OSD superimposition unit 24, a memory 25, a control unit 26, etc., and functions as an image processing device. In the first embodiment, the integrated processing unit 21 is housed in a housing separate from the camera unit. OSD is an abbreviation for On Screen Display.

[0035] The image processing unit 22 acquires the captured image data from the image processing units 11d to 14d and performs image processing on the acquired captured image data. Specifically, the image processing unit 22 performs processing that is not performed by the image processing units 11d to 14d, such as de-Bayer processing, white balance adjustment, gain / offset adjustment, gamma processing, and color matrix processing.

[0036] Furthermore, when lossless compression processing is performed in the imaging processing units 11d to 14d, the image processing unit 22 performs decompression processing. The parameters of the image processing in the image processing unit 22 are set in the image processing unit 22 by the control unit 26 based on the camera information acquired by the control unit 26 from the memory 25.

[0037] The camera information is information for each of the camera units 11 to 14, and is stored in advance in the memory 25. The camera information includes at least part of the number of pixels of the image sensors 11c to 14c, pixel arrangement information, photoelectric conversion characteristics, gamma characteristics, sensitivity characteristics, and image data format information.

[0038] Image processing unit 22 performs image processing corresponding to each of camera units 11 to 14 based on the camera information. Image synthesis unit 23 receives the processed images from image processing unit 22 and synthesizes them to generate a virtual viewpoint image. Here, image synthesis unit 23 functions as virtual viewpoint image generation means that generates a virtual viewpoint image based on a plurality of captured images from a plurality of imaging means mounted on the moving object.

[0039] The parameters for the synthesis process in the image synthesis unit 23 are set in the image synthesis unit 23 by the control unit 26 based on the camera information acquired by the control unit 26 from the memory 25, the operation information acquired by the operation unit 28, and the vehicle information acquired by the vehicle control unit 29.

[0040] The camera information includes at least part of the optical characteristics of the wide-angle lenses 11b to 14b, and the installation position coordinates in the vehicle coordinate system and attitude information of the camera units 11 to 14. The camera information may be stored in a ROM (not shown) in the camera unit and read out by the control unit 26.

[0041] A known method is used to generate the virtual viewpoint image. That is, images after image processing corresponding to each of the camera units 11 to 14 are projected onto a flat or bowl-shaped projection surface with reference to the optical characteristics of the wide-angle lenses 11 b to 14 b. Then, the virtual viewpoint position is determined based on the operation information and the vehicle information, and the virtual viewpoint image is generated.

[0042] The OSD superimposing unit 24 receives the virtual viewpoint image from the image synthesis unit 23 and generates a superimposed image by superimposing an icon, CG, text, etc. representing the host vehicle. The parameters of the superimposing process are set by the control unit 26 to the OSD superimposing unit 24, such as an ID of the icon, etc., and superimposition coordinates on the image, according to the virtual viewpoint position determined by the control unit 26 based on the operation information and the host vehicle information.

[0043] Image data corresponding to the ID, such as an icon, is stored in the memory 25. When the virtual viewpoint is switched, the name of the virtual viewpoint view may be superimposed and displayed.

[0044] Memory 25 stores computer programs executed by a CPU serving as a computer in control unit 26, camera information for each of camera units 11 to 14, and installation coordinates and attitude information in vehicle coordinates for camera units 11 to 14. Furthermore, image data such as icons referenced by OSD superimposition unit 24 is also stored.

[0045] The information in the memory 25 can be read out by the control unit 26. The memory 25 also functions as a frame memory and a work memory for the image synthesis unit 23 to synthesize a virtual vision image. The CPU in the control unit 26 executes the computer programs stored in the memory 25 to perform various controls on the entire image processing system 20.

[0046] The display unit 27 is a display installed in a position that is easily visible to the driver, and receives and displays the superimposed image from the OSD superimposing unit 24. This allows the driver to view the surrounding situation adapted to the situation of the vehicle as a virtual viewpoint image accompanied by an icon of the vehicle. The display unit 27 may also be used as the display of a car navigation system.

[0047] The operation unit 28 is operated when the driver switches the virtual viewpoint, and notifies the control unit 26 of the operation content. Note that the operation unit 28 may also serve as the display unit 27 by providing the display unit 27 with a touch panel or operation buttons, etc.

[0048] The vehicle control unit 29 notifies the control unit 26 of the vehicle state. The vehicle state includes information such as the state of the shift lever, the state of the blinkers, the traveling speed, etc. The vehicle control unit 29 may control the speed and direction of the vehicle as a moving body in accordance with information (distance, direction, position, speed, etc.) of a subject or obstacle recognized by the control unit 26 through image recognition.

[0049] 3 is a flowchart for explaining the processing flow of the image processing method in the integrated processing unit 21 according to embodiment 1. Note that the operation of each step in the flowchart in FIG. 3 is performed sequentially by a CPU or the like serving as a computer in the control unit 26 executing a computer program stored in memory.

[0050] The processing flow of FIG. 3 is executed by the control unit 26 of the integration processing unit 21 when the image processing system 20 is started up.

[0051] In step S31, the control unit 26 acquires the camera information of each of the camera units 11 to 14 from the memory 25, and sets image processing parameters to be performed by the image processing unit 22 in the image processing unit based on the camera information.

[0052] In step S32, control unit 26 identifies an image area in which a structure protruding from the moving object is captured, based on the processed images corresponding to each of camera units 11 to 14. Here, step S32 functions as a identifying step (identifying means) that identifies an area in the captured image in which a structure protruding from the moving object is captured as a protruding area. A detailed example of the processing in step S32 will be described with reference to Fig. 4 and Figs. 5A to 5G.

[0053] Figure 4 is a flowchart illustrating an example of the processing of step S32 for identifying an image area in which a vehicle structure is reflected, and Figures 5A to 5G are figures each showing an example of a UI for the processing for identifying an image area in which a vehicle structure is reflected in embodiment 1.

[0054] The CPU or the like serving as a computer in the control unit 26 executes a computer program stored in memory, thereby sequentially performing the operations of the steps in the flowchart of FIG.

[0055] 4, control unit 26 sets parameters in image synthesis unit 23 and OSD superimposition unit 24, thereby displaying a camera selection screen on display unit 27. FIG. 5A is a diagram showing an example of camera selection screen 51.

[0056] 5A, 51 is a camera selection screen displayed on display unit 27. 52a to 52d are resized images obtained by image processing corresponding to each of camera units 11 to 14, and are arranged in front of, to the right of, behind, and to the left of host vehicle icon 53 according to the arrangement of camera units 11 to 14.

[0057] The image arrangement process is performed by the image synthesis unit 23 in response to instructions from the control unit 26. An operation guide 51a and a cancel button 54a are also arranged on the camera selection screen 51. The drawing process of the host vehicle icon 53, the operation guide 51a, and the cancel button 54a is performed by the OSD superimposition unit 24 in response to instructions from the control unit 26.

[0058] In Figure 5A, for the reflected portion of the vehicle's body, the image area in which the vehicle's structure is reflected is set in a memory (not shown) as a "designated area" in the vehicle structure area list (described later) at the time of shipment from the manufacturer, for example.

[0059] In Fig. 5A, the "designated area" is displayed with hatching. Note that the designated area may be displayed semi-transparently instead of hatched so that the user can see anything reflected in the designated area. The camera image 52c of the camera unit 13 serving as a rear camera reflects the rear mirror, which is a protruding structure of the vehicle.

[0060] 4, control unit 26 accepts any operation of operation unit 28 by the user and determines the content of the operation. If it is determined in step S42 that the user has operated operation unit 28 to select one of camera images 52a to 52d that specifies a structure on the host vehicle, the process proceeds to step S43.

[0061] If it is determined in step S42 that the user has selected the cancel button 54a, the flow for identifying the image area in which the vehicle structure is reflected in FIG. 4 ends, and the process proceeds to step S33 in FIG.

[0062] For example, if the user selects camera image 52c of camera unit 13, which is a rear camera, it is determined in step S42 that a camera has been selected, and the process proceeds to step S43. In step S43, control unit 26 sets parameters in image synthesis unit 23 and OSD superimposition unit 24, thereby displaying area editing screen 55a on display unit 27.

[0063] Fig. 5B is a diagram showing an example of region editing screen 55a in step S43. In Fig. 5B, 55a is region editing screen displayed on display unit 27, and displays camera image 52d selected in step S42, operation guide 55c, add region button 54b, delete region button 54c, and cancel button 54d.

[0064] Next, in step S44 of Fig. 4, control unit 26 accepts a user operation, if any, and determines the content of the operation. When the user operates operation unit 28 to select one of buttons 54b to 54d on region editing screen 55a of Fig. 5B, the process proceeds to the corresponding process according to the selection. For example, if the user selects add region button 54b, the process proceeds to step S45, and region addition screen 56a of Fig. 5C is displayed.

[0065] Fig. 5C is a diagram showing an example of region addition screen 56a in step S45. In Fig. 5C, 56a is the region addition screen displayed on display unit 27. Region addition screen 56a displays camera image 52d of camera unit 13 as the rear camera selected in step S42, operation guide 56c, added region confirmation button 54e, and cancel button 54f.

[0066] At this point, the user has not yet specified an additional area, so the additional area confirmation button 54e is grayed out and cannot be selected.

[0067] 4, control unit 26 accepts a user operation, if any, and determines the content of the operation. By operating operation unit 28, the user can specify a protruding host vehicle structure area (for example, an area reflected in an auxiliary mirror such as a rearview mirror) on camera image 52d in area addition screen 56a in FIG.

[0068] If the operation unit 28 is a touch panel, the user specifies the area to be added by, for example, surrounding the area on the camera image 52d in which the vehicle structure is captured with a finger. Note that the method for specifying the area to be added may also be a method of sequentially specifying the vertex positions of the area using a mouse or an operation button (not shown).

[0069] The area may have any shape. A case where the user operates operation unit 28 to specify area 58a in camera image 52d of camera unit 13 serving as a rear camera will be described with reference to Fig. 5D.

[0070] 5D is a diagram showing an example of an area addition screen 56b in which a portion of an area 58a in a camera image 52d has been designated. In FIG. 5D, the area 58a designated by the user is indicated by a dashed line. Note that the dashed line is not required, and any other method may be used to highlight the designated area 58a in the camera image, such as by flashing, as long as the method makes the position of the designated area 58a easier to understand.

[0071] When the user specifies the area they want to add, the grayed-out status of the Add Area Confirmation button 54e is released and the button becomes selectable. When the user clicks or touches the Add Area Confirmation button 54e, it is determined in step S46 that the add area has been confirmed, and the process proceeds to step S47. In step S47, the coordinates indicating the specified add area, i.e., the specified add area on the camera image that reflects the host vehicle structure, are added to the host vehicle structure area list, and the process returns to step S43.

[0072] The vehicle structure area list may be a list of any structure, but includes the vehicle structure area ID, the camera ID, and information for identifying the area in the camera image. The information for identifying the area may be, for example, a list of the coordinates of each vertex of the area in a coordinate system in which the x and y coordinates of the camera image are normalized to 0 to 1.

[0073] When a protruding area in which the vehicle structure is reflected is added in step S47, the new vehicle structure area ID, camera ID, and information area information for identifying the area within the camera image are added, for example, to the end of the vehicle structure area list, in association with each other.

[0074] Fig. 5E is a diagram showing an example of area editing screen 55b when the user returns to step S43 by selecting added area confirmation button 54e in Fig. 5D. As shown in Fig. 5E, since the area specified by the user in step S46 has been added to the host vehicle structure area list, area 58b where the rearview mirror is reflected is displayed with hatching as a "specified area."

[0075] In step S46 of FIG. 4, if the user selects the cancel button 54f on the region addition screen 56b of FIG. 5D, the process also returns to step S43.

[0076] In step S44 of FIG. 4, if the user selects the area deletion button 54c on the area edit screen 55a of FIG. 5B, the process proceeds to step S48, where an area deletion screen 59a is displayed. 5F is a diagram showing an example of area deletion screen 59a, where 59a is the area deletion screen displayed on display unit 27. The example of area deletion screen 59a in FIG. 5F displays camera image 52d selected in step S42, operation guide 59c, deletion area confirmation button 54g, and cancel button 54h. At this point, because the user has not yet specified an area to be deleted, deletion area confirmation button 54g is grayed out and cannot be selected.

[0077] Next, in step S49 of Fig. 4, control unit 26 accepts a user operation, if any, and determines the content of the operation. At this time, the user operates operation unit 28 to specify an area to be deleted from the vehicle structure area list among the vehicle structure areas displayed as designated areas by hatching on area deletion screen 59a of Fig. 5F.

[0078] If the operation unit 28 is a touch panel, the deletion area can be designated by touching part of the area displayed with hatching as the designated area. Alternatively, one of the designated areas can be designated using a mouse or an operation button (not shown).

[0079] Fig. 5G is a diagram showing an example of area deletion screen 59b in a state in which the user has selected a portion of the designated area by operating operation unit 28. In Fig. 5G, area 58c designated by the user is indicated by a dashed line.

[0080] Any display other than a dashed line may be used as long as it allows the position of the designated area in the camera image to be easily identified. For example, the area 58c designated by the user may be made to blink. In this manner, in this embodiment, an operation means for inputting a user operation is provided, and the protruding area is identified by the user operating the operation means.

[0081] When the deletion area is specified, the deletion area confirmation button 54g is no longer grayed out and becomes selectable, as shown in Fig. 5G. Then, when the user clicks or touches the deletion area confirmation button 54g, it is determined in step S49 that the deletion area has been confirmed, and the process proceeds to step S50, where the information corresponding to the specified deletion area is deleted from the host vehicle structure area list, and the process returns to step S43.

[0082] On the other hand, if the user clicks or touches the cancel button 54h in Fig. 5G in step S49 of Fig. 4, the process also returns to step S43. If the user selects the cancel button 54d in Fig. 5E in step S44, the process returns to step S41.

[0083] Once the processing in Figure 4 is complete, the process proceeds to step S33 in Figure 3, where virtual viewpoint synthesis parameters are generated. The virtual viewpoint synthesis parameters generated in step S33 will be described with reference to Figures 6A to 6E. Figure 6A is a diagram illustrating an example of synthesis processing when a top-view image is generated as a virtual viewpoint image.

[0084] 6A, 61a to 61d are images (source images) in which the designated areas listed in the host vehicle structure area list generated in step S32 are reflected in images obtained after image processing of the images corresponding to each of camera units 11 to 14. Reference numeral 62 denotes a host vehicle structure area corresponding to the rear mirror captured by camera unit 13 as a rear camera, and is an area additionally designated by the user.

[0085] Reference numeral 63 shows a schematic representation of the memory area of ​​the frame memory of m×n pixels for the virtual viewpoint image. The dashed lines indicate which part of the source image each pixel of the virtual viewpoint image refers to. The virtual viewpoint image is obtained by projective transformation based on the relative positions and optical axis directions of the camera units 11-14 with respect to the vehicle, and the optical characteristics of the wide-angle lenses 11b-14b.

[0086] Details of projective transformation are well known and will not be described here. For example, when generating a top-view image as a virtual viewpoint image, the pixel values ​​of each pixel in the upper part of virtual viewpoint image 63 in Figure 6A are obtained by referring to source image 61a corresponding to front camera unit 11.

[0087] 6B is a diagram showing an example of virtual viewpoint synthesis parameters according to embodiment 1. The virtual viewpoint synthesis parameters list which pixel of which camera unit is to be referenced for each pixel of the virtual viewpoint image 63 as a result of the projective transformation.

[0088] Since the coordinates of the source image in the virtual viewpoint synthesis parameters are not necessarily integers, the image synthesis unit 23 calculates the pixel values ​​of the virtual viewpoint image by interpolating from the pixel values ​​of nearby pixels. As shown in Fig. 1, the imaging ranges of the camera units 11 to 14 partially overlap, so that images captured by multiple camera units can be referenced in the overlapping areas.

[0089] In this case, it is possible to refer to the image captured by only one of the camera units, or to alpha blend (weighted addition) the corresponding pixel values ​​of the two camera units. The camera unit selection is determined based on the resolution of each camera unit, the angle of the target area as seen from the optical axis of each camera unit, etc.

[0090] In the virtual viewpoint image 63, in the region near the boundary between images captured by multiple camera units, the seams become noticeable when the camera unit of the source image is switched, so the seams can be smoothed by gradually changing the blending ratio near the boundary. Note that, for example, in the region captured only by camera unit 11, there are no parameters related to camera unit 12.

[0091] Here, the virtual viewpoint synthesis parameters are generated so that the designated areas listed in the vehicle structure area list are not included in the coordinates of the source image. That is, the virtual viewpoint synthesis parameters are generated so that the captured image of the protruding area is not referenced when generating the virtual viewpoint image. If the area is captured by multiple camera units, the captured image of the other camera unit is used as the source image.

[0092] That is, when a captured image from one imaging means includes a protruding region, a virtual viewpoint image that does not include the protruding region can be generated using a captured image from another imaging means that corresponds to the protruding region.

[0093] If the specified area is an area that is only captured by a single camera unit, it is possible to store past images, for example, and refer to the past images by calculating the amount of coordinate change on the image according to the amount of movement of the vehicle.

[0094] The amount of movement of the host vehicle can be acquired from the vehicle control unit 29. In this way, when a protruding region is included in an image captured by one imaging means, a virtual viewpoint image that does not include the protruding region can be generated by referring to a previous image captured by the same imaging means.

[0095] 6C and 6D schematically show examples of virtual viewpoint synthesis parameters when generating a top-view virtual viewpoint image using source images 61a to 61d shown in Fig. 6A. Reference numerals 64a to 64d denote areas that employ source images 61a to 61d from a single camera unit, respectively, and reference numerals 64e to 64h denote overlapping areas where source images from two adjacent camera units are alpha-blended.

[0096] That is, 64e is the overlapping area between source image 61a and source image 61b, 64f is the overlapping area between source image 61b and source image 61c, 64g is the overlapping area between source image 61c and source image 61d, and 64h is the overlapping area between source image 61d and source image 61a.

[0097] In the first embodiment, the host vehicle structure area 62 captured in the source image 61c corresponding to the rear camera unit is not used, and the source image 61d of the left side camera unit is used for this area. 6C is a diagram showing an example of generating virtual viewpoint synthesis parameters such that the host vehicle structure region 62 in FIG. 6A is not used as the blending region, but a peripheral region 64g that avoids the host vehicle structure region 62 is used as the blending region. In this way, the synthesis boundary when synthesizing a plurality of captured images may be configured to have a shape that avoids protruding regions.

[0098] 6D shows an example of generating virtual viewpoint synthesis parameters that do not use the host vehicle structure region 62 while leaving the boundary line of the blending region as a straight line. The region in the center, shown in black, is an area that is not captured in any of the source images 61a to 61d, and a host vehicle icon is displayed superimposed on this region. In this way, the synthesis boundary when combining multiple captured images may be a straight line.

[0099] 3, the control unit 26 sets the virtual viewpoint synthesis parameters generated in step S33 in the image synthesis unit 23. The image synthesis unit 23 synthesizes the source images in accordance with the set virtual viewpoint synthesis parameters, and generates a synthesized image of the virtual viewpoint.

[0100] Here, step S34 functions as a virtual viewpoint image generating step for generating a virtual viewpoint image based on a plurality of captured images from a plurality of imaging means mounted on the moving body. Also, step S34 as a virtual viewpoint image generating step generates a virtual viewpoint image so as not to include the protruding region identified by the identifying step (identifying means).

[0101] Next, in step S35, it is determined whether there has been a change in the state of the vehicle (for example, position), and if there has been no change, step S35 is repeated, and if there has been a change, the process proceeds to step S36. In step S36, the virtual viewpoint is changed in accordance with, for example, the position change, and the process proceeds to step S37. In step S37, it is determined whether the virtual viewpoint synthesis parameters have been generated, and the process returns to step S34, and if not, the process returns to step S33.

[0102] The OSD superimposing unit 24 generates a virtual viewpoint image by superimposing the host vehicle icon on the composite image, and displays the generated image on the display unit 27.

[0103] 6E is a diagram showing an example of a virtual viewpoint image 66 generated by the OSD superimposition unit 24. In FIG. 6E, 65 is a host vehicle icon. Since the user has added and registered the area in which the rearview mirror is reflected to the host vehicle structure area list, an unnatural image caused by host vehicle structures such as the rearview mirror (auxiliary mirror) is not generated in the virtual viewpoint image 66.

[0104] In the above explanation, an example has been shown in which the process of identifying the image area in which the vehicle structure is reflected (step S32 in FIG. 3) is performed every time the image processing system 20 is started up. However, if there is no change in the placement of the vehicle structure, the vehicle structure area list stored in the memory 25 can be read and used.

[0105] In that case, when there is a change in the placement of the vehicle structure, the user can call the flow of Fig. 3 for identifying the image area in which the vehicle structure is reflected, using a UI such as a menu. Alternatively, a timeout period may be set for the process of identifying the image area in which the vehicle structure is reflected, and the flow of Fig. 3 may be terminated if there is no user operation after the image processing system 20 is started.

[0106] In this way, in embodiment 1, images of structures protruding from the vehicle are not synthesized in unnatural positions in the virtual viewpoint image, so it is possible to generate a virtual viewpoint image that the driver will not mistake for an obstacle.

[0107] <Embodiment 2> In the first embodiment, the user individually specified the vehicle structure area by operating the operation unit 28, but in the second embodiment, in an environment where there are no three-dimensional objects around the vehicle, the vehicle structure area is determined by analyzing multiple captured images in response to a user instruction.

[0108] In the second embodiment, the arrangement of the vehicle and camera unit, the configuration of the image processing system, the processing flow, etc., explained in Figures 1 to 3 are almost the same as those in the first embodiment, and explanations of the same parts will be omitted. In the second embodiment, the details of the processing in step S32 in Figure 3 are different from those in the first embodiment, and the differences will be explained using Figures 7 and 8A to 8E.

[0109] Fig. 7 is a flowchart illustrating a processing example for identifying an image area in which a vehicle structure is captured, according to embodiment 2. Note that the operation of each step in the flowchart in Fig. 7 is performed sequentially by a CPU or the like serving as a computer in control unit 26 executing a computer program stored in memory.

[0110] In step S71 of FIG. 7, the control unit 26 sets parameters in the image synthesis unit 23 and the OSD superimposition unit 24, thereby displaying an automatic setting screen as shown in FIG. 8A on the display unit 27.

[0111] 8A is a diagram showing an example of an automatic setting screen 81 according to the second embodiment, and the automatic setting screen 81 is displayed on the display unit 27. Images 82a to 82d are images obtained by processing and resizing images corresponding to the camera units 11 to 14, respectively, and are arranged in front of, to the right of, behind, and to the left of the host vehicle icon 83 according to the arrangement of the camera units 11 to 14. Also displayed on the automatic setting screen 81 are an operation guide 81a, an execute button 84a, and a cancel button 84b.

[0112] Next, in step S72 of Fig. 7, control unit 26 accepts any user operation and determines the content of the operation. When the user operates operation unit 28 to select execute button 84a of Fig. 8A, the process proceeds to step S73. On the other hand, if the user selects cancel button 84b, the process of identifying the image area in which the vehicle structure is reflected is stopped, and the process returns to the flow of Fig. 3 and proceeds to step S33.

[0113] In step S73 of FIG. 7, the control unit generates virtual viewpoint conversion parameters for converting the images captured by the camera units 11 to into virtual viewpoint images of the same viewpoint, and sets the generated parameters in the image synthesis unit .

[0114] The virtual viewpoint conversion parameter is the same concept as the virtual viewpoint synthesis parameter, but rather than synthesizing multiple source images into a single virtual viewpoint image, it is a parameter for converting each source image into an individual virtual viewpoint image.

[0115] 8B is a diagram showing an example of virtual viewpoint transformation parameters for a top view corresponding to the front camera unit 11, and shows the relationship between the virtual viewpoint image coordinates and the source image coordinates. Because the front camera unit 11 does not capture the sides or rear of the vehicle, the source image coordinates corresponding to the coordinates of the sides or rear of the virtual viewpoint image (for example, coordinates (m, n)) are set to "nan."

[0116] Virtual viewpoint conversion parameters corresponding to the right, rear, and left camera units 12 to 14 are also generated in the same manner as in FIG. 8B and set in the image synthesis unit 23.

[0117] 8C shows a top view of vehicle 1 and images 85a to 85d obtained after image processing of the images corresponding to camera units 11 to 14. In image 85c obtained after image processing of the rearward image, rear mirror 86 is reflected as image 87.

[0118] Figure 8D is a diagram showing examples of virtual viewpoint images 88a to 88d that the image synthesis unit 23 generates based on the virtual viewpoint conversion parameters set in step S73 using images 85a to 85d after image processing as input, and stores in memory 25.

[0119] Areas not captured by each camera unit (areas where the source image coordinates of the virtual viewpoint transformation parameters are "nan" as shown in Figure 8B) are shown in black. An image 89 of a rearview mirror 86 is reflected in the rear virtual viewpoint image 88c.

[0120] 7, the control unit 26 reads out the virtual viewpoint images corresponding to the camera units 11 to 14 from the memory 25, compares them at the same coordinates, and detects areas where there are significant differences. Areas not captured by the camera units are not included in the comparison. In this way, the protruding areas can be detected by converting each of the captured images into a virtual viewpoint image and comparing the virtual viewpoint images.

[0121] In the example shown in FIG. 8D, when the four virtual viewpoint images 88a to 88d are compared at the same coordinates, there is a significant difference in the area of ​​the rearview mirror image 89 between the rear virtual viewpoint image 88c and the left virtual viewpoint image 88d.

[0122] 7, the control unit 26 determines that the area of ​​the rearview mirror image 89 reflected in the rear virtual viewpoint image 88c is a candidate area that is a candidate for the host vehicle structure area, taking into consideration the continuity with the surrounding area, etc. Then, in step S76, the control unit 26 displays a host vehicle structure area confirmation screen on the display unit 27.

[0123] 8E is a diagram showing an example of the host vehicle structure area confirmation screen 90 displayed in step S76, and the same components as those in FIG. 8A are assigned the same reference numerals. The host vehicle structure area confirmation screen 90 also displays an operation guide 90a, a confirm button 84c, a retry button 84d, and a cancel button 84e. The rearview mirror area is also displayed as a candidate area 92 to be added as the host vehicle structure area.

[0124] 7, the control unit 26 accepts and determines the content of a user operation, if any. When the user operates the operation unit 28 to select the Confirm button 84c, the process proceeds to step S78, where the candidate area 92 is added to the host vehicle structure area list, and the process proceeds to step S33 in FIG.

[0125] If the user selects the retry button 84d, the candidate area is discarded and the process returns to step S71. If the user selects the cancel button 84e, the process ends the flow in Fig. 7 for identifying the image area in which the vehicle structure is captured, and the process proceeds to step S33 in Fig. 3. The subsequent processes are the same as those in the first embodiment.

[0126] In the above explanation, an example in which there is one host vehicle structure has been used, but even if there are multiple host vehicle structures, the host vehicle structure can be identified in a single process. In this way, in the second embodiment, even if the user does not directly specify the host vehicle structure area on the screen, it is possible to identify structures protruding from the host vehicle, and it is possible to generate a virtual viewpoint image in which the driver will not mistake the image of the host vehicle structure for an obstacle.

[0127] <Embodiment 3> In the third embodiment, a plurality of captured images are acquired from each camera unit while the host vehicle is traveling, and analyzed to determine the host vehicle structure area. In the captured images, the area showing the surrounding environment changes as the host vehicle travels, but the area showing the host vehicle structure does not change. Therefore, by comparing the captured images with past captured images from the same camera unit, the unchanged area is identified as the host vehicle structure area.

[0128] In the third embodiment, the arrangement of the vehicle and camera unit, the configuration of the image processing system, and the processing flow explained in Figures 1 to 3 are the same as those in the first embodiment, so explanations of the same parts will be omitted. In the third embodiment, the content of step S32 in Figure 3 is different from the first and second embodiments.

[0129] 3, control unit 26 acquires the traveling state of the vehicle from vehicle control unit 29, and also acquires images after image processing of images before and after movement corresponding to camera units 11 to 14. The images after image processing are sent via image synthesis unit 23 and temporarily stored in memory 25 without undergoing virtual viewpoint conversion processing.

[0130] The control unit 26 reads out from the memory 25 the images after image processing of each camera unit before and after movement, compares them at the same coordinates, and extracts areas that have not changed due to movement. In this way, protruding areas may be detected by comparing the images taken before and after the movement of the moving object. Note that, because the brightness and color of even the vehicle's structures change depending on the weather and lighting, it is preferable to compare the shapes using known techniques such as edge detection.

[0131] Because the shape of the image changes subtly due to vibrations, etc., it is preferable to use a threshold value based on the shape overlap rate, etc. to make the judgment. Areas that are determined to have no change are then added to the list of structures of the subject vehicle. The subsequent processing is the same as in the first embodiment.

[0132] In this way, in embodiment 3, no user operation is required, and even when there are three-dimensional objects in the surrounding environment, structures protruding from the vehicle can be identified, so a virtual viewpoint image can be generated that will not cause the driver to mistake the image of the vehicle structure for an obstacle.

[0133] The present invention has been described above in detail based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0134] The present invention also includes those that realize the functions of the above-described embodiments using at least one processor or circuit such as a CPU, etc. Also, it is possible to use multiple processors to perform distributed processing.

[0135] In order to realize part or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an image processing system or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the image processing system or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0136] (Configuration 1) An image processing device comprising: a virtual viewpoint image generating means for generating a virtual viewpoint image based on a plurality of captured images from a plurality of imaging means mounted on a moving body; and an identifying means for identifying an area in the captured image in which a structure protruding from the moving body is captured as a protruding area, wherein the virtual viewpoint image generating means generates the virtual viewpoint image so as not to include the protruding area identified by the identifying means.

[0137] (Configuration 2) The image processing device according to configuration 1, wherein the virtual viewpoint image generating means generates a virtual viewpoint synthesis parameter so as not to refer to the captured image of the protruding region when generating the virtual viewpoint image.

[0138] (Configuration 3) The image processing device described in Configuration 1 or 2, characterized in that when the captured image from one of the imaging means includes the protruding region, the virtual viewpoint image generation means generates the virtual viewpoint image that does not include the protruding region using the captured image from another of the imaging means that corresponds to the protruding region.

[0139] (Configuration 4) The image processing device described in any one of configurations 1 to 3, characterized in that when the captured image from one of the imaging means includes the protruding region, the virtual viewpoint image generation means generates the virtual viewpoint image that does not include the protruding region by referring to a past captured image from the same imaging means.

[0140] (Configuration 5) The image processing device according to any one of configurations 1 to 4, wherein the virtual viewpoint image generating means makes a synthesis boundary when synthesizing the plurality of captured images a straight line.

[0141] (Configuration 6) The image processing device described in any one of configurations 1 to 5, characterized in that the virtual viewpoint image generating means is configured to ensure that the synthesis boundary when synthesizing the multiple captured images has a shape that avoids the protruding area.

[0142] (Configuration 7) The image processing device according to any one of configurations 1 to 6, characterized in that the identification means includes an operation means for inputting a user operation, and the protruding region is identified by the user operating the operation means.

[0143] (Configuration 8) The image processing device described in any one of configurations 1 to 7, characterized in that the identification means converts each of the multiple captured images into a virtual viewpoint image and detects the protruding area by comparing the virtual viewpoint images.

[0144] (Configuration 9) The image processing device according to any one of configurations 1 to 8, wherein the specifying means detects the protruding region by comparing the captured images before and after the movement of the moving object.

[0145] (Method) An image processing method comprising: a virtual viewpoint image generation step of generating a virtual viewpoint image based on a plurality of captured images from a plurality of imaging means mounted on a moving body; and an identification step of identifying an area in the captured image in which a structure protruding from the moving body is captured as a protruding area, wherein the virtual viewpoint image generation step generates the virtual viewpoint image so as not to include the protruding area identified by the identification step.

[0146] (Program) A computer program for controlling each means of the image processing device according to any one of configurations 1 to 9 by a computer. [Explanation of symbols]

[0147] 1: Vehicle 11~14: Camera unit 21: Integrated processing unit 22: Image processing unit 23: Image synthesis unit 24: OSD overlay section 25: Memory 26: Control unit 27: Display section 28:Operation unit 29: Vehicle control unit

Claims

1. a virtual viewpoint image generating means for generating a virtual viewpoint image based on a plurality of captured images from a plurality of imaging means mounted on the moving body; and specifying means for specifying, as a protruding region, a region in the captured image in which a structure protruding from the moving body is captured, The image processing device is characterized in that the virtual viewpoint image generating means generates the virtual viewpoint image so as not to include the protruding region identified by the identifying means.

2. 2. The image processing apparatus according to claim 1, wherein the virtual viewpoint image generating means generates the virtual viewpoint synthesis parameters so that the captured image of the protruding region is not referred to when generating the virtual viewpoint image.

3. The image processing device according to claim 1, characterized in that, when the captured image from one of the imaging means includes the protruding region, the virtual viewpoint image generation means generates the virtual viewpoint image that does not include the protruding region using the captured image from another of the imaging means that corresponds to the protruding region.

4. The image processing device according to claim 1, characterized in that, when the captured image from one of the imaging means includes the protruding region, the virtual viewpoint image generation means generates the virtual viewpoint image that does not include the protruding region by referring to a past captured image from the same imaging means.

5. 2. The image processing apparatus according to claim 1, wherein the virtual viewpoint image generating means generates a synthesis boundary when synthesizing the plurality of captured images so that the synthesis boundary is a straight line.

6. 2. The image processing apparatus according to claim 1, wherein the virtual viewpoint image generating means generates a synthesis boundary when synthesizing the plurality of captured images so as to avoid the protruding region.

7. the specifying means includes an operation means for inputting a user operation, 2. The image processing apparatus according to claim 1, wherein the protruding region is identified by a user operating the operating means.

8. 2. The image processing apparatus according to claim 1, wherein the specifying means converts each of the plurality of captured images into a virtual viewpoint image and detects the protruding region by comparing the virtual viewpoint images with each other.

9. 2. The image processing apparatus according to claim 1, wherein the specifying unit detects the protruding region by comparing the captured images before and after the movement of the moving object.

10. a virtual viewpoint image generating step of generating a virtual viewpoint image based on a plurality of captured images from a plurality of imaging means mounted on the moving body; and a specifying step of specifying, as a protruding region, a region in the captured image in which a structure protruding from the moving body is captured, The image processing method, wherein the virtual viewpoint image generating step generates the virtual viewpoint image so as not to include the protruding region identified in the identifying step.

11. A computer program for controlling each unit of the image processing apparatus according to any one of claims 1 to 9 by a computer.

Images