Vehicle assistance method and vehicle assistance device

Abstract

The vehicle assistance device (1) of the present invention comprises: a camera (10) that takes multiple shots of the surroundings of the vehicle at different times; and a controller (20) that processes the multiple images taken by the camera (10) at different times. The controller (20) extracts an area corresponding to a stationary object, i.e., a stationary object area, from each of the multiple images, performs alignment of the multiple images based on the amount of movement of the stationary object in the stationary object area on the image, performs super-resolution processing using the aligned multiple images, generates a super-resolution image having a resolution higher than that of the image taken by the camera (10), and recognizes a road structure based on the super-resolution image.

Description

Technical Field

[0001] The invention relates to a vehicle assistance method and a vehicle assistance device. Background Art

[0002] Conventionally, there is known a road sign recognition technology using camera images (Patent Document 1). The invention described in Patent Document 1 recognizes road signs as road structures by performing template matching on camera images.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2015-191281

[0006] Problems to be solved by the invention

[0007] However, in a region with low image resolution such as a region far from the host vehicle, it is not possible to obtain a desired result by template matching. Summary of the invention

[0008] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a vehicle assistance method and a vehicle assistance device capable of improving the recognition accuracy of road structures in a low-resolution area of ​​an image.

[0009] A vehicle assistance method according to one embodiment of the present invention extracts a region corresponding to a stationary object, i.e., a stationary object region, from each of a plurality of images captured at different times, aligns the plurality of images based on the amount of movement of the stationary object in the stationary object region on the image, performs super-resolution processing using the aligned plurality of images, generates a super-resolution image having a resolution exceeding that of an image captured by a camera, and recognizes road structures based on the super-resolution image.

[0010] Effects of the Invention

[0011] According to the present invention, it is possible to improve the recognition accuracy of road structures in a low-resolution region of an image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a configuration diagram of a vehicle support device 1 according to a first embodiment of the present invention.

[0013] Figure 2 This is a diagram for explaining an example of a stationary object region.

[0014] Figure 3 This is a flowchart for explaining an operation example of the vehicle support device 1 .

[0015] Figure 4It is a configuration diagram of a vehicle support device 2 according to a second embodiment of the present invention.

[0016] Figure 5 It is a diagram for explaining a method of dividing a stationary object region.

[0017] Figure 6 It is a diagram for explaining a method of dividing a stationary object region.

[0018] Fig. 7A This is a flowchart for explaining an operation example of the vehicle support device 2 .

[0019] Figure 7B This is a flowchart for explaining an operation example of the vehicle support device 2 .

[0020] Figure 8 It is a configuration diagram of a vehicle support device 3 according to a third embodiment of the present invention.

[0021] Fig. 9 1 is a diagram for explaining an example of the region of interest 70 .

[0022] Fig.10 This is a diagram for explaining an example of super-resolution processing.

[0023] Fig.11 This is a flowchart for explaining an operation example of the vehicle support device 3 . DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same parts are denoted by the same reference numerals and their description is omitted.

[0025] Reference Figure 1 An example configuration of the vehicle assistance device 1 will be described. Figure 1 As shown, the vehicle assistance device 1 includes a camera 10 , a storage device 11 , a controller 20 , a steering actuator 12 , an accelerator pedal actuator 13 , and a brake actuator 14 .

[0026] The vehicle assistance device 1 can be mounted on a vehicle with an automatic driving function or a vehicle without an automatic driving function. In addition, the vehicle assistance device 1 can also be mounted on a vehicle that can switch between automatic driving and manual driving. In addition, the automatic driving function can also be a driving assistance function that automatically controls only part of the vehicle control functions such as steering control, braking force control, and driving force control to assist the driver in driving. In this embodiment, the case where the vehicle assistance device 1 is mounted on a vehicle with an automatic driving function is described.

[0027] The camera 10 has imaging elements such as CCD (charge-coupled device) and CMOS (complementary metal oxide semiconductor). The location where the camera 10 is installed is not particularly limited. As an example, the camera 10 is installed in front, on the side, or on the rear of the vehicle. The camera 10 continuously captures the surroundings of the vehicle at a specified period. That is, the camera 10 obtains multiple images captured at different times. The camera 10 detects the conditions of objects (pedestrians, bicycles, motorcycles, other vehicles, etc.) existing around the vehicle and the information in front of the vehicle (dividing lines, traffic lights, signs, crosswalks, intersections, etc.). The images captured by the camera 10 are stored in the storage device 11.

[0028] The storage device 11 mainly stores images captured by the camera 10. As an example, the storage device 11 is composed of a HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0029] The controller 20 is a general-purpose microcomputer having a CPU (central processing unit), a memory, and an input / output unit. A computer program for functioning as the vehicle auxiliary device 1 is installed in the microcomputer. By executing the computer program, the microcomputer functions as a plurality of information processing circuits possessed by the vehicle auxiliary device 1. In addition, here, an example of realizing a plurality of information processing circuits possessed by the vehicle auxiliary device 1 by software is shown, but of course, dedicated hardware for performing each information processing shown below can also be prepared to constitute the information processing circuit. In addition, a plurality of information processing circuits can also be constituted by separate hardware. As an example of a plurality of information processing circuits, the controller 20 comprises: an optical flow calculation unit 21, a distance calculation unit 22, a behavior calculation unit 23, a stationary object area determination unit 24, an image movement unit 25, a super-resolution processing unit 26, a road structure recognition unit 27, and a vehicle control unit 28.

[0030] The optical flow calculation unit 21 uses the past image and the current image to calculate the extent to which the object in the image has moved. The optical flow uses a vector to represent the movement of the object in the image. An example of the calculation method is described. The image taken at time T1 is set to image I1, and the image taken at time T2 is set to image I2. Assume that time T1<time T2. The optical flow calculation unit 21 divides the image I1 into arbitrary sizes. The arbitrary size is not particularly limited. Here, the horizontal size of the image is set to N pixels, and the vertical size of the image is set to M pixels. The arbitrary size is N×M. The optical flow calculation unit 21 compares the area (N×M) of image I1 and the area (N×M) of image I2. As comparison methods, SSD (Sum of SquaredDifference), SAD (Sum of Absolute Difference), ZNCC (Zero-mean Normalized Cross-Correlation), etc. are known. These methods are well-known technologies, so the description is omitted. The optical flow calculation unit 21 detects feature points on the current image corresponding to the same object in the real space as the feature points on the past image as related feature points. The optical flow calculation unit 21 calculates a combination of the feature points on the past image and the feature points on the current image that are related to each other as the optical flow.

[0031] The distance calculation unit 22 calculates the distance from the camera 10 to the object corresponding to the feature point based on the visual difference (offset) between the feature point on the mutually related past image and the feature point on the current image. In other words, the distance calculation unit 22 calculates the distance from the vehicle to the object corresponding to the feature point. The distance calculation unit 22 calculates the three-dimensional coordinates in the actual space corresponding to the feature points based on the visual difference between the feature points on the mutually related past image and the feature points on the current image. In addition, the distance calculation unit 22 can also combine a radar or a laser radar with the camera 10 to calculate the distance to the object.

[0032] The behavior calculation unit 23 calculates the three-axis translational motion and three-axis rotational motion of the vehicle. The three-axis translational motion is the motion of the vehicle in the front-back direction, the vehicle width direction, and the up-down direction. The three-axis rotational motion is the rotational motion around three axes including the roll axis, the pitch axis, and the yaw axis. These motions are calculated using wheel speed sensors, acceleration sensors, rotation sensors, etc. Alternatively, these motions are calculated using images. As a calculation method using images, there is known non-patent document 1 "KITT, Bernd; G EIGER, Andreas; LATEGAHN, Henning. Visual odometry based on stereo imagesequences with ransac-based outlier rejection scheme. In: 2010 ieee intelligent vehicles symposium. IEEE, 2010. p. 486-492.".

[0033] The stationary object region determination unit 24 combines the optical flow calculated by the optical flow calculation unit 21, the calculation result of the distance calculation unit 22, and the calculation result of the behavior calculation unit 23, and calculates the three-dimensional movement amount (three-dimensional motion vector) excluding the behavior of the vehicle. The stationary object region determination unit 24 compares the calculated absolute movement amount with the threshold. If the movement amount is below the threshold, it is determined that there is no moving object such as other vehicles in the region. That is, if the movement amount is below the threshold, it is determined that the region is a stationary object region.

[0034] The image moving unit 25 calculates the amount of movement between the images, and moves the images by only the calculated amount of movement for alignment. The amount of movement of the image is the amount of movement between the two images. In such a movement amount calculation, SSD, SAD, ZNCC, etc. are used in the same way as optical flow. Specifically, the image moving unit 25 calculates the similarity by shifting one pixel at a time in the up, down, left, and right directions within the search range, and calculates the amount of movement to the place with the highest similarity as the movement amount. In order to compare two images, an image is required as a reference. Therefore, the image I1 taken at time T1 is used as the reference image. Hereinafter, image I1 is referred to as the reference image R1. The reference image R1 has A pixels in the horizontal direction and B pixels in the vertical direction. As representative coordinates of the reference image R1, the coordinates of the center are set to (X1, Y1).

[0035] Next, in the image I2 captured at a time different from the time T1, that is, at the time T2, a predetermined search range is set based on the coordinates (X1, Y1). For example, if S pixels are set in the top, bottom, left, and right directions, the search range becomes the range from (X1-S, Y1-S) to (X1+S, Y1+S). The coordinates of the center of the image I2 are (a, b). Comparing the reference image R1 and the image I2, the center coordinates (aM, bM) with the highest pixel similarity become the moving destination of (X1, Y1). The moving destination of (X1, Y1) is called (X2, Y2).

[0036] Similarly, in the image I3 captured at time T3 which is different from time T1 and time T2, the search range is set based on (X2, Y2). The reference image R1 and image I3 are compared, and (X3, Y3) is set as the moving destination of (X2, Y2). At this time, the image moving unit 25 may also update the image at that time to the reference image R1 using a predetermined number of frames. In the similarity calculation, all pixels of the reference image R1, i.e., A×B pixels, are used to compare the brightness values ​​of the same coordinates of the reference image R1 and the comparison image. The comparison images are image I2 and image I3.

[0037] If it is SAD, the sum of the absolute values ​​of the brightness differences is calculated, and if it is SSD, the sum of the squares of the brightness differences is calculated. In addition, if the similarity is lower than the threshold value in any center pixel within the search range, the movement amount calculation is terminated. In the present embodiment, in order to calculate the movement amount of a road structure as a stationary object, the area where the moving object exists is excluded to calculate the similarity. That is, the similarity is calculated only in the stationary object area. Here, the stationary object area determined by the stationary object area determination unit 24 in the reference image R1 is defined as a reference area. In addition, the stationary object area determined by the stationary object area determination unit 24 in each comparison image is defined as a comparison area. Pixels overlapping the reference area and the comparison area are used for similarity calculation. As a result, the obtained movement amount is in pixel units. In order to calculate the movement amount of pixels below the decimal point, an approximate calculation using the similarity calculation result is performed. In such an approximate calculation, for example, the method of non-patent document 2 is used. Non-patent document 2: "Motoki Arai, Kazuhiko Washimi, and Takashi Matsuyama, Optimization of correlation function and sub-pixel estimation method in block matching of images, Information Processing Society Research Report Computer Vision and Image Media (CVIM) 2004.40 (2004-CVIM-144), (2004): 33-40."

[0038] The desired magnification is set to N. In this case, the image moving unit 25 rounds the movement amount to one-Nth pixel. The image moving unit 25 sets the movement amount (MX2, MY2) ... (MXn, MYn) from the center coordinates (X1, Y1) at time T1 at each time T2 ... Tn. Then, the image moving unit 25 approximately magnifies the images J1, J2 ... Jn with a size of A pixels in the horizontal direction and B pixels in the vertical direction centered on the center coordinates (X1, Y1) and (X2, Y2) ... (Xn, Yn) at each time by N times. Thus, images K1, K2 ... Kn are generated. As a method of approximate magnification, there can be cited: nearest neighbor interpolation (NearestNeighbor), bilinear interpolation (Bilinear), bicubic interpolation (Bicubic), etc. The movement amount as a unit of one-Nth pixel is multiplied by N times and the approximately magnified image is returned. The image moving unit 25 moves the comparison image relative to the reference image in such a way that the center coordinates are consistent to perform alignment.

[0039] The super-resolution processing unit 26 performs super-resolution processing using the image aligned by the image moving unit 25 to generate an image with high resolution. Here, the number of images required for super-resolution processing is set to Z. The super-resolution processing unit 26 calculates the brightness value of each pixel by adding and averaging the images K1, K2...Kz that are magnified N times by the image moving unit 25 and have the same center coordinates. However, when the super-resolution processing unit 26 cannot track from the reference image R1 without collecting Z images, or when the area for super-resolution processing is completely filled with moving objects, the super-resolution processing is terminated. Super-resolution processing refers to a technology for generating a high-resolution image by referring to multiple images. In this embodiment, the resolution of the reference image R1 is increased by aligning the comparison image on the reference image R1. In other words, an image with a resolution exceeding that of the image taken by the camera 10 is generated. The image generated by the super-resolution processing unit 26 is output to the road structure recognition unit 27.

[0040] The road structure recognition unit 27 uses the image input from the super-resolution processing unit 26 to recognize the road structure. In the present embodiment, the road structure is defined as a stationary object. Specifically, the road structure includes structures such as lanes, parking lanes, crosswalks, road markings such as arrows, roadside strips, curbs, signs, and traffic lights. As an example of a method for recognizing a road structure, semantic segmentation is used to recognize what kind of object each pixel is. The road structure recognition unit 27 outputs the recognition result to the vehicle control unit 28.

[0041] The vehicle control unit 28 controls the steering actuator 12, the accelerator pedal actuator 13, and the brake actuator 14 using the road structure identified by the road structure identification unit 27. The stationary object region becomes clear by super-resolution processing, thereby improving the recognition accuracy of the road structure.

[0042] Next, refer to Figure 2 An example of a stationary object region will be described.

[0043] Figure 2 The image 40 shown is an image captured by the camera 10 while the vehicle is traveling. Figure 2 In the scene shown, the host vehicle is passing through an intersection. From the perspective of the host vehicle, there is another vehicle 42 in front of the left. There is another vehicle 43 in front of the other vehicle 42. There are other vehicles 44 and 45 in the opposite lane. From the perspective of the host vehicle, there is a pedestrian 41 in the left rear.

[0044] The stationary object region determination unit 24 determines the stationary object region using the above-mentioned method. Figure 2 In the example shown, the stationary object region refers to a region excluding other vehicles 42 to 45 and pedestrian 41. When it is determined that a stationary object region exists, the image is processed as a reference image R1. Figure 2 The image 40 shown is set as the reference image R1. Assuming that the image 40 is an image captured at time T1, images captured after time T1, that is, at times T2 ... Tn, are set as comparative images.

[0045] Next, refer to Figure 3 The flowchart shown is for explaining an operation example of the vehicle assistance device 1 .

[0046] In step S101, the surroundings of the vehicle are photographed by the camera 10 mounted on the vehicle. The process proceeds to step S103. If the reference image R1 is set ("Yes" in step S103), the process proceeds to step S111. On the other hand, if the reference image R1 is not set ("No" in step S103), the process proceeds to step S105.

[0047] In step S105, the stationary object region determination unit 24 determines whether the image includes a region where no moving object exists. Optical flow or the like is used in this determination. An example of a moving object is Figure 2 The other vehicles 42 to 45 and the pedestrian 41 shown are shown. If the image contains an area where no moving object exists ("Yes" in step S107), the image is set as the reference image R1 (step S109). On the other hand, if the image does not contain an area where no moving object exists ("No" in step S107), the processing ends. In addition, in step S109, the still image area is set as the reference area.

[0048] In step S111, the image moving unit 25 calculates the similarity between the reference image R1 and a reference image captured at a different time from the reference image R1. As described above, the image moving unit 25 calculates the similarity of an area where the reference area and the comparison area overlap within a specified range. The comparison area refers to a stationary area in each comparison image. The brightness values ​​of the same coordinates of the reference area and the comparison area are compared as the similarity. When the similarity is above the threshold value ("Yes" in step S113), the process proceeds to step S115. On the other hand, when the similarity is less than the threshold value ("No" in step S113), the process ends.

[0049] In step S115, the image moving unit 25 calculates the amount of movement between the images as described above, and moves the images by the calculated amount of movement to perform alignment. In step S117, the super-resolution processing unit 26 generates an image with high resolution using the image aligned by the image moving unit 25. In step S119, the image generated by the super-resolution processing unit 26 is output to the road structure recognition unit 27. In step S121, the road structure recognition unit 27 recognizes the road structure using the image input from the super-resolution processing unit 26. In step S123, the vehicle control unit 28 controls the steering actuator 12, the accelerator pedal actuator 13, and the brake actuator 14 using the road structure recognized by the road structure recognition unit 27. The stationary object area becomes clear through super-resolution processing. As a result, the recognition accuracy of the road structure is improved.

[0050] (Effect)

[0051] As described above, according to the vehicle support device 1 of the first embodiment, the following effects can be obtained.

[0052] The vehicle assistance device 1 includes: a camera 10 that takes multiple shots of the surroundings of the vehicle at different times; and a controller 20 that processes multiple images taken by the camera 10 at different times. The controller 20 extracts a stationary object region, which is a region corresponding to a stationary object, from each of the multiple images. The controller 20 performs alignment of the multiple images based on the amount of movement of the stationary object in the stationary object region on the image. The controller 20 performs super-resolution processing using the aligned multiple images to generate a super-resolution image with a resolution exceeding that of the image taken by the camera 10. The controller 20 recognizes road structures based on the super-resolution image. Thus, the amount of movement between images is calculated in a state where moving objects in the image are excluded. In the background where a road structure as a stationary object exists, a super-resolution image with a resolution exceeding that of the image taken by the camera 10 is generated. Thus, the recognition accuracy of road structures can be improved in areas with low resolution of the image.

[0053] In the above-mentioned prior art, it is considered to perform super-resolution processing in order to detect distant road structures that cannot be detected by template matching. In the case where a moving object exists in the area where super-resolution processing is performed, the moving object is tracked. As a result, super-resolution processing cannot be performed on the road structure to be identified, and there is a possibility that the background is blurred. According to the first embodiment, since the moving object is excluded, super-resolution processing can be performed on the road structure, and the background becomes clear. As a result, the recognition accuracy of the road structure is improved.

[0054] In addition, the controller 20 sets a reference image that serves as a reference when performing super-resolution processing from a plurality of images. The controller 20 calculates the similarity of the reference image and the area where the image other than the reference image overlaps, and performs alignment using the movement amount based on the similarity. Thus, the similarity is calculated using only the stationary object area in both the reference image and the image at each time. Thus, even if the stationary object area blocked by the moving object is different depending on the time, the similarity can be calculated.

[0055] In addition, the controller 20 replaces the image with the super-resolution image for each divided area to generate a single image. Then, the controller 20 recognizes the road structure based on the generated single image. Therefore, if the super-resolution image is generated only for a part, the recognition accuracy of the road structure in the part is improved. In addition, the image to be replaced is not particularly limited, and it is, for example, a reference image.

[0056] In addition, the controller 20 selects a region having a specific amount of movement from the divided plurality of regions. This reduces the computational load required for super-resolution processing. The region having a specific amount of movement is, for example, the right half of the region as viewed from the vehicle.

[0057] In addition, the controller 20 extracts the stationary object area based on the three-dimensional motion vector of the feature point measured by at least one of the camera 10, the laser radar and the radar and the sensor motion of the vehicle. By subtracting the behavior of the host vehicle from the measured three-dimensional motion vector, the area where the moving object exists can be extracted. The stationary object area can be extracted by excluding the area where the moving object exists. In addition, the sensor motion of the host vehicle refers to the behavior of the host vehicle detected by the wheel speed sensor, the acceleration sensor, the rotation sensor, etc.

[0058] In addition, the controller 20 sets the number Z of images to be used for super-resolution processing. This ensures the necessary number of images with a desired magnification. When the number of images captured by the camera 10 is equal to or greater than the set number Z, the controller 20 performs super-resolution processing.

[0059] In addition, the controller 20 controls the driving of the host vehicle based on the recognized road structures. Thus, the driving is controlled based on road markings, traffic lights, signs, etc. far from the host vehicle, thereby achieving high-precision automatic driving.

[0060] (Second Embodiment)

[0061] Next, refer to Figure 4 A vehicle assistance device 2 according to a second embodiment of the present invention is described. The second embodiment differs from the first embodiment in that the controller 20 further includes a feature point calculation unit 29 and a division unit 30. For structures that are repeated in the first embodiment, reference numerals are used and their descriptions are omitted. The following description will focus on the differences.

[0062] The feature point calculation unit 29 calculates the amount of movement of the feature point in the stationary object region. The feature point is a pixel having a feature that can be distinguished from surrounding pixels in any image.

[0063] The division unit 30 divides the stationary object area based on the movement amount calculated by the feature point calculation unit 29. The movement amount of the feature point may not be constant in all the stationary object areas. The desired magnification is set to N. As in the first embodiment, the center coordinates are aligned by moving only the movement amount in units of one-N. The division unit 30 divides the area with a movement amount of 0 into A0, the area with a movement amount of one-N into A1, the area with a movement amount of two-N into A2... and the area with a maximum movement amount of M-N into AM. The image moving unit 25 specifies one or more areas in the divided A0, A1...AM, and aligns the image in a manner that the center coordinates are consistent. The super-resolution processing unit 26 uses the image aligned by the image moving unit 25 to generate an image with high resolution. In the super-resolution processing, a super-resolution image is generated in each divided area, and the generated super-resolution image is used for replacement to generate an image.

[0064] Next, refer to Figures 5-6 The following describes how to divide the stationary object area. Figure 5 The image 50 is a portion of the image 40 shown in the figure. Of course, the entire image 40 can also be divided by the method described below. Figure 5 The image 40 shown is Figure 2 The images 40 shown are identical.

[0065] like Figure 6 As shown in FIG. 5 , a stationary object region is extracted from the image 50. The stationary object region refers to the region excluding other vehicles 43, 44, and 46. The method for extracting the stationary object region is the same as that of the first embodiment. Figure 6On the lower side, symbol 60 represents a set of feature points of other vehicles 43. Symbol 61 represents a set of feature points of other vehicles 44. Symbol 62 represents a set of feature points of other vehicles 46. Symbols 63 and 64 represent a set of feature points of the crosswalk. Symbol 65 represents a set of feature points of the area other than symbols 60 to 64. As described above, since symbols 60, 61, and 62 are removed in the determination of the stationary object area, the feature points remaining in image 50 are symbols 63, 64, and 65.

[0066] At the feature points of the same road structure, the movement amount is substantially the same. The division unit 30 divides the stationary object region using the feature points with the same movement amount calculated by the feature point calculation unit 29. Thus, the stationary object region is divided into a region indicated by a symbol 63, a region indicated by a symbol 64, and a region indicated by a symbol 65. That is, in the image 50, the stationary object region is divided into three regions. The image movement unit 25 performs alignment in each of the divided regions. The subsequent processing is the same as that of the first embodiment.

[0067] Next, refer to Figures 7A-7B The flowchart shown in FIG. 1 is used to explain an example of the operation of the vehicle assistance device 2. The processing of steps S201 to S209, S219 to S225, and S233 to S237 is similar to that of FIG. Figure 3 The processes shown in steps S101 to S123 are the same, and thus description thereof will be omitted.

[0068] In S211, the feature point calculation unit 29 calculates the movement amount of the feature points in the small area of ​​the stationary object area. As an example, the small area is a small area containing several feature points. Let the desired magnification be N. When the difference between the minimum and maximum values ​​of the movement amount is greater than the inverse of the magnification (1 / N) ("Yes" in step S213), the division unit 30 divides the area into regions for every 1 / N of the movement amount. Each divided area is set as a reference image. In addition, the representative coordinates (center coordinates) of the set reference image are set (step S215). Figure 6 The areas indicated by reference numerals 63, 64, and 65 correspond to the divided areas. Figure 6 In step S214, the regions indicated by reference numerals 63, 64, and 65 are set as reference images. On the other hand, in the case of "No" in step S213, all reference regions are set as reference images (step S217).

[0069] In the case of "No" in step S221, the divided areas and the reference image are deleted. In step S227, super-resolution images are generated in each divided area, and the generated super-resolution images are used for replacement to generate a single image. If "Yes" in step S231, the process proceeds to step S233. If "No" in step S231, the process returns to step S219.

[0070] (Effect)

[0071] As described above, according to the vehicle support device 2 of the second embodiment, the following effects can be obtained.

[0072] The controller 20 calculates the movement amount of the feature points on the image in each still object area of ​​the plurality of images taken at different times, and divides the image into each area where the movement amount of the feature points is similar. Then, the controller 20 performs super-resolution processing on each divided area. In the generation of super-resolution images, the image is generated with the best accuracy when the image has the same movement amount as a whole. However, in most cases, the movement of each part is different. Therefore, the controller 20 divides the image into parts with the same movement and performs super-resolution processing. Thus, a high-resolution image is generated as a whole.

[0073] When the desired magnification is defined as N, the controller 20 sets one-Nth pixel in the up, down, left, and right directions of the image as a threshold value of the movement amount, and divides the area when the movement amount is greater than the threshold value. Therefore, for the desired magnification, the area is divided into necessary resolution units, thereby generating a super-resolution image with the desired magnification.

[0074] (Third Embodiment)

[0075] Next, refer to Figure 8 A vehicle assistance device 3 according to a third embodiment of the present invention will be described. The third embodiment differs from the first embodiment in that the controller 20 further includes a region of interest setting unit 31. Configurations that overlap with those of the first embodiment are referenced with reference numerals and their descriptions are omitted. The following description will focus on the differences.

[0076] The region of interest setting unit 31 sets a region of interest. The region of interest is a region to be subjected to super-resolution processing.

[0077] Next, refer to Fig. 9 An example of a region of interest will be described. Fig. 9 The image 40 shown is Figure 2 The images 40 shown are identical. Fig. 9 The symbol 70 shown indicates a region of interest. Fig. 9 It can be seen that the region of interest 70 is smaller than the image 40. In the first embodiment, the super-resolution processing is performed on the entire still object region of the image 40, but in the third embodiment, the super-resolution processing is performed on the still object region of the region of interest 70. This reduces the computational load compared to the first embodiment.

[0078] The method for setting the region of interest 70 is not particularly limited. For example, an area far from the vehicle is set as the region of interest 70. Fig.10As shown, super-resolution processing makes such distant areas clear. Fig.10 The upper side is the image before super-resolution processing, and the lower side is the image after super-resolution processing. Super-resolution processing improves the recognition accuracy of road markings, traffic lights, signs, etc. far away from the vehicle (in Fig.10 In the figure, the road markings become clear. Thus, the vehicle control unit 28 can quickly perform controls suitable for road markings, traffic lights, and signs. Thus, high-precision automatic driving is achieved. In addition, the focus area 70 can also be set in the upper half of the image 40. The upper half of the image 40 means an area farther from the vehicle than the lower half.

[0079] Next, refer to Fig.11 The flowchart shown in FIG. 1 is used to explain an example of the operation of the vehicle assistance device 3. The processing of steps S301, S305 to S325 is similar to that of Figure 3 The processes shown in steps S101 to S123 are the same, and thus description thereof will be omitted.

[0080] In step S303, the region of interest setting unit 31 sets a region of interest 70 smaller than the image 40 (see Fig. 9 ).

[0081] (Effect)

[0082] As described above, according to the vehicle support device 3 of the third embodiment, the following effects can be obtained.

[0083] The controller 20 sets the attention region 70 in the area far from the host vehicle, and extracts the stationary object area in the attention region 70. Thus, only the area to be paid attention to in the image (for example, the area far from the host vehicle) can be made high-resolution.

[0084] The focus area 70 includes at least one of the lane demarcation line indicating the lane in which the host vehicle is traveling, the traffic light in the direction in which the host vehicle is traveling, and the sign in the direction in which the host vehicle is traveling. Thus, by setting the focus area 70 far in the direction in which the host vehicle is traveling, the recognition accuracy of the lane demarcation line, traffic light, sign, etc. far from the host vehicle is improved.

[0085] Each function described in the above embodiments may be implemented by one or more processing circuits. The processing circuit includes a programmed processing device such as a processing device including a circuit. The processing circuit also includes a device such as an application-specific integrated circuit (ASIC) or circuit components configured to perform the described functions.

[0086] As described above, the embodiments of the present invention are described, but it should not be understood that the description and drawings constituting part of this disclosure limit the present invention. Based on this disclosure, various alternative embodiments, examples, and application techniques will be apparent to those skilled in the art.

[0087] As described above, when the number of images required for super-resolution processing is set to Z, images with aligned center pixels are stored during Z frames to perform super-resolution processing. However, when the number of images captured by the camera 10 does not satisfy the set number Z, and when there is no region with a similarity greater than the threshold value within the search range for calculating the movement amount in an image at a time different from the reference image, the super-resolution processing unit 26 may perform super-resolution processing using images that have been aligned up to that time. This can improve the recognition accuracy of road structures.

[0088] In addition, when the number of images captured by the camera 10 does not satisfy the set number Z, and when there is no region having a similarity greater than a threshold value within the search range for calculating the movement amount in an image captured at a different time from the reference image, the super-resolution processing unit 26 may wait for the super-resolution processing for a predetermined time until the set number Z is collected. Thus, even if a moving object passes through the entire region of interest 70, the super-resolution processing is performed without interruption.

[0089] Explanation of symbols

[0090] 1, 2, 3: Vehicle auxiliary devices

[0091] 10: Camera

[0092] 11: Storage device

[0093] 12: Steering gear actuator

[0094] 13: Accelerator pedal actuator

[0095] 14: Brake actuator

[0096] 20: Controller

[0097] 21: Optical flow calculation department

[0098] 22: Distance calculation unit

[0099] 23: Behavioral Computing Department

[0100] 24: Stationary object area determination unit

[0101] 25: Image moving unit

[0102] 26: Super Resolution Processing Unit

[0103] 27: Road structure identification department

[0104] 28: Vehicle Control Department

[0105] 29: Feature point calculation unit

[0106] 30: Division

[0107] 31: Focus area setting department

[0108] 70: Area of ​​concern

Claims

1. A vehicle assistance method, wherein a camera is used to take multiple shots of the surroundings of the vehicle at different times, and a controller is used to process the multiple images taken by the camera at different times, wherein: The controller performs the following processing: extracting a region corresponding to a still object, that is, a still object region, from each of the plurality of images, performing alignment of the plurality of images based on the amount of movement of the stationary object in the stationary object area on the image, performing super-resolution processing using the plurality of aligned images to generate a super-resolution image having a higher resolution than the image captured by the camera, Road structures are recognized based on the super-resolution image.

2. The vehicle assistance method according to claim 1, characterized in that: The controller performs the following processing: calculating the movement amount of the feature points on the images in the stationary object regions of the respective images, The image is divided into regions where the movement amount of the feature points is similar. The super-resolution processing is performed on each divided area.

3. The vehicle assistance method according to claim 1 or 2, characterized in that: The controller performs the following processing: A focus area is set in an area far from the host vehicle. A stationary object region in the region of interest is extracted.

4. The vehicle assistance method according to claim 1, characterized in that: The controller performs the following processing: setting a reference image to be used as a basis for performing the super-resolution processing from among the plurality of images, The similarity between the reference image and an overlapping region of a plurality of images other than the reference image is calculated, and the alignment is performed using a shift amount based on the similarity.

5. The vehicle assistance method according to claim 2, characterized in that: The controller performs the following processing: For each divided area, the image is replaced by the super-resolution image to generate a single image, The road structure is recognized based on the one image.

6. The vehicle assistance method according to claim 2, characterized in that: The controller selects a region having a specific movement amount from among the divided plurality of regions.

7. The vehicle assistance method according to claim 3, characterized in that: The target area includes at least one of a lane dividing line, a traffic light, and a sign indicating a lane in which the host vehicle is traveling.

8. The vehicle assistance method according to claim 1, characterized in that: The controller extracts the stationary object region based on a three-dimensional motion vector of a feature point measured by at least one of the camera, the laser radar, and the radar and a sensor motion of the host vehicle.

9. The vehicle assistance method according to claim 2, characterized in that: When the desired magnification is defined as N, The controller sets one-Nth of the pixels in the up, down, left, and right directions of the image as a threshold value of the movement amount, and divides the area when the movement amount is greater than the threshold value.

10. The vehicle assistance method according to claim 1, characterized in that: The controller performs the following processing: Setting the number of images used in the super-resolution processing, The super-resolution processing is performed when the number of images captured by the camera is equal to or larger than a set number.

11. The vehicle assistance method according to claim 4, characterized in that: The controller performs the following processing: Setting the number of images used in the super-resolution processing, When the number of images taken by the camera does not satisfy the set number, and when there is no area with a similarity above a threshold value within the search range for calculating the movement amount in an image taken at a time different from the reference image, the super-resolution processing is performed using the image that has been aligned until that time.

12. The vehicle assistance method according to claim 4, characterized in that: The controller performs the following processing: Setting the number of images used in the super-resolution processing, When the number of images taken by the camera does not meet the set number, and when there is no area with a similarity above a threshold value within the search range for calculating the movement amount in an image taken at a different time from the reference image, wait for the super-resolution processing for a specified time until the set number of images is collected.

13. The vehicle assistance method according to claim 1 or 2, characterized in that: The controller controls the travel of the host vehicle based on the recognized road structure.

14. A vehicle auxiliary device, characterized in that: have: A camera that takes multiple shots of the surroundings of the vehicle at different times; a controller for processing a plurality of images captured by the camera, The controller performs the following processing: extracting a region corresponding to a still object, that is, a still object region, from each of the plurality of images, performing alignment of the plurality of images based on the amount of movement of the stationary object in the stationary object area on the image, performing super-resolution processing using the plurality of aligned images to generate a super-resolution image having a higher resolution than the image captured by the camera, Road structures are recognized based on the super-resolution image.

Images