Image trimming device and image trimming method

Abstract

To provide an image trimming device and an image trimming method which can prevent variation of quality.SOLUTION: An image processing device 1 which is an image trimming device includes a processing unit 10 which performs processing of an image obtained by image-capturing with a camera 110 placed in a vehicle SC. The processing unit 10 executes: calculation processing which calculates a projective transformation factor between an oblique image obtained by the camera 110 image-capturing a road surface on which the vehicle SC travels and a vertical image relative to the oblique image; acquisition processing which acquires a target oblique image as a target to be processed from the camera 110; and trimming processing which sets as a trimming range Rt a range sandwiched between a pair of y coordinates in a target oblique image corresponding to an upper end and a lower end of the vertical image in a y direction which is a travelling direction of the vehicle SC on the basis of the projective transformation factor and cuts out the trimming range Rt from the target oblique image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image cropping device and an image cropping method. [Background technology]

[0002] Patent document 1 describes that the system receives inputs such as data on images of the road surface taken by an on-board camera, location information (position data) of the camera that took the image, the orientation of the on-board camera, the height of the on-board camera from the road surface, information on the model of the on-board camera, and information on the analyst who will be performing the analysis work, and outputs, as output information, an image of pavement cracks, which is the captured image plus an image showing the shape of the cracks in the pavement. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-107683 Summary of the Invention [Problem to be solved by the invention]

[0004] As mentioned above, in recent years, understanding road surface conditions based on images obtained by capturing road surfaces with vehicle-mounted cameras has become important for purposes such as road maintenance and the realization of automated vehicle driving. However, images captured by vehicle-mounted cameras are perspective projection images, in which distant areas in the direction of vehicle travel are projected small, and the farther away from the capturing point, the greater the influence of lens distortion. Therefore, from the perspective of photogrammetry, it is desirable to use the image captured by the vehicle-mounted camera from the center to the bottom, where lens distortion is minimal. However, it is also necessary to avoid capturing the vehicle body (hood area) in the image depending on the vehicle model used and the tilt of the camera.

[0005] Therefore, in order to effectively utilize images captured by an on-board camera, it is desirable to trim an appropriate range from the image. Patent Document 1 also mentions that the captured image may be an image obtained by trimming a predetermined range from the original image captured by the on-board camera. However, Patent Document 1 does not disclose any specific techniques for trimming on-board camera images, and the details are unclear. Therefore, currently, it is considered that on-board camera images are manually trimmed by an operator, but in this case, the quality of the finished product varies depending on the operator's experience and ability.

[0006] Therefore, an object of the present disclosure is to provide an image trimming device and an image trimming method that can suppress quality fluctuations. [Means for solving the problem]

[0007] The image cropping device according to the present disclosure includes a processing unit that processes images obtained by imaging with a camera installed in a vehicle, and the processing unit performs a calculation process that calculates a projection transformation coefficient between an oblique image including an image of the road surface obtained by imaging the road surface on which the vehicle is traveling with the camera and a vertical image of the road surface corresponding to the oblique image; an acquisition process that acquires a target oblique image, which is the oblique image to be processed, from the camera; and, after the calculation process and the acquisition process, a cropping process that sets the range of the target oblique image sandwiched between a pair of y coordinates of the target oblique image corresponding to the upper and lower ends of the vertical image in the y direction, which is the direction of travel of the vehicle, as a cropping range based on the projection transformation coefficient, and cuts out the cropping range from the target oblique image.

[0008] The image trimming method according to the present disclosure is an image trimming method for performing a trimming process on an image obtained by imaging with a camera installed in a vehicle, and includes a calculation step of calculating a projection transformation coefficient between an oblique image including an image of the road surface obtained by imaging the road surface on which the vehicle is traveling with the camera and a vertical image of the road surface corresponding to the oblique image; an acquisition step of acquiring a target oblique image, which is the oblique image to be processed, from the camera; and a trimming step of, after the calculation step and the acquisition step, setting the range of the target oblique image sandwiched between a pair of y coordinates of the target oblique image corresponding to the upper and lower ends of the vertical image in the y direction, which is the direction of travel of the vehicle, as a trimming range and cutting out the trimming range from the target oblique image based on the projection transformation coefficient.

[0009] According to the inventors' findings, the area in the oblique image corresponding to the range between the upper and lower ends in the y direction (vehicle travel direction) of the vertical image, which is in a projective transformation relationship with the oblique image, is an area near the center of the image with little distortion and where the vehicle body is unlikely to be captured. Therefore, in the device and method disclosed herein, first, a projective transformation coefficient is calculated between an oblique image obtained by capturing a road surface with an onboard camera and a vertical image corresponding to the oblique image. Then, based on the calculated projective transformation coefficient, a cropping range is determined for a target oblique image, which is the oblique image to be actually trimmed, by dividing the range between a pair of y coordinates corresponding to the upper and lower ends of the vertical image in the y direction. This ensures high quality of the product obtained by trimming. Furthermore, by performing quantitative trimming using numerical calculations, the quality of the product is not affected by the experience or ability of the operator, resulting in a stable, high-quality product with reduced quality variations. The oblique image used to calculate the projective transformation coefficient and the target oblique image to be trimmed may be the same or different.

[0010] In the image cropping device according to the present disclosure, the processing unit may, in the calculation process, set four key points, A, B, C, and D, on the oblique image and calculate projective transformation coefficients based on the four key points, and when the vehicle width direction is defined as the x direction, the processing unit may set points A and B of the four key points on a first line parallel to the x axis that defines the x direction, and set points C and D of the four key points on a second line parallel to the x axis and different from the first line. In this case, the y coordinates of points A and B, and points C and D are the same, so the processing load related to the calculation of the projective transformation coefficients is reduced.

[0011] In the image cropping device according to the present disclosure, the processing unit may, in the calculation process, set point C at the bottom end of the oblique image in the y direction and on the vanishing line connecting the vanishing point in the perspective projection and point A, set point D at the end of the oblique image in the x direction and on the same line parallel to the x axis as point C, and set point B on the vanishing line connecting the vanishing point in the perspective projection and point D. In this case, the processing load related to the calculation of the projective transformation coefficients is further reduced.

[0012] In recent years, high-sensitivity cameras with high sensitivity and a wide dynamic range have made it possible to capture high-quality images, even at night, including 4K video. In particular, the ability to capture video at night using high-resolution cameras such as 4K and 8K makes it possible to avoid the effects of sunlight and congestion of people and vehicles, which is expected to improve the efficiency of capturing road surface conditions using in-vehicle cameras. However, one of the light sources used in night-time photography is the car's headlights, and the illumination from the headlights creates a brightness distribution in oblique images. Therefore, to make more effective use of in-vehicle camera images, it is desirable to crop the images taking into account the brightness distribution of night-time oblique images.

[0013] Therefore, in the image trimming device according to the present disclosure, when the target oblique image includes a nighttime image captured by a camera at night, the processing unit may further perform an adjustment process after the trimming process to adjust the trimming range so that the trimming range corresponds to a relatively bright bright area in the nighttime image. In this case, the trimming range is adjusted to correspond to the bright area in the oblique image captured at night. Therefore, quality degradation at night is suppressed.

[0014] In the image trimming device according to the present disclosure, the processing unit may perform a ternarization process before the adjustment process to generate a ternarized image by ternarizing the nighttime image into the brightest bright area, the darkest dark area, and a dim light area between the bright and dark areas. In this case, the bright area can be appropriately extracted by the ternarization process, and the trimming range can be adjusted to correspond to the bright area. This makes it possible to effectively suppress quality degradation at night.

[0015] In the image cropping device according to the present disclosure, the processing unit may perform the following adjustment processes: a labeling process for defining multiple pixel groups by labeling the ternary image; an extraction process for extracting, from a candidate pixel group in the bright area among the multiple pixel groups, a candidate pixel group that includes the center of the nighttime image and has the largest labeled area; and a logical product process for adjusting the cropping range so that the cropping range corresponds to the bright area at least in the y direction by calculating the logical product of the candidate pixel group extracted in the extraction process and the cropping range. In this case, the cropping range can be easily and reliably adjusted. [Effects of the Invention]

[0016] According to the present disclosure, it is possible to provide an image trimming device and an image trimming method that can suppress quality fluctuations. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram showing an example of an image trimming device according to this embodiment. [Figure 2] FIG. 2 is a flowchart showing an example of an image processing method according to this embodiment. [Figure 3] FIG. 3 is a flowchart showing specific steps of the trimming process shown in FIG. [Figure 4] FIG. 4 is a schematic diagram showing the relationship between an oblique image and a vertical image. [Figure 5] FIG. 5 is a diagram for explaining how key points are set. [Figure 6] FIG. 6 is a diagram for explaining how the coordinates of the key points are calculated. [Figure 7] FIG. 7 shows an example in which the trimming range is illustrated on the target oblique image (nighttime image). [Figure 8] FIG. 8 shows the relationship between the focal length (wide-angle: f=24 mm, standard: f=50 mm) and the depression angle in the trimming range when calibration is not taken into consideration. [Figure 9] FIG. 9 shows nighttime images at each processing stage. [Figure 10] FIG. 10 shows nighttime images at each processing stage. [Figure 11] FIG. 11 is a diagram showing a state in which the trimming range is displayed on a nighttime image. [Figure 12] FIG. 12 is a diagram showing a road surface orthoimage. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant description may be omitted.

[0019] FIG. 1 is a diagram showing an example of an image cropping device according to this embodiment. As shown in FIG. 1, the image processing device (image cropping device) 1 includes a processing unit 10. The processing unit 10 processes an image captured by a camera 110 installed in a vehicle SC. An in-vehicle device 100 is installed in the vehicle SC. In addition to the camera 110, the in-vehicle device 100 may include various devices that can be used in an MMS (Mobile Mapping System). As an example, the in-vehicle device 100 may include an IMU (Inertial Measurement Unit), a DMI (Distance Measurement Indicator), a GNSS (Global Navigation Satellite System) receiver, and the like.

[0020] Camera 110 is installed on top of vehicle SC and is directed obliquely downward so as to capture an image of the road surface on which vehicle SC is traveling. Therefore, the image captured by camera 110 is an oblique image including an image of the road surface on which vehicle SC is traveling. The image captured by such camera 110 is an image to which position coordinates are assigned by each device of in-vehicle device 100. Note that camera 110 can be configured to capture high-resolution video such as 4K or 8K. In this case, still images can be acquired at will from the video captured by camera 110.

[0021] The image processing device 1 is physically configured as a computer system (information processor) including a CPU (Central Processing Unit), a RAM (Random Access Memory) as a main memory device, a ROM (Read Only Memory), a communication module as a data transmission / reception device, an output device such as a touch panel display or a liquid crystal display, and an input device as an input device such as an input key or a touch sensor. The processing unit 10 of the image processing device 1 and its functions are realized by loading a predetermined program onto hardware such as the CPU and RAM, thereby operating the communication module, the output device, and the input device under the control of the CPU, and reading and writing data from and to the RAM, etc.

[0022] Next, an image processing method according to this embodiment will be described. Fig. 2 is a flowchart showing an example of the image processing method according to this embodiment. The image processing method according to this embodiment includes the image cropping method according to this embodiment. In other words, the image cropping method according to this embodiment is made up of some of the steps of the image processing method described below.

[0023] 2, in this image processing method, first, the camera 110 is installed relative to the vehicle SC (step S101). In step S101, when the camera 110 is installed, interior orientation data of the camera 110 is acquired and provided to the processing unit 10. The processing unit 10 can acquire data related to the focal length f of the camera 110, and camera specifications such as the image size and sensor size of the camera 110.

[0024] Next, the road surface is photographed (step S102). More specifically, by photographing the vehicle SC with the camera 110 while the vehicle SC is traveling, a moving image photographed during traveling including the road surface on which the vehicle SC is traveling is acquired. In this step S102, exterior orientation data is acquired and provided to the processing unit 10. The processing unit 10 can acquire data relating to the height of the photographing point taken by the camera 110, the approximate tilt ω of the camera 110, etc.

[0025] Next, an image is acquired from the video captured during driving obtained in step S102 (step S103). More specifically, an image with location information is acquired by extracting a still image at a desired timing (frame) from the video captured during driving. As described above, the camera 110 is installed on top of the vehicle SC and is directed obliquely downward toward the road surface. Therefore, the image acquired here is an oblique image including an image of the road surface obtained by the camera 110 capturing an image of the road surface on which the vehicle SC is driving. The acquired oblique image is provided to the processing unit 10. Note that when providing the data acquired by the in-vehicle device 100 including the camera 110 to the processing unit 10, wireless communication or wired communication may be used, or it may be provided via a predetermined recording medium.

[0026] In the next step, image processing such as trimming is performed on the oblique image acquired in step S103. Therefore, the oblique image acquired in step S103 is also a target oblique image that is the subject of image processing. In other words, in step S103, the processing unit 10 of the image processing device 1 executes an acquisition process to acquire the target oblique image, which is the oblique image to be processed, from the camera 110 (the acquisition process is carried out). Then, in the next step, the processing unit 10 of the image processing device 1 performs a trimming process on the target oblique image acquired in step S103 (step S104: trimming process). This point will be described in more detail.

[0027] Fig. 3 is a flowchart showing specific steps of the trimming process shown in Fig. 2. The trimming process shown in Fig. 3 is an example of an image trimming method according to this embodiment. In this image trimming method, the trimming range is the range from the oblique image to the vertical image. Therefore, this point will be explained first.

[0028] FIG. 4 is a schematic diagram showing the relationship between an oblique image and a vertical image. A road surface RL and lane markings CL are shown in FIG. 4. As shown in FIG. 4, the perspective projection image (oblique image) captured by the camera 110 mounted on the vehicle SC is an oblique image captured at the position of the camera center O2 by rotating the camera 110 by an angle ω around the X axis while maintaining a constant distance L from the camera center O1 of the vertical image to a point P on the road surface RL. The X axis defines the X direction, which is the width direction of the vehicle SC.

[0029] In FIG. 4, the positions of both ends of the oblique image in the Y direction are the upper end positions O A and bottom position O C and both ends of the vertical image in the Y direction (the traveling direction of the vehicle SC) are at the top end position V A and bottom position V C In this case, the shooting range of the vertical image is a relatively narrow range near the center of the oblique image, and as will be described later, the bright part in the image is near the center of the image. Therefore, the effective usable range of the oblique image in the environment shown in Figure 4 is assumed to be equivalent to the shooting range of the vertical image.

[0030] In addition, if the oblique image is an image obtained by projectively transforming the vertical image, the trimming range for the oblique image is the upper end position V A and bottom position V C This is the range of the oblique image sandwiched between the y coordinates after projective transformation corresponding to the y coordinates. Within the range of a single image, the area near the center of the image has the least distortion. Therefore, in this image cropping method, the top edge position V A -Lower end position V C is designated as the "reliable range in terms of accuracy" and as the trimming range for the oblique image.

[0031] For the above reasons, in this image cropping method, a projective transformation coefficient between an oblique image and a vertical image is calculated. To this end, key points are first set (step S201). More specifically, in consideration of ease of calculation, four key points, namely, points A, B, C, and D, are set in the oblique image as shown in FIG. 5(a), and quantitative cropping is made possible based on the theory of photogrammetry. In FIG. 5, the y-axis direction is the traveling direction of the vehicle SC, and the vehicle SC is assumed to be traveling in the left lane of a two-lane road (lane width is about 3 m). The image size is 2S x ×2S y Let's say.

[0032] In photogrammetry, as shown in Figure 5(b), generally, if the ground coordinates of the photographing point O are (X0, Y0, Z0), the ground coordinates for point P are (X, Y, Z), and the photographic coordinates for image point p of point P are (x, y), the photographic coordinates of image point p are derived from the collinearity condition shown in the following equation (1) using the focal length f.

number

[0033] where a 11 =cosφ·cosκ, a 12 = -cosφ·sinκ, a 13 = sinφ, a 21 =cosω·sinκ+sinω·sinφ·cosκ, a 22 =cosω·cosκ-sinω·sinφ·sinκ, a 23 =-sinω·cosφ, a 31 =sinω·sinκ-cosω·sinφ·cosκ, a 32 =sinω·cosκ+cosω·sinφ·sinκ, a 33 =cosω·cosφ.

[0034] If camera calibration has been performed in advance, a in the above formula (1) ij, X0, Y0, Z0, and focal length f are known quantities, but even if camera calibration has not been performed, this process can be performed even if the other parameters are set to 0, as long as the camera specifications (focal length f, image size, and sensor size), the approximate tilt (angle ω) of the camera 110, and the height of the shooting point (Z0) can be estimated.

[0035] Next, we will explain in detail how to select each of the key points and how to calculate the coordinates of each key point. Note that in the following, the Z coordinates for each key point are all set to 0. [Point A]

[0036] The position of point A in Euclidean coordinates is assumed to be TX to the left of the camera position, and the X coordinate of this point is X A Let X0-TX. Next, if points A and B are placed on the same line (first line) parallel to the x-axis, the y coordinates of points A and B will be equal. Also, let that value be y a Then, the Y coordinate of point A can be calculated from the following formula (2) which is derived from the second formula of the above formula (1). A Then, the x coordinate of point A can be derived from the following equation (3) using the collinearity condition.

number

number

[0037] Note that TX is only required to be an approximate value that fits within the imaging space, and does not need to be an exact value. On the other hand, the positions of points A and B, i.e., the y coordinates of these points, are also arbitrary values. For example, if they are selected on the x axis, then the y coordinates are a (=y b )=0.0mm, but (y b is the y coordinate of point B), the results are slightly affected by this value due to the misalignment between the optical axis and the center of the screen. [Point C]

[0038] On the other hand, if point C is assumed to be on the vanishing line connecting the vanishing point and point A due to the characteristics of perspective projection, the X coordinate of point C is equal to the X coordinate of point A, and the relationship of the following equation (4) holds.

number

[0039] Furthermore, if point C is placed at the bottom of the image, y c =-S y Next (y c is the y coordinate of point C), and the Y coordinate of point C is the y coordinate of the above equation (2). a instead of -S y It is calculated using the following formula (5).

number

[0040] In addition, the x coordinate of point C is the same as the x coordinate of point A, calculated by the above formula (5) Y c is used, the collinearity condition is used to calculate the following equation (6).

number

[0041] Next, if the position of point D is taken as the bottom right corner of the image, the photograph coordinate of point D (x d ,y d ) is (S x ,-S y ) Furthermore, since points C and D are on the same line (second line different from the first line) parallel to the x-axis (bottom edge of the image), their Y coordinates satisfy the relationship of the following formula (7).

number

[0042] Furthermore, the X coordinate for point D is the photograph coordinate (x d ,y d ) and Z D =0 and is calculated using the following formula (8).

number

[0043] As for point B, since points A and B are taken on the same line parallel to the x-axis, the y coordinates and Y coordinates of points A and B are equal, and the relationship is expressed by the following formula (9).

number

[0044] Furthermore, if point B, like point C, is assumed to be on the vanishing line connecting the vanishing point and point D due to the characteristics of perspective projection, the X coordinate of point B is given by the following equation (10).

number

[0045] Furthermore, the x coordinate of point B is the X B and Y B Using Z B = 0, it is calculated by the following formula (11) using the collinearity condition.

number

[0046] As a result, the ground coordinates and photographic coordinates of points A, B, C, and D are calculated. Thus, in this image cropping method, in step S201, processing unit 10 sets points A and B on the same line parallel to the x-axis that defines the x-direction, sets point C at the bottom end of the oblique image in the y-direction on the vanishing line connecting point A and the vanishing point in the perspective projection, sets point D on the same line parallel to point C and the x-axis on the end (right corner) of the oblique image in the x-direction, and also sets point B on the vanishing line connecting point D and the vanishing point in the perspective projection.

[0047] In this image trimming method, following step S201, the processing unit 10 calculates the coordinates of each key point as described above (step S202).

[0048] In the next step, the processing unit 10 executes a calculation process to calculate a projective transformation coefficient between the oblique image and the vertical image of the road surface corresponding to the oblique image, based on the coordinates calculated in step S202 (step S203, calculation step). This point will be described in more detail.

[0049] As shown in Figure 6, to convert a vertical image to an oblique image, it is necessary to obtain the image coordinates of points C' and D' on the vertical image. The x-coordinate of point C' is x c x' and D' are the x coordinates of points d Regarding ', the relationship of the following formula (12) is obtained as shown in FIG.

number

[0050] On the other hand, the y coordinates of points C' and D' are c '(=y d ') is the actual length between points A and B. X Let the length on the image be l x Similarly, the actual length between points C and D is L Y Let the length on the image be l y Then, the following formula (13) is obtained, and the value is calculated by the following formula (14) using ly calculated by the following formula (13).

number

number

[0051] Here, using the image coordinates of the four points (points A, B, C', and D') before the projection transformation and the four points (points A, B, C, and D) after the projection transformation, the projection transformation coefficients (a1 to a8) are calculated using the quadratic projection transformation formula shown in the following formula (15).

number

[0052] The trimming range in this embodiment is the upper end position V A (y=S y ), and bottom position V C (y=-S y ) is the range of the oblique image between the y coordinates of the oblique image after the projective transformation (oblique image) corresponding to the y coordinates in the second equation of the above equation (15), respectively. y and -S y It is calculated as:

[0053] If φ=κ=0, then a 11 =1, a 12 =a 13 =0, a 21 =0, a 22 =cosω, a 23 = sinω, a 31 =0, a 32 = sinω, a 33 =cosω, and the result is not affected by X0 and Y0, so if X0=Y0=0, then Y C From the above equation (5), the following equation (16) is obtained.

number

[0054] where Z0 is the camera position (height) and 2S x ×2S y is the image size, f is the focal length, and ω is the tilt (depression angle) of the camera. Also, the ground coordinate Y A In the above formula (16), S y =y a= 0, the following equation (17) is obtained.

number

[0055] On the other hand, the ground coordinates for points A and B are obtained from the above equation (8) as the following equation (18). Furthermore, in the below equation (18), by setting ya=yb=0 and rearranging the above equation (13) using the above equations (16) to (18), the following equation (19) is obtained.

number

number

[0056] From the above equation (19), the position of point C on the vertical image (y c '=l y It is understood that ω is not affected by the shooting height (height of the camera 110) and is a function of the depression angle ω and the focal length f.

[0057] As described above, in step S203, the projective transformation coefficients between the oblique image and the vertical image are calculated. In the subsequent step, the processing unit 10 acquires an arbitrary oblique image (target oblique image) that is the actual processing target (step S204). In other words, the processes in steps S201 to S203 do not have to be performed on the actual oblique image to be processed, and can be calculated in advance based on a previously acquired oblique image. In this embodiment, since the target oblique image is acquired in step S103, the processes in steps S201 to S203 are performed using the target oblique image that is the actual processing target. In this case, step S204 is performed prior to step S201.

[0058] Next, the processing unit 10 determines the cropping range in the target oblique image (step S205: cropping process, cropping step). More specifically, in step S205, the processing unit 10 determines the top end position V of the vertical image in the y direction, which is the traveling direction of the vehicle SC, based on the projective transformation coefficients calculated in steps S201 to S203. A (y=S y ), and bottom position V C (y=-S y The range of the oblique target image between the pair of y coordinates of the oblique target image corresponding to the y coordinates of the oblique target image is set as the trimming range.

[0059] At the same time, the processing unit 10 adjusts the nighttime image (step S206: adjustment process). More specifically, in this step S206, when the target oblique image is a nighttime image captured at night by the camera 110, the processing unit 10 adjusts the cropping range determined in step S205 so that the cropping range corresponds to a relatively bright bright area in the nighttime image. The bright area includes an area illuminated by the headlights of the vehicle SC.

[0060] FIG. 7 shows an example of the trimming range illustrated in a target oblique image (nighttime image). (a) of FIG. 7 shows the case with camera calibration, and (b) of FIG. 7 shows the case without camera calibration. In FIG. 7, the trimming range Rt is illustrated as a light gray band-like area. The initial input in determining the trimming range Rt is the y coordinate (y a ) and the TX value for point A. The TX value only needs to be an approximate value, and this value does not affect the results, whereas the trimming range Rt is affected by the deviation between the optical axis and the center of the screen in the y direction.

[0061] Therefore, in this embodiment, for the ternary image of the target oblique image (nighttime image), the input value of the y coordinate for point A is gradually shifted (for example, by 0.5 mm) from the center of the image, and the y coordinate when the number of pixels corresponding to the bright part in the acquired trimming range Rt is the maximum value is adopted (this point will be described in detail later). In FIG. 7, the y coordinate obtained by the above trimming process is a 7 shows the trimming range Rt at θ = -2.0 mm. It can be seen from Fig. 7 that the bright area in the target oblique image has been trimmed.

[0062] Incidentally, Figure 7(a) uses the results of camera calibration (f = 24.558 mm, Z0 = 2.370 m, ω = 12° 28' 19"), while Figure 7(b) shows the result without camera calibration, using the nominal focal length and the results of simplified measurements of the height and tilt of the camera 110 (f = 24 mm, Z0 = 2.4 m, ω = 13°). From Figure 7, it can be seen that the cropping range Rt is hardly affected by camera calibration. In other words, even if camera calibration has not been performed, the above method is feasible as long as the camera specifications (focal length, image size, sensor size) and the approximate tilt (angle ω) of the camera 110 can be estimated.

[0063] On the other hand, as described above, the trimming range Rt is affected by the focal length f and the depression angle, and the trimming range Rt for a wide-angle lens increases in proportion to the angle of view. Also, the smaller the depression angle, the closer the camera 110 is to being horizontal with respect to the ground, and therefore the narrower the trimming range Rt.

[0064] Figure 8 shows the relationship between focal length (wide-angle: f=24 mm, standard: f=50 mm) and depression angle in the cropping range when calibration is not taken into consideration. The range between the top and bottom ends at each depression angle is the cropping range, and the colored areas in the figure correspond to the cropping range Rt in Figure 7. As shown in Figure 8, it can be seen that the cropping range Rt widens as the depression angle increases, but when focusing on the bright areas in the center of the image, a depression angle of 10° to 15° is deemed appropriate.

[0065] As described above, the processing according to this embodiment enables automated quantitative trimming regardless of whether camera calibration is performed. One of its features is that points A, B, C, and D are each located on the same line parallel to the x-axis, and points C and D are positioned distinctively on the screen, enabling calculation of the ground coordinates and image coordinates of each key point based on photogrammetry theory. Another feature is the use of geometric features in perspective projection images. Specifically, point C lies on the vanishing line connecting the vanishing point and point A, and its X coordinate is the same as the X coordinate for point A. The same is true for the X coordinate of point B.

[0066] In particular, by utilizing the geometric features of perspective projection images, it is possible to associate oblique images with vertical images by projective transformation even for monotonous images without texture or feature points, and the top edge position V of the vertical image can be calculated. A ·Lower end position V C By calculating the y-coordinates of the corresponding points in the oblique image, trimming in the oblique image becomes possible. On the other hand, the trimming range Rt is automatically calculated from the proportion of bright areas within the trimming range Rt in the ternary image, making quantitative determination possible.

[0067] Here, the adjustment process of the processing unit 10 in step S206 will be described in more detail. FIG. 9 shows nighttime images at each processing stage. (a) of FIG. 9 shows a nighttime image (nighttime illumination driving image) input from the camera 110 (on-board device 100). In step S206, the processing unit 10 first performs a ternarization process to generate a ternarized image Qt by ternarizing the nighttime image Ql into the brightest bright area BP, the darkest dark area DP, and the dim light area MP whose brightness is between the bright area BP and the dark area DP (see (b) of FIG. 9). For example, Otsu's multilevel thresholding method can be used for the ternarization in this case.

[0068] Next, as shown in Fig. 9(c), the processing unit 10 performs a labeling process on the ternarized image Qt to define multiple pixel groups. At the same time, the processing unit 10 performs an extraction process to extract a candidate pixel group CU that includes the center of the nighttime image Ql and has the largest labeled area from the candidate pixel groups in the bright area of ​​the multiple pixel groups. Fig. 9(c) shows the nighttime image QL, with the candidate pixel group CU in the bright area depicted in light gray.

[0069] 10(a) and 10(b), the processing unit 10 performs a logical AND operation to adjust the trimming range Rt so that the trimming range Rt corresponds to the bright area BP at least in the y direction by calculating the logical AND between the candidate pixel group CU extracted in the extraction operation and the trimming range Rt. This results in the adjusted trimming range. In FIG. 10(b), the trimming range Rt before adjustment and the trimming range Rv after adjustment that takes the bright area BP into account are shown for the original nighttime image Ql.

[0070] FIG. 11 is a diagram showing a state in which a trimming range is displayed on a nighttime image. FIG. 11(a) shows a trimming range Rm set by an operator, and FIG. 11(b) shows a trimming range Rv automatically generated by the image trimming method according to this embodiment. As shown in FIG. 11, the trimming range Rm has a shape that suppresses stripes between frames and sets the maximum range necessary to ensure brightness. In contrast, the trimming range Rv sets a sufficient range necessary to ensure brightness, and it can be seen that it is comparable to the trimming range Rm.

[0071] Thereafter, the processing unit 10 cuts out the cropping area Rv from the target oblique image (here, the nighttime image Ql), thereby completing the cropping process. Note that if the target oblique image is not a nighttime image, step S206 is omitted, and the cropping area Rt is cut out from the target oblique image.

[0072] This completes the image cropping method according to this embodiment. In the image processing method according to this embodiment, the processing unit 10 then performs projective transformation of the image (step S105). Here, since the cropping range Rv (or cropping range Rt, the same applies below) has already been extracted from the target oblique image in step S104, the target of projective transformation is only the cropping range Rv (cropped image). Therefore, the processing load is reduced compared to when the entire target oblique image is projectively transformed. The processing unit 10 then joins together the multiple cropped images after projective transformation to create an orthoimage that conforms to the road surface (road surface orthoimage) (step S106).

[0073] Fig. 12 is a diagram showing road surface orthoimages. Fig. 12(a) shows a road surface orthoimage IA generated by joining the cropping ranges Rm shown in Fig. 11(a), and Fig. 12(b) shows a road surface orthoimage IB generated by joining the cropping ranges Rv automatically generated by the image cropping method according to this embodiment. Comparing the road surface orthoimage IA and the road surface orthoimage IB, the boundaries between frames are linear in the road surface orthoimage IB, making it less likely to feel unnatural.

[0074] As described above, in the image processing device (image cropping device) 1 according to this embodiment and the image cropping method performed by the image processing device 1, first, a projective transformation coefficient between an oblique image obtained by capturing an image of a road surface with the in-vehicle camera 110 and a vertical image corresponding to the oblique image is calculated. Then, based on the calculated projective transformation coefficient, the upper end position V of the vertical image in the y direction is calculated for the target oblique image, which is the oblique image that is actually the target of cropping. A and bottom position V C The range between the pair of y-coordinates corresponding to the image is set as the trimming range Rt and is trimmed out. This ensures that the quality of the product obtained by trimming is high. Furthermore, by performing trimming quantitatively using numerical calculations, there is no room for the experience or ability of the worker to affect the quality of the product, and the quality of the product is stabilized at a high quality and quality fluctuations are suppressed. Note that the oblique image used to calculate the projective transformation coefficients and the target oblique image to be trimmed may be the same or different.

[0075] Furthermore, in the image processing device 1 according to this embodiment, the processing unit 10 sets four key points, A, B, C, and D, on the oblique image in the calculation process, and calculates projective transformation coefficients based on the four key points. In particular, when the vehicle width direction of the vehicle SC is defined as the x direction, the processing unit 10 sets points A and B of the four key points on a first line parallel to the x axis that defines the x direction, and sets points C and D of the four key points on a second line parallel to the x axis and different from the first line. Therefore, the y coordinates of points A and B, and points C and D are the same, and the processing load related to the calculation of projective transformation coefficients is reduced.

[0076] Furthermore, in the image processing device 1 according to this embodiment, in the calculation process, the processing unit 10 sets point C at the bottom end of the oblique image in the y direction, on the vanishing line connecting the vanishing point in the perspective projection and point A, sets point D at the end of the oblique image in the x direction, and sets point B on the vanishing line connecting the vanishing point in the perspective projection and point D. In this case, the processing load related to the calculation of the projective transformation coefficients is further reduced.

[0077] Furthermore, in the image processing device 1 according to this embodiment, when the target oblique image includes a nighttime image captured by the camera 110 at night, the processing unit 10 further performs an adjustment process after the trimming process to adjust the trimming range Rt so that the trimming range Rt corresponds to a relatively bright bright area BP in the nighttime image. Therefore, in the oblique image captured at night, the trimming range Rt is adjusted so that it corresponds to the bright area BP. This suppresses quality degradation at night.

[0078] Furthermore, in the image processing device 1 according to this embodiment, the processing unit 10 performs a ternarization process before the adjustment process to generate a ternarized image Qt from the nighttime image Ql, which is ternarized into the brightest bright area BP, the darkest dark area DP, and the dim light area MP, which is between the bright area BP and the dark area DP. Therefore, the ternarization process can appropriately extract the bright area BP, and then adjust the trimming range Rt to correspond to the bright area BP. This makes it possible to effectively suppress quality degradation at night.

[0079] Furthermore, in the image processing device 1 according to this embodiment, the processing unit 10 performs the following adjustment processes: a labeling process for defining multiple pixel groups by labeling the ternary image Qt; an extraction process for extracting a candidate pixel group CU that includes the center of the nighttime image Ql and has the largest labeled area from candidate pixel groups in the bright areas of the multiple pixel groups after the labeling process; and a logical AND process for adjusting the trimming range Rt so that the trimming range Rt corresponds to the bright areas at least in the y direction by taking the logical AND of the candidate pixel group CU extracted in the extraction process and the trimming range Rt. This allows the trimming range Rt to be adjusted easily and reliably.

[0080] The above embodiment has described one embodiment of the image cropping device and image cropping method according to the present disclosure. Therefore, the image cropping device and image cropping method according to the present disclosure can be modified.

[0081] For example, in the above embodiment, the processing unit 10 reduces the processing load related to the calculation of the projective transformation coefficients by setting four characteristic key points, namely, points A, B, C, and D, in step S201. However, the method of selecting the key points is not limited to the above embodiment and may be any method.

[0082] Furthermore, in the above embodiment, the processing unit 10 adjusts the trimming range Rt in accordance with the bright part BP in step S206, but if the target oblique image is not a nighttime image, step S206 can be omitted. [Explanation of symbols]

[0083] 1...image processing device (image cropping device), 10...processing unit, 110...camera, BP...bright area, DP...dark area, MP...twilight area, CU...candidate pixel group, Rt...cropping range, Rv...cropping range, Ql...nighttime image, Qt...ternarized image.

Claims

1. a processing unit that processes images captured by a camera installed in the vehicle; The processing unit a calculation process for calculating a projective transformation coefficient between an oblique image including an image of the road surface on which the vehicle is traveling, obtained by capturing an image of the road surface with the camera, and a vertical image of the road surface corresponding to the oblique image; an acquisition process of acquiring a target oblique image, which is the oblique image to be processed, from the camera; after the calculation process and the acquisition process, a trimming process is performed to set a trimming range of the oblique target image between a pair of y coordinates of the oblique target image corresponding to the upper and lower ends of the vertical image in the y direction, which is the traveling direction of the vehicle, based on the projective transformation coefficient, and to cut out the trimming range from the oblique target image; The vertical image is an image that would be captured by the camera when the camera is rotated around an x-axis that is parallel to the road surface and perpendicular to the optical axis, while maintaining a constant distance between the camera and the road surface on the optical axis that passes through the center of the camera, until the optical axis faces vertically. Image cropping device.

2. the processing unit, in the calculation process, sets four key points, namely, points A, B, C, and D, on the oblique image, and calculates the projective transformation coefficients based on the four key points; When the vehicle width direction of the vehicle is defined as an x ​​direction, the processing unit sets the points A and B of the four key points on a first line parallel to the x axis that defines the x direction, and sets the points C and D of the four key points on a second line parallel to the x axis and different from the first line. The image cropping device according to claim 1 .

3. In the calculation process, the point C is set by the processing unit at the lower end of the oblique image in the y direction, on a vanishing line connecting a vanishing point in perspective projection and the point A; the point D is set by the processing unit to an end of the oblique image in the x direction, The point B is set by the processing unit on a vanishing line connecting the vanishing point in perspective projection and the point D. The image cropping device according to claim 2 .

4. When the target oblique image includes a nighttime image captured by the camera at night, the processing unit further performs, after the trimming process, an adjustment process of adjusting the trimming range so that the trimming range corresponds to a relatively bright bright area in the nighttime image. The image trimming device according to any one of claims 1 to 3.

5. The processing unit, before the adjustment processing, executes a ternarization processing to generate a ternarized image in which the nighttime image is ternarized into the brightest bright area, the darkest dark area, and a dim light area having a brightness between the bright area and the dark area.

5. The image cropping device according to claim 4.

6. In the adjustment process, the processing unit a labeling process for defining a plurality of pixel groups by labeling the ternarized image; an extraction process for extracting, from the candidate pixel group in the bright area among the plurality of pixel groups, a candidate pixel group that includes the center of the nighttime image and has the largest labeled area, after the labeling process; a logical product process for adjusting the trimming range so that the trimming range corresponds to the bright area at least in the y direction by calculating a logical product of the candidate pixel group extracted in the extraction process and the trimming range; To execute 6. The image cropping device according to claim 5.

7. An image trimming method for performing a trimming process on an image captured by a camera installed in a vehicle, a calculation step of calculating a projective transformation coefficient between an oblique image including an image of the road surface on which the vehicle is traveling, obtained by capturing an image of the road surface with the camera, and a vertical image of the road surface corresponding to the oblique image; an acquisition step of acquiring a target oblique image, which is the oblique image to be processed, from the camera; a cropping step of, after the calculation step and the acquisition step, determining a cropping range of the oblique target image between a pair of y coordinates of the oblique target image corresponding to the upper and lower ends of the vertical image in the y direction, which is the traveling direction of the vehicle, based on the projective transformation coefficients, and cutting out the cropping range from the oblique target image; Equipped with The vertical image is an image that would be captured by the camera when the camera is rotated around an x-axis that is parallel to the road surface and perpendicular to the optical axis, while maintaining a constant distance between the camera and the road surface on the optical axis that passes through the center of the camera, until the optical axis faces vertically. Image cropping methods.

Images