Image trimming device, and image trimming method
The image trimming device and method address quality variability in in-vehicle camera images by calculating projective transformation coefficients and adjusting for brightness, ensuring consistent high-quality trimming.
Patent Information
- Application Number
- JP2024156971
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-28
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing methods for trimming images captured by in-vehicle cameras lack a systematic approach, leading to variable quality due to reliance on human expertise, and do not adequately address lens distortion and brightness distribution issues, particularly at night.
An image trimming device and method that calculates projective transformation coefficients to define a trimming range based on key points, adjusts for brightness distribution in nighttime images, and uses ternarization to stabilize image quality.
The method ensures high-quality image trimming by minimizing distortion and brightness-related quality issues, providing consistent results through automated quantitative calculations.
Smart Images

Figure 2024166321000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image cropping device and an image cropping method. [Background technology]
[0002] Patent document 1 describes that the system receives inputs such as image data of the road surface taken by an on-board camera, location information (position data) of the camera that took the image, the orientation of the on-board camera, the height of the on-board camera from the road surface, information such as the model of the on-board camera, and information about the analyst performing the analysis work, and outputs, as output information, an image of pavement cracks, which is the captured image plus an image showing the shape of the cracks in the pavement. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2021-107683 A Summary of the Invention [Problem to be solved by the invention]
[0004] In recent years, as described above, understanding road surface conditions based on images obtained by photographing the road surface with an on-board camera has become important for the purpose of road maintenance and management, realizing automatic driving of vehicles, and the like. However, images photographed by on-board cameras are perspective projection images, and distant areas in the direction of travel of the vehicle are projected small, and the influence of lens distortion increases the farther away from the photographing point. Therefore, from the viewpoint of photogrammetry, it is desirable to use the images photographed by on-board cameras from the center to the bottom of the image, which have less lens distortion, but it is also necessary to avoid capturing the vehicle body (bonnet part) depending on the type of vehicle used and the inclination of the camera.
[0005] Therefore, in order to effectively use an image captured by an on-board camera, it is desirable to trim an appropriate range from the image. Patent Document 1 also mentions that the captured image may be an image obtained by trimming a predetermined range from the original image captured by the on-board camera. However, Patent Document 1 does not disclose a specific method for trimming the on-board camera image, and is unclear. Therefore, at present, it is considered that the trimming of the on-board camera image is performed manually by an operator, but in that case, the quality of the finished product varies depending on the experience and ability of the operator.
[0006] Therefore, an object of the present disclosure is to provide an image trimming device and an image trimming method that are capable of suppressing quality fluctuations. [Means for solving the problem]
[0007] The image cropping device of the present disclosure includes a processing unit that processes images obtained by imaging with a camera installed in a vehicle, and the processing unit executes a calculation process that calculates a projection transformation coefficient between an oblique image including an image of the road surface obtained by imaging the road surface on which the vehicle is traveling with the camera and a vertical image of the road surface corresponding to the oblique image, an acquisition process that acquires a target oblique image, which is the oblique image to be processed, from the camera, and a cropping process that, after the calculation process and the acquisition process, sets the range of the target oblique image sandwiched between a pair of y coordinates of the target oblique image corresponding to the upper and lower ends of the vertical image in the y direction, which is the direction of travel of the vehicle, as a cropping range based on the projection transformation coefficient, and cuts out the cropping range from the target oblique image.
[0008] The image cropping method according to the present disclosure is an image cropping method for performing a cropping process on an image obtained by imaging with a camera installed in a vehicle, and includes a calculation step of calculating a projection transformation coefficient between an oblique image including an image of the road surface obtained by imaging the road surface along which the vehicle is traveling with the camera and a vertical image of the road surface corresponding to the oblique image, an acquisition step of acquiring from the camera a target oblique image which is the oblique image to be processed, and a cropping step of cutting out the cropping range from the target oblique image based on the projection transformation coefficient after the calculation step and the acquisition step.
[0009] According to the findings of the present inventors, the range in the oblique image corresponding to the range between the upper end and the lower end in the y direction (vehicle travel direction) of the vertical image in the projective transformation relationship with the oblique image is an area in which distortion is small near the center of the image and the vehicle body is unlikely to be reflected. Therefore, in the device and method according to the present disclosure, first, a projective transformation coefficient between the oblique image obtained by the vehicle-mounted camera capturing the road surface and the vertical image corresponding to the oblique image is calculated. Then, based on the calculated projective transformation coefficient, the range between a pair of y coordinates corresponding to the upper end and the lower end of the vertical image in the y direction is set as the trimming range for the target oblique image, which is the oblique image that is actually the target of trimming, and is cut out. This ensures that the quality of the product obtained by trimming is high quality. In addition, by quantitatively performing trimming by numerical calculation, there is no room for the experience or ability of the worker to be involved in the quality of the product, and the quality of the product is stabilized at high quality and quality fluctuation is suppressed. Note that the oblique image used to calculate the projective transformation coefficient and the target oblique image to be trimmed may be the same or different.
[0010] In the image trimming device according to the present disclosure, the processing unit may set four key points, namely points A, B, C, and D, on the oblique image in the calculation process, and calculate a projection transformation coefficient based on the four key points, and when the vehicle width direction of the vehicle is defined as the x direction, the processing unit may set points A and B of the four key points on a first line parallel to the x axis that defines the x direction, and set points C and D of the four key points on a second line parallel to the x axis and different from the first line. In this case, the y coordinates of points A and B, and points C and D are the same, so that the processing load related to the calculation of the projection transformation coefficient is reduced.
[0011] In the image trimming device according to the present disclosure, the processing unit may, in the calculation process, set point C at the bottom end of the oblique image in the y direction on the vanishing line connecting the vanishing point in the perspective projection and point A, set point D at the end of the oblique image in the x direction on the same straight line parallel to the x axis as point C, and set point B on the vanishing line connecting the vanishing point in the perspective projection and point D. In this case, the processing load related to the calculation of the projective transformation coefficients is further reduced.
[0012] In recent years, high-sensitivity cameras that have achieved high sensitivity and a wide dynamic range have made it possible to capture high-quality images even at night and in 4K video. In particular, the fact that video can now be captured at night using high-resolution cameras such as 4K and 8K makes it possible to avoid the effects of sunlight and congestion of people and cars, and is expected to improve the efficiency of road surface condition acquisition work using car-mounted cameras. On the other hand, one of the light sources in night-time photography is the car's headlights, and the illumination of the headlights causes a brightness distribution in the oblique image. Therefore, in order to make more effective use of car-mounted camera images, it is desirable to trim the images taking into account the brightness distribution of the oblique images taken at night.
[0013] Therefore, in the image trimming device according to the present disclosure, when the target oblique image includes a nighttime image captured by a camera at night, the processing unit may further perform an adjustment process to adjust the trimming range after the trimming process so that the trimming range corresponds to a relatively bright bright area in the nighttime image. In this case, the trimming range is adjusted in the oblique image captured at night so as to correspond to the bright area. Thus, quality degradation at night is suppressed.
[0014] In the image trimming device according to the present disclosure, the processing unit may perform a ternarization process before the adjustment process, which generates a ternarized image of the nighttime image into the brightest bright area, the darkest dark area, and a dim light area between the bright area and the dark area. In this case, the bright area is appropriately extracted by the ternarization process, and the trimming range can be adjusted to correspond to the bright area. Therefore, it is possible to suitably suppress quality degradation at night.
[0015] In the image cropping device according to the present disclosure, the processing unit may perform, in the adjustment process, a labeling process for defining a plurality of pixel groups by labeling the ternary image, an extraction process for extracting a candidate pixel group that includes the center of the nighttime image and has the largest label area from a candidate pixel group in the bright area among the plurality of pixel groups after the labeling process, and a logical product process for adjusting the cropping range so that the cropping range corresponds to the bright area at least in the y direction by taking the logical product of the candidate pixel group extracted in the extraction process and the cropping range. In this case, the cropping range can be easily and reliably adjusted. Effect of the Invention
[0016] According to the present disclosure, it is possible to provide an image trimming device and an image trimming method capable of suppressing quality fluctuations. [Brief description of the drawings]
[0017] [Figure 1] FIG. 1 is a diagram showing an example of an image trimming device according to the present embodiment. [Diagram 2] FIG. 2 is a flowchart showing an example of an image processing method according to this embodiment. [Diagram 3] FIG. 3 is a flow chart showing specific steps of the trimming process shown in FIG. [Figure 4] FIG. 4 is a schematic diagram showing the relationship between an oblique image and a vertical image. [Diagram 5] FIG. 5 is a diagram for explaining how key points are set. [Figure 6] FIG. 6 is a diagram for explaining how the coordinates of the key points are calculated. [Figure 7] FIG. 7 shows an example in which the trimming range is illustrated on a target oblique image (nighttime image). [Figure 8] FIG. 8 shows the relationship between the focal length (wide-angle: f=24 mm, standard: f=50 mm) and the depression angle in the trimming range when calibration is not taken into consideration. [Figure 9] FIG. 9 shows nighttime images at each processing stage. [Figure 10] FIG. 10 shows nighttime images at each processing stage. [Figure 11] FIG. 11 is a diagram showing a state in which the trimming range is displayed on a nighttime image. [Figure 12] FIG. 12 is a diagram showing a road surface orthoimage. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and duplicated explanations may be omitted.
[0019] FIG. 1 is a diagram showing an example of an image cropping device according to the present embodiment. As shown in FIG. 1, an image processing device (image cropping device) 1 includes a processing unit 10. The processing unit 10 processes an image captured by a camera 110 installed in a vehicle SC. An in-vehicle device 100 is installed in the vehicle SC. In addition to the camera 110, the in-vehicle device 100 can include various devices that can be used in a Mobile Mapping System (MMS). As an example, the in-vehicle device 100 can include an Inertial Measurement Unit (IMU), a Distance Measurement Indicator (DMI), and a Global Navigation Satellite System (GNSS) receiver, etc.
[0020] Camera 110 is installed on the upper part of vehicle SC and is directed obliquely downward so as to capture an image of the road surface on which vehicle SC is traveling. Therefore, the image captured by camera 110 is an oblique image including an image of the road surface on which vehicle SC is traveling. The image captured by such camera 110 is an image to which position coordinates are assigned by each device of in-vehicle device 100. Camera 110 can capture high-resolution video such as 4K / 8K. In this case, still images can be acquired arbitrarily from the video captured by camera 110.
[0021] The image processing device 1 is physically configured as a computer system (information processor) including a CPU (Central Processing Unit), a RAM (Random Access Memory) as a main memory device, a ROM (Read Only Memory), a communication module as a data transmission / reception device, an output device such as a touch panel display, a liquid crystal display, and an input device as an input device such as an input key, a touch sensor, etc. The processing unit 10 of the image processing device 1 and the functions of the processing unit 10 are realized by loading a predetermined program onto hardware such as the CPU and RAM, and by operating the communication module, the output device, and the input device under the control of the CPU, and by reading and writing data in the RAM, etc.
[0022] Next, an image processing method according to this embodiment will be described. Fig. 2 is a flowchart showing an example of the image processing method according to this embodiment. The image processing method according to this embodiment includes the image cropping method according to this embodiment. In other words, the image cropping method according to this embodiment is composed of some steps of the image processing method described below.
[0023] 2, in this image processing method, first, the camera 110 is installed with respect to the vehicle SC (step S101). In step S101, when the camera 110 is installed, interior orientation data of the camera 110 is acquired and provided to the processing unit 10. The processing unit 10 can acquire data related to the focal length f of the camera 110, and camera specifications such as the image size and sensor size of the camera 110.
[0024] Next, the road surface is photographed (step S102). More specifically, while the vehicle SC is traveling, the camera 110 photographs the road surface on which the vehicle SC is traveling, thereby acquiring a moving image photographed during traveling, including the road surface on which the vehicle SC is traveling. In this step S102, exterior orientation data is acquired and provided to the processing unit 10. The processing unit 10 can acquire data related to the height of the photographing point by the camera 110, the approximate tilt ω of the camera 110, and the like.
[0025] Next, an image is acquired from the video captured during driving obtained in step S102 (step S103). More specifically, an image with location information is acquired by extracting a still image at a desired timing (frame) from the video captured during driving. As described above, the camera 110 is installed on the upper part of the vehicle SC and is directed diagonally downward toward the road surface. Therefore, the image acquired here is an oblique image including an image of the road surface obtained by the camera 110 capturing an image of the road surface on which the vehicle SC is traveling. The acquired oblique image is provided to the processing unit 10. Note that when providing the data acquired by the in-vehicle device 100 including the camera 110 to the processing unit 10, wireless communication or wired communication may be used, or a predetermined recording medium may be used.
[0026] In the next step, image processing such as trimming is performed on the oblique image acquired in step S103. Therefore, the oblique image acquired in step S103 is also a target oblique image that is the subject of image processing. In other words, in step S103, the processing unit 10 of the image processing device 1 executes an acquisition process for acquiring the target oblique image, which is the oblique image to be processed, from the camera 110 (the acquisition process is carried out). Then, in the next step, the processing unit 10 of the image processing device 1 performs a trimming process on the target oblique image acquired in step S103 (step S104: trimming process). This point will be described in more detail.
[0027] Fig. 3 is a flow chart showing specific steps of the trimming process shown in Fig. 2. The trimming process shown in Fig. 3 is an example of an image trimming method according to this embodiment. In this image trimming method, the trimming range is a range from an oblique image to a vertical image. Therefore, this point will be described first.
[0028] Fig. 4 is a schematic diagram showing the relationship between an oblique image and a vertical image. A road surface RL and a lane marking CL are shown in Fig. 4. As shown in Fig. 4, a perspective projection image (oblique image) captured by a camera 110 mounted on a vehicle SC is an oblique image captured at the position of a camera center O2 by rotating the camera 110 by an angle ω around the X axis while keeping a constant distance L from a camera center O1 of the vertical image to a point P on the road surface RL. The X axis defines the X direction, which is the vehicle width direction of the vehicle SC.
[0029] In FIG. 4, the positions of both ends of the oblique image in the Y direction are the upper end positions O A and bottom position O C The two ends of the vertical image in the Y direction (the traveling direction of the vehicle SC) are the upper end positions V A and bottom position V C In this case, since the shooting range of the vertical image is a relatively narrow range near the center of the oblique image, and since the bright parts in the image are near the center of the image as described below, the effective usage range of the oblique image in the environment shown in Figure 4 is assumed to be equivalent to the shooting range of the vertical image.
[0030] In addition, if the oblique image is an image obtained by projectively transforming the vertical image, the trimming range for the oblique image is the upper end position V A and bottom position V C The range of the oblique image between the y coordinates after projective transformation corresponding to the y coordinates is the range of the oblique image between the y coordinates after projective transformation corresponding to ... A -Lower end position V C is designated as the "reliable range in terms of accuracy" and as the cropping range for the oblique image.
[0031] For the above reasons, in this image cropping method, a projective transformation coefficient between an oblique image and a vertical image is calculated. To this end, first, key points are set (step S201). More specifically, in consideration of ease of calculation, four key points, namely points A, B, C, and D, are set in the oblique image as shown in FIG. 5(a), enabling quantitative cropping based on the theory of photogrammetry. In FIG. 5, the y-axis direction is the traveling direction of the vehicle SC, and the vehicle SC is assumed to be traveling in the left lane of a two-lane road (lane width is approximately 3 m). The image size is 2S x x2S y Let us assume that.
[0032] In photogrammetry, as shown in FIG. 5(b), generally, if the ground coordinates of the photographing point O are (X0, Y0, Z0), the ground coordinates for point P are (X, Y, Z), and the photographic coordinates for image point p of point P are (x, y), then the photographic coordinates of image point p of point P are derived from the collinearity condition shown in the following equation (1) using the focal length f.
number
[0033] Here, a 11 = cosφ cosκ, a 12 = -cosφ sinκ, a 13 = sinφ, a 21 =cosω·sinκ+sinω·sinφ·cosκ, a 22 =cosω·cosκ-sinω·sinφ·sinκ, a 23 = -sinω cosφ, a 31 =sinω·sinκ-cosω·sinφ·cosκ, a 32 =sinω·cosκ+cosω·sinφ·sinκ, a 33 = cosω cosφ.
[0034] If camera calibration has been performed in advance, a in the above formula (1) ij, X0, Y0, Z0, and focal length f are known quantities, but even if camera calibration has not been performed, this process can be executed even if the other parameters are set to 0, as long as the camera specifications (focal length f, image size, and sensor size), the approximate tilt (angle ω) of the camera 110, and the height of the shooting point (Z0) can be estimated.
[0035] Next, a method for selecting each of the above key points and a method for calculating the coordinates of each of the key points will be specifically described. Note that in the following, the Z coordinates of each key point are all set to 0. [Point A]
[0036] The position of point A in Euclidean coordinates is TX to the left of the camera position, and the X coordinate of this point is X A Let X0-TX be the coordinate system of the x-axis. Next, if points A and B are placed on the same line (first line) parallel to the x-axis, the y-coordinates of points A and B will be equal. Also, let that value be y a Then, the Y coordinate of point A is calculated by the following formula (2) which is derived from the second formula of the above formula (1). A Then, the x coordinate of point A can be derived from the following equation (3) using the collinearity condition.
number
number
[0037] Note that TX does not have to be an exact value, but it is sufficient if it is an approximate value that falls within the shooting space. On the other hand, the positions of points A and B, that is, the y coordinates of these points, are also arbitrary values. For example, if they are selected on the x axis, then y a (=y b )=0.0mm, but (y b is the y coordinate of point B). The result is slightly affected by this value due to the misalignment between the optical axis and the center of the screen. [Point C]
[0038] On the other hand, if point C is assumed to be on the vanishing line connecting the vanishing point and point A due to the characteristics of perspective projection, the X coordinate of point C is equal to the X coordinate of point A, and the relationship of the following equation (4) is established.
number
[0039] Furthermore, if point C is placed at the bottom of the image, y c =-S y Next (y c is the y coordinate of point C), and the Y coordinate of point C is the y a Instead of -S y It is calculated using the following formula (5).
number
[0040] In addition, the x-coordinate of point C is calculated by the above formula (5) in the same way as the x-coordinate of point A. c If the collinearity condition is used, the following formula (6) is used to calculate the
number
[0041] Next, if we take the position of point D at the bottom right corner of the image, the photograph coordinates of point D (x d ,y d ) is (S x ,-S y ) In addition, since points C and D are on the same line (a second line different from the first line) parallel to the x-axis (the bottom edge of the image), their Y coordinates satisfy the relationship of the following formula (7).
number
[0042] Furthermore, the X coordinate for point D is the photographic coordinate (x d ,y d ) and Z D = 0 and calculated using the following formula (8).
number
[0043] As for point B, since points A and B are located on the same line parallel to the x-axis, the y coordinates and Y coordinates of points A and B are equal, and the relationship is expressed by the following formula (9).
number
[0044] Furthermore, if point B, like point C, is assumed to be on the vanishing line connecting the vanishing point and point D due to the characteristics of perspective projection, the X coordinate of point B is given by the following equation (10).
number
[0045] Furthermore, the x coordinate of point B is the X B and Y B Using Z B = 0, it is calculated from the following formula (11) using the collinearity condition.
number
[0046] In this way, the ground coordinates and photographic coordinates of points A, B, C, and D are calculated. Thus, in this image trimming method, in step S201, the processing unit 10 sets points A and B on the same line parallel to the x-axis that defines the x-direction, sets point C at the bottom end of the oblique image in the y-direction on the vanishing line connecting the vanishing point in the perspective projection and point A, sets point D on the same line parallel to the x-axis with point C and at the end (right corner) in the x-direction of the oblique image, and further sets point B on the vanishing line connecting the vanishing point in the perspective projection and point D.
[0047] In this image cropping method, following step S201, the processing unit 10 calculates the coordinates of each key point as described above (step S202).
[0048] In the next step, the processing unit 10 executes a calculation process to calculate a projective transformation coefficient between the oblique image and the vertical image of the road surface corresponding to the oblique image based on the coordinates calculated in step S202 (step S203, calculation step). This point will be described in more detail.
[0049] As shown in FIG. 6, in order to convert a vertical image into an oblique image, it is necessary to obtain the image coordinates of points C' and D' on the vertical image. The x-coordinate of point C' is x c ' and D' are the x coordinates of points x d Regarding ', the relationship of the following formula (12) is obtained as shown in FIG.
number
[0050] On the other hand, the y coordinates of points C' and D' are c '(=y d ') is the actual length between points A and B, L X Let the length on the image be l x Similarly, the actual length between points C and D is L Y Let the length on the image be l y Then, as shown in the following formula (13), ly calculated by the following formula (13) is used to calculate by the following formula (14).
number
number
[0051] Here, using the image coordinates of the four points (points A, B, C', and D') before the projective transformation and the four points (points A, B, C, and D) after the projective transformation, the projective transformation coefficients (a1 to a8) are calculated according to the quadratic projective transformation formula shown in the following formula (15).
number
[0052] The trimming range in this embodiment is the upper end position V A (y=S y ), and bottom position V C (y=-S y ) is the range of the oblique image between the y coordinates of the oblique image after the projective transformation (oblique image), and these coordinates are expressed as S y and -S y It is calculated as:
[0053] If φ=κ=0, then a 11 =1, a 12 =a 13 =0, a 21 =0, a 22 =cosω, a 23 = sinω, a 31 =0, a 32 = sinω, a 33 =cosω. Furthermore, the result is not affected by X0 and Y0, so if X0=Y0=0, then Y C From the above formula (5), we obtain the following formula (16).
number
[0054] where Z0 is the camera position (height) and 2S x x2S y is the image size, f is the focal length, and ω is the camera tilt (depression angle). Also, the ground coordinate Y A In the above formula (16), S y =y aIf we set =0, we get the following equation (17).
number
[0055] On the other hand, the ground coordinates for points A and B are obtained from the above equation (8) as shown in the following equation (18). Furthermore, in the following equation (18), by setting ya=yb=0 and rearranging the above equation (13) using the above equations (16) to (18), the following equation (19) is obtained.
number
number
[0056] From the above equation (19), the position of point C on the vertical image (y c '=l y It is understood that θ is not affected by the shooting height (height of the camera 110) and is a function of the depression angle ω and the focal length f.
[0057] As described above, in step S203, the projective transformation coefficient between the oblique image and the vertical image is calculated. In the next step, the processing unit 10 acquires an arbitrary oblique image (target oblique image) that is the actual processing target (step S204). In other words, each process of steps S201 to S203 does not have to be performed on the actual processing target oblique image, and can be calculated in advance based on a previously acquired oblique image. In this embodiment, since the target oblique image is acquired in step S103, each process of steps S201 to S203 is performed using the target oblique image that is the actual processing target. In this case, step S204 is performed prior to step S201.
[0058] Next, the processing unit 10 determines a cropping range in the target oblique image (step S205: cropping process, cropping step). More specifically, in step S205, the processing unit 10 determines the top end position V of the vertical image in the y direction, which is the traveling direction of the vehicle SC, based on the projective transformation coefficients calculated in steps S201 to S203. A (y=S y ), and bottom position V C (y=-S y ) is set as the cropping range.
[0059] At the same time, the processing unit 10 adjusts the nighttime image (step S206: adjustment process). More specifically, in this step S206, when the target oblique image is a nighttime image captured by the camera 110 at night, the processing unit 10 adjusts the cropping range determined in step S205 so that the cropping range corresponds to a relatively bright bright area in the nighttime image. The bright area includes a portion illuminated by the headlights of the vehicle SC.
[0060] FIG. 7 shows an example of the trimming range illustrated in a target oblique image (nighttime image). FIG. 7(a) shows the case with camera calibration, and FIG. 7(b) shows the case without camera calibration. In FIG. 7, the trimming range Rt is illustrated as a light gray band-like area. The initial input in determining the trimming range Rt is the y coordinate (y a ) and the TX value for point A. The TX value only needs to be an approximate value, and furthermore, this value does not affect the results, whereas the trimming range Rt is affected by the deviation between the optical axis and the center of the screen in the y direction.
[0061] Therefore, in this embodiment, for the ternary image of the target oblique image (nighttime image), the input value of the y coordinate for point A is gradually shifted (for example, by 0.5 mm) from the center of the image, and the y coordinate when the number of pixels corresponding to the bright part in the acquired trimming range Rt is the maximum value is adopted (this point will be described in detail later). In FIG. 7, the y coordinate obtained by the above trimming process is a 7 shows the trimming range Rt at θ = -2.0 mm. It can be seen from Fig. 7 that the bright area in the target oblique image has been trimmed.
[0062] Incidentally, FIG. 7(a) uses the results of camera calibration (f=24.558 mm, Z0=2.370 m, ω=12°28'19"), while FIG. 7(b) shows a case where no camera calibration is performed and the focal length is the nominal value and the height and tilt of the camera 110 are the results of simplified measurement (f=24 mm, Z0=2.4 m, ω=13°). It can be seen from FIG. 7 that the trimming range Rt is hardly affected by camera calibration. In other words, even if camera calibration is not performed, the above method is executable as long as the camera specifications (focal length, image size, sensor size) and the approximate tilt (angle ω) of the camera 110 can be estimated.
[0063] On the other hand, as described above, the trimming range Rt is affected by the focal length f and the depression angle, and the trimming range Rt for a wide-angle lens increases in proportion to the angle of view. Also, the smaller the depression angle, the closer the camera 110 is to being horizontal with respect to the ground, so the trimming range Rt becomes narrower.
[0064] Fig. 8 shows the relationship between focal length (wide angle: f=24mm, standard: f=50mm) and depression angle in the cropping range when calibration is not taken into consideration. The range between the upper and lower ends at each depression angle is the cropping range, and the colored parts in the figure correspond to the cropping range Rt in Fig. 7. As shown in Fig. 8, it can be seen that the cropping range Rt becomes wider as the depression angle increases, but when focusing on the bright area in the center of the image, a depression angle of 10° to 15° is deemed appropriate.
[0065] As described above, the process according to this embodiment enables the automation of quantitative trimming regardless of the presence or absence of camera calibration. One of its features is that points A, B, C, and D are each placed on the same line parallel to the x-axis, and points C and D are further arranged characteristically on the screen, making it possible to calculate the ground coordinate values and image coordinates of each key point based on the theory of photogrammetry. Another feature is the use of geometric features in perspective projection images. That is, it is noted that point C exists on the vanishing line connecting the vanishing point and point A, and its X coordinate is the same as the X coordinate for point A. The same is true for the X coordinate of point B.
[0066] In particular, by utilizing the geometric features of perspective projection images, it is possible to match oblique and vertical images by projective transformation even for monotonous images without texture or feature points, and the top edge position V A ·Lower end position V C By calculating the y coordinates of the corresponding points in the oblique image, it becomes possible to trim the oblique image. On the other hand, the trimming range Rt is automatically calculated from the ratio of bright areas in the trimming range Rt in the ternary image, making it possible to quantitatively determine the trimming range.
[0067] Here, the adjustment process of the processing unit 10 in step S206 will be described in more detail. FIG. 9 is a diagram showing a nighttime image at each processing stage. FIG. 9(a) shows a nighttime image (nighttime illumination driving image) input from the camera 110 (vehicle-mounted device 100). In step S206, the processing unit 10 first executes a ternarization process to generate a ternarized image Qt in which the nighttime image Ql is ternarized into the brightest bright area BP, the darkest dark area DP, and a dim light area MP with a brightness between the bright area BP and the dark area DP (see FIG. 9(b)). For the ternarization at this time, for example, Otsu's multilevel thresholding method can be used.
[0068] Next, as shown in Fig. 9(c), the processing unit 10 performs a labeling process for defining a plurality of pixel groups by labeling the ternary image Qt. At the same time, the processing unit 10 performs an extraction process for extracting a candidate pixel group CU that includes the center of the nighttime image Ql and has the largest label area from a candidate pixel group in the bright area among the plurality of pixel groups. Fig. 9(c) shows the nighttime image QL in which the candidate pixel group CU in the bright area is illustrated in light gray.
[0069] Then, as shown in (a) and (b) of Fig. 10, the processing unit 10 performs a logical AND process to adjust the trimming range Rt so that the trimming range Rt corresponds to the bright part BP at least in the y direction by taking the logical AND of the candidate pixel group CU extracted in the extraction process and the trimming range Rt. This results in the adjusted trimming range. Fig. 10(b) shows the trimming range Rt before adjustment and the trimming range Rv after adjustment that takes into account the bright part for the original nighttime image Ql.
[0070] Fig. 11 is a diagram showing a state in which a trimming range is displayed on a nighttime image. Fig. 11(a) shows a trimming range Rm set by an operator, and Fig. 11(b) shows a trimming range Rv automatically generated by the image trimming method according to the present embodiment. As shown in Fig. 11, the trimming range Rm has a shape that suppresses stripes between frames, and the maximum range for ensuring brightness is set. In contrast, the trimming range Rv has a sufficient range for ensuring brightness, and it can be seen that it is comparable to the trimming range Rm.
[0071] Thereafter, the processing unit 10 cuts out the cropping area Rv from the target oblique image (here, the nighttime image Ql), thereby completing the cropping process. If the target oblique image is not a nighttime image, step S206 is omitted, and the cropping area Rt is cut out from the target oblique image.
[0072] This completes the image cropping method according to this embodiment. In the image processing method according to this embodiment, the processing unit 10 then performs projective transformation of the image (step S105). Here, since the cropping range Rv (or cropping range Rt, the same applies below) has already been cut out from the target oblique image in step S104, the target of projective transformation is only the cropping range Rv (cropped image). This reduces the processing load compared to when the entire target oblique image is projectively transformed. Thereafter, the processing unit 10 joins together a plurality of cropped images after projective transformation to create an orthoimage that conforms to the road surface (road surface orthoimage) (step S106).
[0073] Fig. 12 is a diagram showing a road surface orthoimage. Fig. 12(a) shows a road surface orthoimage IA generated by joining the cropping ranges Rm shown in Fig. 11(a), and Fig. 12(b) shows a road surface orthoimage IB generated by joining the cropping ranges Rv automatically generated by the image cropping method according to this embodiment. Comparing the road surface orthoimage IA and the road surface orthoimage IB, the boundaries between frames in the road surface orthoimage IB are linear, making it less likely that an unnatural feeling will occur.
[0074] As described above, in the image processing device (image cropping device) 1 according to the present embodiment and the image cropping method implemented by the image processing device 1, first, a projection transformation coefficient between an oblique image obtained by capturing an image of a road surface with the on-vehicle camera 110 and a vertical image corresponding to the oblique image is calculated. Then, based on the calculated projection transformation coefficient, the top end position V of the vertical image in the y direction is calculated for the target oblique image, which is the oblique image that is actually the subject of cropping. A and bottom position V C The range between a pair of y coordinates corresponding to is set as the trimming range Rt and is trimmed out. This ensures that the quality of the final product obtained by trimming is high. Furthermore, by quantitatively trimming using numerical calculations, there is no room for the experience or ability of the worker to affect the quality of the final product, and the quality of the final product is stabilized at a high quality and quality fluctuations are suppressed. Note that the oblique image used to calculate the projective transformation coefficients and the target oblique image to be trimmed may be the same or different.
[0075] In the image processing device 1 according to the present embodiment, the processing unit 10 sets four key points, namely, points A, B, C, and D, on the oblique image in the calculation process, and calculates a projection transformation coefficient based on the four key points. In particular, when the vehicle width direction of the vehicle SC is the x direction, the processing unit 10 sets points A and B of the four key points on a first line parallel to the x axis that defines the x direction, and sets points C and D of the four key points on a second line parallel to the x axis and different from the first line. Therefore, the y coordinates of points A and B, and points C and D are the same, and the processing load related to the calculation of the projection transformation coefficient is reduced.
[0076] In the image processing device 1 according to the present embodiment, the processing unit 10, in the calculation process, sets point C at the lower end of the oblique image in the y direction, on the vanishing line connecting the vanishing point in the perspective projection and point A, sets point D at the end of the oblique image in the x direction, and sets point B on the vanishing line connecting the vanishing point in the perspective projection and point D. In this case, the processing load related to the calculation of the projective transformation coefficients is further reduced.
[0077] Furthermore, in the image processing device 1 according to the present embodiment, when the target oblique image includes a nighttime image captured by the camera 110 at night, the processing unit 10 further performs an adjustment process to adjust the trimming range Rt after the trimming process so that the trimming range Rt corresponds to a relatively bright bright part BP in the nighttime image. Therefore, in the oblique image captured at night, the trimming range Rt is adjusted so as to correspond to the bright part BP. This suppresses quality degradation at night.
[0078] In addition, in the image processing device 1 according to this embodiment, the processing unit 10 performs a ternarization process before the adjustment process to generate a ternarized image Qt that is ternarized into the brightest bright area BP, the darkest dark area DP, and the dim light area MP that is between the bright area BP and the dark area DP. Therefore, after the bright area BP is appropriately extracted by the ternarization process, the trimming range Rt can be adjusted to correspond to the bright area BP. Therefore, it is possible to suitably suppress quality degradation at night.
[0079] Furthermore, in the image processing device 1 according to the present embodiment, the processing unit 10 performs, in the adjustment process, a labeling process for defining a plurality of pixel groups by labeling the ternary image Qt, an extraction process for extracting a candidate pixel group CU that includes the center of the nighttime image Ql and has the largest label area from the candidate pixel group in the bright area among the plurality of pixel groups after the labeling process, and a logical AND process for adjusting the trimming range Rt so that the trimming range Rt corresponds to the bright area at least in the y direction by taking the logical AND between the candidate pixel group CU extracted in the extraction process and the trimming range Rt. This makes it possible to easily and reliably adjust the trimming range Rt.
[0080] The above embodiment describes one embodiment of the image cropping device and image cropping method according to the present disclosure. Therefore, the image cropping device and image cropping method according to the present disclosure can be modified.
[0081] For example, in the above embodiment, the processing unit 10 reduces the processing load related to the calculation of the projective transformation coefficients by setting four characteristic key points, namely, points A, B, C, and D, in step S201. However, the method of selecting the key points is not limited to the above embodiment and may be any method.
[0082] Furthermore, in the above embodiment, the processing unit 10 adjusts the trimming range Rt in accordance with the bright part BP in step S206. However, if the target oblique image is not a nighttime image, step S206 can be omitted. [Explanation of symbols]
[0083] 1...image processing device (image cropping device), 10...processing unit, 110...camera, BP...bright area, DP...dark area, MP...twilight area, CU...candidate pixel group, Rt...cropping range, Rv...cropping range, Ql...nighttime image, Qt...ternary image.
Claims
1. A processing unit is provided for processing images captured by a camera installed in the vehicle, The processing unit includes: A calculation process of calculating a projective transformation coefficient between an oblique image including an image of the road surface on which the vehicle is traveling, which is obtained by capturing an image of the road surface with the camera, and a vertical image of the road surface corresponding to the oblique image; an acquisition process for acquiring a target oblique image, which is the oblique image to be processed, from the camera; a trimming process is executed after the calculation process and the acquisition process, to determine a trimming range of the oblique target image between a pair of y coordinates of the oblique target image corresponding to an upper end and a lower end of the vertical image in a y direction, which is a traveling direction of the vehicle, based on the projective transformation coefficient, and to cut out the trimming range from the oblique target image. Image cropping device.
2. the processing unit, in the calculation process, sets four key points, namely, points A, B, C, and D, on the oblique image, and calculates the projective transformation coefficients based on the four key points; When a vehicle width direction of the vehicle is defined as an x direction, the processing unit sets the points A and B of the four key points on a first straight line parallel to an x axis that defines the x direction, and sets the points C and D of the four key points on a second straight line that is parallel to the x axis and different from the first straight line. The image cropping device according to claim 1 .
3. The processing unit, in the calculation process, point C is set on a vanishing line connecting a vanishing point in a perspective projection and point A at a lower end of the oblique image in the y direction; The point D is set at an end of the oblique image in the x direction; The point B is set on the vanishing line connecting the vanishing point in the perspective projection and the point D. The image cropping device according to claim 2 .
4. When the target oblique image includes a nighttime image captured by the camera at night, the processing unit further performs, after the trimming process, an adjustment process of adjusting the trimming range so that the trimming range corresponds to a relatively bright bright area in the nighttime image. The image cropping device according to any one of claims 1 to 3.
5. The processing unit executes a ternarization process before the adjustment process to generate a ternarized image in which the nighttime image is ternarized into the brightest bright area, the darkest dark area, and a dim light area having a brightness between the bright area and the dark area.
5. The image cropping device according to claim 4.
6. In the adjustment process, the processing unit A labeling process for defining a plurality of pixel groups by labeling the ternarized image; an extraction process for extracting a candidate pixel group that includes a center of the nighttime image and has a maximum label area from the candidate pixel group in the bright area among the plurality of pixel groups after the labeling process; a logical product process for adjusting the trimming range so that the trimming range corresponds to the bright area at least in the y direction by calculating a logical product between the candidate pixel group extracted in the extraction process and the trimming range; Execute The image cropping device according to claim 5.
7. An image cropping method for performing a cropping process on an image captured by a camera installed in a vehicle, comprising: a calculation step of calculating a projective transformation coefficient between an oblique image including an image of the road surface on which the vehicle is traveling, which is obtained by capturing an image of the road surface with the camera, and a vertical image of the road surface corresponding to the oblique image; an acquisition step of acquiring a target oblique image, which is the oblique image to be processed, from the camera; a trimming step of, after the calculation step and the acquisition step, determining a trimming range of the oblique target image between a pair of y coordinates of the oblique target image corresponding to an upper end and a lower end of the vertical image in a y direction, which is a traveling direction of the vehicle, based on the projective transformation coefficient, and cutting out the trimming range from the oblique target image; A method for cropping an image comprising:
Citation Information
Patent Citations
Crack analysis data editing device, crack analysis data editing method, and crack analysis data editing program
JP2021107683A
