Camera system for work machinery and calibration method for camera parameters

Abstract

To accurately configure multiple imaging apparatuses without using markers.SOLUTION: A controller acquires a first captured image and a second captured image. The controller generates a first transformed image and a second transformed image by partially deforming each of the first captured image and the second captured image such that overlapping portions of their imaging ranges are continuous with each other. The controller extracts feature points where feature quantities correspond between the first captured image and the second captured image, and between the first transformed image and the second transformed image. The controller calibrates external parameters on the basis of multiple corresponding points with aligned coordinate systems.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to a camera system for a work machine and a method for calibrating camera parameters. [Background technology]

[0002] Patent Document 1 discloses a technique for configuring multiple cameras mounted on a vehicle. According to the technique described in Patent Document 1, a vehicle is parked on a grid drawn on the ground, and each camera is calibrated based on images of the grid captured by the multiple cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-211707 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 requires that grid-shaped markers be prepared in advance, making it difficult to perform calibration at any location, resulting in low calibration accuracy. An object of the present disclosure is to provide a camera system for a work machine and a method for calibrating camera parameters that can configure multiple imaging devices with high accuracy without using marks. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, a camera system for a work machine includes a first imaging device and a second imaging device that capture images of the outside of the work machine and have imaging ranges that partially overlap each other, and a controller that outputs images to a display device based on a first captured image captured by the first imaging device and a second captured image captured by the second imaging device and external parameters of the first imaging device and the second imaging device, and the controller acquires the first captured image and the second captured image, and distorts and deforms each of the first captured image and the second captured image so that the overlapping portions of the imaging ranges are continuous with each other to generate a first deformed image and a second deformed image, and extracting a plurality of corresponding imaging points, which are feature points whose feature amounts correspond to each other in an overlapping portion of the imaging range, from the extracted corresponding imaging points, extracting a plurality of corresponding deformation points, which are feature points whose feature amounts correspond to each other in an overlapping portion of the imaging range of the first deformed image and the second deformed image, converting positions of the plurality of corresponding deformation points to positions in the first captured image or the second captured image, or converting positions of the plurality of corresponding imaging points to positions in the first deformed image or the second captured image, thereby aligning coordinate systems of the plurality of corresponding imaging points and the plurality of corresponding deformation points, and calibrating the external parameters based on the plurality of corresponding imaging points and the plurality of corresponding deformation points whose coordinate systems have been aligned. [Effects of the Invention]

[0006] According to the above aspect, the camera system can accurately configure the plurality of image capturing devices without using marks. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a perspective view showing the appearance of a work machine according to a first embodiment. [Figure 2] 1 is a schematic diagram showing the internal configuration of a work machine according to a first embodiment. [Figure 3] 3 is a diagram showing the imaging ranges of a plurality of imaging devices provided in the work machine according to the first embodiment. FIG. [Figure 4]FIG. 2 is a schematic block diagram showing the configuration of a controller according to the first embodiment. [Figure 5] 5A to 5C are diagrams illustrating examples of deformation of a captured image according to the first embodiment. [Figure 6] 3A and 3B are diagrams illustrating examples of captured corresponding points and deformation corresponding points according to the first embodiment. [Figure 7] 10 is a flowchart (part 1) showing a process of calibrating external parameters of the imaging device by a controller according to the first embodiment. [Figure 8] 10 is a flowchart (part 2) showing the process of calibrating the external parameters of the imaging device by the controller according to the first embodiment. [Figure 9] FIG. 10 is a diagram showing the imaging ranges of a plurality of imaging devices provided on a work machine according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] First Embodiment <Configuration of work machine 100> Hereinafter, the embodiments will be described in detail with reference to the drawings. FIG. 1 is a perspective view showing the appearance of a work machine according to a first embodiment. The work machine 100 according to the first embodiment is, for example, a bulldozer. The work machine 100 includes a vehicle body 110, a traveling device 120, a work implement 130, and a driver's cab 140.

[0009] The traveling device 120 is provided on the bottom of the vehicle body 110. The traveling device 120 has a pair of crawlers 121. The left and right crawlers 121 rotate, respectively, causing the work machine 100 to travel. The work machine 130 is used to excavate and transport excavation targets such as earth and sand. The work machine 130 has a blade 131 for pushing earth and sand, and a lift cylinder 132 for driving the blade 131 up and down. The blade is disposed in front of the vehicle body 110.

[0010] FIG. 2 is a schematic diagram showing the internal configuration of the work machine according to the first embodiment. The body 110 of the work machine 100 is equipped with an engine 111 , a hydraulic pump 112 , a control valve 113 , and a transmission 114 . The engine 111 is a prime mover that drives the hydraulic pump 112. The hydraulic pump 112 is a variable displacement pump that is driven by the engine 111 and discharges hydraulic oil. The control valve 113 controls the flow rate of the hydraulic oil supplied from the hydraulic pump 112. The lift cylinder 132 is driven by the hydraulic oil that is supplied from the hydraulic pump 112 and whose flow rate is controlled by the control valve 113. The transmission 114 transmits the driving force of the engine 111 to the traveling device 120. Examples of the transmission 114 include an HST (Hydraulic Static Transmission) and a torque converter. The transmission 114 rotates the left and right crawlers 121 of the traveling device 120 independently. For example, the work machine 100 may be provided with separate transmissions 114 for the left and right crawlers 121, or the power transmitted from a single transmission 114 may be distributed to the left and right crawlers 121 by a differential device.

[0011] The cab 140 is a space where an operator sits and operates the work machine 100. The cab 140 is provided on top of the vehicle body 110. Inside the cab 140 are provided a seat for the operator to sit in, levers and pedals for operating the work machine 100, and a control system 150 that controls the work machine 100. Levers are provided on the left and right of the seat, and receive travel commands for the traveling gear 120 and operation commands for the work implement 130. The control system 150 is provided with a monitor 143 for displaying the status of the work machine 100. The monitor 143 is provided in a position inside the cab 140 that can be seen by the operator seated in the seat. The monitor 143 displays at least an overhead image of the work machine 100, which will be described later.

[0012] FIG. 3 is a diagram showing the imaging ranges of the multiple imaging devices 142 provided on the work machine 100 according to the first embodiment. Multiple imaging devices 142 are provided above the cab 140. Specifically, above the cab 140, there are provided an imaging device 142a that images the forward range Ra, an imaging device 142b that images the right range Rb, an imaging device 142c that images the rear range Rc, and an imaging device 142d that images the left range Rd. The imaging direction (line of sight) of each imaging device 142 is directed diagonally downward, and the imaging range includes at least the ground surface near the work machine 100. The imaging ranges of the multiple imaging devices 142 cover the entire periphery of the work machine 100. In other words, each of the multiple imaging devices 142 is an imaging device that images the outside of the work machine 100.

[0013] The imaging ranges of the multiple imaging devices 142 partially overlap with one another. Specifically, the imaging range Ra of imaging device 142a partially overlaps with the imaging range Rb of imaging device 142b and the imaging range Rd of imaging device 142d. The imaging range Rb of imaging device 142b partially overlaps with the imaging range Ra of imaging device 142a and the imaging range Rc of imaging device 142c. The imaging range Rc of imaging device 142c partially overlaps with the imaging range Rb of imaging device 142b and the imaging range Rd of imaging device 142d. The imaging range Rd of imaging device 142d partially overlaps with the imaging range Ra of imaging device 142a and the imaging range Rc of imaging device 142c.

[0014] Control system configuration FIG. 4 is a schematic block diagram showing the configuration of a control system 150 according to the first embodiment. The control system 150 includes a controller 141 , an imaging device 142 , and a monitor 143 . The controller 141 controls the operation of the work machine 100 and the display on the monitor 143. An operation panel, gauges, levers, pedals, and switches are attached to the controller 141. The controller 141 receives lever inputs and controls the drive of the engine 111, hydraulic pump 112, and transmission 114. The controller 141 displays the operation of the operation panel and measurement information from the gauges on the monitor 143. The operator can check the status of the work machine 100 by visually checking the monitor 143.

[0015] Furthermore, the controller 141 transforms and combines the captured images captured by the multiple imaging devices 142 to generate a bird's-eye image of the work site viewed from above in plan, with the work machine 100 at the center, and displays this on the monitor 143. That is, the controller 141 transforms each of the multiple captured images into a bird's-eye coordinate system and combines the transformed captured images (deformed images) to generate the bird's-eye image. That is, the control system 150 according to the first embodiment is an example of a camera system. The control system 150 may display the overhead image and the measurement information of the instruments on one monitor 143, or may have separate monitors 143 for displaying the overhead image and for displaying the measurement information, and display the overhead image and the measurement information separately on each monitor 143.

[0016] The controller 141 is a computer that includes a processor 210 , a main memory 230 , a storage 250 , and an interface 270 .

[0017] Storage 250 is a non-transitory tangible storage medium. Examples of storage 250 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), and semiconductor memory. Storage 250 may be an internal medium directly connected to the bus of controller 141, or may be an external medium connected to controller 141 via interface 270 or a communication line. Storage 250 stores a program for implementing monitoring of the surroundings of work machine 100.

[0018] The storage 250 also records external parameters of each image capture device 142. The external parameters of the image capture device 142 are parameters for converting coordinates in a coordinate system (camera coordinate system) based on the image capture device 142 into coordinates in a vehicle body coordinate system based on the work machine 100. One axis (for example, the X-axis) of the camera coordinate system extends in the line of sight direction of the image capture device 142. The external parameters represent the three-dimensional position and tilt (line of sight direction) of the image capture device 142 in the vehicle body coordinate system. In other words, the external parameters represent the amount of parallel translation of each axis and the amount of rotation around each axis for converting the camera coordinate system into the vehicle body coordinate system.

[0019] The program may be for realizing some of the functions to be performed by the controller 141. For example, the program may be combined with other programs already stored in the storage 250 or other programs implemented in other devices to perform the functions. Note that the controller 141 according to other embodiments may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor may be realized by the integrated circuit.

[0020] By executing the program, the processor 210 functions as an image acquisition unit 211, an image deformation unit 212, a feature calculation unit 213, a corresponding point extraction unit 214, a coordinate conversion unit 215, a calibration unit 216, an image generation unit 217, and a display control unit 218.

[0021] The image acquisition unit 211 acquires a captured image P1 from each imaging device 142. Hereinafter, the captured image captured by imaging device 142a will be referred to as P1a, the captured image captured by imaging device 142b will be referred to as P1b, the captured image captured by imaging device 142c will be referred to as P1c, and the captured image captured by imaging device 142d will be referred to as P1d.

[0022] The image transformation unit 212 transforms each captured image P1 acquired by the image acquisition unit 211 into an image of the site viewed from above in a plan view. The captured image P1 transformed by the image transformation unit 212 is called a transformed image P2. More specifically, an image obtained by transforming captured image P1a is called a transformed image P2a, an image obtained by transforming captured image P1b is called a transformed image P2b, an image obtained by transforming captured image P1c is called a transformed image P2c, and an image obtained by transforming captured image P1d is called a transformed image P2d. The transformed image P2 generated by the image transformation unit 212 is used to generate an overhead image. For example, the image transformation unit 212 performs viewpoint conversion processing on each captured image P1 based on external parameters recorded in the storage 250 and parameters of the viewpoint of the overhead image (parameters of the overhead coordinate system) specified in advance.

[0023] FIG. 5 is a diagram showing an example of deformation of the captured image P1 according to the first embodiment. The image deformation unit 212 performs deformation processing (distortion deformation) to stretch the captured image P1 in the vertical direction and stretch the portion showing a distant object in the horizontal direction. The image deformation unit 212 deforms the image by projective transformation. As shown in FIG. 3, the imaging ranges of the imaging devices 142 partially overlap, so the same subject is captured in a pair of deformed images P2 generated from the captured images P1 of adjacent imaging devices 142. The image deformation unit 212 generates a pair of deformed images P2 in which the overlapping portions of the imaging ranges are continuous with each other. However, if the external parameters recorded in the storage 250 are not configured, the overlapping portions of the imaging ranges in the pair of deformed images P2 may not necessarily be continuous. Note that the deformed image P2 shown in FIG. 5 is a cropped image of only the vicinity of the work machine 100 for convenience of illustration, but the deformed image P2 may also include portions showing distant views such as trees and the sky. Note that the image deformation unit 212 according to other embodiments may perform deformation processing such as enlargement processing or rotation processing.

[0024] The feature amount calculation unit 213 extracts feature points from each captured image P1 and each deformed image P2 and calculates the feature amount of each feature point. The feature amount calculation unit 213 may simultaneously extract feature points and calculate feature amounts. Feature points are characteristic points in an image. Examples of feature points include points that represent the characteristic shape of an object, as determined by changes in color or brightness in the image, and boundaries between objects. The feature amount calculation unit 213 extracts feature points based on local image features, for example. The feature amount calculation unit 213 extracts points in the image where the local image feature amount is a maximum or minimum as feature points. The values ​​of the local image features do not change significantly depending on changes in scale or rotation. Therefore, feature amounts of a common subject appearing in captured images P1 captured by different image capture devices 142 are calculated as similar values. The feature amount calculation unit 213 may also calculate other local image features, such as SIFT, AKAZE, and SuperPoint.

[0025] The corresponding point extraction unit 214 extracts corresponding points, which are corresponding feature points, from a pair of captured images P1 captured by adjacent imaging devices 142. The corresponding point extraction unit 214 also extracts corresponding points, which are corresponding feature points, from a pair of deformed images P2 generated from the captured images P1 captured by the adjacent imaging devices 142. Hereinafter, the corresponding points extracted from the captured image P1 will be referred to as captured corresponding points, and the corresponding points extracted from the deformed image P2 will be referred to as deformed corresponding points. In particular, the corresponding point extraction unit 214 extracts deformed corresponding points from elongated portions in the deformed image P2. The corresponding point extraction unit 214 extracts corresponding points using a feature point matching algorithm such as LightGlue, SuperGlue, or LoFTR.

[0026] FIG. 6 is a diagram showing examples of imaged corresponding points and deformed corresponding points according to the first embodiment. Because the same subject is captured in areas where the image capture ranges of a pair of captured images overlap, the corresponding point extraction unit 214 can extract corresponding points from areas where the image capture ranges overlap. Even if the image capture ranges of a pair of captured images do not overlap, for example, if an object (such as a white line) exists across the pair of captured images, there is a possibility that corresponding points representing the object will appear. In other words, there is a possibility that corresponding points can be extracted from areas where the image capture ranges do not overlap. As shown in FIG. 6, imaged corresponding points and deformed corresponding points appear in different locations on the image. In the example shown in FIG. 6, it can be seen that imaged corresponding points appear frequently in the vicinity of the work machine 100 and in the distance, while deformed corresponding points appear frequently on the road surface somewhat distant from the work machine 100.

[0027] The coordinate transformation unit 215 transforms the positions of the deformed corresponding points extracted by the corresponding point extraction unit 214 into positions in the captured image. The coordinate transformation unit 215 can transform the coordinates of the deformed corresponding points by performing an inverse transformation of the transformation performed by the image transformation unit 212.

[0028] The calibration unit 216 calibrates the external parameters of the imaging device 142 based on the captured corresponding points extracted from the captured image P1 of the imaging device 142 and the deformed corresponding points extracted from the deformed image P2 and converted to positions in the captured image. The calibration unit 216 calibrates the external parameters by, for example, the bundle method (bundle adjustment).

[0029] The image generating unit 217 synthesizes the transformed image P2 generated using the configured external parameters to generate an overhead image. The display control unit 218 causes the overhead image generated by the image generation unit 217 to be displayed on the display.

[0030] <<Calibration Process of External Parameters of Imaging Device 142>> FIG. 7 is a flowchart (part 1) showing a process of calibrating the external parameters of the imaging device 142 by the controller 141 according to the first embodiment. FIG. 8 is a flowchart (part 2) showing a process of calibrating the external parameters of the imaging device 142 by the controller 141 according to the first embodiment. The controller 141 according to the first embodiment uses the imaging device 142a, which captures an image of the front, of the multiple imaging devices 142 as a reference and calibrates the external parameters of the other imaging devices 142b-d. Therefore, in the first embodiment, it is preferable to separately calibrate the external parameters of the imaging device 142a of the multiple imaging devices 142 in advance.

[0031] When an operator issues an instruction to calibrate the imaging device 142 to the controller 141, the image acquisition unit 211 of the controller 141 acquires the captured images P1a and P1b from the imaging devices 142a and 142b (step S1). The image deformation unit 212 performs deformation processing on the acquired captured images P1a and P1b using the external parameters of the imaging devices 142a and 142b recorded in the storage 250, and obtains the deformed images P2a and P2b (step S2).

[0032] The feature amount calculation unit 213 extracts feature amounts from the captured images P1a and P1b (step S3). The corresponding point extraction unit 214 extracts captured corresponding points, which are points having common feature amounts in the captured images P1a and P1b (step S4).

[0033] Furthermore, the feature amount calculation unit 213 extracts feature amounts from the deformed images P2a and P2b (step S5). The corresponding point extraction unit 214 extracts deformed corresponding points, which are points having common feature amounts in the deformed images P2a and P2b (step S6). The coordinate conversion unit 215 converts the positions of the deformed corresponding points in the deformed images P2a and P2b extracted by the corresponding point extraction unit 214 into positions in the captured images P1a and P1b, respectively (step S7).

[0034] The calibration unit 216 calculates the relative relationship (amount of translation and amount of rotation) between the image capturing device 142a and the image capturing device 142b, for example, by the bundle method, based on the captured corresponding points obtained in step S4 and the deformed corresponding points obtained in step S7 (step S8). The calibration unit 216 determines the extrinsic parameters of the image capturing device 142b based on the extrinsic parameters of the image capturing device 142a recorded in the storage 250 and the relative relationship between the image capturing device 142a and the image capturing device 142b calculated in step S8 (step S9). The calibration unit 216 then records the extrinsic parameters of the image capturing device 142b in the storage 250 (step S10).

[0035] Next, the image acquisition unit 211 acquires the captured images P1a and P1d from the imaging devices 142a and 142d (step S11). The image deformation unit 212 performs deformation processing on the acquired captured images P1a and P1d using the external parameters of the imaging devices 142a and 142d recorded in the storage 250, thereby obtaining deformed images P2a and P2d (step S12).

[0036] The feature amount calculation unit 213 extracts feature points from the captured images P1a and P1d (step S13). The corresponding point extraction unit 214 extracts captured corresponding points, which are feature points common to the captured images P1a and P1d (step S14).

[0037] Furthermore, the feature amount calculation unit 213 extracts feature points from the deformed images P2a and P2d (step S15). The corresponding point extraction unit 214 extracts deformed corresponding points, which are feature points common to the deformed images P2a and P2d (step S16). The coordinate conversion unit 215 converts the positions of the deformed corresponding points in the deformed images P2a and P2d extracted by the corresponding point extraction unit 214 into positions in the captured images P1a and P1d, respectively (step S17).

[0038] The calibration unit 216 calculates the relative positional relationship between the imaging devices 142a and 142d based on the imaged corresponding points obtained in step S14 and the deformed corresponding points obtained in step S17 (step S18). The calibration unit 216 determines the extrinsic parameters of the imaging device 142d based on the extrinsic parameters of the imaging device 142a recorded in the storage 250 and the relative positional relationship between the imaging devices 142a and 142d calculated in step S18 (step S19). Then, the calibration unit 216 records the extrinsic parameters of the imaging device 142d in the storage 250 (step S20).

[0039] Next, the image acquisition unit 211 acquires captured images P1b, P1c, and P1d from the imaging devices 142b, 142c, and 142d (step S21). The image deformation unit 212 performs deformation processing on the acquired captured images P1b, P1c, and P1d using the external parameters of the imaging devices 142b, 142c, and 142d recorded in the storage 250, and obtains deformed images P2b, P2c, and P2d (step S22).

[0040] The feature amount calculation unit 213 extracts feature points from the captured images P1b, P1c, and P1d (step S23). The corresponding point extraction unit 214 extracts captured corresponding points, which are feature points common to the captured images P1b, P1c, and P1d (step S24).

[0041] Furthermore, the feature amount calculation unit 213 extracts feature points from the deformed images P2b, P2c, and P2d (step S25). The corresponding point extraction unit 214 extracts deformed corresponding points between the deformed images P2b and P2c, and between the deformed images P2c and P2d (step S26). The coordinate conversion unit 215 converts the positions of the deformed corresponding points extracted by the corresponding point extraction unit 214 in the deformed images P2b, P2c, and P2d into positions in the captured images P1b, P1c, and P1d, respectively (step S27).

[0042] The calibration unit 216 calculates the relative positional relationship between the imaging devices 142b and 142c and the relative positional relationship between the imaging devices 142c and 142d based on the imaged corresponding points obtained in step S24 and the deformed corresponding points obtained in step S27 (step S28). The calibration unit 216 determines the extrinsic parameters of the imaging device 142c based on the extrinsic parameters of the imaging devices 142b and 142c recorded in the storage 250 and the relative positional relationship between the imaging devices 142b and 142c and the relative positional relationship between the imaging devices 142c and 142d calculated in step S28 (step S29). The calibration unit 216 then records the extrinsic parameters of the imaging device 142c in the storage 250 (step S30).

[0043] This allows the controller 141 to calibrate the external parameters of all the image capture devices 142 with the image capture device 142a as the reference.

[0044] Thereafter, image deformation unit 212 generates deformed images using the external parameters updated by the above-described calibration process, and image generation unit 217 synthesizes the deformed images to generate an overhead image. Since the external parameters of each image capture device 142 are calibrated using corresponding points extracted from the deformed images used to generate the overhead image, it is guaranteed that the joints between the deformed images in the overhead image will be continuous.

[0045] Actions and Effects According to the first embodiment, the controller 141 calibrates external parameters for a first imaging device and a second imaging device (e.g., imaging device 142a and imaging device 142b) whose imaging ranges partially overlap each other, using the following procedure. The controller 141 acquires a first captured image captured by the first imaging device and a second captured image captured by the second imaging device. The controller 141 distorts and deforms the first captured image and the second captured image so that the overlapping portions of their imaging ranges are continuous with each other, thereby generating a first deformed image and a second deformed image. The controller 141 extracts a plurality of corresponding imaging points from the first captured image and the second captured image. The controller 141 extracts a plurality of deformed corresponding points from the first deformed image and the second deformed image. The controller 141 aligns the coordinate systems of the plurality of corresponding imaging points and the plurality of deformed corresponding points by converting the positions of the plurality of deformed corresponding points to positions in the first captured image or the second captured image. The controller 141 calibrates external parameters based on the plurality of corresponding imaging points and the plurality of deformed corresponding points whose coordinate systems have been aligned.

[0046] This allows the controller 141 to extract corresponding points from both the captured image and the deformed image. This allows the controller 141 to increase the number of corresponding points that may be insufficient with only the captured image. This is because the feature amounts (local image feature amounts) used to extract corresponding points are not easily affected by enlargement or rotation, but their values ​​change when the image is distorted or deformed, such as by stretching it. This allows the external parameters to be calibrated with high accuracy without using landmarks such as markers.

[0047] Furthermore, according to the first embodiment, by extracting corresponding points from the deformed image, it is possible to create a deformed image so that the correspondence relationship of overlapping portions in the overhead image matches, thereby allowing the controller 141 to guarantee that the joints between the deformed images in the overhead image are continuous.

[0048] Other Embodiments Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design modifications are possible. That is, in other embodiments, the order of the above-described processes may be changed as appropriate. Furthermore, some processes may be executed in parallel. The controller 141 according to the above-described embodiments may be configured by a single computer, or the configuration of the controller 141 may be divided among multiple computers that work together to function as the controller 141. In this case, some of the computers that make up the controller 141 may be mounted inside the work machine, and other computers may be provided outside the work machine. Furthermore, the monitor 143 according to other embodiments may be provided outside the work machine 100.

[0049] Although the controller 141 according to the above embodiment converts the positions of the deformation corresponding points into positions corresponding to the captured image, this is not limiting. For example, the controller 141 according to other embodiments may perform calibration by converting the positions of the captured corresponding points into positions corresponding to the deformed image.

[0050] According to the embodiment described above, the controller 141 calibrates the external parameters of the image capturing devices 142b-d using the image capturing device 142a as a reference, but this is not limiting. For example, in another embodiment, the controller 141 may calibrate the external parameters of the image capturing devices 142a-d so that changes from the pre-calibration external parameters recorded in the storage 250 are minimized.

[0051] The work machine 100 according to the embodiment described above is a bulldozer, but is not limited to this. For example, the work machine 100 according to other embodiments may be other work machines such as a hydraulic excavator, a motor grader, or a dump truck. Furthermore, the work machine 100 according to other embodiments may not be a traveling machine, as long as it has the function of generating overhead images from a plurality of imaging devices.

[0052] 9 is a diagram showing the imaging ranges of multiple imaging devices 142 provided on a work machine 100 according to another embodiment. The work machine 100 according to another embodiment may be a hydraulic excavator. The work machine 100 according to another embodiment is provided with multiple imaging devices 142 that capture images of the surroundings. Specifically, the work machine 100 is provided with an imaging device 142e that images the left rear range Re of the surroundings of the work machine 100, an imaging device 142f that images the rear range Rf of the surroundings of the work machine 100, an imaging device 142g that images the right rear range Rg of the surroundings of the work machine 100, and an imaging device 142h that images the right front range Rh of the surroundings of the work machine 100.

[0053] The imaging ranges of the multiple imaging devices 142 cover the entire circumference of the work machine 100 excluding the left front range Ri that is visible from the cab 140. At this time, the controller 141 calibrates the external parameters of each imaging device 142 in the following procedure. For example, the controller 141 calibrates the external parameters of imaging device 142f based on imaging device 142e. Next, the controller 141 calibrates the external parameters of imaging device 142g based on the external parameters of the calibrated imaging device 142f. Then, the controller 141 calibrates the external parameters of imaging device 142h in order based on the external parameters of the calibrated imaging device 142g.

[0054] The work machine 100 according to the embodiment described above extracts corresponding points and calibrates external parameters based on captured images and deformed images taken at a certain point in time, but this is not limited to this. For example, a work machine 100 according to another embodiment may extract corresponding points and calibrate external parameters based on a time series of captured images. For example, the controller 141 extracts corresponding points and calibrates external parameters based on the time series in the following procedure. The controller 141 acquires captured images captured by the pair of imaging devices and extracts corresponding points (captured corresponding points and deformed corresponding points). The controller 141 extracts corresponding points each time a change in the captured image is detected. A change in the captured image is detected, for example, when a certain amount of time has passed, when the work machine 100 has moved, or when the amount of movement of a feature point calculated by tracking the feature point has exceeded a predetermined distance. If the number of corresponding points extracted each time a change in the captured image exceeds a predetermined amount, the controller 141 calibrates the external parameters using the collected corresponding points. This makes it possible to calibrate the external parameters using a sufficient number of corresponding points even if only a small number of feature amounts are extracted at a given time. [Explanation of symbols]

[0055] DESCRIPTION OF SYMBOLS 100... Work machine 110... Body 120... Traveling device 130... Work machine 140... Cab 141... Controller 142... Imaging device 210... Processor 211... Image acquisition unit 212... Image deformation unit 213... Feature amount calculation unit 214... Corresponding point extraction unit 215... Coordinate conversion unit 216... Calibration unit 217... Image generation unit 218... Display control unit 230... Main memory 250... Storage 270... Interface P1... Captured image P2... Deformed image

Claims

1. a first imaging device and a second imaging device that capture images of the outside of the work machine and whose imaging ranges partially overlap each other; a controller that outputs an image to a display device based on a first captured image captured by the first imaging device, a second captured image captured by the second imaging device, and external parameters of the first imaging device and the second imaging device; Equipped with The controller acquiring a first captured image and a second captured image; generating a first deformed image and a second deformed image by deforming a portion of the first captured image and the second captured image so that the first captured image and the second captured image are continuous with each other; extracting a plurality of corresponding image points, the feature amounts of which correspond to each other, from the first captured image and the second captured image; extracting a plurality of corresponding deformation points in the imaging range of the first deformed image and the second deformed image; converting the positions of the plurality of deformation corresponding points into positions in the first captured image or the second captured image, or converting the positions of the plurality of image corresponding points into positions in the first deformed image or the second captured image, thereby aligning the coordinate systems of the plurality of image corresponding points and the plurality of deformation corresponding points; The external parameters are calibrated based on the plurality of captured corresponding points and the plurality of deformed corresponding points whose coordinate systems are aligned. Camera systems for work machines.

2. the deformation includes a process of stretching an overlapping portion of the imaging range; The controller extracts the plurality of deformation corresponding points from the stretched portion. The camera system for a work machine according to claim 1 .

3. The stretched portion is a portion showing the ground around the work machine. The camera system for a work machine according to claim 2 .

4. The controller After the external parameters of the first imaging device are determined, a positional relationship of the second imaging device with respect to the first imaging device is identified based on the plurality of imaged corresponding points and the plurality of deformed corresponding points whose coordinate systems are aligned, and the external parameters of the second imaging device are calibrated based on the external parameters of the first imaging device and the positional relationship of the second imaging device with respect to the first imaging device. The camera system for a work machine according to claim 1 .

5. a third imaging device whose imaging range partially overlaps with that of the second imaging device; The controller acquiring a third captured image captured by the third imaging device; generating a third deformed image by deforming a portion of the third captured image so as to be continuous with the second deformed image; extracting a plurality of corresponding imaging points in the imaging range from the second captured image and the third captured image; extracting a plurality of deformation corresponding points, which are feature points whose feature amounts correspond to each other, in an overlapping portion of the imaging range of the second deformed image and the third deformed image; converting the positions of the plurality of deformation corresponding points into positions in the third captured image, or converting the positions of the plurality of imaged corresponding points into positions in the third deformed image, thereby aligning the coordinate systems of the plurality of imaged corresponding points and the plurality of deformation corresponding points; After the external parameters of the second imaging device are determined, a positional relationship of the third imaging device with respect to the second imaging device is identified based on the plurality of imaged corresponding points and the plurality of deformed corresponding points whose coordinate systems are aligned, and the external parameters of the third imaging device are calibrated based on the external parameters of the second imaging device and the positional relationship of the third imaging device with respect to the second imaging device. The camera system for a work machine according to claim 1 or 4.

6. A method for calibrating camera parameters of a work machine that is equipped with a plurality of imaging devices that capture images of the outside of the work machine and a controller, comprising: The controller acquiring a plurality of captured images captured by the plurality of imaging devices; generating a plurality of deformed images by deforming a portion of the plurality of captured images so that the captured images are continuous with each other; extracting a plurality of corresponding image points having corresponding feature amounts from the plurality of captured images; extracting a plurality of deformation corresponding points from the plurality of deformed images, the feature amounts of which correspond to each other; converting the positions of the plurality of deformation corresponding points into positions in any one of the plurality of captured images, or converting the positions of the plurality of imaged corresponding points into positions in any one of the plurality of deformed images, thereby aligning the coordinate systems of the plurality of imaged corresponding points and the plurality of deformation corresponding points; At least one external parameter of the plurality of image capturing devices is calibrated based on the plurality of image capturing corresponding points and the plurality of deformed corresponding points whose coordinate systems are aligned. A method for calibrating camera parameters of a work machine.

Images