Control system and method for construction machinery

Abstract

The present invention discloses a control system for a construction machine, comprising: an upper camera which is provided on a cab of a rear vehicle body and which captures an image of the front of the cab; a lower camera which is provided on a front vehicle body rotatably connected to the rear vehicle body and which captures an image of the front of the front vehicle body; an object recognition unit for recognizing an object in a first image and a second image captured by the upper camera and the lower camera, respectively, and determining warning information based on the position of the recognized object; an image processing unit for synthesizing the first image and the second image into one image, transparentizing at least one of the first image and the second image in a transparentizing region in the synthesized image, and tagging warning information of the object; and a display device for displaying the composite image marked with the warning information.

Description

Technical Field

[0001] The present invention relates to a control system and method for construction machinery. More specifically, it relates to a control system for recognizing a front obstacle when construction machinery such as a wheel loader or an excavator is operating or traveling, and a control method for construction machinery using the same. Background Art

[0002] Construction machinery such as excavators and wheel loaders are widely used for operations such as excavating and transporting loads such as soil and sand at a construction site and loading them onto trucks such as dump trucks. These operations are performed by driving working devices provided in the construction machinery, such as a bucket and an arm. However, the above-mentioned working devices may block or limit the operator's forward view during driving or traveling, and the obstruction of the operator's forward view by the above-mentioned working devices may cause safety accidents. Summary of the Invention

[0003] Technical Problem

[0004] One object of the present invention is to provide a control system for construction machinery that can improve the forward view restricted by the working device.

[0005] Another object of the present invention is to provide a control method for construction machinery using the above control system.

[0006] Technical Solution

[0007] The control system of construction machinery according to an exemplary embodiment for achieving one object of the present invention includes: an upper camera provided on the cab of the rear body to photograph the front of the cab; a lower camera provided on the front body rotatably connected to the rear body to photograph the front of the front body; an object recognition unit for recognizing an object in a first image and a second image respectively photographed by the upper camera and the lower camera and determining warning information based on the position of the recognized object; an image processing unit for synthesizing the first image and the second image into one image, performing transparency processing on at least one of the first image and the second image in a transparency processing area in the synthesized image, and performing label processing on the warning information of the object; and a display device for displaying the synthesized image marked with the warning information.

[0008] In an exemplary embodiment, the object recognition unit may include: a first shape recognition unit for recognizing the object in the first image and determining the position of the object; a second shape recognition unit for recognizing the object in the second image and determining the position of the object; and a warning information determination unit for determining the warning information of the recognized object based on the position of the object.

[0009] In an exemplary embodiment, when the recognized object is located outside a first region corresponding to the transparency processing region in the first image, the warning information determination unit may determine the warning information with a first label; when the recognized object is located inside a second region corresponding to the transparency processing region in the second image, the warning information determination unit may determine the warning information with a second label.

[0010] In an exemplary embodiment, when the recognized object is located outside the first region in the first image, the image processing unit may perform a display process on the recognized object with a bounding box of a first color; when the recognized object is located inside the second region in the second image, the image processing unit may perform a display process on the recognized object with a bounding box of a second color different from the first color.

[0011] In an exemplary embodiment, the image processing unit may cut a part of the second image based on the first image and then synthesize the cut image of the second image into the first image.

[0012] In an exemplary embodiment, the image processing unit may perform a transparency process on a first region corresponding to the transparency processing region in the first image in a manner having a first transparency; determine a second region corresponding to the transparency processing region in the second image as a cut region, and perform a transparency process on the determined second region in a manner having a second transparency; and synthesize the cut image extracted from the second image into the first image.

[0013] In an exemplary embodiment, the first transparency and the second transparency may be set to be the same or different from each other.

[0014] In an exemplary embodiment, the image processing unit may set a part of the first image as a first synthesis region, set a part of the second image as a second synthesis region, perform a semi-transparency process on the second synthesis region of the second image, and synthesize the semi-transparency processed second synthesis region into the first synthesis region of the first image to produce a synthesized composite image.

[0015] In an exemplary embodiment, the control system of the construction machinery may further include a working device posture detection unit for detecting the posture of a front working device, and the image processing unit may perform a transparency process on at least one of the first image and the second image in the transparency processing region according to the posture of the front working device detected by the working device posture detection unit.

[0016] In an exemplary embodiment, when at least a part of the front working device encroaches on a set position, the image processing unit may make the first image transparent in the transparency processing area, and when the front working device does not encroach on the set position, make the second image transparent in the transparency processing area.

[0017] In an exemplary embodiment, the control system of the construction machine may further include an input unit for setting image processing conditions in the image processing unit.

[0018] In an exemplary embodiment, the image processing conditions may include the transparency conversion time points of the first image and the second image or the area occupied by the transparency processing area in the entire display area of the display device.

[0019] In an exemplary embodiment, the image processing unit may process in the transparency processing area to display the outline of the boom or bucket appearance that is captured and made transparent in the first image or the second image with lines or dotted lines, etc.

[0020] In an exemplary embodiment, the outline of the transparent boom or bucket may be displayed by making the boom or bucket image actually captured in the first image or the second image transparent.

[0021] In an exemplary embodiment, the image processing unit may selectively make the boom or bucket images combined with the front vehicle body in the first image and the second image transparent in the transparency processing area.

[0022] In a control method of a construction machine according to an exemplary embodiment for achieving another object of the present invention, a first image in front of the cab is obtained from an upper camera provided on the cab of the rear vehicle body. A second image in front of the front vehicle body is obtained from a lower camera provided on the front vehicle body rotatably connected to the rear vehicle body. An object is recognized in the first image and the second image, and warning information based on the position of the recognized object is determined. The first image and the second image are synthesized into one image. At least one of the first image and the second image is made transparent in the transparency processing area in the synthesized image, and the warning information of the object is labeled. The synthesized image marked with the warning information is displayed through a display device.

[0023] In an exemplary embodiment, the steps of identifying an object in the first image and the second image and determining warning information based on the position of the identified object may include: identifying the object in the first image and determining the position of the object; identifying the object in the second image and determining the position of the object; determining the warning information of the identified object based on the position of the object.

[0024] In an exemplary embodiment, the step of determining the warning information of the identified object based on the position of the object may include: when the identified object is located outside a first area corresponding to the transparency processing area in the first image, determining the warning information with a first label; when the identified object is located inside a second area corresponding to the transparency processing area in the second image, determining the warning information with a second label.

[0025] In an exemplary embodiment, the step of labeling the warning information of the object may include: when the identified object is located outside the first area in the first image, performing a display process on the identified object with a bounding box of a first color; when the identified object is located inside the second area in the second image, performing a display process on the identified object with a bounding box of a second color different from the first color.

[0026] In an exemplary embodiment, the step of performing transparency processing on at least one of the first image and the second image may include: after cropping a part of the second image based on the first image, synthesizing the cropped image of the second image into the first image.

[0027] In an exemplary embodiment, the step of performing transparency processing on at least one of the first image and the second image may include: performing transparency processing on a first area corresponding to the transparency processing area in the first image with a first transparency; determining a second area corresponding to the transparency processing area in the second image as a cropping area, and performing transparency processing on the determined second area with a second transparency; synthesizing the cropped image extracted from the second image into the first image.

[0028] In an exemplary embodiment, the control method of the construction machinery may further include: detecting the posture of the front working device, and the step of performing transparency processing on at least one of the first image and the second image in the transparency processing area may include: performing transparency processing on at least one of the first image and the second image according to the detected posture of the front working device.

[0029] In an exemplary embodiment, the control method of the construction machinery may further include: setting image processing conditions for making the first image and the second image transparent.

[0030] In an exemplary embodiment, the image processing conditions may include the transparency conversion time points of the first image and the second image or the area occupied by the transparency processing region in the entire display area of the display device.

[0031] Effects of the Invention

[0032] According to an exemplary embodiment, the first image and the second image captured by an upper camera disposed on the cab of a wheel loader and a lower camera disposed on the front body can be synthesized into one image, at least one of the first image and the second image is made transparent within the transparency processing region, and the image subjected to the transparency processing is displayed through a display device.

[0033] Meanwhile, an object can be identified in the first image and the second image, and warning information of the identified object can be determined. The warning information of the identified object can be labeled in the synthesized image subjected to the transparency processing. When the identified object is at a position with a high collision probability, the identified object can be displayed with a red bounding box. In such a situation with a relatively high collision probability, a red image can be synthesized inside the transparency processing region in the synthesized image, or a flashing process can be performed in a manner of making the red image flash.

[0034] Therefore, an operator can identify an object with a relatively high collision probability in front of the wheel loader through the flashing of the red bounding box or the red image within the transparency processing region. Accordingly, the cognitive ability of the operator can be increased, stability can be ensured, and thus safety accidents can be prevented in advance.

[0035] However, the effects of the present invention are not limited to those described above, and various expansions can be made without departing from the spirit and scope of the present invention. Brief Description of the Drawings

[0036] Figure 1 It is a side view of a construction machinery showing an exemplary embodiment.

[0037] Figure 2 It shows Figure 1 a side view of the bucket rising position corresponding to the rotation angle of the boom of

[0038] Figure 3 It shows Figure 1 a block diagram of the control system of the construction machinery of

[0039] Figure 4 is a block diagram of an image processing device showing Figure 3 .

[0040] Figure 5 is a diagram showing warning information of an object recognized from a first image captured by an upper camera by an image processing device through Figure 4 .

[0041] Figure 6 is a diagram showing warning information of an object recognized from a second image captured by a lower camera by an image processing device through Figure 4 .

[0042] Figure 7 is a diagram showing an image synthesized from a first image of Figure 4 and a second image of Figure 5 by an image processing device of Figure 6 .

[0043] Figure 8 is a sequence diagram showing a control method of a wheel loader of an exemplary embodiment.

[0044] Figure 9 is a sequence diagram showing steps of determining warning information of an identified object in a control method of a wheel loader of Figure 8 . DETAILED DESCRIPTION

[0045] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0046] In each drawing of the present invention, the dimensions of the structure are enlarged for clarity of the present invention.

[0047] In the present invention, terms such as first and second may be used to describe various components, but these components should not be limited by these terms. These terms are only for the purpose of distinguishing one component from another.

[0048] The terms used in the present invention are only for explaining specific embodiments and are not intended to limit the present invention. Unless otherwise defined clearly in the context, singular expressions include plural expressions. In the present application, terms such as "including" or "having" should be understood as designating the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, without precluding the presence or additivity of one or more other features, numbers, steps, actions, components, or combinations thereof.

[0049] For the embodiments of the present invention disclosed herein, the specific structural or functional descriptions are merely illustrative for the purpose of explaining the embodiments of the present invention. The embodiments of the present invention can be implemented in various forms and should not be construed as limited to the embodiments described herein.

[0050] That is, the present invention can be variously modified and can have various forms. Specific embodiments will be illustrated in the drawings and will be described in detail in the text. However, this is not to limit the present invention to the specific disclosed forms, but rather it should be understood to include all modifications, equivalents, or alternatives falling within the spirit and technical scope of the present invention.

[0051] Figure 1 It is a side view of a construction machine showing exemplary embodiments. Figure 1 A wheel loader 10 is shown in, but the control device of the construction machine of the exemplary embodiment is not thereby limited to use only in wheel loaders and can also be applied to various industrial vehicles. Hereinafter, for ease of explanation, only the wheel loader 10 will be described.

[0052] Refer to Figure 1 , the construction machine 10 may include a body 12, 14, a cab 40, and a front working device. Taking Figure 1 the wheel loader 10 as an example, the body 12, 14 may include a front body 12 and a rear body 14 that are rotatably connected to each other. The front body 12 may include a front working device and front wheels 70. The rear body 14 may include a cab 40, an engine room 50, and rear wheels 72.

[0053] The front working device may include a boom 20 and a bucket 30. The boom 20 may be rotatably connected to the front body 12, and the bucket 30 may be rotatably connected to one end of the boom 20. The boom 20 is connected to the front body 12 by a pair of boom cylinders 22, and the boom 20 can rotate in the vertical direction by the drive of the boom cylinders 22. A tilt bar 34 is connected in a manner that can freely rotate at almost the center of the boom 20. One end of the tilt bar 34 and the front body 12 are connected by a pair of bucket cylinders 32, and the bucket 30 connected to the other end of the tilt bar 34 through the tilt bar can rotate in the vertical direction (dump or dig) by the drive of the bucket cylinders 32.

[0054] In addition, the front body 12 and the rear body 14 are rotatably connected to each other by a center pin 16, and the front body 12 can bend left and right with respect to the rear body 14 by the expansion and contraction of a steering cylinder (not shown).

[0055] A traveling device for driving the wheel loader 10 can be mounted on the rear body 14. An engine (not shown) can be disposed in the engine room and supply power output to the traveling device. The traveling device can include a torque converter, a transmission, a propeller shaft, an axle, etc. The power output of the engine is transmitted to the front wheels 70 and the rear wheels 72 through the torque converter, the transmission, the propeller shaft, and the axle, enabling the wheel loader 10 to travel.

[0056] A hydraulic pump (not shown) for supplying pressurized oil to the boom cylinder 22 and the bucket cylinder 32 of the work device can be mounted on the rear body 14. The hydraulic pump can be driven using a part of the power output from the engine. For example, the output of the engine can drive the hydraulic pump for the work device and the hydraulic pump for steering through a power transmission device such as a gear system connected to the engine.

[0057] The hydraulic pump can drive the work device by supplying working oil and can be classified into a variable capacity type and a fixed capacity type. A pump control device is connected to the variable capacity type hydraulic pump, and the discharge flow rate of the variable capacity type hydraulic pump can be controlled by the pump control device. The can be connected to a main control valve (MCV) including a boom control valve and a bucket control valve through a predetermined hydraulic circuit. The discharged oil of the hydraulic pump can be supplied to the boom cylinder 22 and the bucket cylinder 32 through the boom control valve and the bucket control valve of the main control valve (MCV). The main control valve (MCV) can supply the working oil discharged from the hydraulic pump to the boom cylinder 22 and the bucket cylinder 32 according to a pilot pressure signal based on the operation input of the operation lever. Accordingly, the boom 20 and the bucket 30 can be driven by the hydraulic pressure of the working oil discharged from the hydraulic pump.

[0058] The cab 40 is provided on the body of the construction machinery and can be provided on the upper part of the rear body 14 in the case of a wheel loader. The cab 40 can be provided with driving operation devices. The driving operation devices can include a traveling pedal, a brake pedal, an FNR traveling lever, an operation lever for operating work cylinders such as the boom cylinder 22 and the bucket cylinder 32, and a steering device such as a steering wheel for operating the steering cylinder.

[0059] As described above, the wheel loader 10 can include a traveling system for driving the traveling device through a power transmission device and a hydraulic device system for driving front work devices such as the boom 20 and the bucket 30.

[0060] The control system of the construction machinery will be described below taking the wheel loader as an example.

[0061] Figure 2 It shows Figure 1Side view of the bucket raising position corresponding to the rotation angle of the boom. Figure 3 is a block diagram showing Figure 1 the control system of the construction machinery. Figure 4 is a block diagram showing Figure 3 the image processing device.

[0062] Figure 5 is a diagram showing warning information of an object recognized from a first image captured by an upper camera by Figure 4 the image processing device. Figure 6 is a diagram showing warning information of an object recognized from a second image captured by a lower camera by Figure 4 the image processing device. Figure 7 is a diagram showing Figure 4 the first image synthesized by Figure 5 the image processing device and Figure 6 the second image.

[0063] Referring to Figures 1 to 7 , the control system of the wheel loader may include: a camera head 100 mounted on the wheel loader 10 for photographing the front of the wheel loader 10; an image processing device 200 for real-time processing of images from the camera head 100; and a display device 300 for displaying the images processed by the image processing device 200. In addition, the control system of the wheel loader may further include: a working device posture detection unit for detecting the posture of the front working device connected to the front body 12; and an input unit 400 for setting the image processing conditions in the image processing device 200.

[0064] The image processing device 200 of the wheel loader 10 may be mounted on the rear body 14 as part of an engine control unit (ECU) or a vehicle control unit (VCU) or as a separate controller. The image processing device 200 may include dedicated hardware, software, and circuits for performing the functions described herein. These components may be physically implemented by circuits such as logic circuits, microprocessors, storage devices, etc.

[0065] In an exemplary embodiment, the camera head 100 is used to monitor the front of the wheel loader 10 when the wheel loader 10 is traveling or operating and may include a plurality of cameras. Specifically, the camera head 100 may include: an upper camera 110 disposed above the cab 40 for photographing the front of the cab 40 to obtain a first image IM1; and a lower camera 120 disposed on the front body 12 for photographing the front of the front body 12 to obtain a second image IM2. For example, the upper camera and the lower camera may be fisheye cameras equipped with fisheye lenses.Figure 1 and Figure 2 As shown in Figure 1 and Figure 2 , an upper camera and a lower camera are shown, but are not limited thereto, and multiple upper cameras and multiple lower cameras may be provided.

[0066] The upper camera 110 may have a first vertical Field of View (FoV) θv1 and a first horizontal FoV θh1 based on the front direction of the wheel loader. For example, the first vertical FoV and the first horizontal FoV may have an angular range of 60 degrees to 120 degrees. The lower camera 120 may have a second vertical FoV θv2 and a second horizontal FoV θh2. For example, the second vertical FoV and the second horizontal FoV may have an angular range of 60 degrees to 120 degrees.

[0067] The first image may be an image captured by the upper camera 110 with the front upper part as the focus, and the second image may be an image captured by the lower camera 120 with the front lower part as the focus.

[0068] It may be set such that the first vertical FoV θv1 of the upper camera 110 and the second vertical FoV θv2 of the lower camera 120 partially overlap, and the first horizontal FoV θh1 of the upper camera 110 and the second horizontal FoV θh2 of the lower camera 120 partially overlap, so that the first image and the second image may partially overlap each other.

[0069] In an exemplary embodiment, the image processing device 200 may synthesize the first image IM1 and the second image IM2 captured by the upper camera 110 and the lower camera 120 into one image, identify an object in the first image IM1 and the second image IM2, determine warning information of the identified object, perform a transparency process on at least one of the first image and the second image in the synthesized image, and perform a tagging process on the warning information of the object. The image processing device 200 may include an image data receiving unit 210, an object identifying unit 220, and an image processing unit 230. The image processing device 200 may be provided in the form of a control device of a construction machine or a control device built in a display device.

[0070] Specifically, the image data receiving unit 210 may include a first data receiving unit 212 and a second data receiving unit 214. The first data receiving unit 212 may receive the image data of the first image IM1 captured by the upper camera 110. The second data receiving unit 214 may receive the image data of the second image IM2 captured by the lower camera 120.

[0071] The object identifying unit 220 may include a first shape identifying unit 222, a second shape identifying unit 224, and a warning information determining unit 226.

[0072] The first shape recognition unit 222 can recognize an object based on the first image data input from the first data receiving unit 212 and determine the position of the object. The first shape recognition unit 222 can recognize a person, construction equipment (such as an excavator, a wheel loader, etc.), a vehicle, etc. as an object in the first image IM1 through a predetermined first algorithm and output the recognition data of the object. The recognition data may include data on the position of the object in the image coordinate system of the first image IM1.

[0073] The second shape recognition unit 224 can recognize an object based on the second image data input from the second data receiving unit 214 and determine the position of the object. The second shape recognition unit 224 can recognize a person, construction equipment (such as an excavator, a wheel loader, etc.), a vehicle, etc. as an object in the second image IM2 through a predetermined second algorithm and output the recognition data on the position of the object in the image coordinate system of the second image IM2. The recognition data may include data on the position of the object in the image coordinate system of the second image IM2.

[0074] The first shape recognition unit 222 and the second shape recognition unit 224 compare the actual images in the first image and the second image with the learned images of the objects stored in the storage unit of the image processing device 200, and if the actual image is the same as the image of the object, it can be recognized as the object. Here, the learned images of the objects may include images obtained by performing machine learning on various shapes captured by a camera and storing them. Machine Learning is a field of artificial intelligence and can refer to an algorithm that enables a processing device such as a computer to learn.

[0075] The first shape recognition unit 222 and the second shape recognition unit 224 can use the same learning model, but the image data sets used for learning in the first image and the second image may be different from each other. The first shape recognition unit 222 and the second shape recognition unit 224 can determine the position of the object by obtaining the positions of the pixels in the first image and the second image where the object is located and output the determined position data.

[0076] The warning information determination unit 226 can determine the warning information of the recognized object based on the recognition data of the object from the first shape recognition unit 222 and the second shape recognition unit 224.

[0077] Specifically, the warning information determination unit 226 may label the warning information to be displayed for the object based on the position data of the object in the first image from the first shape recognition unit 222. For example, when the recognized object is outside the first region R1 corresponding to the transparency processing region in the first image IM1, the warning information determination unit 226 may determine the warning information with the first label OL1.

[0078] The warning information determination unit 226 may label the warning information to be displayed for the object based on the position data of the object in the second image from the second shape recognition unit 224. When the position of the recognized object is inside the second region R2 corresponding to the transparency processing region in the second image IM2, the warning information determination unit 226 may determine the warning information with the second label OL2.

[0079] The image processing unit 230 may synthesize the first image IM1 and the second image IM2 into one image from the first image and the second image data input from the image data receiving unit 210. The image processing unit 230 may match the overlapping images in the first image and the second image captured by the upper camera 110 and the lower camera 120 to synthesize them into one image. The image processing unit 230 may perform transparency processing on at least one of the first image and the second image in the transparency processing region R3. The transparency processing region R3 may be defined as a region that includes the area that blocks the operator's view when the front working device (at least a part of the boom and the bucket) encroaches within the entire display area of the display device 300. For example, the transparency processing region may be determined to have a width dimension in the vehicle width of the wheel loader or in the horizontal direction of the bucket when the wheel loader travels forward.

[0080] The transparency processing may remove or semi-transparently process the first image or the second image within the transparency processing region of the synthesized image so that it overlaps with the background image, or two-dimensionally display the outline of the appearance with lines or dotted lines so that only the shape can be recognized. For example, the Alpha Blending technique or the like may be used to remove the first image or the second image within the transparency processing region from the synthesized image.

[0081] The image processing unit 230 may receive the warning information of the identified object from the warning information determination unit 226, and perform tagging processing on the warning information of the identified object in the synthesized image subjected to the transparency processing. The image processing unit 230 may mark the warning information of the identified object in the synthesized image by means of a bounding box, a box shadow, or other markings. In addition, the image processing unit 230 may perform a blinking process on the bounding box or use an oval bar at the lower end of the object to mark the warning information.

[0082] In an exemplary embodiment, when the identified object is located outside the first region R1 in the first image IM1, the image processing unit 230 may reflect the warning information of the first label OL1, and perform display processing on the identified object in the synthesized image with a bounding box of a first color (e.g., blue). When the identified object is located inside the second region R2 in the second image IM2, the image processing unit 230 may reflect the warning information of the second label OL2, and perform display processing on the identified object in the synthesized image with a bounding box of a second color (e.g., red). At this time, compared with the first color (blue), the second color (red) may inform the operator that the possibility of a more dangerous situation occurring is higher. In addition, the image processing unit 230 may synthesize a red image inside the transparency processing region R3 in the synthesized image or perform a blinking process in a manner that makes the red image blink in such a situation where the possibility of danger occurring is high.

[0083] As Figures 5 to 7 shown, the image processing unit 230 may crop a part of the second image M2 based on the first image M1 and then synthesize the cropped image of the second image M2 into the first image M1.

[0084] First, transparency processing can be performed in such a way that a first region R1 corresponding to a transparency processing region R3 in the first image M1 has a first transparency. A second region R2 corresponding to the transparency processing region R3 in the second image M2 can be determined as a cutting region, and the determined second region can be transparently processed in such a way that it has a second transparency. The first transparency and the second transparency can be set to be the same or different from each other. The first transparency and the second transparency can be adjusted, for example, to a level of 150 / 255 to 230 / 255. At this time, the transparency of the cut image extracted from the second image can be not adjusted. Next, the cut image extracted from the second image M2 can be synthesized into the transparency processing region R3 of the first image M1. At this time, when the area of the cutting region is smaller than the area of the first region R1 in the first image, the cut image can be enlarged and the enlarged cut image can be synthesized into the first region R1 of the first image.

[0085] When there is a vehicle in front of the wheel loader, in Figure 5 the first image, the boom or the bucket blocks the view, and the vehicle cannot be accurately confirmed. However, in Figure 6 the second image, the vehicle can be confirmed. The cut image extracted from Figure 6 the second image can include an image of the vehicle. Since the cut image is synthesized into the first region R1 corresponding to the transparency processing region of the first image, the image of the vehicle extracted from Figure 8 can be confirmed in the transparency processing region R3 of the synthesized image in Figure 6 . Accordingly, the forward field of view restricted by the working device can be improved.

[0086] Meanwhile, the object recognition unit 220 can recognize an object from the first image M1 and the second image M2 and determine warning information for the recognized object. The image processing unit 230 can perform label processing on the warning information of the recognized object in the transparently processed synthesized image.

[0087] As Figure 5 shown, when the object recognition unit 220 recognizes an excavator in the first image M1 and the recognized excavator is outside the first region R1 in the first image IM1, the warning information for the recognized excavator can be determined with a first label OL1.

[0088] As Figure 6 shown, when the object recognition unit 220 recognizes a vehicle in the second image M2 and the recognized vehicle is outside the second region R2 in the second image IM2, the warning information for the recognized vehicle can be determined with a second label OL2.

[0089] As Figure 7As shown, the image processing unit 230 may perform display processing on the recognized blue bounding box A for the excavator in the synthesized image, and perform display processing on the recognized red bounding box B for the vehicle. In addition, in such a situation with a high likelihood of collision, the image processing unit 230 may synthesize a red image inside the transparency processing area R3 in the synthesized image, or perform a flashing process in a manner that causes the image to flash.

[0090] Accordingly, the operator can identify an object with a relatively high likelihood of collision in front of the wheel loader through the red bounding box B or the flashing of the red image in the transparency processing area R3.

[0091] In an exemplary embodiment, the working device posture detection unit may detect whether the front working device infringes on the transparency processing area R3 within the display area of the display device 300. Based on the posture of the front working device detected by the working device posture detection unit, the image processing unit 230 may perform transparency processing in response to whether at least a part of the front working device infringes on the position corresponding to the transparency processing area.

[0092] The posture of the front working device may include the position of the bucket 30 (the height of the bucket from the ground) or the posture of the boom 20 (the rotation angle of the boom). For this purpose, the working device posture detection unit may include a boom angle sensor 24 for detecting the position of the bucket 30 or the posture of the boom 20. At the same time, a bucket angle sensor (not shown) for detecting the relative rotation angle between the boom 20 and the bucket 30 may be included. Instead of the boom angle sensor 24, a displacement sensor for detecting the stroke of the cylinder driving the boom 20 may be included.

[0093] In addition, the working device posture detection unit may include an image analysis device that analyzes an image of the front working device captured by a camera and determines the posture of the front working device (for example, a shape recognition unit).

[0094] The boom angle sensor 24 may detect the rotation angle of the boom 20 and provide information about the position of the bucket 30 based on this. As Figure 2 shown, the rotation angle of the boom 20 may be the angle θ between the extension line L at the lowest position (0%) of the boom 20 (bucket 30) and the extension line R of the boom 20. The rotation angle of the boom 20 at the highest position (max boom height) of the boom 20 is θmax.height, and at this time, the boom (bucket) position may be the maximum height (100%).

[0095] The image processing unit 230 can receive the detected value of the rotation angle of the boom and determine the first transparency in the first region R1 of the first image M1 and the second transparency in the second region R2 of the second image M2 based on this.

[0096] For example, when it can be determined that at least a part of the front working device does not infringe on the bucket or boom position of the transparency processing area and is lower than the set position (transparency conversion position), the image processing unit 230 can make the second image transparent in the synthesized image. At the same time, when it can be determined that the bucket or boom position of the front working device that infringes on the transparency processing area is in a state higher than the set position (transparency conversion position), the image processing unit 230 can make the first image transparent in the synthesized image. For example, the set position of the boom can be set so that the rotation angle θ of the boom 20 is in the range of 15 degrees to 20 degrees.

[0097] When the bucket 30 is between the lowest position (0%) and the set bucket position, that is, the transparency conversion position which is the boundary of the transparency processing area, by performing transparency processing on the second image captured by the lower camera 120, it can be displayed so that the subject achieved by the upper camera 110 is emphasized. When the bucket 30 is in a relatively low position, in the second image captured by the lower camera 120, the front view of the front working device may be blocked by the front body 12 including the boom 20 and the bucket 30. The image processing unit 230 can prevent the field of view from being blocked by the front working device by performing transparency processing on the second image and displaying the first image as the emphasis.

[0098] When the bucket 30 rises or falls and passes through the set bucket position (transparency conversion position), the image within the transparency processing area where the image processing unit 230 performs transparency processing can be converted from the second image to the first image, or from the first image to the second image.

[0099] Differently, when the rotation angle θ of the boom is within the first angle range, the image processing unit 230 can make the second image transparent in the synthesized image; when the rotation angle θ of the boom is within the second angle range, the image processing unit 230 can make the first image and the second image transparent in the transparency processing area of the synthesized image; when the rotation angle θ of the boom is within the third angle range, the image processing unit 230 can make the first image transparent in the synthesized image. For example, the first angle range can be set to be within the range of 0 degrees to 15 degrees; the second angle range can be set to be within the range of 15 degrees to 25 degrees; the third angle range can be set to be within the range of 25 degrees to 45 degrees.

[0100] In an exemplary embodiment, image processing conditions in the image processing device 200 can be set through the input unit 400. For example, the image processing conditions can be for the area of the transparency processing region in the entire display region. With the determination of the transparency processing region, the transparency conversion positions of the first image and the second image, the transparency processing region in the entire display region of the display device 300, etc. can be set. For example, the transparency conversion position can refer to the boundary position of the transparency processing region. When the bucket 30 moves and is located at the boundary of the transparency processing region, it can be regarded that the bucket 30 is located at the set position for transparency conversion. The area and position of the transparency processing region, the transparency conversion time point, etc. can be fixedly set by the manufacturer according to the device model, and can be freely changed and set by the operator or maintenance personnel.

[0101] For example, the input unit 400 can be implemented in the form of a dashboard option, and the operator can change the transparency conversion time point, the transparency processing region, etc. through the input unit 400.

[0102] As described above, when the transparency processing region and the transparency conversion time point are set, the display device 300 can be divided into a transparency processing region R3 for transparently processing and displaying the captured image of the imaging head and an external region of the transparency processing region R3 for displaying an image. The display device 300 can additionally display the outline of the transparency processing region R3 in a manner that can distinguish the transparency processing region R3, and can connect and display the transparently processed image with the image of the external region of the transparency processing region R without the outline.

[0103] In addition, in the external region of the transparency processing region R3, the display device 300 can display the first image; in the internal region of the transparency processing region R3, as the transparency processing progresses, the display device 300 can display the first image, the second image, or a transparency image in which at least one of the first image and the second image is prominently displayed.

[0104] For example, when the bucket 30 is located in the external region of the transparency processing region R3, the display device 300 can only display the first image that connects the transparency processing region R3 and the external region of the transparency processing region R3 to each other. In addition, when at least a part of the bucket 30 is located inside the transparency processing region R3, for the inside of the transparency processing region R3, the display device 300 can display a transparency processed image in which the second image is prominently displayed or the second image; for the external region of the transparency processing region R3, the display device 300 can display only the first image excluding the image inside the transparency processing region R3.

[0105] Next, the use ofFigure 4 The control system of a construction machine and a method for controlling a construction machine will be described. The following description will also be based on a wheel loader as in the method described above.

[0106] Figure 8 It is a sequence diagram showing a control method of a wheel loader showing an exemplary embodiment. Figure 9 It shows Figure 8 It is a sequence diagram of the step of determining warning information of an identified object in the control method of the wheel loader.

[0107] Referring to Figures 1 to 8 , first, a first image IM1 and a second image IM2 can be obtained through an upper camera 110 and a lower camera 120 provided on the wheel loader 10 (S10).

[0108] In an exemplary embodiment, the upper camera 110 provided on the cab 40 can be used to obtain a first image IM1 of the front of the cab 40. The lower camera 120 provided on the front body 12 can be used to obtain a second image IM2 of the front of the front body 12.

[0109] The first image may be an image taken by the upper camera 110 with the front upper part as the focus, and the second image may be an image taken by the lower camera 120 with the front lower part as the focus. It can be set so that the first vertical viewing angle θv1 of the upper camera 110 and the second vertical viewing angle θv2 of the lower camera 120 partially overlap, and the first horizontal viewing angle θh1 of the upper camera 110 and the second horizontal viewing angle θh2 of the lower camera 120 partially overlap, so that the first image and the second image can partially overlap with each other.

[0110] Next, recognition data for the identified object can be obtained from the first image IM1 and the second image IM2 (S20), and warning information for the identified object can be determined (S30).

[0111] In an exemplary embodiment, the object recognition unit 220 can recognize an object in the first image IM1 and the second image IM2, determine the position of the object, and determine warning information for the identified object based on the position data of the object.

[0112] The object in the first image IM1 obtained from the upper camera 110 and the second image IM2 obtained from the lower camera 120 can be displayed as corresponding pixels among a plurality of pixels. Here, the working space photographed by the upper camera 110 and the lower camera 120 can be represented by grids of the same size, and whether there is an object can be displayed on each grid.

[0113] The first shape recognition unit 222 and the second shape recognition unit 224 of the object recognition unit 220 can recognize a person, construction equipment (such as an excavator, a wheel loader, etc.), a vehicle, etc. as an object in the first image IM1 and the second image IM2 through a pre-determined algorithm, and output the recognition data of the object. The recognition data may include data on the position of the object in the image coordinate system of each of the first image IM1 and the second image IM2.

[0114] Compare the actual images in the first image and the second image with the learned images of the objects stored in the storage unit of the image processing device 200. If the actual image is the same as the image of the object, it can be recognized as the object. Here, the learned images of the objects may include images obtained by performing machine learning on various shapes captured by a camera and storing them. Machine Learning is a field of artificial intelligence and can refer to an algorithm that enables a processing device such as a computer to learn.

[0115] The first shape recognition unit 222 and the second shape recognition unit 224 may use the same learning model, but the image data sets used for learning in the first image and the second image may be different from each other. The first shape recognition unit 222 and the second shape recognition unit 224 can determine the position of the object by obtaining the positions of the pixels in the first image and the second image where the object is located, and output the determined position data.

[0116] After that, the warning information of the recognized object can be determined based on the recognition data of the object.

[0117] As Figures 5 to 7 and Figure 9 shown, the recognition data of the object can be input from the first shape recognition unit 222 and the second shape recognition unit 224 (S310), and it is determined whether an object is detected in the first image and the second image (S320). It is determined whether the recognized object is outside the first region R1 corresponding to the transparency processing region in the first image from the upper camera 110 (S330). When the recognized object is outside the first region R1 in the first image IM1, the warning information can be determined with the first label OL1 (S340). It is determined whether the object recognized in the second image from the lower camera 120 is inside the second region R2 corresponding to the transparency processing region (S332). When the recognized object is inside the second region R2 in the second image IM2, the warning information can be determined with the second label OL2 (S342). After that, the warning information of the recognized object can be output (S350).

[0118] Thereafter, the first image IM1 and the second image IM2 can be synthesized into one image (S40), at least one of the first image and the second image is subjected to transparency processing within the transparency processing area R3 (S50), warning information of the identified object is marked in the synthesized image subjected to transparency processing (S60), and the synthesized image marked with the warning information is displayed (S70).

[0119] In an exemplary embodiment, the image processing unit 230 can synthesize the first image IM1 and the second image IM2 into one image from the first image and the second image data input from the image data receiving unit 210. The image processing unit 230 can match the overlapping images in the first image and the second image captured by the upper camera 110 and the lower camera 120 to synthesize them into one image. The image processing unit 230 can perform transparency processing on at least one of the first image and the second image in the transparency processing area R3.

[0120] The transparency processing can remove or semi-transparently process the first image or the second image within the transparency processing area of the synthesized image so that it overlaps with the background image, or two-dimensionally display the outline of the appearance with lines or dotted lines so that only the shape can be recognized. For example, the Alpha Blending technology or the like can be used to remove the first image or the second image within the transparency processing area from the synthesized image.

[0121] As Figures 5 to 7 shown, the image processing unit 230 can crop a part of the second image M2 based on the first image M1 and then synthesize the cropped image of the second image M2 into the first image M1.

[0122] First, transparency processing can be performed in such a way that the first region R1 corresponding to the transparency processing area R3 in the first image M1 has a first transparency. The second region R2 corresponding to the transparency processing area R3 in the second image M2 can be determined as the cropping area, and the determined second region can be subjected to transparency processing so that it has a second transparency. The first transparency and the second transparency can be set to be the same or different from each other. The first transparency and the second transparency can be adjusted to a level of 150 / 255 to 230 / 255, for example. At this time, the transparency of the cropped image extracted from the second image can be not adjusted. Then, the cropped image extracted from the second image M2 can be synthesized into the transparency processing area R3 of the first image M1. At this time, when the area of the cropping area is smaller than the area of the first region R1 in the first image, the cropped image can be enlarged and the enlarged cropped image can be synthesized into the first region R1 of the first image.

[0123] When there is a vehicle in front of the wheel loader, in Figure 5 the first image of Figure 6 , the boom or bucket blocks the view, making it impossible to accurately identify the vehicle. However, in Figure 6 the second image of Figure 8 , the vehicle can be identified. The cropped image extracted from Figure 6 the second image may include an image of the vehicle. Since the cropped image is synthesized into the first region R1 corresponding to the transparency processing area of the first image, the image of the vehicle extracted from

[0124] can be confirmed in the transparency processing area R3 of the synthesized image. Accordingly, the front view restricted by the working device can be improved.

[0125] At the same time, the object recognition unit 220 can recognize an object from the first image M1 and the second image M2 and determine warning information for the recognized object. The image processing unit 230 can label the warning information of the recognized object in the synthesized image that has been subjected to transparency processing. Figure 5 As

[0126] shown, when the object recognition unit 220 recognizes an excavator in the first image M1 and the recognized excavator is outside the first region R1 in the first image IM1, the warning information for the recognized excavator can be determined with the first label OL1. Figure 6 As

[0127] shown, when the object recognition unit 220 recognizes a vehicle in the second image M2 and the recognized vehicle is outside the second region R2 in the second image IM2, the warning information for the recognized vehicle can be determined with the second label OL2. Figure 7 As

[0128] shown, the image processing unit 230 can display the recognized excavator with a blue bounding box A in the synthesized image and display the recognized vehicle with a red bounding box B. In addition, in such a situation with a high possibility of collision, the image processing unit 230 can synthesize a red image inside the transparency processing area R3 in the synthesized image or perform a flashing process in a way that makes the image flash.

[0129] Accordingly, the operator can identify an object with a high possibility of collision in front of the wheel loader through the flashing of the red bounding box B or the red image in the transparency processing area R3.

[0130] For example, when it can be determined that at least a part of the front working device does not violate the bucket or boom position of the transparency processing area and is lower than a set position (transparency conversion position), the image processing unit 230 can make the second image transparent in the synthesized image. At the same time, when it can be determined that the bucket or boom position of the front working device that violates the transparency processing area is in a state higher than the set position (transparency conversion position), the first image can be made transparent in the synthesized image. For example, the set position of the boom can be set so that the rotation angle θ of the boom 20 is in the range of 15 degrees to 20 degrees.

[0131] In an exemplary embodiment, image processing conditions for making the first image and the second image transparent can be set. The image processing conditions in the image processing device 200 can be set through the input unit 400. For example, the image processing conditions can include the transparency conversion time points of the first image and the second image, the area occupied by the transparency processing area in the entire display area of the display device 300, and the like. The transparency conversion time points of the first image and the second image can be determined according to the positions of the bucket 30 or the boom 20 and the set positions of the bucket or the boom. The transparency processing area can be selected according to the model of the equipment.

[0132] For example, the input unit 400 can be implemented in the form of a dashboard option, and the operator can change the transparency conversion time point, the area of the transparency processing area, etc. through the input unit 400. The input unit 400 can be provided in the form of a separate operating device provided inside the cab, an operating device integrally provided on the display device, or a touch screen constituting the display screen of the display device. Accordingly, the operator can set various image processing conditions, such as setting the area around an object that needs attention during operation as the transparency processing area.

[0133] As described above, the first image and the second image captured by the upper camera 110 provided on the cab 40 of the wheel loader 10 and the lower camera 120 provided on the front body 12 can be synthesized into one image, at least one of the first image and the second image is made transparent in the transparency processing area R3, and the image after the transparency processing is displayed through the display device 300.

[0134] Meanwhile, an object can be recognized in the first image and the second image, and warning information for the recognized object can be determined. The warning information for the recognized object can be tagged in the synthesized image subjected to the transparency processing. When the recognized object is at a position with a high likelihood of collision, the recognized object can be displayed with a red bounding box. In such a situation with a relatively high likelihood of collision, a red image can be synthesized inside the transparency processing area R3 in the synthesized image, or a flashing process can be performed in such a way that the red image flashes.

[0135] Therefore, the operator can recognize that there is an object with a relatively high likelihood of collision in front of the wheel loader through the flashing of the red bounding box or the red image in the transparency processing area R3. Accordingly, the operator's awareness can be enhanced and stability can be ensured, thereby preventing safety accidents from occurring.

[0136] Although the above has been described with reference to embodiments of the present invention, those skilled in the art of this technology will understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention described in the following claims.

[0137] Reference Numerals

[0138] 10: Wheel loader, 12: Front body, 14: Rear body, 20: Boom, 22: Boom cylinder, 24: Boom angle sensor, 30: Bucket, 32: Bucket cylinder, 34: Tilt bar, 40: Cab, 70: Front wheel, 100: Camera head, 110: First camera, 120: Second camera, 150: Angle information detection unit, 200: Image processing device, 210: Image data receiving unit, 212: First data receiving unit, 214: Second data receiving unit, 220: Object recognition unit, 222: First shape recognition unit, 224: Second shape recognition unit, 226: Warning information determination unit, 230: Image processing unit, 300: Display device, 400: Input unit.

Claims

1. A control system for construction machinery, characterized in that, comprising: an upper camera, which is arranged on the cab of the rear body and shoots the front of the cab; a lower camera, which is rotatably connected to the front body of the rear body and shoots the front of the front body; an object recognition unit, which is used to recognize an object in the first image and the second image respectively shot by the upper camera and the lower camera, and determine warning information based on the position of the recognized object; an image processing unit, which is used to synthesize the first image and the second image into one image, perform transparency processing on at least one of the first image and the second image in a transparency processing area in the synthesized image, and perform label processing on the warning information of the object; and a display device, which is used to display the synthesized image marked with the warning information.

2. The control system for construction machinery according to claim 1, characterized in that, the object recognition unit comprises: a first shape recognition unit, which is used to recognize the object in the first image and determine the position of the object; a second shape recognition unit, which is used to recognize the object in the second image and determine the position of the object; and a warning information determination unit, which is used to determine the warning information of the recognized object based on the position of the object.

3. The control system for construction machinery according to claim 2, characterized in that, when the recognized object is outside a first area corresponding to the transparency processing area in the first image, the warning information determination unit determines the warning information with a first label; when the recognized object is inside a second area corresponding to the transparency processing area in the second image, the warning information determination unit determines the warning information with a second label.

4. The control system for construction machinery according to claim 3, characterized in that, when the recognized object is outside the first area in the first image, the image processing unit performs display processing on the recognized object with a bounding box of a first color; when the recognized object is inside the second area in the second image, the image processing unit performs display processing on the recognized object with a bounding box of a second color different from the first color.

5. The control system for construction machinery according to claim 1, characterized in that, the image processing unit cuts a part of the second image based on the first image, and then synthesizes the cut image of the second image into the first image.

6. The control system for construction machinery according to claim 1, characterized in that, the image processing unit performs transparency processing on a first area corresponding to the transparency processing area in the first image with a first transparency; determines a second area corresponding to the transparency processing area in the second image as a cut area, and performs transparency processing on the determined second area with a second transparency; and synthesizes the cut image extracted from the second image into the first image.

7. The control system of the construction machinery according to claim 6, characterized in that, the first transparency and the second transparency are set to be the same as or different from each other.

8. The control system of the construction machinery according to claim 1, characterized in that, the image processing unit sets a part of the first image as a first synthesis area, sets a part of the second image as a second synthesis area, performs a semi-transparency process on the second synthesis area of the second image, and synthesizes the semi-transparency processed second synthesis area into the first synthesis area of the first image to produce the synthesized composite image.

9. The control system of the construction machinery according to claim 1, characterized in that, it further includes a working device posture detection unit for detecting the posture of the front working device, and the image processing unit performs a transparency process on at least one of the first image and the second image in the transparency process area according to the posture of the front working device detected by the working device posture detection unit.

10. The control system of the construction machinery according to claim 9, characterized in that, the image processing unit performs transparency on the first image in the transparency process area when at least a part of the front working device invades a set position, and performs transparency on the second image in the transparency process area when the front working device does not invade the set position.

11. The control system of the construction machinery according to claim 1, characterized in that, it further includes an input unit for setting image processing conditions in the image processing unit.

12. The control system of the construction machinery according to claim 11, characterized in that, the image processing conditions include the transparency conversion time points of the first image and the second image or the area occupied by the transparency process area in the entire display area of the display device.

13. The control system of the construction machinery according to claim 1, characterized in that, the image processing unit processes to display the outline of the boom or bucket appearance that is photographed and made transparent in the first image or the second image with a line or a dotted line, etc. in the transparency process area.

14. The control system of the construction machinery according to claim 13, characterized in that, the outline of the transparent boom or bucket is displayed by performing a transparency process on the boom or bucket image actually photographed in the first image or the second image.

15. The control system of the construction machinery according to claim 1, characterized in that, the image processing unit selectively performs a transparency process on the boom or bucket images combined with the front body in the first image and the second image in the transparency process area.

16. A control method for construction machinery, characterized in that, it includes: obtaining a first image in front of the cab from an upper camera provided on the cab of the rear body; obtaining a second image in front of the front body from a lower camera provided on the front body rotatably connected to the rear body; Identify an object in the first image and the second image, and determine warning information based on the position of the identified object; Combine the first image and the second image into one image; Perform transparency processing on at least one of the first image and the second image within a transparency processing area in the combined image, and label the warning information of the object; Display the combined image marked with the warning information through a display device.

17. The control method of the construction machinery according to claim 16, wherein, The step of identifying an object in the first image and the second image and determining warning information based on the position of the identified object includes: Identify the object in the first image and determine the position of the object; Identify the object in the second image and determine the position of the object; Determine the warning information of the identified object based on the position of the object.

18. The control method of the construction machinery according to claim 17, wherein, The step of determining the warning information of the identified object based on the position of the object includes: When the identified object is located outside a first area corresponding to the transparency processing area in the first image, determine the warning information with a first label; When the identified object is located inside a second area corresponding to the transparency processing area in the second image, determine the warning information with a second label.

19. The control method of the construction machinery according to claim 18, wherein, The step of labeling the warning information of the object includes: When the identified object is located outside the first area in the first image, perform display processing on the identified object with a bounding box of a first color; When the identified object is located inside the second area in the second image, perform display processing on the identified object with a bounding box of a second color different from the first color.

20. The control method of the construction machinery according to claim 16, wherein, The step of performing transparency processing on at least one of the first image and the second image includes: After cropping a part of the second image based on the first image, synthesize the cropped image of the second image into the first image.

21. The control method of the construction machinery according to claim 16, wherein, The step of performing transparency processing on at least one of the first image and the second image includes: Perform transparency processing on a first area corresponding to the transparency processing area in the first image in a manner with a first transparency; Determine a second area corresponding to the transparency processing area in the second image as a cropping area, and perform transparency processing on the determined second area in a manner with a second transparency; Synthesize the cropped image extracted from the second image into the first image.

22. The control method of the construction machinery according to claim 16, wherein, The step of performing transparency processing on at least one of the first image and the second image includes: Set a part of the first image as a first composite area; Set a part of the second image as a second composite area; Perform a semi-transparency process on the second composite area of the second image; and Composite the semi-transparency processed second composite area onto the first composite area of the first image to produce the composite image.

23. The control method of a construction machine according to claim 16, characterized in that, further comprising: detect the posture of the front working device, The step of performing a transparency process on at least one of the first image and the second image within the transparency process area includes: Performing a transparency process on at least one of the first image and the second image according to the detected posture of the front working device.

24. The control method of a construction machine according to claim 16, characterized in that, further comprising: Set image processing conditions for making the first image and the second image transparent.

25. The control method of a construction machine according to claim 24, characterized in that, The image processing conditions include the transparency conversion time points of the first image and the second image or the area occupied by the transparency process area in the entire display area of the display device.

Images