Lookup table-based detected object display device and lookup table-based detected object display method

Abstract

According to an embodiment of the present invention, a lookup table-based detected object display device may comprise a processing device and a plurality of cameras installed at preconfigured locations of a construction machine, wherein the processing device: acquires a plurality of corresponding images from the plurality of cameras, respectively; performs deep learning-based object detection on the respective plurality of images; if an object is detected from at least one image among the plurality of images, determines center coordinates of the detected object; performs coordinate transformation on the center coordinates of the detected object by using a lookup table; generates a composite image by synthesizing the acquired plurality of images; and displays the composite image and a preconfigured image so that the preconfigured image is overlaid with the detected object in the composite image.

Description

Lookup table-based detection object display device and lookup table-based detection object display method

[0001] The following embodiments relate to a lookup table-based detection object display device and a lookup table-based detection object display method.

[0002]

[0003] In general, construction machinery such as dozers, loaders, excavators, graders, dump trucks, and articulated dump trucks are primarily utilized in earthmoving environments. In particular, dump trucks with hinged front and rear frames and steering cylinders that allow left-right directional control are called articulated dump trucks.

[0004] While a typical dump truck turns by turning its front wheels, an articulated dump truck turns by turning its front end, reducing the turning radius and enabling it to move easily in narrow work sites and secure an efficient moving route.

[0005] Construction equipment, such as articulated dump trucks, are increasingly being manufactured larger for reasons such as increased efficiency. These larger machines create a blind spot around the cab, obscuring the view from the operator's seat.

[0006]

[0007] The present invention was invented to solve the above-mentioned problem, and its purpose is to provide a look-up table-based detection object display device and a look-up table-based detection object display method capable of displaying a detected object and a preset object in a composite image generated by synthesizing a plurality of images.

[0008] In addition, another object of the present invention is to provide a look-up table-based detection object display device and a look-up table-based detection object display method that can convert the center coordinates determined for an object detected in an image into coordinates using a look-up table.

[0009]

[0010] According to one embodiment of the present invention, a lookup table-based detection object display device includes a plurality of cameras installed at preset locations of a construction machine and a processing device, wherein the processing device acquires a plurality of corresponding images from each of the plurality of cameras, performs deep learning-based object detection on each of the plurality of images, detects an object in at least one of the plurality of images, determines a center coordinate for the detected object, transforms the center coordinate for the detected object into a coordinate using a lookup table, synthesizes the acquired plurality of images to generate a composite image, and displays the composite image and the preset image so that the preset image is overlaid with the detected object in the composite image.

[0011] In addition, the processing device can generate a distortion-corrected image for each of the plurality of images by correcting the distortion of each of the plurality of images obtained based on a preset distortion correction value.

[0012] Additionally, the processing device can obtain a bounding box surrounding the detected object and determine the center coordinates of the bounding box based on the upper left coordinate and the lower right coordinate of the bounding box.

[0013] Additionally, the processing device can transform the center coordinates of the bounding box determined by using the lookup table.

[0014] In addition, the processing device can generate the composite image by synthesizing a distortion correction image for each of the plurality of images.

[0015] Additionally, the processing device can display a preset image so as to be overlaid with the detected object.

[0016] According to another embodiment of the present invention, a method for displaying a detected object based on a lookup table may include the steps of: obtaining a plurality of images corresponding to each of a plurality of cameras installed at preset locations of a construction machine; performing deep learning-based object detection on each of the plurality of images; determining a center coordinate for the detected object when an object is detected in at least one of the plurality of images; converting the center coordinate for the detected object into coordinates using a lookup table; generating a composite image by synthesizing the obtained plurality of images; and displaying the composite image and the preset image so that the preset image is overlaid with the detected object in the composite image.

[0017] In addition, the step of acquiring the plurality of images may include a step of generating a distortion-corrected image for each of the plurality of images by correcting the distortion of each of the acquired plurality of images based on a preset distortion correction value.

[0018] Additionally, the step of determining the center coordinates for the detected object may include the step of obtaining a bounding box surrounding the detected object, and the step of determining the center coordinates of the bounding box based on the upper left coordinate and the lower right coordinate of the bounding box.

[0019] Additionally, the step of converting the center coordinates of the detected object into coordinates using a lookup table may include a step of converting the center coordinates of the determined bounding box into coordinates using the lookup table.

[0020] Additionally, the displaying step may include a step of generating the composite image by synthesizing a distortion correction image for each of the plurality of images.

[0021] Additionally, the displaying step may further include a step of displaying a preset image so as to be overlaid with the detected object.

[0022]

[0023] According to one embodiment of the present invention, there is an effect of being able to display a detected object and a preset object in a composite image created by synthesizing a plurality of images.

[0024] Additionally, it has the effect of being able to convert the center coordinates determined for objects detected in the image into coordinates using a lookup table.

[0025]

[0026] FIG. 1 is a diagram showing the configuration of a lookup table-based detection object display device according to one embodiment.

[0027] FIGS. 2A and 2B are drawings showing a camera included in a lookup table-based detection object display device mounted on a construction machine according to one embodiment.

[0028] FIG. 3 is a drawing showing an object detected in an image being displayed in a synthetic image according to an embodiment.

[0029] FIG. 4 is a flow chart illustrating a detection object display method based on a lookup table according to an embodiment.

[0030]

[0031] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0032] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes all modifications, equivalents, or alternatives falling within the spirit and technical scope of the present invention.

[0033] While terms such as "first" or "second" may be used to describe various components, these components should not be limited by these terms. These terms are intended solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component, without departing from the scope of the invention.

[0034] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0035] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0036] In this specification, it should be understood that terms such as “comprise” or “have” are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0038] Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0039] In the following description, identical identifiers indicate identical configurations, and unnecessary redundant descriptions and explanations of known technologies are omitted.

[0040] Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the attached drawings.

[0041] FIG. 1 is a diagram showing the configuration of a lookup table-based detection object display device according to one embodiment.

[0042] A lookup table-based detection object display device (100) can be combined with a construction machine as a separate, independent device.

[0043] A lookup table-based detection object display device (100) can be communicatively connected to a construction machine as a separate, independent device.

[0044] A lookup table-based detection object display device (100) can be included in a construction machine.

[0045] Referring to FIG. 1, a lookup table-based detection object display device (100) according to one embodiment includes a first camera (110), a second camera (120), a third camera (130), a fourth camera (140), a fifth camera (150), a sixth camera (160), a processing device (170), and a display device (180).

[0046] The first camera (110), the second camera (120), the third camera (130), the fourth camera (140), the fifth camera (150), the sixth camera (160), the processing device (170), and the display device (180) included in the lookup table-based detection object display device (100) are interconnected and can transmit data to each other.

[0047] The first camera (110) can be installed on the upper part of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0048] The first camera (110) can be installed on the upper part of the driver's seat vertically above the head of the driver riding in the driver's seat of the construction machine (e.g., an articulated dump truck, etc.).

[0049] The first camera (110) can be installed on the upper part of the driver's seat of the construction machine (e.g., an articulated dump truck, etc.) so that the direction of view is the same as the direction of the driver's gaze riding in the driver's seat of the construction machine (e.g., an articulated dump truck, etc.).

[0050] The first camera (110) can generate a first image corresponding to the field of view of a driver in the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0051] The second camera (120) can be installed on the front bumper of a construction machine (e.g., an articulated dump truck, etc.).

[0052] The second camera (120) may be installed on the front bumper of a construction machine (e.g., an articulated dump truck, etc.) to have a preset angle (e.g., 45 degrees, etc.). Here, the preset angle is an angle that can be changed in consideration of the shape, size, etc. of the construction machine.

[0053] The second camera (120) may be installed on the front bumper of a construction machine (e.g., an articulated dump truck) located on the same line as the first camera (110). In this way, if the first camera (110) and the second camera (120) are located on the same line, camera calibration is easy, and since the cameras are arranged based on the same axis, there is a physical reference coordinate system for camera parameter modeling, which can be facilitated.

[0054] The second camera (120) can generate a second image corresponding to the front image of the construction machine (e.g., an articulated dump truck, etc.).

[0055] The third camera (130) can be installed on the left outside of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0056] The third camera (130) may be installed on the left outside of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.) so as to be on the same line as the first camera (110) and the fourth camera (140). In this way, if the third camera (130) is positioned on the same line as the first camera (110) and the fourth camera (140), camera calibration is easy, and since the cameras are arranged based on the same axis, there is a physical reference coordinate system for camera parameter modeling, which can be facilitated.

[0057] The third camera (130) can generate a third image corresponding to the left external image of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0058] The fourth camera (140) can be installed on the outside right side of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0059] The fourth camera (140) may be installed on the right outside of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.) so as to be on the same line as the first camera (110) and the third camera (130). In this way, if the fourth camera (140) is positioned on the same line as the first camera (110) and the second camera (130), camera calibration is easy, and since the cameras are arranged based on the same axis, there is a physical reference coordinate system for camera parameter modeling, which can be facilitated.

[0060] The fourth camera (140) can generate a fourth image corresponding to the right external image of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0061] The fifth camera (150) can be installed in a location where the blind area of ​​the construction machine can be photographed (e.g., outside the cabin included in the front frame when the construction machine is an articulated dump truck).

[0062] The fifth camera (150) can be installed vertically so as to be able to look down from above at a location where a blind area of ​​the construction machine (e.g., between a cabin included in the front frame of an articulated dump truck and a dump included in the rear frame) can be photographed (e.g., outside a cabin included in the front frame when the construction machine is an articulated dump truck).

[0063] The fifth camera (150) can generate a fifth image corresponding to a blind area of ​​the construction machine (e.g., between a cabin included in the front frame of an articulated dump truck and a dump included in the rear frame, etc.).

[0064] The sixth camera (160) can be installed at the rear of the construction machine (e.g., if the construction machine is an articulated dump truck, a dump included in the rear frame, etc.).

[0065] The sixth camera (160) can be installed in a location where the rear bottom of the construction machine can be photographed (e.g., if the construction machine is an articulated dump truck, under the dump included in the rear frame, etc.).

[0066] The sixth camera (160) can generate a sixth image corresponding to the rear lower image of the construction machine.

[0067] The processing device (170) can obtain the first image generated by the first camera (110).

[0068] The processing device (170) can obtain a second image generated by the second camera (120).

[0069] The processing device (170) can obtain a third image generated by the third camera (130).

[0070] The processing device (170) can obtain the fourth image generated by the fourth camera (140).

[0071] The processing device (170) can obtain the fifth image generated by the fifth camera (150).

[0072] The processing device (170) can obtain the sixth image generated by the sixth camera (160).

[0073] The processing device (170) can correct distortion of each of the first to sixth images corresponding to each of the first to sixth cameras acquired based on a preset distortion correction value. Here, the distortion correction value can vary according to a 3D model implemented in advance for synthesizing the first to sixth images.

[0074] The processing device (170) can generate a first distortion-corrected image to a sixth distortion-corrected image by correcting the distortion of the first image to the sixth image.

[0075] The processing device (170) can perform deep learning-based object detection for each of the first to sixth images.

[0076] The processing device (170) can perform deep learning-based object detection for each frame constituting the first to sixth images.

[0077] The processing device (170) can perform deep learning-based object detection using a bounding box for each frame constituting the first to sixth images.

[0078] The processing device (170) can perform deep learning-based object detection using a bounding box for each frame constituting the first to sixth images and detect an object (hereinafter referred to as “object”) included in each frame constituting the first to sixth images.

[0079] When the processing device (170) detects an object in any one of the first to sixth images, it can determine coordinates for the object detected in the image in which the object is detected.

[0080] According to one embodiment, when the processing device (170) detects an object, it may set a bounding box to surround the detected object, obtain upper left coordinates (e.g., x1, y1, etc.) and lower right coordinates (e.g., x2, y2, etc.) of the bounding box, and determine the center coordinates of the bounding box based on the upper left coordinates and lower right coordinates of the bounding box.

[0081] According to another embodiment, the processing device (170) may obtain the upper right coordinates (e.g., x1, y2, etc.) and the lower left coordinates (e.g., x2, y1, etc.) of the bounding box, and determine the center coordinates of the bounding box based on the upper right coordinates and the lower left coordinates of the bounding box.

[0082] The processing device (170) can determine the center coordinates of the bounding box as the center coordinates of the detected object.

[0083] The processing unit (170) can use a lookup table to minimize the load of the system on a chip (SoC).

[0084] The processing device (170) can convert coordinates of an object detected in any one of the first to sixth images using a lookup table.

[0085] The processing device (170) can transform the center coordinates of the bounding box determined using the lookup table.

[0086] The processing device (170) can set the size of the table for the resolution area of ​​the output image. Here, the resolution of the output image may be 1280x720, but the resolution of the output image is not limited thereto.

[0087] The processing device (170) can set the size of the table by considering the resolution area of ​​the output image and the precision of the output result.

[0088] The processing device (170) can convert the center coordinates of the bounding box using a lookup table so that an object detected in any one of the first to sixth images can be displayed at an accurate location in the composite image.

[0089] The processing device (170) can generate a composite image by synthesizing the first to sixth images.

[0090] The processing device (170) can generate a composite image by synthesizing the first distortion correction image to the sixth distortion correction image.

[0091] The processing device (170) can generate a composite image by synthesizing the first distortion correction image to the sixth distortion correction image based on a preset 3D model.

[0092] The processing device (170) can display an object detected in any one of the first to sixth images in a synthetic image based on the center coordinates of a bounding box whose coordinates are transformed using a lookup table.

[0093] The processing device (170) can display an image preset to be overlaid with an object detected in any one of the first to sixth images displayed in the composite image on the composite image.

[0094] According to one embodiment, the processing device (170) may display a preset image on the composite image so that the preset image is displayed at the center coordinates of the bounding box whose coordinates are transformed using a lookup table. Here, the preset image may be a warning image, but the preset image is not limited thereto.

[0095] The processing device (170) can output a composite image in which an object detected in any one of the first to sixth images is displayed using the display device (180).

[0096] The processing device (170) can output a composite image in which an object detected in one of the first to sixth images and a preset image is overlaid on the detected object using the display device (180).

[0097] The display device (180) may be a display device equipped in the driver's seat of a construction machine, a head-up display (HUD) device, a head-mounted display (HMD) device, an image output device capable of outputting a composite image, etc., but the display device (180) is not limited thereto.

[0098] The display device (180) may include a monitor, a touch screen, a mouse, an electronic pen, a microphone, a keyboard, a speaker, earphones, headphones, or a touchpad.

[0099] The display device (180) can obtain user input.

[0100] The display device (180) may include a port (e.g., a USB port) for connecting to an external device.

[0101] The display device (180) can be provided as an integrated unit with a lookup table-based detection object display device.

[0102] The display device (180) may be provided separately from the lookup table-based detection object display device.

[0103] The display device (180) may be a separate device that is communicatively connected to the lookup table-based detection object display device.

[0104] It is apparent to those skilled in the art that the term 'device or module' used herein refers to a logical configuration unit and not necessarily a physically distinct component.

[0105] FIGS. 2A and 2B are drawings showing a camera included in a lookup table-based detection object display device mounted on a construction machine according to one embodiment.

[0106] According to one embodiment, FIG. 2a may illustrate a position where a camera is mounted as seen from the side of the construction machine, and FIG. 2b may illustrate a position where a camera is mounted as seen from the top of the construction machine.

[0107] Referring to FIGS. 2a and 2b, each of the plurality of cameras included in the lookup table-based detection object display device can be installed at a preset location of the construction machine.

[0108] Among the multiple cameras included in the lookup table-based detection object display device, the first camera (201, 231) can be installed on the upper part of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0109] Among the plurality of cameras included in the lookup table-based detection object display device, the first camera (201, 231) may be installed above the driver's seat vertically above the head of the driver riding in the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0110] According to one embodiment, the first camera (201, 231) may be installed on the upper part of the driver's seat of the construction machine (e.g., an articulated dump truck, etc.) so that the direction of the driver's gaze is the same as the direction of the driver's gaze when riding in the driver's seat of the construction machine (e.g., an articulated dump truck, etc.).

[0111] The first camera (201, 231) can generate a first image corresponding to the field of view of a driver in the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0112] Among the multiple cameras included in the lookup table-based detection object display device, the second camera (202, 232) can be installed on the front bumper of a construction machine (e.g., an articulated dump truck, etc.).

[0113] According to one embodiment, the second camera (202, 232) may be installed on the front bumper of a construction machine (e.g., an articulated dump truck, etc.) at a preset angle (e.g., 45 degrees, etc.). Here, the preset angle may be changed depending on the shape, size, etc. of the construction machine.

[0114] Additionally, the second camera (202, 232) can be installed on the front bumper of a construction machine (e.g., an articulated dump truck, etc.) located on the same line as the first camera (201, 231).

[0115] The second camera (202, 232) can generate a second image corresponding to the front image of the construction machine (e.g., an articulated dump truck, etc.).

[0116] Among the multiple cameras included in the lookup table-based detection object display device, the third camera (203, 233) may be installed on the left outside of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0117] According to one embodiment, the third camera (203, 233) may be installed on the left outside of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.) so as to be on the same line as the first camera (201, 231) and the fourth camera (204, 234).

[0118] The third camera (203, 233) can generate a third image corresponding to the left external image of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0119] Among the multiple cameras included in the lookup table-based detection object display device, the fourth camera (204, 234) may be installed on the outside right side of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0120] According to one embodiment, the fourth camera (204, 234) may be installed on the right outside of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.) so as to be on the same line as the first camera (201, 231) and the third camera (203, 233).

[0121] The fourth camera (204, 234) can generate a fourth image corresponding to the right external image of the driver's seat of a construction machine (e.g., an articulated dump truck, etc.).

[0122] Among the plurality of cameras included in the lookup table-based detection object display device, the fifth camera (205, 235) can be installed at a location where a blind area of ​​the construction machine can be photographed (e.g., outside a cabin included in a front frame when the construction machine is an articulated dump truck).

[0123] According to one embodiment, the fifth camera (205, 235) may be installed vertically so as to be able to look down from above at a location where a blind area of ​​the construction machine (e.g., between a cabin included in a front frame of an articulated dump truck and a dump included in a rear frame) can be captured (e.g., outside a cabin included in a front frame when the construction machine is an articulated dump truck).

[0124] The fifth camera (205, 235) can generate a fifth image corresponding to a blind area of ​​the construction machine (e.g., between a cabin included in the front frame of an articulated dump truck and a dump included in the rear frame, etc.).

[0125] Among the plurality of cameras included in the lookup table-based detection object display device, the sixth camera (206, 236) may be installed at the rear of the construction machine (e.g., in the case where the construction machine is an articulated dump truck, a dump included in the rear frame, etc.).

[0126] According to one embodiment, the sixth camera (206, 236) may be installed in a position where the rear bottom of the construction machine can be photographed (e.g., under the dump included in the rear frame when the construction machine is an articulated dump truck).

[0127] The sixth camera (206, 236) can generate a sixth image corresponding to the rear lower image of the construction machine.

[0128] FIG. 3 is a drawing showing an object detected in an image being displayed in a synthetic image according to an embodiment.

[0129] Referring to FIG. 3, a lookup table-based detection object display device can obtain first to sixth image frames (300) corresponding to the first to sixth cameras, respectively, from the first to sixth cameras.

[0130] A lookup table-based detection object display device can perform deep learning-based object detection for each of the first to sixth images.

[0131] A lookup table-based detection object display device can perform deep learning-based object detection for each frame (300) constituting the first to sixth images.

[0132] A lookup table-based detection object display device can perform deep learning-based object detection using a bounding box (310) for each frame (300) constituting the first to sixth images.

[0133] A lookup table-based detection object display device can detect an object (320) included in each of the frames (300) constituting the first to sixth images by performing deep learning-based object detection using a bounding box for each of the frames (300) constituting the first to sixth images.

[0134] A lookup table-based detection object display device can correct distortion of each of the first to sixth images corresponding to each of the first to sixth cameras acquired based on a preset distortion correction value. Here, the distortion correction value can vary according to a 3D model implemented in advance for synthesizing the first to sixth images.

[0135] A lookup table-based detection object display device can generate a first distortion-corrected image to a sixth distortion-corrected image by correcting the distortion of the first image to the sixth image.

[0136] If an object (340) is included in one of the first distortion-corrected image to the sixth distortion-corrected image (330) of the detection object display device based on the lookup table, the object (340) may also be distortion-corrected. In this case, the recognizability of the distortion-corrected object (340) may be significantly reduced.

[0137] A detection object display device based on a lookup table can determine the center coordinates of a bounding box (310) based on the upper left coordinates (e.g., x1, y1, etc.) and the lower right coordinates (e.g., x2, y2, etc.) of the bounding box (310) for the acquired object.

[0138] A lookup table-based detection object display device can convert the center coordinates of a bounding box (310) using a lookup table so that an object detected in any one of the first to sixth images can be displayed at an accurate location in a synthetic image.

[0139] A lookup table-based detection object display device can generate a composite image by synthesizing the first distortion correction image to the sixth distortion correction image.

[0140] A lookup table-based detection object display device can display an object (340) detected in any one of the first to sixth images in a synthetic image (330) based on the center coordinates of a bounding box (310) whose coordinates are transformed using the lookup table.

[0141] A lookup table-based detection object display device can display an image (360) preset to be overlaid with an object (370) detected in any one of the first to sixth images displayed in the composite image (350) on the composite image (350).

[0142] A lookup table-based detection object display device can display a synthetic image (350) including a preset image (360) so that the preset image (360) is displayed at the center coordinates of a bounding box (310) whose coordinates are converted using the lookup table.

[0143] FIG. 4 is a flow chart illustrating a detection object display method based on a lookup table according to an embodiment.

[0144] Referring to FIG. 4, a lookup table-based detection object display method according to an embodiment includes a step (S400) in which a lookup table-based detection object display device acquires a plurality of images corresponding to each of a plurality of cameras installed at preset locations of a construction machine, a step (S410) in which the lookup table-based detection object display device performs deep learning-based object detection on each of the plurality of images, a step (S420) in which the lookup table-based detection object display device determines whether an object is detected in at least one image among the plurality of images, a step (S430) in which the lookup table-based detection object display device determines coordinates for the detected object when the object is detected in at least one image among the plurality of images, a step (S440) in which the lookup table-based detection object display device converts coordinates for the detected object using a lookup table, and a step (S450) in which the lookup table-based detection object display device synthesizes the plurality of images acquired to generate a synthesized image and displays the detected object in the synthesized image generated based on the coordinate transformation.

[0145] In step S400, the lookup table-based detection object display device can obtain first to sixth images generated by the first to sixth cameras.

[0146] A lookup table-based detection object display device can correct distortion of each of the first to sixth images based on a preset distortion correction value. Here, the distortion correction value can vary according to a 3D model implemented in advance for synthesizing the first to sixth images.

[0147] A lookup table-based detection object display device can generate a first distortion-corrected image to a sixth distortion-corrected image by correcting the distortion of the first image to the sixth image.

[0148] In step S410, the lookup table-based detection object display device can perform deep learning-based object detection for each frame constituting the first to sixth images.

[0149] A lookup table-based detection object display device can perform deep learning-based object detection using a bounding box for each frame constituting the first to sixth images.

[0150] In step S420, if the lookup table-based detection object display device performs deep learning-based object detection using a bounding box for each of the frames constituting the first to sixth images and fails to detect an object included in each of the frames constituting the first to sixth images, the lookup table-based detection object display device may return to step S410 and perform deep learning-based object detection using a bounding box for each of the frames constituting the first to sixth images.

[0151] When a lookup table-based detection object display device performs deep learning-based object detection using a bounding box for each frame constituting the first to sixth images and detects an object included in each frame constituting the first to sixth images, the lookup table-based detection object display device can perform step S430.

[0152] In step S430, the lookup table-based detection object display device can obtain the upper left coordinates (e.g., x1, y1, etc.) and the lower right coordinates (e.g., x2, y2, etc.) of the bounding box for the detected object.

[0153] A lookup table-based detection object display device can determine the center coordinates of a bounding box based on the upper left coordinates and lower right coordinates of the bounding box for the acquired object.

[0154] In step S440, the lookup table-based detection object display device can transform the center coordinates of the bounding box determined using the lookup table.

[0155] In step S440, the lookup table-based detection object display device can convert the center coordinates of the bounding box using the lookup table so that the object detected in any one of the first to sixth images can be displayed at an accurate location in the synthesized image.

[0156] In step S450, the lookup table-based detection object display device can generate a composite image by synthesizing the first distortion correction image to the sixth distortion correction image.

[0157] A lookup table-based detection object display device can display an object detected in any one of the first to sixth images in a synthetic image based on the center coordinates of a bounding box whose coordinates are transformed using the lookup table.

[0158] A lookup table-based detection object display device can display a preset image on a synthetic image so that the preset image is displayed at the center coordinates of a bounding box whose coordinates are transformed using the lookup table. Here, the preset image may be a warning image, but is not limited thereto.

[0159] In step S450, the lookup table-based detection object display device can output a composite image in which an object detected in any one of the first to sixth images and a preset image is overlaid on the detected object using the display device (180).

[0160]

[0161] Although all components constituting the embodiments of the present invention have been described as being combined or operated in combination as one, the present invention is not necessarily limited to such embodiments. That is, within the scope of the present invention, all components may be selectively combined and operated at least as one.

[0162] Additionally, while each of these components may be implemented as a single, independent piece of hardware, some or all of these components may be selectively combined and implemented as a computer program having program modules that perform some or all of the functions of one or more hardware components. The codes and code segments that constitute the computer program will be readily inferred by those skilled in the art.

[0163] These computer programs can be stored on computer-readable media and read and executed by a computer, thereby implementing embodiments of the present invention. Storage media for the computer program may include magnetic recording media, optical recording media, etc.

[0164] In addition, terms such as “include,” “comprise,” or “have” described above, unless specifically stated to the contrary, should be interpreted to mean that the corresponding component may be included, and thus should not be interpreted to exclude other components, but rather to include other components.

[0165] All terms, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains, unless otherwise defined. Commonly used terms, such as terms defined in dictionaries, should be interpreted in a way consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0166] The above description is merely an example of the technical idea of ​​the present invention, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present invention.

[0167] Accordingly, the embodiments disclosed in the present invention are intended to illustrate, rather than limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical concepts within the scope equivalent thereto should be construed as being included within the scope of the present invention.

Claims

1. In a lookup table-based detection object display device, Multiple cameras installed at preset locations on the construction machine; and including a processing device, The above processing device, Acquire corresponding multiple images from each of the multiple cameras, Deep learning-based object detection is performed on each of the above multiple images, When an object is detected in at least one of the above multiple images, the center coordinates of the detected object are determined, The center coordinates of the above-detected object are converted into coordinates using a lookup table, Generate a composite image by synthesizing the acquired multiple images, Displaying the synthetic image and the preset image so that the preset image is overlaid with the detected object in the synthetic image, A detection object display device based on a lookup table.

2. In paragraph 1, The above processing device, Correcting the distortion of each of the plurality of images obtained based on a preset distortion correction value to generate a distortion-corrected image for each of the plurality of images. A detection object display device based on a lookup table.

3. In paragraph 1, The above processing device, Obtain a bounding box surrounding the detected object, Determining the center coordinates of the bounding box based on the upper left coordinates and lower right coordinates of the bounding box, A detection object display device based on a lookup table.

4. In paragraph 3, The above processing device, Converting the center coordinates of the bounding box determined using the above lookup table into coordinates, A detection object display device based on a lookup table.

5. In paragraph 1, The above processing device, A lookup table-based detection object display device that generates the composite image by synthesizing a distortion correction image for each of the plurality of images.

6. In paragraph 1, The above processing device, A lookup table-based detection object display device that displays a preset image to be overlaid with the detected object.

7. A step of acquiring multiple images corresponding to each of the multiple cameras installed at preset locations of the construction machine; A step of performing deep learning-based object detection for each of the plurality of images; A step of determining the center coordinates of the detected object when an object is detected in at least one image among the plurality of images; A step of converting the center coordinates of the detected object into coordinates using a lookup table; A step of generating a composite image by synthesizing the acquired plurality of images; and A lookup table-based detection object display method, comprising the step of displaying the synthetic image and the preset image so that the preset image is overlaid with the detected object in the synthetic image.

8. In paragraph 7, The step of acquiring the above multiple images is: A lookup table-based detection object display method, comprising the step of generating a distortion-corrected image for each of the plurality of images by correcting the distortion of each of the plurality of images based on a preset distortion correction value.

9. In paragraph 7, The step of determining the center coordinates for the above-detected object is: A step of obtaining a bounding box surrounding the detected object; A lookup table-based detection object display method, comprising a step of determining the center coordinates of the bounding box based on the upper left coordinates and the lower right coordinates of the bounding box.

10. In paragraph 9, The step of converting the center coordinates of the above-detected object into coordinates using a lookup table is as follows: A lookup table-based detection object display method, comprising a step of converting the center coordinates of the bounding box determined using the lookup table.

11. In paragraph 8, The above displaying steps are: A lookup table-based detection object display method, comprising a step of generating the composite image by synthesizing a distortion correction image for each of the plurality of images.

12. In paragraph 7, The above displaying steps are: A lookup table-based detection object display method further comprising the step of displaying a preset image so as to be overlaid with the detected object.

Images