System for segmenting object domain based on object detection feature map and thereof

Abstract

An object detection feature map-based object region segmentation system and method are provided. An object detection feature map-based object region segmentation system and method according to an embodiment of the present invention comprises: a ground truth extraction unit that extracts ground truth data of an object and surrounding terrain features from a two-dimensional image taken from above; an object detection unit that detects and classifies objects within the image based on the ground truth data extracted by the ground truth extraction unit to detect objects; and an object region segmentation unit that segments the area of ​​an object within the image using the ground truth data extracted by the ground truth extraction unit and object information detected by the object detection unit.

Description

Technology Field

[0001] The present invention relates to an object detection feature map-based object region segmentation system and method, and more particularly, to an object detection feature map-based object region segmentation system and method that detects and segments an object and surrounding terrain features using vision data of an aerial view. Background Technology

[0002] Existing one-stage detection models perform object detection, classification, and bounding box regression without going through the process of proposing candidate regions, so while object detection is fast, there is a disadvantage of low accuracy. On the other hand, two-stage detection models propose candidate regions and then perform object detection, classification, and bounding box regression within the proposed candidate regions to improve accuracy, but there is a limitation in that the speed is slow.

[0003] Meanwhile, the self-attention distillation method is a partitioning model designed to improve the representation of each layer, inspired by the existing knowledge distillation method. Applying the self-attention distillation method to existing one-stage detection models can improve accuracy, which is pointed out as a disadvantage compared to two-stage detection models.

[0004] However, methods designed for object segmentation models are difficult to apply to object detection models without modification. Therefore, there is a need for a new object detection model that improves accuracy without compromising speed compared to a first-stage detection model. Prior art literature

[0005] (Patent Document 0001) KR 10-2511315 B1 The problem to be solved

[0006] In order to solve the problems of the conventional technology described above, one embodiment of the present invention aims to provide an object detection feature map-based object region segmentation system and method capable of segmenting an object region within an artificial intelligence detection and segmentation model by combining an object detection model that detects and classifies shape information of objects and surrounding terrain features using two-dimensional image information captured from above as input, and an object segmentation model that segments an object region using a feature map generated through the object detection model and captured image information as input. means of solving the problem

[0007] According to one aspect of the present invention for solving the above problems, an object detection feature map-based object region segmentation system and method are provided, comprising: a ground truth extraction unit for extracting ground truth data of an object and surrounding terrain features from a two-dimensional image taken from above; an object detection unit for detecting and classifying objects within the image based on the ground truth data extracted by the ground truth extraction unit to detect objects; and an object region segmentation unit for segmenting the area of ​​an object within the image using the ground truth data extracted by the ground truth extraction unit and object information detected by the object detection unit.

[0008] In one embodiment, the object detection unit may include: an object detection unit that detects objects within the two-dimensional image based on actual data extracted by the actual data extraction unit; an object classification unit that classifies the objects detected by the object detection unit; and an object location recognition unit that recognizes the location of the objects detected by the object detection unit.

[0009] In one embodiment, the object detection unit may include a feature map generation unit that generates a feature map using actual measurement data extracted from the actual measurement data extraction unit as input values.

[0010] In one embodiment, the feature map generation unit extracts a feature map using a Convolutional Neural Network (CNN), processes the extracted feature map through a Path Aggregation Feature Pyramid Network (PAFPN), and provides it to the object classification unit and the object location recognition unit. The object classification unit classifies the meaning of an object detected by the object detection unit using the feature map generated by the feature map generation unit, and the object location recognition unit derives a bounding box region using the feature map generated by the feature map generation unit and performs bounding box regression based on the derived bounding box region to recognize the location of an object detected by the object detection unit.

[0011] In one embodiment, the object region segmentation unit may include: an attention map generation unit that generates an attention map based on a feature map generated by the feature map generation unit; and a segmentation mask generation unit that generates a segmentation mask based on the attention map generated by the attention map generation unit.

[0012] In one embodiment, the attention map generation unit may generate an attention map by converting the feature map and provide it to the segmentation mask generation unit.

[0013] In one embodiment, the segmentation mask generation unit divides objects within an image into pixel units to label the objects, corrects the labeling by comparing it with actual data extracted by the actual data extraction unit, and can learn so that the segmentation mask of the lower layer mimics the segmentation mask of the upper layer.

[0014] According to another aspect of the present invention, an object detection feature map-based object region segmentation method is provided, comprising: a step of extracting actual measurement data of an object and surrounding topographic features from a two-dimensional image taken from above; a step of detecting objects by detecting and classifying objects within the image based on the extracted actual measurement data; and a step of segmenting the region of an object within the image using the extracted actual measurement data and the detected object information.

[0015] In one embodiment, the step of detecting the object may include: a step of detecting objects in an image based on the extracted actual measurement data; a step of classifying the detected object; and a step of recognizing the location of the detected object.

[0016] In one embodiment, the step of detecting the object may include the step of generating a feature map using the extracted actual measurement data as input values.

[0017] In one embodiment, the step of generating the feature map includes: a step of extracting the feature map using a convolutional neural network (CNN); and a step of processing the extracted feature map through a path aggregation function extraction network (PAFPN); and the processed feature map is provided to a step of classifying an object and a step of recognizing the object location, wherein the step of classifying the object classifies the meaning of the detected object using the generated feature map, and the step of recognizing the object location derives a bounding box region based on the generated feature map and performs regression of the bounding box based on the derived bounding box region to recognize the location of the detected object.

[0018] In one embodiment, the step of dividing the object region may include: a step of generating a attention map based on the generated feature map; a step of generating a segmentation mask using the generated attention map; and a step of mimicking the generated segmentation masks with each other.

[0019] In one embodiment, the step of generating the attention map may generate the attention map by transforming the feature map.

[0020] In one embodiment, the step of generating the segmentation mask may include dividing objects within an image into pixel units to label the objects, correcting the labeling by comparing it with the extracted actual measurement data, and the step of mimicking the segmentation mask may include a process of learning so that the segmentation mask of the upper lower layer mimics the segmentation mask of the upper layer. Effects of the invention

[0021] An object detection feature map-based object region segmentation system and method according to one embodiment of the present invention can provide an object detection-based object region segmentation system with faster detection and region segmentation speeds and improved accuracy by efficiently combining an artificial intelligence-based object detection model and an object segmentation model.

[0022] In addition, the object detection feature map-based object region segmentation system and method according to one embodiment of the present invention can realize effective safety management by utilizing video data, such as CCTVs already established at construction sites, by rapidly and accurately detecting and segmenting objects from two-dimensional images taken from above. Brief explanation of the drawing

[0023] FIG. 1 is a configuration diagram of an object detection feature map-based object region segmentation system according to one embodiment of the present invention. FIG. 2 is a block diagram of an object detection unit of an object region segmentation system based on an object detection feature map according to an embodiment of the present invention. FIG. 3 is a block diagram of an object region segmentation unit of an object region segmentation system based on an object detection feature map according to an embodiment of the present invention. FIG. 4 is a configuration diagram illustrating an artificial intelligence object detection and region segmentation model of an object region segmentation system based on an object detection feature map according to an embodiment of the present invention. FIG. 5 is a diagram showing the results of a demonstration experiment of an object region segmentation system based on an object detection feature map according to an embodiment of the present invention. FIG. 6 is a diagram of the time and result generated through an object detection feature map-based object region segmentation system according to an embodiment of the present invention. FIG. 7 is a flowchart of an object region segmentation method based on an object detection feature map according to an embodiment of the present invention. FIG. 8 is a flowchart of the object detection step of an object region segmentation method based on an object detection feature map according to an embodiment of the present invention. FIG. 9 is a flowchart of the object region segmentation step of an object region segmentation method based on an object detection feature map according to an embodiment of the present invention. Specific details for implementing the invention

[0024] Hereinafter, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. The present invention may be embodied in various different forms and is not limited to the embodiments described herein. In the drawings, parts unrelated to the explanation have been omitted to clearly explain the present invention, and the same reference numerals have been used for identical or similar components throughout the specification.

[0025] Hereinafter, an object detection feature map-based object region segmentation system and method according to one embodiment of the present invention will be described in more detail with reference to the drawings.

[0026] FIG. 1 is a configuration diagram of an object detection feature map-based object region segmentation system (100) according to one embodiment of the present invention.

[0027] Referring to FIG. 1, an object detection feature map-based object region segmentation system (100) according to one embodiment of the present invention may include a measurement data extraction unit (110), an object detection unit (120), and an object region segmentation unit (130).

[0028] The actual measurement data extraction unit (110) can extract and store two-dimensional image data captured from above. Here, the two-dimensional image data can be collected from a camera attached to the end of construction equipment such as a crane. As another example, the two-dimensional image data can be collected from a closed-circuit camera (CCTV) installed at a construction site. At this time, the collected two-dimensional image data can be provided to the object detection unit (120).

[0029] The object detection unit (120) can detect and classify objects within an image by analyzing two-dimensional image data provided by the actual measurement data extraction unit (110). Here, the object detection unit (120) can generate a feature map to detect and classify objects within the image. The generated feature map can be used to classify the meaning of the object and to recognize the location of the object.

[0030] The object region segmentation unit (130) can segment the object region by finely analyzing the object detected and classified by the object detection unit (120) in pixel units. Here, the object region segmentation unit (130) can generate a pattern for segmenting the object region by utilizing the feature map generated by the object detection unit (120). Additionally, the object region segmentation unit (130) can use actual measurement data extracted by the actual measurement data extraction unit (110). At this time, the actual measurement data extracted by the actual measurement data extraction unit (110) can be used to reinforce the pattern for segmenting the object region.

[0031] FIG. 2 is a block diagram of an object detection unit (120) of an object region segmentation system based on an object detection feature map according to an embodiment of the present invention.

[0032] Referring to FIG. 2, the object detection unit (120) may include an object detection unit (122), an object classification unit (124), and an object location recognition unit (126). Here, the object detection unit (120) can detect objects within an image by training an artificial intelligence object detection model. At this time, a first-stage detection model may be utilized as the artificial intelligence object detection model. For example, YOLO, which is a type of first-stage detection model, may be utilized as the artificial intelligence object detection model. The first-stage detection model can improve the speed of object detection by having a relatively simple data processing pipeline and can predict the meaning and location of objects in real time.

[0033] The object detection unit (122) can detect objects within an image based on actual measurement data extracted from the actual measurement data extraction unit (110). Here, the object detection unit (122) may include a feature map generation unit (123). The feature map generation unit (123) can extract and process a feature map and provide it to the object classification unit (124) and the object location recognition unit (126).

[0034] More specifically, a feature map refers to a pattern extracted from the features of the shape of each object when detecting objects in image data. The feature map generation unit (123) can extract a feature map from input actual data using a Convolutional Neural Network (CNN). At this time, the feature map generation unit (123) can use a kernel having the same dimension as the actual data extracted from the image data to extract the feature map. The kernel represents the features of the shape of each object in the same dimension as the actual data extracted from the image data. The feature map generation unit (123) can extract a feature map by convolution of the kernel.

[0035] Additionally, the feature map generation unit (123) can process the extracted feature map through a Path Aggregation Feature Pyramid Network (PAFPN). The Path Aggregation Feature Pyramid Network (PAFPN) can generate a pyramid of feature maps by refining, reconstructing, and mixing multiple layered feature maps from which the features of each object's shape have been extracted. The pyramid of feature maps generated by the Path Aggregation Feature Pyramid Network (PAFPN) can be provided to the object classification unit (124) and the object location recognition unit (126) to be used for detecting objects of various sizes. The Path Aggregation Feature Pyramid Network (PAFPN) can be replaced with other network models having similar processing capabilities. For example, the feature map generation unit (123) can use Bi-FPN or PANet to process the feature map.

[0036] The object classification unit (124) can classify the meaning (class) of an object detected by the object detection unit (122) using a feature map generated by the feature map generation unit (123). The meaning of an object is to find the common noun of the object. For example, the object classification unit (124) can perform calculations to distinguish between 'cars' and 'people' in a photo where cars and pedestrians coexist on a road. The object classification unit (124) can determine the meaning of an object as a probability by applying a feature map to objects of various sizes within the image. For example, the object classification unit (124) can calculate the probability that an object is a car and the probability that it is a person in a photo where cars and pedestrians coexist on a road, and if the probability that it is a car is 0.9 and the probability that it is a person is 0.1, the object can be classified as a car.

[0037] The object location recognition unit (126) can recognize the location of an object detected by the object detection unit (122) using a feature map generated by the feature map generation unit (123). To recognize the location of an object, the object location recognition unit (126) can first set a bounding box area based on the feature map. A bounding box area is defined as the area occupied by the object detected by the object detection unit (122) as a rectangular coordinate area. The object location recognition unit (126) can perform bounding box regression based on the bounding box area set using the feature map. Here, bounding box regression is the process of comparing the bounding box area set using the feature map with actual measurement data of the image data to calculate the error in the probability that an object exists within the bounding box area, and performing a regression operation to reduce the error. Through this, the object location recognition unit (126) can improve the accuracy of object location recognition.

[0038] FIG. 3 is a block diagram of an object region segmentation unit (130) of an object region segmentation system based on an object detection feature map according to an embodiment of the present invention.

[0039] Referring to FIG. 3, the object region segmentation unit (130) may include an attention map generation unit (132) and a segmentation mask generation unit (134). Here, the object region segmentation can detect objects within an image by segmenting them into pixel units to complement the object detection unit (120). At this time, the object region segmentation unit (130) may use a self-attention distillation model as an artificial intelligence object region segmentation model for learning and performing object region segmentation. Additionally, the attention map generation unit (132) may utilize a feature map generated by the feature map generation unit (123), and the segmentation mask generation unit (134) may utilize actual data extracted by the actual data extraction unit (110).

[0040] More specifically, the attention map generation unit (132) can convert the feature map generated by the object detection unit (122) into an attention map. The attention map can visualize which areas of the feature map have a significant influence on recognizing the object area by computation. For example, the attention map can visualize areas containing key features for object recognition in a feature map containing image data extracted from actual measurement data. At this time, an attention generator can be used to generate the attention map from the feature map.

[0041] The segmentation mask generation unit (134) can generate a segmentation mask based on the attention map generated by the attention map generation unit (132). The segmentation mask is a label that indicates the class of an object by dividing an object in an image into pixel units. Here, the segmentation mask can be corrected by comparing the actual measurement data extracted by the actual measurement data extraction unit (110) with the label indicating the class of the object. Additionally, the segmentation mask generation unit (134) can increase the accuracy of the object region by reducing the error through the lower-level segmentation mask mimicking the upper-level segmentation mask.

[0042] FIG. 4 is a configuration diagram illustrating an artificial intelligence object detection and region segmentation model according to an embodiment of the present invention. Here, (a) represents a self-attention distillation model, (b), (c), and (d) represent each component of a first-stage detection model, and (e) represents two-dimensional image data that is input as actual measurement data to the above artificial intelligence object detection and region segmentation model and to which the detection and segmentation model is applied.

[0043] Referring to FIG. 4, the first stage detection model can start with (b) a backbone structure. (b) The backbone structure can extract feature maps by using actual data extracted from the actual data extraction unit (110) as input values ​​and passing through operation stages (convolution layers) from S2 to S5. The feature maps extracted at each operation stage in (b) the backbone structure may have resolutions of various sizes at each stage. The feature maps extracted with various resolutions can be applied to objects of different sizes.

[0044] (c) The neck structure can process feature maps extracted at various resolutions. (c) The neck structure can refine, mix, and reconstruct feature maps by up-scaling feature maps extracted at low resolution to increase resolution and concatenating each feature map. At this time, the feature maps refined, mixed, and reconstructed in the (c) neck structure can be used in the (d) head structure to classify the semantics of objects and recognize their locations.

[0045] (d) The head structure can classify the meaning of objects in a 2D image and recognize their location using feature maps. Here, the feature maps used in the (d) head structure are extracted by size in the (b) backbone structure and can be applied to objects of various resolutions and sizes through a processing process in the (c) neck structure. As a result, the (d) head structure can effectively detect objects of various sizes contained in a 2D image.

[0046] Additionally, referring to Fig. 4, (a) is the structure of a self-attention distillation model, which is used for object region segmentation. Here, the self-attention distillation model (a) can utilize feature maps extracted for each resolution from the backbone structure (b). Furthermore, the self-attention distillation model (a) may include an attention generator internally. The attention generator can convert the feature maps extracted for each resolution from the backbone structure (b) into attention maps by modeling the relationships between object regions within a 2D image. Segmentation masks can be generated for each resolution by referencing the attention maps and actual measurement data. The segmentation masks generated for each resolution can segment object regions within a 2D image at the pixel level based on the object detection results classified and localized by the first-stage detection model. Additionally, the generated segmentation masks can improve accuracy by reducing the error in object region segmentation, as the segmentation mask of the lower stage mimics the segmentation mask of the upper stage.

[0047] The artificial intelligence detection and segmentation model used in the object detection feature map-based object region segmentation system (100) may not require additional supervision. The artificial intelligence detection and segmentation model can internally learn a process to improve accuracy by using internal output values ​​as input values ​​again.

[0048] FIG. 5 is a diagram showing the results of an empirical experiment of an object region segmentation system based on an object detection feature map according to an embodiment of the present invention. Here, (a) shows actual measurement data, (b) shows the object region segmented using only a 1-stage detection model, and (c) shows the object region segmented using a combination of a 1-stage detection model and a self-attention distillation model.

[0049] More specifically, referring to FIG. 5, (b) is a segmentation of object regions by applying only the feature maps for each resolution extracted from the object detection unit (120) to a 2D image. (b)-(i) and (b)-(ii) are segments of object regions by applying the feature maps to a 2D image that illuminates the foreground at a low magnification. The low-magnification feature maps distinguish object regions by bright colors and can have a resolution level that corresponds to the actual measurement data in the 2D image when viewed with the naked eye. However, object region segmentation using only feature maps becomes indistinguishable as the 2D image is magnified and the magnification of the actual measurement data increases ((b)-(iii), (b)-(iv)).

[0050] On the other hand, (c) is the result of object region segmentation output through an object region segmentation model in an object region segmentation unit (130) using the object detection result output through an object detection model in an object detection unit (120). When the attention map is applied to a 2D image that illuminates the foreground at a low magnification ((c)-(i), (c)-(ii)), the object region segmentation form is not significantly different from when only the feature map is applied to a 2D image at the same magnification ((b)-(i), (b)-(ii)), but as the magnification of the 2D image increases ((c)-(iii), (c)-(iv)), the part separated as an object region becomes more clearly highlighted when compared with (b), and the accuracy of object region segmentation increases.

[0051] As such, there are clear limitations to dividing object regions using only an object detection model, such as a first-stage detection model. Therefore, the accuracy of object region division can be improved regardless of resolution by an object detection feature map-based object region division system (100) that merges an object region division model, such as a self-attention distillation model, with an object detection model. In addition, the object detection feature map-based object region division system (100) can provide an object detection and region division system with high accuracy while maintaining high speed by using a first-stage detection model with a fast detection speed for object detection prior to object region division.

[0052] FIG. 6 is a diagram of a visualization result generated through an object detection feature map-based object region segmentation system according to an embodiment of the present invention.

[0053] Referring to FIG. 6, (a) shows the result of projecting a segmentation mask onto actual measurement data. Here, objects within the 2D image of (a) can be distinguished by color and line according to the meaning and recognized location of each object.

[0054] (b) shows the result of projecting the segmented object region onto the actual data. In the procedure (b), the region of interest among the object regions segmented in the procedure (a) is designated and projected onto the actual data, thereby allowing the object of interest to be tracked and observed regardless of changes in the position and size of the object within the 2D image.

[0055] FIG. 7 is a flowchart of an object region segmentation method (200) based on an object detection feature map according to an embodiment of the present invention.

[0056] Referring to FIG. 7, an object detection feature map-based object region segmentation method (200) according to one embodiment of the present invention includes a measurement data extraction step (S210), an object detection step (S220), and an object region segmentation step (S230).

[0057] To explain in more detail, as illustrated in FIG. 7, the object detection feature map-based object region segmentation method (200) extracts and stores actual measurement data from two-dimensional image data captured from above (step S210). Here, the two-dimensional image data can be collected from a camera attached to the end of construction equipment such as a crane. As another example, the two-dimensional image data can be collected from a closed-circuit camera (CCTV) installed at a construction site. At this time, the collected two-dimensional image data can be provided to the object detection step (S220).

[0058] Next, the object detection feature map-based object region segmentation method (200) detects objects within an image by analyzing two-dimensional image data from extracted actual measurement data (step S220). Here, the object detection feature map-based object region segmentation method (200) can generate a feature map to detect and classify objects within an image. The generated feature map can be used to classify the meaning of objects and to recognize the location of objects.

[0059] Next, the object detection feature map-based object region segmentation method (200) detects and classifies the detected objects and analyzes them in detail at the pixel level to segment the object region (step S230). Here, the object detection feature map-based object region segmentation method (200) can generate a pattern for segmenting the object region by utilizing the generated feature map. Additionally, the object detection feature map-based object region segmentation method (200) can use actual measurement data. In this case, the actual measurement data can be used to reinforce the pattern for segmenting the object region.

[0060] FIG. 8 is a flowchart of the object detection step (S220) of the object detection feature map-based object region segmentation method according to one embodiment of the present invention.

[0061] Referring to FIG. 8, the object detection step (S220) includes a feature map extraction step (S222), a feature map processing step (S224), an object classification step (S226), and an object location recognition step (S228). Here, the object detection step (S220) can detect objects within an image by training an artificial intelligence object detection model. At this time, a first-stage detection model can be utilized as the artificial intelligence object detection model. For example, YOLO, which is a type of first-stage detection model, can be utilized as the artificial intelligence object detection model.

[0062] To explain in more detail, as illustrated in FIG. 9, the object detection feature map-based object region segmentation method (200) extracts a feature map based on actual measurement data (step S222). Here, the feature map refers to a pattern extracted from the features of the shape of each object when detecting objects in image data. The object detection feature map-based object region segmentation method (200) can extract a feature map from the input actual measurement data using a convolutional neural network (CNN). At this time, the object detection feature map-based object region segmentation method (200) may use a kernel having the same dimension as the actual measurement data extracted from the image data to extract the feature map. The kernel represents the features of the shape of each object in the same dimension as the actual measurement data extracted from the image data. The object detection feature map-based object region segmentation method (200) can extract a feature map by performing operations on the kernel.

[0063] Next, the object detection feature map-based object region segmentation method (200) processes the extracted feature map through a path aggregation function extraction network (PAFPN) (step S224). The path aggregation function extraction network (PAFPN) can generate a pyramid of feature maps by refining, reconstructing, and mixing multiple layered feature maps from which features of the shape of each object have been extracted. The pyramid of feature maps generated by the path aggregation function extraction network (PAFPN) can be provided to the object classification step (S226) and the object location recognition step (S228) to be used for detecting objects of various sizes. The path aggregation function extraction network (PAFPN) can be replaced with other network models having similar processing capabilities. For example, Bi-FPN or PANet may be used to process the feature map in the feature map processing step (S224).

[0064] Next, the object detection feature map-based object region segmentation method (200) classifies the meaning (class) of an object using the generated feature map (step S226). The meaning of an object is to find the common noun of the object. For example, the object detection feature map-based object region segmentation method (200) can be computed to distinguish between 'cars' and 'people' in a photo where cars and pedestrians coexist on a road. The object detection feature map-based object region segmentation method (200) can determine the meaning of an object as a probability by applying feature maps to objects of various sizes within an image. For example, the object detection feature map-based object region segmentation method (200) computes the probability that an object is a car and the probability that it is a person in a photo where cars and pedestrians coexist on a road, and if the probability that it is a car is 0.9 and the probability that it is a person is 0.1, the object can be classified as a car.

[0065] Next, the object detection feature map-based object region segmentation method (200) recognizes the location of a detected object using a feature map (step S228). To recognize the location of an object, the object detection feature map-based object region segmentation method (200) can first set a bounding box area based on the feature map. A bounding box area is defined as the area of ​​a rectangle's coordinates that occupies the space occupied by the detected object. The object detection feature map-based object region segmentation method (200) can perform bounding box regression based on the bounding box area set using the feature map. Here, bounding box regression is the process of comparing the bounding box area set using the feature map with actual measurement data of the image data to calculate the error in the probability that an object exists within the bounding box area, and performing regression operations to reduce the error. Through this, the object detection feature map-based object region segmentation method (200) can improve the accuracy of object location recognition.

[0066] FIG. 9 is a flowchart of the object region segmentation step of an object region segmentation method based on an object detection feature map according to an embodiment of the present invention.

[0067] Referring to FIG. 9, the object region segmentation step (S230) includes an attention map generation step (S232), a segmentation mask generation step (S234), and a segmentation mask imitation step (S236). Here, object region segmentation can detect objects within an image by segmenting them pixel by pixel to complement the object detection step (S220). At this time, the object region segmentation step (S230) may use a self-attention distillation model as an artificial intelligence object region segmentation model to learn and perform object region segmentation. Additionally, the attention map generation step (S232) may utilize the feature map generated in the feature map extraction step (S222), and the segmentation mask generation step (S234) may utilize the actual data extracted in the actual data extraction step (S210).

[0068] To explain in more detail, the object detection feature map-based object region segmentation method (200) generates a caution map by converting the feature map into a caution map (step S232). The caution map can visualize which region of the feature map has a significant influence on recognizing the object region by computation. For example, the caution map can visualize the region containing key features for object recognition in a feature map containing image data extracted from actual measurement data. At this time, a caution generator can be used to generate the caution map from the feature map.

[0069] Next, the object detection feature map-based object region segmentation method (200) generates a segmentation mask based on the attention map (step S234). The segmentation mask is a label that indicates the meaning of an object by segmenting an object within an image in pixel units. Here, the segmentation mask can be corrected by comparing actual measurement data with the label indicating the meaning of the object. Additionally, the object detection feature map-based object region segmentation method (200) can increase the accuracy of the object region by reducing the error through the lower-layer segmentation mask mimicking the upper-layer segmentation mask.

[0070] Accordingly, an object detection feature map-based object region segmentation system and method according to one embodiment of the present invention utilize a shooting system equipped with an image sensor in a manufacturing industry and construction site environment to capture and collect images from above, refine them to construct training data for detecting lifted objects and surrounding obstacles, and provide an artificial intelligence-based object detection and segmentation model that segments object regions within the object detection and segmentation model using actual measurement data of the detection model and feature maps extracted from the object detection model as inputs without additional supervision.

[0071] The above methods can be implemented by an object detection feature map-based object region segmentation system (100) as illustrated in FIG. 1, and in particular, can be implemented by a software program that performs these steps, in which case such programs can be stored on a computer-readable recording medium or transmitted by a computer data signal combined with a carrier wave on a transmission medium or communication network.

[0072] At this time, the computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored, and may be, for example, ROM, RAM, CD-ROM, DVD-ROM, DVD-RAM, magnetic tape, floppy disk, hard disk, optical data storage device, etc.

[0073] Although an embodiment of the present invention has been described above, the scope of the present invention is not limited to the embodiments presented in this specification. Those skilled in the art who understand the scope of the present invention may easily propose other embodiments within the same scope by adding, changing, deleting, or adding components, and such are also to be considered to fall within the scope of the present invention. Explanation of the symbols

[0075] 100: Object Detection Feature Map-Based Object Region Segmentation System 110: Actual measurement data extraction unit 120: Object detection unit 122: Object detection unit 123: Feature map generation unit 124: Object Classification Unit 126: Object Location Recognition Unit 130: Object Area Partitioning Unit 132: Attention Map Generation Unit 134 : Split Mask Generation Unit

Claims

Claim 1 An object detection feature map-based object region segmentation system comprising: a ground truth extraction unit that extracts ground truth data regarding an object and surrounding terrain features from the two-dimensional camera image captured from above; an object detection unit that generates a feature map using the ground truth data regarding an object and surrounding terrain features extracted by the ground truth extraction unit as input values, classifies the meaning of the object using the feature map, and detects the object by recognizing the location of the object through bounding box regression; and an object region segmentation unit that converts the feature map into an attention map reflecting the importance of the object region using the feature map generated by the object detection unit and the ground truth data together, and generates a segmentation mask that segments the object in pixel units based on the attention map, wherein the object detection unit and the object region segmentation unit perform object detection and object region segmentation within a single linked processing flow using the feature map in common, and the object region segmentation unit is configured to correct the segmentation mask by comparing it with the ground truth data. Claim 2 An object detection feature map-based object region segmentation system according to claim 1, wherein the object detection unit includes a feature map generation unit, and the feature map generation unit extracts feature maps using a Convolutional Neural Network (CNN), refines, reconstructs, and mixes feature maps extracted from multiple layers to generate a feature map pyramid, and the feature map pyramid is provided to the object detection unit for object classification and object location recognition to correspond to objects having different resolutions, and is commonly used as an input for generating a attention map and a segmentation mask in the object region segmentation unit. Claim 3 In paragraph 2, the object detection feature map-based object region segmentation system is characterized in that the feature map generation unit generates a feature map corresponding to objects of multiple sizes by upscaling and combining feature maps of different resolutions using a path aggregation-based feature fusion network. Claim 4 An object detection feature map-based object region segmentation system according to claim 1, wherein the object region segmentation unit comprises a caution map generation unit that generates a caution map emphasizing a region with high contribution to object region recognition from the feature map, and a segmentation mask generation unit that labels objects in pixel units using the caution map and the actual measurement data. Claim 5 An object detection feature map-based object region segmentation system according to claim 1, wherein the object region segmentation unit is configured to improve the accuracy of object region segmentation by performing learning such that a segmentation mask of a lower layer mimics a segmentation mask of an upper layer among a plurality of segmentation masks generated at different resolutions or layers. Claim 6 A method for detecting and segmenting objects in a two-dimensional camera image taken from above, comprising: a step of extracting ground truth data regarding an object and surrounding terrain features from the two-dimensional camera image; a step of generating a feature map using the ground truth data and the two-dimensional camera image as inputs; an object detection step of detecting an object by classifying the meaning of the object using the feature map and recognizing the location of the object through bounding box regression; and an object region segmentation step of converting the feature map into an attention map reflecting the importance of the object region using the feature map generated in the object detection step and the ground truth data together, and generating a segmentation mask that segments the object in pixel units based on the attention map, wherein the object detection step and the object region segmentation step are performed within a single linked processing flow using the feature map in common, and the object region segmentation step includes a step of correcting the segmentation mask by comparing it with the ground truth data. Claim 7 In claim 6, the step of generating the feature map comprises the step of generating a feature map corresponding to multiple sizes of objects by upscaling and combining feature maps of different resolutions using a path aggregation-based feature fusion network. Claim 8 The object detection feature map-based object region segmentation method according to claim 6, wherein the step of generating the feature map comprises: a step of extracting a feature map using a convolutional neural network (CNN); a step of generating a feature map pyramid by refining, reconstructing, and mixing the feature maps extracted from multiple layers; and a step of providing the feature map pyramid for semantic classification and location recognition of an object, and using it commonly as an input for generating an attention map and a segmentation mask for object region segmentation. Claim 9 In claim 6, the object region segmentation step comprises: a step of generating a caution map that emphasizes a region with a high contribution to object region recognition from the feature map; and a step of generating a segmentation mask that labels objects in pixel units using the caution map and the actual measurement data, characterized in that the object detection feature map-based object region segmentation method. Claim 10 The object detection feature map-based object region segmentation method according to claim 6, wherein the object region segmentation step comprises a step of improving the accuracy of object region segmentation by performing learning such that a segmentation mask of a lower layer mimics a segmentation mask of an upper layer among a plurality of segmentation masks generated at different resolutions or layers. Claim 11 A computer-readable recording medium having a program recorded thereon for performing the method of any one of paragraphs 6 through 10. Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete

Images