Method and device for detecting AVPS marker

Abstract

A method and apparatus for detecting an AVPS marker. In one example, a method of detecting a coded marker includes receiving an image and pre-processing the image; extracting a feature map of the image through a backbone network; extracting a central point heat map of the coding mark; using a plurality of head networks to obtain a first feature map of the width and height of a bounding box, a second feature map used for adjusting the offset of the center point and a third feature map of the corner point of the coding mark from the feature map, and generating candidate detection information including the bounding box and the corner based on the center point heat map, the first feature map, the second feature map, and the third feature map, and outputting final detection information by performing modified non-maximum suppression on the candidate detection information.

Description

[0001] Related Applications

[0002] This application claims the benefit of Korean Patent Application No. 10-2023-0127919, filed on September 25, 2023, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates to a method and apparatus for detecting AVPS markers. More specifically, the present disclosure relates to techniques for accurately detecting bounding boxes and corner points of markers in images. Background Art

[0004] The following description merely provides background information related to the present disclosure and does not constitute prior art.

[0005] For parking convenience, an automatic valet parking system (AVPS) is being developed. AVPS automatically operates a vehicle so that when a driver gets off the vehicle at a drop-off area in a parking facility, the vehicle moves to an empty parking space and parks. In addition, AVPS automatically moves the parked vehicle to a pick-up area upon request by the driver, allowing the driver to board the vehicle.

[0006] For safe and reliable AVPS, level 4 or higher autonomous driving is required. AVPS must not only identify other objects to prevent collisions, but also determine empty parking spaces and driving routes and automatically move, park, and exit the vehicle. To this end, positioning technology for estimating the current position of the target vehicle is important.

[0007] AVPS adopts vision-based positioning using high-definition maps and cameras as positioning technology. AVPS uses coded markers specified in ISO 23374 for positioning. Coded markers are a type of fiducial markers and can be installed in parking facilities and recognized by vehicles. Fiducial markers are artificial markers, objects placed in the field of view of the imaging system, which appear in the generated image and are used as reference points or measurements. High-definition maps may include information about the identification (ID), location, orientation, etc. of the coded markers installed in the parking facility. The accuracy of coded marker recognition affects the vehicle positioning performance in AVPS.

[0008] Related technologies for fiducial marker detection include methods for detecting ArUco markers using general image processing techniques or deep learning. However, the related technologies have the following problems: small markers cannot be detected, markers may be detected incorrectly, or the marker recognition rate may be very low for highly distorted images. Summary of the invention

[0009] Embodiments of the present disclosure provide a method and apparatus for detecting AVPS markers in an image.

[0010] Embodiments of the present disclosure provide a method for accurately detecting bounding boxes and corner points of markers in an image.

[0011] The embodiments of the present disclosure are not limited to the above-mentioned embodiments, and other embodiments not mentioned above will be clearly understood by a person of ordinary skill in the art through the following description.

[0012] At least one embodiment of the present disclosure provides a method for detecting a coded marker, comprising: receiving an image and preprocessing the image, extracting a feature map of the image through a backbone network, extracting a center point heat map for a coded marker, a first feature map for a width and a height of a bounding box, a second feature map for adjusting an offset of the center point, and a third feature map of a corner point of the coded marker from the feature map using multiple head networks, generating candidate detection information including the bounding box and the corner point based on the center point heat map, the first feature map, the second feature map, and the third feature map, and outputting final detection information by performing modified non-maximum suppression on the candidate detection information.

[0013] Another embodiment of the present disclosure provides a device, including one or more processors and a memory operably connected to the one or more processors, wherein the memory stores instructions, the instructions causing the one or more processors to perform operations in response to the one or more processors executing the instructions, and the operations include: receiving an image and preprocessing the image, extracting a feature map of the image through a backbone network, extracting a center point heat map for an encoded marker, a first feature map of the width and height of a bounding box, a second feature map for adjusting the offset of the center point, and a third feature map for the corner points of the encoded marker from the feature map using multiple head networks, generating candidate detection information including the bounding box and the corner points based on the center point heat map, the first feature map, the second feature map, and the third feature map, and outputting final detection information by performing modified non-maximum suppression on the candidate detection information.

[0014] According to an embodiment of the present disclosure, the vehicle localization performance in an AVPS may be improved by accurately detecting the bounding boxes and corner points of markers in an image.

[0015] The effects of the embodiments of the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a configuration diagram of a vehicle positioning system using an AVPS tag according to an embodiment of the present disclosure.

[0017] Figure 2 is a schematic diagram of the coding mark.

[0018] Figure 3 is an exemplary diagram showing coded markings installed in a parking facility.

[0019] Figure 4 is an exemplary diagram showing bounding boxes and corner points of encoded markers detected in an image according to an embodiment of the present disclosure.

[0020] Figure 5 The present invention is a flowchart of a method for detecting AVPS markers according to an embodiment of the present disclosure.

[0021] Figure 6 is an exemplary diagram illustrating size adjustment in image preprocessing according to an embodiment of the present disclosure.

[0022] Figure 7 is a diagram illustrating the structure of a deep learning network for detecting AVPS markers according to an embodiment of the present disclosure.

[0023] Figure 8 is a diagram illustrating a process of extracting information about a preset number of center points from a center point heat map according to an embodiment of the present disclosure.

[0024] Fig. 9 is a diagram illustrating a process of extracting a width and height of a bounding box, an offset, and corner point information of an encoded marker based on extracted center point information according to an embodiment of the present disclosure.

[0025] Fig.10 is a diagram illustrating a process of performing modified non-maximum suppression on candidate detection information according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the attached illustrative drawings. In the following description, the same reference numerals preferably represent the same elements, although the elements are shown in different drawings. In addition, in the following description of some embodiments, for the purpose of clarity and brevity, detailed descriptions of related known components and functions will be omitted when it is considered to obscure the subject matter of the embodiments of the present disclosure.

[0027] Various ordinal or letter codes, such as first, second, i), ii), a), b), etc., are prefixed only to distinguish one component from another, and do not imply or suggest the substance, order or sequence of the components. Throughout this specification, when a component "includes" or "comprises" a component, the component is intended to further include other components, not excluding these components, unless otherwise specifically stated. Terms such as "unit", "module", etc. refer to one or more units for processing at least one function or operation, which can be implemented by hardware, software or a combination thereof.

[0028] The following description of the embodiments of the present disclosure in conjunction with the accompanying drawings is intended to describe exemplary embodiments of the present disclosure, and is not intended to represent only embodiments in which the technical concept of the present disclosure can be practiced.

[0029] In this specification, the term "AVPS tag" may be used interchangeably with the term "coding tag" and may be simply referred to as the term "tag".

[0030] A vehicle positioning system using an AVPS tag (hereinafter, referred to as a “vehicle positioning system”) according to an embodiment of the present disclosure will first be described.

[0031] refer to Figure 1 , the vehicle positioning system 10 according to the embodiment of the present disclosure obtains the current position of the vehicle. The vehicle positioning system 10 can recognize the coded marker in the image, estimate the position of the coded marker using the coded marker recognition result and the mileage information of the vehicle, and then perform map matching based on the estimated coded marker position and the AVPS parking facility marking information 30 to obtain the current position of the vehicle. To this end, the vehicle positioning system 10 includes all or some of the marker recognition unit 100, the mileage extraction unit 120, the marker position estimation unit 140, and the vehicle positioning unit 160.

[0032] The marker recognition unit 100 recognizes a coded marker in a parking facility based on images obtained by photographing the front, left, right, and rear sides of a vehicle using a plurality of cameras 20a, 20b, 20c, and 20d. The marker recognition unit 100 outputs a coded marker recognition result including an ID, a bounding box, and corner points of the coded marker in the image.

[0033] To this end, the marker recognition unit 100 may perform marker detection, marker classification, and marker tracking. Marker detection is the process of simultaneously detecting the bounding box and corner points of an object belonging to a category corresponding to a coded marker within each input image frame, and a deep learning network trained by multi-task learning may be used, but the present disclosure is not limited thereto. Marker classification is a process of extracting a region of interest (ROI) corresponding to a coded marker in each image frame using the detection result and identifying the coded marker ID by performing decoding on the ROI. Marker tracking is the process of tracking the position change of a coded marker within a series of image frames, and coded marker tracking may be performed by additionally utilizing the identified coded marker ID information.

[0034] The mileage extraction unit 120 periodically calculates the longitudinal and lateral movement distances of the vehicle over time through a wheel odometer, and stores the longitudinal and lateral movement distances in a buffer. The wheel odometer refers to estimating the change in the position of the vehicle over time using vehicle specification information and vehicle sensor information. The vehicle specification information may include wheelbase, the number of gear teeth, wheel size, etc. The vehicle sensor information may include wheel speed signals, wheel pulse signals, and steering angle signals received through a chassis controller area network (CAN).

[0035] The marker position estimation unit 140 estimates the position and orientation of the coded marker based on the center of the vehicle (hereinafter referred to as "vehicle-based coded marker position"). Here, "vehicle-based" means indicating the relative position and orientation of the coded marker with the center of the vehicle as a reference point. First, the camera-based coded marker position and orientation can be obtained using the coded marker recognition result, the actual size of the coded marker, and the internal / external parameters of the camera. Here, "camera-based" means indicating the relative position and orientation of the coded marker with the camera as a reference point. Then, by using the camera's installation position and vehicle specifications, the reference point of the camera-based coded marker position and orientation is moved to the center of the vehicle, and the vehicle-based coded marker position can be estimated.

[0036] At the same time, a delay time is generated from time t1 when the marker recognition unit 100 receives the image to time t2 when the marker recognition unit 100 outputs the coded marker recognition result. When the delay is long, the accuracy of the vehicle-based coded marker position estimation based on the coded marker recognition result may deteriorate due to the movement of the vehicle. Therefore, the marker position estimation unit 140 can obtain the longitudinal and lateral movement distances of the vehicle between time t1 and time t2 by using the buffer data stored by the mileage extraction unit 120, and correct the vehicle-based coded marker position estimation result by using the obtained movement distance.

[0037] The vehicle positioning unit 160 obtains the current position of the vehicle in the parking facility based on the coded marker identification information (the ID of the coded marker and the information on the vehicle-based coded marker position) and the AVPS parking facility marker information 30. The vehicle positioning unit 160 can find the marker ID that is the same as the ID of the recognized coded marker from the AVPS parking facility marker information 30, and obtain the current position of the vehicle in the parking facility using the vehicle-based coded marker position.

[0038] For example, it is assumed that a marker with ID 45 is installed at a position of 100 m in the longitudinal direction and 50 m in the lateral direction relative to the entry point of the parking facility. When the ID of the recognized coded marker is 45 and the vehicle-based coded marker position is estimated to be 3 m in the longitudinal direction and -1 m in the lateral direction, by searching for the marker corresponding to ID 45 in the AVPS parking facility marker information 30, based on the absolute position of the marker in the parking facility (100, 50) and the vehicle-based coded marker position (3, -1), it can be seen that the vehicle is located in the parking facility (103, 49).

[0039] The AVPS parking facility marking information 30 includes information about the ID, location, orientation, size, etc. of each of the plurality of coded markings installed in the parking facility. When a vehicle enters the parking facility, the AVPS parking facility marking information 30 may be included in a high-definition map transmitted from a parking facility control center. Here, the parking facility control center may be an R subsystem specified in ISO23374, but is not limited thereto.

[0040] The present disclosure relates to a method and apparatus for detecting an AVPS marker performed by the marker recognition unit 100. Although conventional marker detection methods only output bounding box information, the present disclosure provides a method for simultaneously detecting not only bounding box information of a marker but also corner point information of a marker using a deep learning network trained by multi-task learning.

[0041] First, the AVPS flag will be described.

[0042] Figure 2 is a schematic diagram of the coding mark.

[0043] Reference Figure 2 , shows a coded mark in the form of a square with a black and white binary pattern. This is for high contrast, simple geometry and ease of information encoding.

[0044] For easy identification, the coded mark contains coded information within a square consisting of 6×6 grids on a white background of size 40cm×40cm. The borders of the 6×6 square are filled with black, and the information is encoded in a black and white binary pattern in the inner 4×4 grid. In the square consisting of the inner 4×4 grid, the grids corresponding to the four corners are used for directional bits, and the other grids are used for eight data bits and four parity bits.

[0045] The orientation bit is used to determine the orientation of the encoded mark, and only the grid corresponding to the upper left corner has a value of 1, and the other grids have a value of 0.

[0046] The data bits are used to identify the ID of the coded tag, and a total of 255 ID values ​​can be expressed. Values ​​0 to 250 are used only once within a particular parking facility, and values ​​251 to 255 can be reused to allow for required coverage in large parking facilities.

[0047] The parity bits are used for error checking.

[0048] Information about the locations where the coded markers are installed in the parking facility must be correctly entered into the high definition map managed by the parking facility control center.The coded markers are preferably installed on a flat surface for detection and identification. Figure 3 An example of coded markings installed in a parking facility is shown in FIG.

[0049] Figure 4 is an exemplary diagram showing bounding boxes and corner points of encoded markers detected in an image according to an embodiment of the present disclosure.

[0050] The coded mark included in the images obtained by photographing the front, left, right, and rear sides of the vehicle acquired from the plurality of cameras 20a, 20b, 20c, and 20d may appear in various postures (e.g., positions and orientations) according to the installation position in the parking facility, and may appear in various scales and modified forms according to the distance from the vehicle, etc. Figure 4 , the coded mark in this image may have a non-rectangular form, and shows a bounding box 41 and corner points 42a, 42b, 42c, and 42d which are the result of the coded mark detection in the mark recognition unit 100. That is, the bounding box 41 is a box including the coded mark with a white background, and the corner points 42a, 42b, 42c, and 42d are the respective corners of a 6×6 square.

[0051] The detected bounding box information can be used for different purposes. Specifically, the bounding box has the meaning of a region of interest (ROI) where the coded marker is present in the image. Since the entire shape of the coded marker is contained within the region in the image where the bounding box is located, tasks such as classification and tracking can be performed using the image within the bounding box.

[0052] The detected corner point information is used for decoding to identify the coded marker ID. Since the detected corner point information includes the coordinates of each corner of the 6×6 square, and the corner points of the actual coded marker installed in the parking facility correspond to the corresponding corners of the square, it can be used to obtain the homography matrix, and the image can be bent on the plane using the homography matrix and converted into a normalized coded marker image. The converted coded marker image is easy to decode, and this can improve the vehicle positioning performance in the AVPS.

[0053] Further, each corner point is assigned a corresponding number based on its position. The number can be determined based on the training data of the deep learning network. For example, when the deep learning network is trained using training data configured with the number 1 assigned to the upper left end, the number 2 assigned to the upper right end, the number 3 assigned to the lower right end, and the number 4 assigned to the lower left end, the coordinates of the corner points of the corresponding numbers in the corresponding channels of the feature map output from the corner point head network can be obtained, which will be described later. Further, the number assigned to each corner point can be used to identify the orientation of the coding mark. For example, when the corner point with the number 1 is the upper left end, this indicates the position with the value 1 in the four orientation bits of the coding mark. Using this, the orientation of the coding mark can be determined.

[0054] Will refer to Figures 5 to 10 A method of detecting an AVPS marker according to an embodiment of the present disclosure is described.

[0055] The marker recognition unit 100 receives images from the plurality of cameras 20a, 20b, 20c and 20d and performs preprocessing on the images (S510). This is intended to make the received images suitable for input into a deep learning network for detecting coded markers (hereinafter referred to as a "deep learning network"), which will be described later. To this end, the received images may be cropped, resized, normalized and / or standardized.

[0056] First, the size of the received image is adjusted. During the training process of the deep learning network, the size of the input image is determined, and the size of the received image is adjusted to the determined size. For example, when the size of the input image is determined to be 640×448, the size of the received image may be adjusted to a multiple of 640×448 to minimize distortion. In this case, the size adjustment may be performed after cropping the lower portion of the received image corresponding to the trim and bottom surface of the vehicle and adjusting the magnification. Figure 6 , showing a received image having a size of 1280×944 and an image resized to a size of 640×448.

[0057] Normalization and standardization may be performed on the resized image. Normalization is a process of converting each pixel value of the image into a value between 0 and 1, and may be performed using Equation 1, but is not limited thereto. Standardization is a process of converting the pixel values ​​of the image into a standard distribution having a mean of 0 and a standard deviation of 1, and may be performed using Equation 2.

[0058] [Equation 1]

[0059]

[0060] In equation 1, x 归-化 represents the normalized pixel value, x represents the unnormalized pixel value, and x 最小值 Represents the minimum value of the pixel value, x 最大值 Indicates the maximum value among the pixel values.

[0061] [Equation 2]

[0062]

[0063] In equation 2, x 标准化 is the normalized pixel value, x is the unnormalized pixel value, is the average pixel value, and σ i is the standard deviation of the pixel values.

[0064] In this case, the mean and standard deviation are determined from the training data during the training of the deep learning network.

[0065] The marker recognition unit 100 inputs the preprocessed image into a deep learning network to extract a center point heat map of the encoded marker, a first feature map for the width and height of the bounding box, a second feature map for adjusting the offset of the position of the center point, and a third feature map for the corner points of the encoded marker.

[0066] A deep learning network is trained and used to detect coded tags via multi-task learning. Figure 7 ,The deep learning network can be composed of a backbone network and four head networks.

[0067] The marker recognition unit 100 extracts a feature map of the preprocessed image through the backbone network 710 (S520). The backbone network 710 includes a CNN (convolutional neural network), which includes several convolutional layers, activation functions, skip connections, etc.

[0068] For example, the backbone network of the present disclosure can be configured by combining the upsampling technology in DLA (deep layer aggregation) with MobileNetV2, but the present disclosure is not limited thereto. MobileNet V2 is a lightweight network characterized by the use of an inverse residual block structure. The inverse residual block has a structure opposite to that of the residual block of the prior art, and has the effect of reducing the amount of computation and memory usage. DLA is a technique for improving the performance of a deep learning model through iterative and hierarchical deep aggregation, which iteratively and hierarchically merges feature layers.

[0069] The marker recognition unit 100 receives the feature map of the image using the plurality of head networks 720, 730, 740, and 750, and extracts a heat map of the center point of the coding marker, a first feature map, a second feature map, and a third feature map for the corner points of the coding marker (S530). Figure 7 As shown in , the center point heat map, the first feature map, the second feature map, and the third feature map for encoding the corner points of the markers have the same width and height, but are different in the number of channels. Each head network can be implemented using the head structure of a keypoint-based object detection model (e.g., CenterNet).

[0070] The center point heat map head 720 is a head network trained to output a center point heat map of a category belonging to a coding mark. Here, the center point refers to the center point of the bounding box used to detect the coding mark. The center point heat map has one channel.

[0071] The dimension head 730 is a head network trained to output a first feature map of the width and height of the bounding box corresponding to the center point. The first feature map has two channels, one for each of the width and the height.

[0072] The offset head 740 is a head network trained to output a second feature map for adjusting the offset of the center point according to the scale of the input image. The second feature map has two channels, one for the x-coordinate offset and one for the y-coordinate offset.

[0073] The corner head 750 is a head network trained to output a third feature map encoding the relative positions of the marked corner points relative to the center point. The third feature map has eight channels, one for each of the x-coordinates and the y-coordinates of the four corner points.

[0074] The landmark recognition unit 100 generates candidate detection information including a bounding box and corner points based on the center point heat map, the first feature map, the second feature map, and the third feature map (S540).

[0075] First, a preset number of coordinates and confidence scores are extracted from the center point heat map in the order of the highest confidence score among the coordinates with peak pixel values. Here, the coordinates with peak pixel values ​​refer to coordinates with pixel values ​​equal to or greater than the pixel values ​​of the eight adjacent coordinates. That is, since the center point heat map is a feature map composed of confidence scores for the center point of the encoded mark, the top K x-coordinates and y-coordinates with high confidence scores among all coordinates with maximum pixel values ​​within the surrounding 3×3 window are selected, and the confidence scores are output. Here, K is a preset number (e.g., 100). Reference Figure 8 Describe in detail the process of extracting K center points from the center point heat map.

[0076] A 3×3 window sliding is performed on the center point heat map ( S810 ).

[0077] It is determined whether the pixel value located at the center of the 3×3 window is the maximum within the 3×3 window (S820).

[0078] When the pixel value is the maximum value, it is determined whether the pixel value is within the top K in descending order of confidence scores (S830), and when the pixel value is not the maximum value, the process proceeds to process S850.

[0079] When the pixel value is within the top K, the corresponding pixel is collected. That is, the pixel is included in the top K center points (S840). When the pixel value is not within the top K, the process proceeds to process S850.

[0080] It is determined whether the window is the last window (S850), and when the last window has been reached, the coordinates and confidence scores of the first K collected center points are output.

[0081] The first feature map, the second feature map, and the third feature map are analyzed based on the coordinates of the first K center points extracted from the center point heat map. Fig. 9 Describes the process of extracting the width and height of the bounding box, the offset, and the corner point information of the encoded marker based on the extracted center point information.

[0082] Since the first feature map represents the width and height values ​​of a bounding box with each pixel as a center point, the width values ​​and height values ​​corresponding to the coordinates of the first K center points extracted from the center point heat map are extracted and matched with the corresponding first K center points.

[0083] Since the second feature map represents the offset value of the x coordinate and the offset value of the y coordinate of the center point used to encode the mark, the offset value of the x coordinate and the offset value of the y coordinate corresponding to the coordinates of the first K center points extracted from the center point heat map are extracted and matched with the corresponding first K center points.

[0084] Since the third feature map represents the relative x-coordinate values ​​and y-coordinate values ​​of the corner points relative to the center point of the encoded mark, the relative x-coordinate values ​​and y-coordinate values ​​of the corner points corresponding to the first K center point coordinates extracted from the center point heat map are extracted and matched with the corresponding first K center points.

[0085] At the same time, since the coordinate values ​​of the first K center points, the width / height values ​​of the corresponding bounding box, the corresponding offset values, and the relative coordinate values ​​of the corner points are values ​​of the size of the feature map of the reference input image, the values ​​must be multiplied by the scaling factor s. Here, the scaling factor is the ratio between the size of the input image and the size of the feature map extracted by the backbone network. In other words, the scaling factor can be used to obtain the position information of the bounding box and corner points in the input image.

[0086] The landmark identification unit 100 outputs final detection information by performing modified non-maximum suppression on the candidate detection information ( S550 ).

[0087] Fig.10 The process of performing modified non-maximum suppression on candidate detection information according to an embodiment of the present disclosure is shown. Each IoU (Intersection over Union) between the bounding boxes included in the first K candidate detection information is calculated. For bounding boxes whose IoU is equal to or greater than the threshold, the average coordinate value of the center point, the average width value, the average height value, the average confidence score value, and the average coordinate value of the corner points are calculated to obtain the final detection information. That is, the method of performing modified non-maximum suppression according to an embodiment of the present disclosure does not remove non-maximum bounding boxes, but uses the average value of bounding boxes whose IoU is equal to or greater than the threshold to create a single detection result.

[0088] First, the top K candidate detection information can be rearranged in order of high confidence scores.

[0089] The i-th bounding box is matched with each of the (i+1)-th to K-th bounding boxes ( S1010 ), and the IoU is calculated ( S1020 ).

[0090] It is determined whether the IoU is equal to or greater than a threshold value (S1030).

[0091] When the IoU is equal to or greater than the threshold, the jth bounding box is collected (S1040). This is to determine that the i-th bounding box and the collected j-th bounding box correspond to the same encoding tag, remove duplication and output a single detection result.

[0092] When the IoU is less than the threshold, it is determined whether the above process has been performed until the K-th bounding box (S1050). When the above process has not been performed until the K-th bounding box, the process proceeds to process S1010, and when the above process is performed until the K-th bounding box, the average value between the i-th bounding box and the collected bounding box is calculated to obtain a single detection result (S1060), and the i-th bounding box and the collected bounding box are removed from the candidate detection information.

[0093] The processes S1010 to S1060 are repeatedly performed until there is no bounding box included in the candidate detection information. This makes it possible to obtain the final coded marker detection result.

[0094] Each component of the device or method according to the embodiment of the present disclosure can be implemented in hardware, software or a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can be implemented to perform the software function corresponding to each component.

[0095] Different implementations of the systems and techniques described herein may include digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These different implementations may include implementations with one or more computer programs that can be executed on a programmable system. A programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) combined to receive data and instructions from a storage system and to send data and instructions to the storage system, at least one input device, and at least one output device. A computer program (also referred to as a program, software, software application, or code) includes instructions for a programmable processor and is stored in a "computer-readable storage medium."

[0096] Computer-readable storage media include all kinds of storage devices that store data that can be read by a computer system. Computer-readable storage media may include non-volatile or non-temporary media (such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, and storage device), and further include temporary media (such as data transmission media). In addition, computer-readable storage media can be distributed in computer systems connected via a network, and computer-readable codes can be stored and executed in a distributed manner.

[0097] In the flowcharts in this specification, each process is described to occur sequentially, but this is only an example of the technology of the embodiments of the present disclosure. In other words, a person skilled in the art of the embodiments of the present disclosure can make various modifications and changes by changing the order described in the flowcharts in this specification or by going through one or more processes in parallel within the basic features of the embodiments of the present disclosure, so the flowcharts in this specification are not limited to the time series order.

[0098] Although exemplary embodiments of the present disclosure have been described for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible without departing from the concept and scope of the claimed invention. Therefore, for the sake of brevity and clarity, exemplary embodiments of the present disclosure have been described. The scope of the technical concept of the embodiments of the present disclosure is not limited by the diagrams. Therefore, those of ordinary skill will understand that the scope of the claimed invention is not limited by the embodiments explicitly described above, but by the claims and their equivalents.

Claims

1. A method for detecting a coding mark, the method comprising: receiving an image and preprocessing the image; Extracting a feature map of the image through a backbone network; Extracting a center point heat map of the encoded marker, a first feature map for a width and a height of a bounding box, a second feature map for an offset for adjusting the center point, and a third feature map for corner points of the encoded marker from the feature map using a plurality of head networks; Generate candidate detection information including the bounding box and the corner point based on the center point heat map, the first feature map, the second feature map, and the third feature map; as well as Final detection information is output by performing modified non-maximum suppression on the candidate detection information.

2. The method according to claim 1, wherein: Preprocessing the image includes: performing at least one of cropping, resizing, normalizing, and standardizing on the image.

3. The method according to claim 1, wherein: The backbone network is a convolutional neural network using an inverse residual block structure and deep aggregation, and the backbone network is trained to extract feature maps of input images.

4. The method according to claim 1, wherein: The multiple head networks include: a center point heat map head, which is trained to output the center point heat map of the category corresponding to the encoded label from the feature map; a dimension head, which is trained to output the first feature map for the width and height of the bounding box corresponding to the center point; an offset head, which is trained to output the second feature map for the offset used to adjust the center point; and a corner point head, which is trained to output the third feature map for the relative position of each of the corner points relative to the center point.

5. The method according to claim 1, wherein: The multiple head networks are trained through multi-task learning.

6. The method according to claim 1, wherein: Generating the candidate detection information includes: extracting coordinates and confidence scores of a preset number of center points in order of highest confidence scores among coordinates having peak pixel values ​​from the center point heat map; Extracting width and height values ​​of bounding boxes corresponding to the preset number of center points from the first feature map; Extracting offset values ​​corresponding to the preset number of center points from the second feature map; Extracting the coordinates of corner points corresponding to the preset number of center points from the third feature map; and Bounding box information is generated using the center point coordinates, the width value, the height value, the offset value, and the scaling factor, and corner point information is generated using the center point coordinates, the corner point coordinates, and the scaling factor.

7. The method according to claim 6, wherein: A coordinate having a peak pixel value is a coordinate having a pixel value equal to or greater than the pixel values ​​of eight adjacent coordinates.

8. The method according to claim 6, wherein: The scaling factor is the ratio between the size of the image and the size of the feature map.

9. The method according to claim 1, wherein: Outputting the final detection information includes: Calculating each intersection-over-union ratio between bounding boxes included in the candidate detection information; For bounding boxes whose intersection-over-augment ratio is equal to or greater than the threshold, calculate the average of center point coordinates, the average of width values, the average of height values, the average of confidence score values, and the average of corner point coordinates; and The average value of the center point coordinates, the average value of the width values, the average value of the height values, the average value of the confidence score values, and the average value of the corner point coordinates are output as the final detection information.

10. A device comprising one or more processors and a memory, the memory being operably connected to the one or more processors, wherein: The memory stores instructions that cause the one or more processors to perform operations in response to execution of the instructions by the one or more processors, and the operations include: receiving an image and preprocessing the image; Extracting a feature map of the image through a backbone network; Extracting a center point heat map of the encoded marker, a first feature map for a width and a height of a bounding box, a second feature map for an offset for adjusting the center point, and a third feature map for corner points of the encoded marker from the feature map using a plurality of head networks; Generate candidate detection information including the bounding box and the corner point based on the center point heat map, the first feature map, the second feature map, and the third feature map; and Final detection information is output by performing modified non-maximum suppression on the candidate detection information.

Images