Method and device for detecting object

Abstract

A device according to one aspect of the present invention comprises at least one memory and at least one processor. The at least one processor: generates integrated information in which image information and coordinate information about an object are integrated by matching the image information and the coordinate information about the object; performs either density-based clustering or distance-based clustering on the basis of the integrated information; and detects the object by using a center point of a representative cluster determined on the basis of the clustered result.

Description

Method and device for detecting objects

[0001] The present disclosure relates to a method and device for detecting an object. More specifically, the present disclosure relates to a method and device for detecting a dynamic object.

[0002] Recently, with the advancement of autonomous ship navigation technology, the development of autonomous ships is continuously progressing. Autonomous ship navigation technology enables ships to set their own routes and navigate without human intervention. To achieve this, technology capable of accurately recognizing and analyzing the surrounding environment is essential.

[0003] In particular, for autonomous ships to navigate safely, technology that detects various objects, such as obstacles and ships, around the ship in real time and precisely analyzes them to prevent collisions is important.

[0004] However, there was a problem that it was difficult to efficiently and accurately detect various objects around the ship.

[0005] The present invention provides a method and device for detecting an object. Furthermore, the present invention provides a computer-readable recording medium containing a program for executing the method on a computer. The technical problems to be solved are not limited to the technical problems described above, and other technical problems may exist.

[0006] According to one aspect of the present disclosure, a method for detecting a dynamic object can be provided, including: performing clustering on point data of an object using image information and coordinate information for the same object; determining a representative cluster among a plurality of clusters based on a result of performing the clustering; and detecting the object by extracting a center point of the determined representative cluster.

[0007] According to another aspect of the present disclosure, a device includes a memory storing at least one program; and at least one processor executing the at least one program, wherein the at least one processor generates integrated information in which the image information and the coordinate information for the object are integrated by corresponding image information and coordinate information for the object, performs either density-based clustering or distance-based clustering based on the integrated information, and detects the object using the center point of a representative cluster determined based on a result of performing the clustering.

[0008] A computer-readable recording medium according to another aspect of the present disclosure includes a recording medium having recorded thereon a program for executing the above-described method on a computer.

[0009] By utilizing data acquired from multiple sensors, various objects around the ship can be detected more efficiently and precisely.

[0010] Additionally, it can reduce maritime accidents by detecting various objects in real time and preventing collisions.

[0011] Additionally, accurate object detection can help plan optimal routes and reduce operating costs by minimizing fuel consumption.

[0012] However, the effects of the embodiments are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art from the description of the present invention.

[0013] FIG. 1 is a drawing for explaining an example of a method for detecting an object according to one embodiment.

[0014] FIG. 2 is a schematic diagram illustrating an example of a device for detecting an object according to one embodiment.

[0015] FIG. 3 is a flowchart illustrating an example of a method for detecting a static object according to one embodiment.

[0016] FIG. 4 is a drawing for explaining an example of a method for obtaining point data of an object according to one embodiment.

[0017] FIG. 5 is a diagram illustrating an example of a method for performing filtering of point data of an object according to one embodiment.

[0018] FIG. 6 is a diagram illustrating an example of a method for determining candidate point data according to one embodiment.

[0019] FIG. 7 is a diagram illustrating an example of a method for detecting an object by determining candidate point data as representative point data according to one embodiment.

[0020] FIG. 8 is a flowchart illustrating an example of a method for detecting a dynamic object according to one embodiment.

[0021] FIG. 9 is a drawing for explaining an example of a method for obtaining image information of an object according to one embodiment.

[0022] FIG. 10 is a drawing for explaining an example of a method for obtaining coordinate information of an object according to one embodiment.

[0023] FIG. 11 is a drawing for explaining an example of a method for generating integrated information by corresponding image information and coordinate information of an object according to one embodiment.

[0024] FIG. 12 is a diagram illustrating an example of a method for performing clustering according to one embodiment.

[0025] FIG. 13 is a diagram illustrating an example of a method for detecting an object using a representative cluster according to one embodiment.

[0026] A device according to one aspect comprises at least one memory; and at least one processor; wherein the at least one processor generates integrated information in which image information and coordinate information for an object are associated with each other, and performs either density-based clustering or distance-based clustering based on the integrated information, and detects the object using the center point of a representative cluster determined based on a result of performing the clustering.

[0027] The terms used in the examples are selected from widely used, current terms, as much as possible. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in the specification should be defined based on their intended meaning and the overall content of the specification, rather than simply their names.

[0028] When a part of the specification is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "unit" and "module" used throughout the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0029] Additionally, terms including ordinal numbers, such as "first" or "second," used in the specification may be used to describe various components, but the components should not be limited by the terms. The terms may be used to distinguish one component from another.

[0030] The present disclosure will be described in detail with reference to the attached drawings. However, the embodiments may be implemented in various different forms and are not limited to the examples described herein.

[0031] FIG. 1 is a drawing for explaining an example of a method for detecting an object according to one embodiment.

[0032] Referring to Fig. 1, the charity (1) may encounter various objects (10) during operation.

[0033] For example, charity (1) may be manually operated or autonomously operated, and object (10) may be any object (10) without limitation in type, such as a bridge, a large ship, a small ship, a fishing boat, a reef, a buoy, a person, a jet ski, a yacht, etc.

[0034] Charity (1) may encounter various objects (10) during operation, and in this case, charity (1) may extract information about the object (10) to prevent collision with the object (10).

[0035] For example, charity (1) can use various sensors to detect an object (10) and extract information about the object (10).

[0036] For example, charity (1) can detect an object (10) and extract information about the object (10) by acquiring an image of the object (10) using a sensor that acquires an image.

[0037] In addition, charity (1) can detect an object (10) and extract information about the object (10) by obtaining location information, speed information, etc. of the object (10) using a sensor that obtains information about the object (10).

[0038] In addition, charity (1) can detect an object (10) and extract information on the object (10) by obtaining location information, track information, etc. of the object (10) using an automatic identification system (AIS).

[0039] For example, charity (1) can detect an object (10) and extract information about the object (10) by integrating information related to the object (10) obtained using various sensors.

[0040] In this case, charity (1) can detect the object (10) and extract information of the object (10) in different ways depending on whether the object (10) is a dynamic object or a static object.

[0041]

[0042] FIG. 2 is a schematic diagram illustrating an example of an object detection device according to one embodiment. Here, the object detection device according to one embodiment may include a device for detecting static objects and a device for detecting dynamic objects.

[0043] Referring to FIG. 2, an object detection device (hereinafter referred to as "device") (200) may include a communication unit (210), a processor (220), a memory (230), a display unit (240), and a control unit (not shown). Only components related to the embodiment are shown in the device (200) of FIG. 2. Therefore, it is obvious to a person skilled in the art that other general components may be included in addition to the components shown in FIG. 2.

[0044] The communication unit (210) may include one or more components that enable wired / wireless communication with an external server or external device. For example, the communication unit (210) may include a short-range communication unit (not shown) and a mobile communication unit (not shown) for communication with an external server or external device.

[0045] The processor (220) controls the overall operation of the device (200). For example, the processor (220) can control the input unit (not shown), the display (not shown), the communication unit (210), the memory (230), etc., by executing programs stored in the memory (230).

[0046] The processor (220) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0047] The processor (220) can control the operation of the device (200) by executing programs stored in the memory (230). For example, the processor (220) can perform at least a part of the method for detecting a static object described with reference to FIGS. 3 to 7 and at least a part of the method for detecting a dynamic object described with reference to FIGS. 8 to 13.

[0048] The memory (230) is hardware that stores various data processed within the device (200), and can store a program for processing and controlling the processor (220).

[0049] For example, the memory (230) may store various data such as LiDAR data, image data, ship specification information, ship navigation information, ship navigation image information, weather forecast information, maritime information, ship control information, object information, and data generated according to the operation of the processor (220). In addition, the memory (230) may store an operating system (OS) and at least one program (e.g., a program required for the processor (220) to operate).

[0050] The memory (230) may include random access memory (RAM) such as dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

[0051] The display unit (240) may be hardware that displays data generated according to the operation of the processor (220) or the control unit (not shown).

[0052] For example, the display unit (240) may include a smartphone, tablet PC, PC, smart TV, media player, navigation, kiosk, etc. In addition, the display unit (240) may provide a monitoring image or interface to the user. Here, the monitoring image may be an image acquired by a predetermined image acquisition device, a previously stored image, a real-time image, an electronic chart, a map, a route guidance image, etc.

[0053] The control unit (not shown) may have the same configuration as the processor (220), but may also have a separate configuration.

[0054] As an example, the control unit (not shown) can perform the same operation as the processor (220). For example, the control unit (not shown) can control the operation of the device (200) by executing programs stored in the memory (230). For example, the control unit (not shown) can perform at least a part of the method for detecting a static object described with reference to FIGS. 3 to 7 and at least a part of the method for detecting a dynamic object described with reference to FIGS. 8 to 13.

[0055] As another example, the control unit (not shown) can control the vessel using information about detected objects. For example, the control unit (not shown) can detect various objects in real time to prevent collisions, thereby reducing maritime accidents. Furthermore, the control unit (not shown) can plan an optimal route through accurate object detection and reduce operating costs by minimizing fuel consumption.

[0056]

[0057] FIG. 3 is a flowchart illustrating an example of a method for detecting a static object according to one embodiment.

[0058] Referring to FIG. 3, a method for detecting a static object may include operations 310 to 330. However, the present invention is not limited thereto, and other general operations may be further included in the method for detecting a static object in addition to the operations illustrated in FIG. 3. Furthermore, as described above with reference to FIGS. 1 and 2, at least one of the operations in the flowchart illustrated in FIG. 3 may be processed by a processor.

[0059] In operation 310, the processor can generate a first region including at least one point data of the object.

[0060] For example, the processor can obtain point data within a range set based on information about the charity.

[0061] For example, the processor may filter point data. For example, the processor may filter water surface reflection data included in the point data. As another example, the processor may filter point data based on the specific information of the charity.

[0062] For example, the processor may generate a first region based on the location information of the charity. More specifically, the processor may generate the first region based on whether the charity is located in the ocean or on the coast.

[0063] In operation 320, the processor can extract candidate point data from among point data included in the first area based on the distance from the charity.

[0064] For example, the processor may extract candidate point data that is close to the charity among the point data included in the first area, and determine candidate point data that satisfies a preset criterion among the candidate point data as representative point data.

[0065] In operation 330, the processor can detect the object by determining representative point data that satisfies a preset criterion among candidate point data.

[0066] For example, the processor may extract candidate point data that is close to the charity among the point data included in the first area, and determine candidate point data that satisfies a preset criterion among the candidate point data as representative point data.

[0067] More specifically, the processor may determine a second area including candidate point data, calculate the number of point data included in the second area, and determine the candidate point data as representative point data based on a result of determining whether the number of calculated point data exceeds a reference value.

[0068] As an example, the processor may determine candidate point data as representative point data based on a result of determining that the number of produced point data exceeds a reference value.

[0069] As another example, the processor may determine candidate point data as noise based on the determination that the number of generated point data does not exceed a reference value. Furthermore, the processor may extract new candidate point data based on the determination that the number of generated point data does not exceed a reference value.

[0070] As an additional example, the processor may determine that data among candidate points assigned a class is a dynamic object. Furthermore, the processor may delete candidate points determined to be dynamic objects, thereby not using them as data for detecting static objects.

[0071]

[0072] Hereinafter, with reference to FIGS. 4 to 7, the method for detecting static objects described above with reference to FIG. 3 will be described in more detail.

[0073] FIG. 4 is a drawing for explaining an example of a method for obtaining point data of an object according to one embodiment.

[0074] Hereinafter, with reference to FIG. 4, an example of a method for a processor to obtain point data of an object is described.

[0075] Referring to FIG. 4, the processor can obtain point data of an object existing within a certain range around the charity (4).

[0076] For example, the processor can acquire point data within a range set based on information of the charity (4). Here, the information of the charity (4) may include, but is not limited to, size information of the charity (4), specification information of the charity (4), speed information of the charity (4), and navigation route information of the charity (4). In addition, the range set based on the information of the charity (4) is a range for controlling a sensor that acquires point data, and point data can be acquired within the set range.

[0077] For example, the processor can set a wider range as the size of the charity (4) increases, and a narrower range as the size of the charity (4) decreases. In addition, the processor can set a wider range as the speed of the charity (4) increases, and a narrower range as the speed of the charity (4) decreases.

[0078] Meanwhile, the range for acquiring point data can be set manually according to input from the user or navigator, and can be set to a range of 120° in front of the ship (4) or 360° around the ship (4).

[0079] Therefore, the processor can acquire point data within a set range.

[0080] Meanwhile, the processor can acquire point data using LiDAR (Light Detection And Ranging). Here, LiDAR (Light Detection And Ranging) is a sensor that detects objects by irradiating a laser, hitting the object, and measuring the reflected laser, and acquires information about the detected object. Information about the object can be acquired in the form of point cloud data.

[0081] For example, the processor may create a first area for acquiring point data within a set range. Here, the first area is an area for acquiring point data, and at least one first area may be included in a range set based on the information of the charity (4).

[0082] For example, the processor can generate the first area based on the location information of the charity (4).

[0083] For example, the processor may generate a first region based on whether the location of the charity (4) is oceanic or coastal.

[0084] For example, if the location of the charity (4) is in the ocean, the probability of detecting an object is lower than in the coastal area, and thus the processor may generate fewer first areas than in the coastal area. As another example, if the location of the charity (4) is in the coastal area, the probability of detecting an object is higher than in the ocean, and thus the processor may generate more first areas than in the ocean.

[0085] Meanwhile, the first area can also be generated based on user or navigator input. That is, the user or navigator can determine the number of first areas for acquiring point data and create as many first areas as the determined number of first areas.

[0086] Accordingly, the processor can detect an object using the point data included in the generated first area.

[0087] Meanwhile, the processor may perform filtering of the point data before using the point data included in the first area.

[0088]

[0089] FIG. 5 is a diagram illustrating an example of a method for performing filtering of point data of an object according to one embodiment.

[0090] Hereinafter, with reference to FIG. 5, an example of a method in which a processor performs filtering of point data of an object is described.

[0091] Referring to FIG. 5, the processor can perform filtering of point data.

[0092] For example, the processor can filter the water reflection data included in the point data and perform filtering of the point data based on the specification information of the charity (5).

[0093] First, the processor can perform filtering of the water surface reflection data included in the point data.

[0094] For example, since the water reflection data included in the point data is noise and not valid data, the processor can remove the water reflection data by filtering the water reflection data included in the point data.

[0095] As an example, the processor may perform filtering of surface reflection data based on reflection intensity.

[0096] For example, since water surface reflection data typically has low reflection intensity, the processor can filter the water surface reflection data included in the point data by determining whether the reflection intensity of the point data is below a preset threshold. In other words, if the reflection intensity of the point data is below a preset threshold, the processor can determine the point data as water surface reflection data and remove the determined water surface reflection data.

[0097] As another example, the processor may perform filtering of surface reflection data based on reflectivity.

[0098] For example, since water surface reflection data typically has a low reflectivity, the processor can filter the water surface reflection data included in the point data by determining whether the reflectivity of the point data is below a preset threshold. In other words, if the reflectivity of the point data is below the preset threshold, the processor can determine the point data as water surface reflection data and remove the determined water surface reflection data.

[0099] For example, the processor can perform filtering of point data based on the specification information of the charity (5).

[0100] For example, the processor can perform filtering of point data using height (50) information of charity included in the specification information of charity (5).

[0101] For example, the processor may filter point data measured at a location higher than the height of the charity (50) among the point data. In other words, the processor may classify point data acquired from a location that is the height of the charity (50) from the water surface as noise and remove it.

[0102] For example, point data such as bridges, birds, and airplanes located higher than the height of the charity (50) have a low possibility of collision with the charity (5), and thus may not correspond to objects to be detected to prevent collision with the charity (5).

[0103] Therefore, the processor can accurately detect an object by filtering point data acquired from a location higher than the height of the charity (50) and extracting only valid point data.

[0104] For example, a processor may be included in a first region to detect objects using filtered point data.

[0105] Specifically, the processor determines representative point data based on the point data included in the first area and the distance from the charity (5), and can detect an object using the representative point data.

[0106] For example, the processor can determine candidate point data that is close to the charity (5) among the point data included in the first area.

[0107]

[0108] FIG. 6 is a diagram illustrating an example of a method for determining candidate point data according to one embodiment.

[0109] Hereinafter, with reference to FIG. 6, an example of a method by which a processor determines candidate point data is described.

[0110] Referring to FIG. 6, the processor can determine candidate point data (61a, 61b, 62, 63) from among point data included in each of a plurality of first areas (601, 602, 603).

[0111] For example, the processor can extract candidate point data based on the point data included in the first area (601, 602, 603) and the distance to the charity (6).

[0112] For example, the processor can extract point data closest to charity (6) for each first area (601, 602, 603) and determine the point data closest to charity (6) extracted as candidate point data (61a, 61b, 62, 63).

[0113] For example, the processor may extract at least one point data closest to the charity (6) among the point data included in each of the first regions (601, 602, 603) for each of the first regions (601, 602, 603). Accordingly, the processor may determine the point data closest to the charity (6) extracted for each of the first regions (601, 602, 603) as candidate point data (61a, 61b, 62, 63).

[0114] Meanwhile, if there are multiple candidate point data (61a, 61b) extracted from a first area (601), the processor can assign a rank to each of the multiple candidate point data (61a, 61b).

[0115] For example, the processor may give priority to candidate point data (61a) that is closer to the charity (6) among multiple candidate point data (61a, 61b), and give priority to candidate point data (61b) that is further from the charity (6).

[0116] Therefore, the processor can detect an object using the first-priority candidate point data (61a) before the second-priority candidate point data (61b).

[0117] For example, the processor can determine candidate point data that satisfies a preset criterion among the extracted candidate point data (61a, 61b, 62, 63) as representative point data and use this to detect an object.

[0118]

[0119] FIG. 7 is a diagram illustrating an example of a method for detecting an object by determining candidate point data as representative point data according to one embodiment.

[0120] Hereinafter, with reference to FIG. 7, an example of a method for detecting an object by having a processor determine candidate point data as representative point data is described.

[0121] Referring to FIG. 7, the processor can determine candidate point data that satisfies a preset criterion among candidate point data as representative point data (71a, 72, 73).

[0122] For example, the processor may determine a second region (710a, 720, 730) containing candidate point data.

[0123] For example, the processor can determine the second region (710a, 720, 730) based on each candidate point data.

[0124] As an example, the processor can determine a second area (710a, 720, 730) of a preset width. In other words, the processor can determine a second area (710a, 720, 730) of the same width.

[0125] As another example, the processor can determine the second areas (710a, 720, 730) having an area set based on the distance between each candidate point data and the charity (7). In other words, the processor can determine the second areas (710a, 720, 730) having different areas. For example, the processor can determine the size of the second areas (710a, 720, 730) to be smaller for candidate point data that are closer to the charity (7), and to be larger for candidate point data that are farther from the charity (7). Therefore, even if the density of point data is low for candidate point data that are farther from the charity (7), the processor can determine the size of the second areas (710a, 720, 730) to be larger, thereby including a lot of point data, so as not to miss detection of objects that are far from the charity (7). That is, as described later, the object is detected using the candidate point data only when the number of all point data included in the second area (710a, 720, 730) exceeds the reference value, so that the processor can prevent a situation in which the object is not detected because the number of point data is small even though the distance between the charity (7) and the candidate point data is the closest.

[0126] Additionally, the processor can calculate the number of all point data included in the determined second area (710a, 720, 730) and determine whether the number of all calculated point data exceeds a reference value.

[0127] For example, the processor may determine candidate point data as representative point data (71a, 72, 73) based on the result of determining whether the number of all point data included in the second area (710a, 720, 730) exceeds a reference value.

[0128] As an example, the processor may determine candidate point data as representative point data (71a, 72, 73) based on a result of determining that the number of produced point data exceeds a reference value.

[0129] As another example, the processor may determine candidate point data as noise based on the result that the number of generated point data does not exceed a reference value. Furthermore, if the candidate point is determined to be noise, the processor may extract new candidate point data using the method described above with reference to FIG. 6.

[0130] Meanwhile, if there are multiple candidate point data included in the first area, and the candidate point data with the highest priority among them is not determined as the representative point data, i.e., if the candidate point data with the highest priority is determined as noise, the processor may determine the second area based on the candidate point data with the lowest priority, and determine the representative point data by determining whether the number of all point data included in the determined second area exceeds a reference value.

[0131] As an additional example, the processor may determine that data among candidate point data that has been assigned a class is a dynamic object. Here, the class may refer to the classification level of the object, and may be set based on the object's type information or object's status information.

[0132] For example, based on the object type information or the object status information, the class of the candidate point data can be assigned as a fishing state, a restricted state, a rapidly moving state, an object with lost control, an object at anchor, etc.

[0133] Therefore, the processor can determine that the candidate point data to which a class is assigned is a dynamic object, and thus the candidate point data is a dynamic object.

[0134] For example, the processor can detect an object and obtain information about the object using the determined representative point data (71a, 72, 73).

[0135] For example, the processor can detect a static object by determining that the representative point data (71a, 72, 73) is an actual static object and not noise.

[0136] Accordingly, the processor can provide a user with a monitoring image displaying information about a detected static object. Here, the monitoring image may refer to a display unit.

[0137] Additionally, the processor can generate a grid centered on the charity (7) using the determined representative point data (71a, 72, 73) and generate a 3D cluster.

[0138] Meanwhile, the processor may also detect dynamic objects using data acquired from LiDAR, as described above with reference to FIG. 4.

[0139]

[0140] FIG. 8 is a flowchart illustrating an example of a method for detecting a dynamic object according to one embodiment.

[0141] Hereinafter, with reference to FIG. 8, an example of a method for a processor to detect a dynamic object is described.

[0142] Referring to FIG. 8, in step 810, the processor may generate integrated information in which the image information and coordinate information for the object are integrated by corresponding the image information and coordinate information for the object. Here, the image information for the object may include outline information of the object.

[0143] For example, the processor can use an image acquisition sensor to acquire an image of an object and extract contour information contained in the image. Furthermore, the processor can use a coordinate information acquisition sensor to acquire three-dimensional coordinate information of the object and convert the three-dimensional coordinate information into two-dimensional coordinate information.

[0144] For example, the processor can generate integrated information about an object by matching contour information, which is image information about the object, with two-dimensional coordinate information, which is coordinate information about the object.

[0145] For example, the processor can determine the order in which to generate integrated information based on the position of one side of a bounding box containing an object, and generate integrated information by corresponding two-dimensional coordinate information of the bounding box and the object according to the determined order.

[0146] At step 820, the processor may perform either density-based clustering or distance-based clustering based on the integrated information.

[0147] For example, the processor may perform density-based clustering or distance-based clustering based on the location of the charity.

[0148] For example, the processor can perform clustering on point data included in the integrated information to obtain multiple clusters.

[0149] In step 830, the processor can detect an object using the center point of a representative cluster determined based on the result of performing clustering.

[0150] For example, a processor can determine a representative cluster among multiple clusters.

[0151] For example, the processor may extract the number of expected point data based on the resolution of the device acquiring the point data, determine the number of reference point data using the number of expected point data, and determine a representative cluster based on the determined number of reference point data. Here, the processor may weight the number of expected point data based on the angle between the self-reference and the object, and determine the number of reference point data based on the weight.

[0152] Additionally, the processor may obtain point data from a cluster close to the charity among multiple clusters, and determine a representative cluster based on whether the obtained point data exceeds the number of reference point data.

[0153] For example, the processor may perform clustering on point data included in the integrated information based on the location of the charity, obtain a plurality of clusters as a result of performing the clustering, and determine a representative cluster among the plurality of clusters.

[0154] For example, if the ship is located offshore, the processor may perform clustering based on the density of point data. If the ship is located in the ocean, the processor may perform clustering based on the distance between point data. As another example, if the ship is located in the ocean, the processor may perform reclustering on representative clusters to determine representative points, and use the representative points to detect objects.

[0155] For example, the processor can extract the center point of a representative cluster and estimate the center point as the object's location. Furthermore, the processor can estimate the object's type, direction, and speed based on the image information and coordinate information contained in the representative cluster.

[0156]

[0157] Hereinafter, with reference to FIGS. 9 to 13, the method for detecting the dynamic object described above with reference to FIG. 8 will be described in more detail.

[0158] FIG. 9 is a drawing for explaining an example of a method for obtaining image information of an object according to one embodiment, FIG. 10 is a drawing for explaining an example of a method for obtaining coordinate information of an object according to one embodiment, and FIG. 11 is a drawing for explaining an example of a method for generating integrated information by corresponding image information and coordinate information of an object according to one embodiment.

[0159] Hereinafter, with reference to FIGS. 9 to 11, an example of a method in which a processor generates integrated information by matching image information and coordinate information for an object will be described.

[0160] Referring to FIG. 9, the processor can acquire image information of an object using an image acquisition sensor. Here, the image acquisition sensor can be a camera, specifically, at least one of a PTZ camera, an EO camera, and an IR camera.

[0161] For example, the processor can acquire an image of an object existing in the direction of movement of the charity (9). Here, the object can include all objects such as large ships, small ships, fishing boats, jet skis, buoys, reefs, people, bridges, facilities, etc.

[0162] For example, the processor can use an image acquisition sensor to acquire an image of an object and extract image information of the object.

[0163] For example, the processor can detect objects contained in continuously acquired images and extract contour information of the detected objects.

[0164] As an example, the processor can extract contour information using the edge lines of an object.

[0165] For example, the processor may extract contour information by distinguishing objects and their outlines with different colors. That is, the processor may extract contour information by distinguishing objects in red and their outlines in green, but is not limited thereto.

[0166] As another example, the processor can extract contour information by generating a bounding box (910, 920, 930) containing the object.

[0167] In the following, for the convenience of explanation, the description focuses on using the bounding box (910, 920, 930) as the outline information of the object. However, it is obvious to those skilled in the art that the same method as using the bounding box (910, 920, 930) can be applied to cases where the edge line of the object is used.

[0168] Accordingly, the processor can extract contour information of an object by detecting an object included in an image and generating a bounding box (910, 920, 930) containing the detected object.

[0169]

[0170] Referring to FIG. 10, the processor can obtain two-dimensional coordinate information (1010, 1020, 1030) of the object.

[0171] For example, a processor can acquire 3D coordinate information of an object using a coordinate information acquisition sensor. More specifically, the processor can acquire 3D coordinate information of an object using LiDAR. Here, LiDAR (Light Detection And Ranging) is a sensor that detects an object by irradiating a laser, hitting the object, and measuring the reflected laser, and acquires information about the detected object, and can acquire information about the object in the form of point cloud data.

[0172] Additionally, the processor can convert three-dimensional coordinate information into two-dimensional coordinate information (1010, 1020, 1030).

[0173] For example, the processor can use lidar to obtain point data coordinates of an object in a three-dimensional space, and convert the obtained point data coordinates of the object in the three-dimensional space into pixel coordinates on a two-dimensional image.

[0174] Accordingly, the processor can obtain the two-dimensional coordinate information (1010, 1020, 1030) of the object by converting the three-dimensional data point data coordinates into two-dimensional data pixel coordinates.

[0175]

[0176] Referring to FIG. 11, the processor can generate integrated information (1110, 1120, 1130) about an object.

[0177] For example, the processor can generate integrated information (1110, 1120, 1130) that integrates image information and coordinate information of an object by corresponding image information and coordinate information of the object. In other words, the processor can generate integrated information (1110, 1120, 1130) by corresponding bounding box (910, 920, 930), which is image information of the object, and two-dimensional coordinate information (1010, 1020, 1030), which is coordinate information.

[0178] For example, the processor may determine the order in which to generate the integrated information (1110, 1120, 1130) based on the position of one side (1111, 1121, 1131) of a bounding box (910, 920, 930) containing the object. Here, one side (1111, 1121, 1131) of the bounding box (910, 920, 930) may mean the lower side (or bottom side) of the bounding box (910, 920, 930), i.e., the side on which the object touches the sea surface.

[0179] For example, since the closer an object is to charity (9, 10, 11), the greater the risk of collision, the processor can generate integrated information (1110, 1120, 1130) starting from objects closer to charity (9, 10, 11). In this case, the processor can determine that an object is closer to charity (9, 10, 11) as the one whose side (1111, 1121, 1131) of the bounding box (910, 920, 930) is located at the bottom.

[0180] Accordingly, the processor can determine the order of generation of integrated information (1110, 1120, 1130), with the object located at the bottom of one side (1111, 1121, 1131) of the bounding box (910, 920, 930) having a higher priority, and the object located at the top having a lower priority.

[0181] For example, the processor can generate integrated information (1110, 1120, 1130) by matching bounding boxes (910, 920, 930) and two-dimensional coordinate information (1010, 1020, 1030) of objects in a determined order. That is, the processor can generate integrated information (1110, 1120, 1130) by matching bounding boxes (910, 920, 930) and two-dimensional coordinate information (1010, 1020, 1030) of objects from the highest priority.

[0182] For example, the processor can generate integrated information (1110, 1120, 1130) in which two-dimensional coordinate information (1010, 1020, 1030) is assigned to each bounding box (910, 920, 930) of an object. That is, the integrated information (1110, 1120, 1130) can include the bounding box (910, 920, 930) of the object and two-dimensional coordinate information (1010, 1020, 1030), and the bounding box (910, 920, 930) can include point data.

[0183] Meanwhile, the processor can filter out an object if the object is determined to be noise such as land, a pillar, or a building based on at least one of the object's image or coordinates.

[0184] For example, the processor can determine that an object is a static object based only on the object's image information or coordinate information, and can filter out objects determined to be static objects by classifying them as noise.

[0185] For example, the processor can perform clustering based on the generated integrated information (1110, 1120, 1130) and determine a representative cluster to detect dynamic objects.

[0186] For example, the processor may perform clustering based on the resolution of a device that acquires point data, i.e., a lidar, and determine a representative cluster to detect a dynamic object. As another example, the processor may perform clustering based on the location of the magnetometer and determine a representative cluster to detect a dynamic object.

[0187]

[0188] FIG. 12 is a diagram illustrating an example of a method for performing clustering according to one embodiment.

[0189] Hereinafter, with reference to FIG. 12, an example of a method by which a processor performs clustering is described.

[0190] Referring to FIG. 12, the processor can perform clustering on point data included within a bounding box (1210).

[0191] For example, the processor may perform density-based clustering or distance-based clustering on point data. As one example, the processor may perform density-based clustering, which defines a densely populated area of ​​point data within the bounding box (1210) as a single cluster. As another example, the processor may perform distance-based clustering, which defines an area of ​​point data within the bounding box (1210) with a similar distance between the point data and the target point as a single cluster. This improves clustering accuracy and reduces the computational load of the computing device.

[0192] For example, the processor can obtain multiple clusters (1211, 1212, 1213) by performing density-based clustering using coordinate information on point data included in a bounding box (1210) of integrated information.

[0193] For example, the processor may define an area as a cluster if the number of point data within that area is greater than a preset number.

[0194] Alternatively, the processor can obtain multiple clusters (1211, 1212, 1213) by performing clustering on point data included in the integrated information based on the location of the charity.

[0195] For example, the processor may determine either a density-based clustering method or a distance-based clustering method based on the location of the charity, and perform clustering on the point data using the determined clustering method.

[0196] First, the processor can determine the location of the charity.

[0197] For example, a processor can use GPS information to determine the location of a ship. For example, the processor can use map information or GPS information to determine whether the ship is located coastal or oceanic.

[0198] As another example, the processor can determine the location of the ship using the number of point data. For example, if the number of point data is greater than or equal to a preset number, the processor can determine the ship's location as coastal, and if the number of point data is less than the preset number, the processor can determine the ship's location as ocean.

[0199] As another example, the processor can determine the location of the ship using image information of the object. For example, the processor can convert the entire image containing the object into a depth image and determine the location of the ship using the converted depth image. Here, the depth image is an image that provides information about the distance from the camera to the object. If there is a discontinuity in the depth image, i.e., an edge, the presence of a different object can be determined. Accordingly, the processor can determine the location of the ship as a coast if the number of edges included in the depth image of the object is greater than or equal to a preset number, and as an ocean if the number of edges is less than the preset number.

[0200] Meanwhile, the processor can use lidar to generate images similar to depth images.

[0201] For example, the processor can convert 3D point data acquired using LiDAR into a 2D depth image. In other words, the processor can generate a depth image by converting the coordinates of the 3D point data into 2D coordinates and depth values.

[0202] The processor can determine the location of the charity using at least one of the methods for determining the location of the charity described above, and perform clustering based on the determined location of the charity.

[0203] Accordingly, the processor can determine a method for performing clustering based on the location of the charity, and perform clustering on the point data according to the determined clustering method.

[0204] For example, if the ship is located offshore, the processor may perform clustering based on the density of point data. As another example, if the ship is located in the ocean, the processor may perform clustering based on the distance between point data.

[0205] For example, the processor can perform density-based clustering or distance-based clustering on point data to obtain multiple clusters (1211, 1212, 1213) and determine a representative cluster among the multiple clusters (1211, 1212, 1213).

[0206] For example, the processor can determine a representative cluster among multiple clusters (1211, 1212, 1213) using the number of expected point data and the number of actual point data included in the bounding box (1210).

[0207] Specifically, the processor can extract expected point data based on the resolution of a device that acquires point data, i.e., a lidar, determine the number of reference point data using the expected point data, and determine a representative cluster based on the determined number of reference point data.

[0208] First, the processor can extract the number of expected point data included within the bounding box (1210) based on the resolution information of the lidar.

[0209] For example, the higher the resolution of the lidar, the greater the number of expected point data, and the processor can extract the number of expected point data included in the bounding box (1210) by using resolution information such as the horizontal resolution, vertical resolution, laser rotation speed, and sampling speed of the lidar.

[0210] Additionally, the processor can determine the number of reference point data using the number of expected point data.

[0211] For example, the processor may weight the number of expected point data based on the angle between the charity and the object, and determine the number of reference point data based on the weights.

[0212] For example, the processor can calculate the angle between the charity and the object using multiple clusters (1211, 1212, 1213). That is, the processor can determine each of the multiple clusters (1211, 1212, 1213) as an object and calculate the angle between the charity and the object. Here, the angle between the charity and the object may mean bearing, but is not limited thereto.

[0213] Additionally, the processor may weight the number of predicted point data based on the calculated angle. For example, the processor may assign a lower weight to the number of predicted point data when the calculated angle is larger (i.e., the object is located to the side relative to the direction of travel of the ship). As another example, the processor may assign a higher weight to the number of predicted point data when the calculated angle is smaller (i.e., the object is located to the front relative to the direction of travel of the ship).

[0214] Therefore, the processor can determine the number of reference point data using the number of expected point data and the determined weights.

[0215] For example, the processor may determine a representative cluster based on the number of determined reference point data. Specifically, the processor may calculate the number of actual point data from a cluster closest to the charity among multiple clusters (1211, 1212, 1213) and determine whether the number of calculated actual point data exceeds the number of reference point data to determine the representative cluster.

[0216] For example, the processor may determine a cluster as a representative cluster if the number of actual point data included in the cluster closest to the charity among multiple clusters (1211, 1212, 1213) exceeds the number of reference point data.

[0217] In addition, if the number of actual point data included in the cluster closest to the charity among the plurality of clusters (1211, 1212, 1213) does not exceed the number of reference point data, the processor calculates the number of actual point data included in the cluster next closest to the charity, and determines whether the number of calculated actual point data exceeds the number of reference point data, thereby determining a representative cluster.

[0218] Therefore, the processor can detect objects (specifically, dynamic objects) using representative clusters determined by performing clustering.

[0219]

[0220] FIG. 13 is a diagram illustrating an example of a method for detecting an object using a representative cluster according to one embodiment.

[0221] Hereinafter, with reference to FIG. 13, an example of a method for a processor to detect an object using a representative cluster is described.

[0222] Referring to FIG. 13, the processor can extract the center point (1310) of the representative cluster (1300).

[0223] For example, the processor can extract a first center point (1310) of a representative cluster (1300) and estimate the extracted first center point (1310) as a representative location of the object.

[0224] Additionally, the processor can use the extracted first center point (1310) to determine the type of object within the bounding box and estimate information about the object, including the direction of the object, the speed of the object, etc.

[0225] For example, the processor can estimate the type of object, the direction of the object, and the speed of the object based on the image information and coordinate information included in the representative cluster (1300).

[0226] As an example, the processor can extract the first center point (1310) of the representative cluster (1300) to estimate information about the object, such as the type of object, the direction of the object, and the speed of the object. As another example, the processor can estimate information about the object by comparing all point data included in the representative cluster (1300) with previous point data. As yet another example, the processor can estimate information about the object by using an area such as a 3D bounding box, a 2D bounding box, a circle, or an ellipse that includes the representative cluster (1300).

[0227] Meanwhile, if the charity's location is in the ocean, the processor can perform reclustering to remove noise and improve accuracy.

[0228] For example, if the location of the charity is the ocean, the processor can perform re-clustering on the representative cluster (1300) to determine a representative point and detect an object using the representative point.

[0229] First, the processor may determine that the location of the charity is the ocean based on at least one of the methods for determining the location of the charity described above.

[0230] Additionally, if the processor determines that the location of the charity is in the ocean, it can perform re-clustering on the representative cluster (1300) to determine the representative point.

[0231] For example, the processor can calculate the number of point data of the representative cluster (1300) to derive the size of the representative cluster (1300). In addition, the processor can determine an area in which reclustering is to be performed based on the size of the representative cluster (1300).

[0232] For example, the processor may determine an area equal to a preset multiple of the size of the representative cluster (1300) based on the first center point (1310) of the representative cluster (1300) as an area for performing reclustering. Here, the preset multiple may be set based on the distance between the point data included in the representative cluster (1300) and the self-study. That is, the further the distance between the point data included in the representative cluster (1300) and the self-study, the smaller the preset multiple may be set, and the closer the distance, the larger the preset multiple may be set.

[0233] For example, the processor may perform reclustering using a distance-based clustering method on point data included in the area where reclustering is to be performed.

[0234] Additionally, the processor can perform reclustering to derive a second center point of the acquired cluster.

[0235] Accordingly, the processor can determine the representative point using the first center point (1310) of the representative cluster (1300) and the second center point of the cluster obtained by performing reclustering.

[0236] For example, the processor may determine a representative point by assigning weights to the first center point (1310) and the second center point. Here, the processor may assign a weight to the first center point (1310) that is less than the weight to the second center point, but is not limited thereto.

[0237] For example, the processor may determine any one point around the first center point (1310) and the second center point as a representative point based on the weights assigned to each of the first center point (1310) and the second center point.

[0238] Accordingly, the processor can use the determined representative points to determine the type of object within the bounding box and estimate information about the object, including the direction of the object, the speed of the object, etc.

[0239] Additionally, the processor may provide a user with a monitoring image displaying information about a detected dynamic object. Here, the monitoring image may refer to a display unit.

[0240] Meanwhile, the processor can generate an avoidance path to avoid collision between the detected static or dynamic object and the vehicle, and control the vehicle to navigate along the generated avoidance path.

[0241] For example, the processor may generate at least one collision avoidance path based on information about detected static or dynamic objects. Furthermore, the processor may control navigation-related devices of the vessel so that the vessel can navigate along the generated collision avoidance path.

[0242] For example, when the ship is operating autonomously, the processor may control the autonomous operation processing unit so that the autonomous operation processing unit calculates control values ​​for autonomous operation. In this case, the processor may be a processor included in a separate device from the autonomous operation processing unit, but is not limited thereto, and may also be a processor included in the autonomous operation processing unit. As another example, when the ship is operating manually, the processor may provide the generated collision avoidance path to the user (or navigator) and control the navigation-related devices of the ship so that the ship follows the collision avoidance path and navigates based on the user's input.

[0243] Additionally, the processor can provide the generated collision avoidance path to the user using a monitoring image (or display unit).

[0244] Additionally, the processor can apply information of detected static or dynamic objects to a charity-centered grid to generate 3D clusters.

[0245] For example, the processor may generate a grid centered on a charity. Furthermore, the processor may obtain information about dynamic or static objects surrounding the charity and, based on the obtained information about the dynamic or static objects, place the dynamic or static objects in the generated grid. Furthermore, the processor may generate source data for the dynamic or static objects placed in the grid and, based on the generated source data, generate a graphical interface, i.e., a 3D cluster.

[0246] As described above, the present invention can detect static objects and / or dynamic objects using a predetermined sensor.

[0247] For example, the present invention can estimate an object using a sensor that acquires data of the object.

[0248] As an example, the present invention can estimate a static object using a lidar sensor.

[0249] For example, the present invention can acquire point data of an object using a lidar sensor and divide the acquired point data into at least one region. Furthermore, the present invention can determine representative point data for each divided region and estimate a static object using the determined representative point data.

[0250] As another example, the present invention can estimate dynamic objects using a lidar sensor and a camera.

[0251] For example, the present invention can acquire point data of an object using a lidar sensor and image data of the object using a camera. Furthermore, the processor can associate the point data and image data of the object to generate integrated information that integrates each data. Furthermore, the present invention can perform clustering based on the integrated information to determine representative clusters and estimate dynamic objects using the determined representative clusters.

[0252] In addition, the present invention can also generate 3D clusters by applying information of estimated static or dynamic objects to a charity-centered grid.

[0253] Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented on a general-purpose digital computer that runs the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0254] Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from the essential characteristics of the above-described invention. Therefore, the disclosed methods should be considered illustrative rather than restrictive. The scope of the claims, not the foregoing description, is defined by the scope of the patent, and should be interpreted to encompass all differences within the scope equivalent thereto.

Claims

1. Memory in which at least one program is stored; and comprising at least one processor executing at least one program; At least one processor, By matching image information and coordinate information about an object, integrated information is generated in which the image information and the coordinate information about the object are integrated, and based on the integrated information, either density-based clustering or distance-based clustering is performed, and based on the result of performing the clustering, the object is detected using the center point of a representative cluster determined. A device for detecting dynamic objects.

2. In paragraph 1, The image information for the above object includes outline information of the object, At least one processor, A device that acquires an image of the object using an image acquisition sensor and extracts the contour information included in the image.

3. In paragraph 1, At least one processor, A device that acquires three-dimensional coordinate information of the object using a coordinate information acquisition sensor and converts the three-dimensional coordinate information into two-dimensional coordinate information.

4. In paragraph 1, At least one processor, A device that determines the order in which the integrated information is to be generated based on the position of one side of a bounding box including the object, and generates the integrated information by corresponding two-dimensional coordinate information of the bounding box and the object according to the determined order.

5. In paragraph 1, At least one processor, A device that performs clustering on point data included in the integrated information to obtain multiple clusters and determines the representative cluster among the multiple clusters.

6. In paragraph 5, At least one processor, A device that extracts the number of expected point data based on the resolution of the device that acquires the point data, determines the number of reference point data using the number of expected point data, and determines the representative cluster based on the determined number of reference point data.

7. In paragraph 6, At least one processor, A device that weights the number of the expected point data based on the angle between the charity and the object, and determines the number of the reference point data based on the weight.

8. In paragraph 6, At least one processor, A device that acquires the point data from a cluster close to charity among the plurality of clusters, and determines the representative cluster based on whether the acquired point data exceeds the number of the reference point data.

9. In paragraph 1, At least one processor, A device that performs clustering on point data included in the integrated information based on the location of the charity, obtains a plurality of clusters as a result of performing the clustering, and determines the representative cluster among the plurality of clusters.

10. In paragraph 9, At least one processor, A device that performs density-based clustering based on the density of the point data when the location of the charity is coastal, and performs distance-based clustering based on the distance of the point data when the location of the charity is ocean.

11. In paragraph 9, At least one processor, A device that determines a representative point by performing re-clustering on the representative cluster when the location of the above charity is the ocean, and detects the object using the representative point.

12. In paragraph 1, At least one processor, A device that extracts the center point of the representative cluster and estimates the center point as the location of the object.

13. In paragraph 12, At least one processor, A device that estimates information of the object, including the type of the object, the direction of the object, and the speed of the object, based on the image information and the coordinate information included in the representative cluster, and provides a monitoring image that displays the information of the object.

14. An operation of performing clustering on point data of the object using image information and coordinate information for the same object; An operation of determining a representative cluster among multiple clusters based on the result of performing the above clustering; and An operation of detecting the object by extracting the center point of the representative cluster determined above; How to detect dynamic objects.

15. A computer-readable recording medium recording a program for executing the method of Article 14 on a computer.

Images