Electronic device for identifying marine target on basis of coordinate conversion of marine sensor information, and operation method thereof

Abstract

An operation method of an electronic device, according to various embodiments, may be provided, the operation method comprising the operations of: obtaining an image by using a camera; obtaining first location information indicating a target by using at least one other detection device other than the camera; converting the first location information into second location information of a coordinate system of the camera on the basis of at least one piece of conversion information, wherein the at least one piece of conversion information includes at least one among first conversion information associated with an electronic navigational chart device and a radar, second conversion information associated with the radar and a lidar, or third conversion information associated with the lidar and the camera; obtaining information on the location of the target on the basis of inputting a partial area of the image corresponding to the second location information into a vision AI; and displaying a screen including at least one graphic object indicating the location of the target.

Description

Electronic device for identifying maritime targets based on coordinate transformation of maritime sensor information, and method of operation thereof

[0001] Various embodiments of the present invention relate to an electronic device for identifying a maritime target based on coordinate transformation of maritime sensor information, and a method of operation thereof.

[0002]

[0003] Autonomous navigation is achieved by automating situation awareness regarding the external environment, decision-making, and appropriate actions based on those decisions. In particular, accurate situation awareness of the surroundings is the most critical element of safe autonomous navigation.

[0004] To ensure situational awareness, vessels operating at sea utilize various devices such as radar, lidar, and cameras to identify buoys, including other vessels, and based on the recognition results, proceed with setting paths to avoid collisions.

[0005] At this point, the conventional method has a problem in that the radar and camera each acquire data separately and are limited to providing their own object recognition results. In other words, since the radio-based radar and the optical imaging-based camera operate as separate sensors to acquire independent data and provide object recognition results, this acts as a factor that degrades the management of separated data and reduces the accuracy and reliability of object recognition and tracking.

[0006] For example, radar is generally well-suited for long-range detection and is effective at identifying relatively large objects. However, it has limitations when dealing with small, distant objects or situations requiring precise details. In other words, there was a problem in identifying details such as small vessels or floating objects from a distance, and visual identification by a navigator is necessary to compensate for this.

[0007] Furthermore, camera-based object recognition faces the problem that real-time processing is difficult because analyzing high-resolution images or videos requires significant computing resources. Additionally, since the detection range of cameras is limited compared to radar, there was a problem in that preemptive object detection and tracking were impossible based solely on camera-based object recognition results.

[0008]

[0009] Conventional ship equipment is implemented to recognize a target within an image captured by a camera and to provide instruction information to indicate the recognized target along with the image on a display. However, due to limitations in image information, such as resolution, there is a problem in that it is difficult for the ship equipment to recognize targets within the image. An electronic device and a method of operation thereof, according to various embodiments, can be implemented to identify a target within an image based on information about a target received from the outside (e.g., electronic chart information, AIS (automatic identification system) messages, etc.) and to provide instruction information for the identified target. Accordingly, the accuracy of target recognition, which is degraded due to limitations in image information, can be improved.

[0010] Conventional autonomous navigation equipment operates different sensors, such as cameras, radar, and lidar, individually. Consequently, there was a problem where data from the same point in time could not be aligned due to inconsistencies in sensor formats and coordinate systems. The electronic device and its operation method according to various embodiments can resolve the coordinate inconsistency problem between heterogeneous sensors by converting sensor data into a common format and performing fusion processing through an integrated recognition module. As a result, the accuracy of situational awareness based on composite sensors can be improved.

[0011] In addition, conventionally, there was a problem in recognizing objects within an image because degradation occurred due to external environmental conditions during image capture, making it difficult to recognize objects within the image. For example, in backlit situations where strong light is provided within the camera's shooting area, the brightness value of a part of the image becomes excessive, making it difficult to recognize objects located in that part. An electronic device and a method of operation according to various embodiments can improve the accuracy of object recognition by performing object recognition by adjusting the brightness value of the image based on the result of comparing the brightness value of a part of the image with a threshold value.

[0012]

[0013] According to various embodiments, a method of operation of an electronic device may be provided, comprising: an operation of acquiring an image using a camera; an operation of acquiring first position information representing a target using at least one detection device other than the camera; an operation of converting the first position information into second position information of the camera's coordinate system based on at least one conversion information, wherein the at least one conversion information includes at least one of first conversion information associated with an electronic chart device and a radar, second conversion information associated with the radar and a lidar, or third conversion information associated with the lidar and the camera, and an operation of acquiring information about the position of the target based on inputting a portion of the image corresponding to the second position information into a vision AI; and an operation of displaying a screen including at least one graphic object representing the position of the target.

[0014] According to various embodiments, an electronic device may be provided comprising: a control device; wherein the control device acquires an image using a camera, acquires first position information representing a target using at least one detection device other than the camera, converts the first position information into second position information in the coordinate system of the camera based on at least one conversion information, wherein the at least one conversion information includes at least one of first conversion information associated with an electronic chart device and a radar, second conversion information associated with the radar and a lidar, or third conversion information associated with the lidar and the camera, acquires information regarding the position of the target based on inputting a portion of the image corresponding to the second position information into a vision AI, and is configured to display a screen including at least one graphic object representing the position of the target.

[0015] The means for solving the problem according to various embodiments are not limited to the means for solving the problem described above, and unmentioned means for solving the problem will be clearly understood by those skilled in the art from this specification and the attached drawings.

[0016]

[0017] According to various embodiments, an electronic device and a method of operation thereof may be provided, which are implemented to identify a target within an image based on information about a target received from an external source (e.g., electronic chart information, AIS messages, etc.) and to provide indication information for the identified target. Accordingly, the accuracy of target recognition, which is degraded due to limitations in image information, can be improved.

[0018] According to various embodiments, an electronic device and a method of operation thereof can be provided, which can improve the accuracy of object recognition by performing object recognition by adjusting the brightness value of an image based on the result of comparing the brightness value of a part of the area with a threshold value.

[0019]

[0020] FIG. 1 is a drawing showing examples of components of an autonomous navigation (or autonomous navigation support) system according to various embodiments.

[0021] FIG. 2 is a drawing showing examples of components of a navigation support device according to various embodiments.

[0022] FIG. 3 is a drawing for illustrating examples of the configuration of an integrated recognition module according to various embodiments.

[0023] FIG. 4 is a diagram illustrating examples of the operation of a conversion module according to various embodiments.

[0024] FIG. 5 is a diagram illustrating examples of the operation of a conversion module according to various embodiments.

[0025] FIG. 6 is a diagram illustrating examples of the operation of an AI module according to various embodiments.

[0026] FIG. 7 is a diagram illustrating an example of situational awareness of a navigation support device based on an integrated recognition module according to various embodiments.

[0027] FIG. 8 is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation support device) for coordinate transformation-based target identification according to various embodiments.

[0028] FIG. 9a is a diagram illustrating an example of an operation for recognizing a target based on at least one conversion information according to various embodiments.

[0029] FIG. 9b is a diagram illustrating offset information reflected in at least one conversion information according to various embodiments.

[0030] FIG. 10 is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation aid) for generating conversion information between a radar and other world sensors according to various embodiments.

[0031] FIG. 11a is a drawing for illustrating examples of offset information in the yawing, pitching, and rolling directions according to various embodiments.

[0032] FIG. 11b is a diagram illustrating an example of an offset information generation module for acquiring offset information of a lidar and a motion sensor for a radar according to various embodiments.

[0033] FIG. 11c is a diagram illustrating an example of operation in which offset information is consequently reflected in conversion information between radar information and motion sensor information according to various embodiments.

[0034] FIG. 12a is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation support device) for generating conversion information associated with a radar and a marine sensor according to various embodiments.

[0035] FIG. 12b is a drawing for illustrating an example of an operation to generate first bearing grid information of an information acquisition device (e.g., electronic chart) according to various embodiments.

[0036] FIG. 12c is a diagram illustrating an example of an operation to obtain first offset information based on a comparison of bearing grid information according to various embodiments.

[0037] FIG. 13a is a flowchart illustrating an example of an operation method of an electronic device (e.g., a navigation support device) for generating conversion information associated with radar and lidar according to various embodiments.

[0038] FIG. 13b is a diagram illustrating an example of an operation to acquire third offset information for a pitching direction according to various embodiments.

[0039] FIG. 14 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device) for conversion information calibration according to various embodiments.

[0040] FIG. 15 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation aid) for multi-crop-based target recognition according to various embodiments.

[0041] FIG. 16 is a diagram illustrating an example of an operation in which multiple regions on an image captured by a camera are multi-cropped and input into a vision AI according to various embodiments.

[0042] FIG. 17 is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation aid) for tracking a target according to various embodiments.

[0043] FIG. 18 is a diagram illustrating an example of an operation to track a target by setting a tracking area on an image according to various embodiments.

[0044] FIG. 19 is a flowchart illustrating an example of an operation for displaying external information-based target information of an electronic device (e.g., a ship device) according to various embodiments.

[0045] FIG. 20 is a diagram illustrating an example of an operation to obtain indication information indicating a target within camera information based on ECDIS information according to various embodiments.

[0046] FIG. 21 is a diagram illustrating an example of an operation to obtain indicator information representing a target within camera information based on a message received from the outside, according to various embodiments.

[0047] FIG. 22 is a flowchart for further illustrating an example of an operation to acquire first information of an electronic device (e.g., a ship device) according to various embodiments.

[0048] FIG. 23 is a diagram illustrating an example of an operation to set a monitoring area from an electronic chart based on a conversion module according to various embodiments.

[0049] FIG. 24 is a flowchart for further illustrating examples of operations for identifying specific regions within an image of an electronic device (e.g., a ship device) according to various embodiments.

[0050] FIG. 25 is a flowchart illustrating an example of a target information display operation based on brightness values ​​in an image of an electronic device (e.g., a ship device) according to various embodiments.

[0051] FIG. 26 is a diagram illustrating an example of a case where the brightness value of at least a portion of an image is an abnormal range (or not a normal range) according to various embodiments.

[0052] FIG. 27 is a flowchart illustrating an example of operation in a first mode (e.g., high visibility mode) when the brightness value of an electronic device (e.g., a ship device) is in an abnormal range according to various embodiments.

[0053] FIG. 28 is a diagram illustrating an example of an operation for recognizing an object within an area having an abnormal brightness based on images of different shutter speeds according to various embodiments.

[0054] FIG. 29 is a flowchart illustrating an example of operation in a second mode (e.g., low visibility mode) when the brightness value of an electronic device (e.g., navigation support device (200)) is in an abnormal range according to various embodiments.

[0055] FIG. 30a is a diagram illustrating an example of an operation for identifying information about a target (e.g., bounding box) using an AI module according to various embodiments.

[0056] FIG. 30b is a diagram illustrating an example of an operation to preprocess and display an abnormal section identified using a display module according to various embodiments.

[0057] FIG. 31 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device) for performing target location and target information display operations according to various embodiments.

[0058] FIG. 32 is a drawing showing an example of an image (or screen) including target location and target information according to various embodiments.

[0059] FIG. 33 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device) for obtaining target information associated with a target location according to various embodiments.

[0060] FIG. 34 is a diagram illustrating an example of an operation to extract information of an information acquisition device based on a comparison of pattern information according to various embodiments.

[0061] FIG. 35 is a diagram illustrating an example of an operation to extract information of an information acquisition device based on a comparison of pattern information according to various embodiments.

[0062] FIG. 36 is a flowchart illustrating an example of an operation method of an electronic device (e.g., a navigation support device) for acquiring coordinate system transformation-based target information according to various embodiments.

[0063] FIG. 37 is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation aid) for displaying information intensity / size of a detection device as target information according to various embodiments.

[0064] FIG. 38 is a drawing showing an example of information intensity / size of a detection device according to various embodiments.

[0065] FIG. 39 is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation aid) for displaying distance and direction as target information according to various embodiments.

[0066] FIG. 40 is a drawing showing an example in which the direction and distance of a target are displayed according to various embodiments.

[0067] FIG. 41 is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation aid) for displaying distance and direction as target information according to various embodiments.

[0068] FIG. 42 is a drawing showing an example of a wake being displayed around a target according to various embodiments.

[0069] FIG. 43 is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation support device) for displaying S-100 information selection-based target information according to various embodiments.

[0070] FIG. 44 is a drawing for explaining an example in which target location and target information are displayed according to the selection of a specific category among a plurality of categories associated with S-100 according to various embodiments.

[0071] FIG. 45 is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation support device) for indicating a fish farm according to various embodiments.

[0072] FIG. 46 is a drawing for explaining an example of an operation to modify and display the location of a fish farm according to various embodiments.

[0073] FIG. 47 is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation support device) that displays target information depending on whether the detection device identifies a target, according to various embodiments.

[0074] FIG. 48 is a drawing for illustrating various object information display examples according to various embodiments.

[0075]

[0076] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0077] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0078] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0079] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., internal memory) or external memory that is readable by a machine (e.g., an electronic device). For example, a processor (e.g., a processor) of a machine (e.g., an electronic device) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by a machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0080] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0081] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0082] According to various embodiments, a method of operation of an electronic device may be provided, comprising: an operation of acquiring an image using a camera; an operation of acquiring first position information representing a target using at least one detection device other than the camera; an operation of converting the first position information into second position information of the camera's coordinate system based on at least one conversion information, wherein the at least one conversion information includes at least one of first conversion information associated with an electronic chart device and a radar, second conversion information associated with the radar and a lidar, or third conversion information associated with the lidar and the camera, and an operation of acquiring information about the position of the target based on inputting a portion of the image corresponding to the second position information into a vision AI; and an operation of displaying a screen including at least one graphic object representing the position of the target.

[0083] According to various embodiments, a method of operation of a ship device may be provided, comprising: an operation of acquiring first information from the outside; an operation of acquiring an image captured by a camera installed on the ship; and an operation of acquiring instruction information of a specific area corresponding to the location of a target included in the first information within the image, based on at least a portion of the first information, wherein the instruction information includes information for describing the target, and an operation of controlling a display to display an image including the instruction information in the specific area.

[0084]

[0085] 1. Autonomous navigation system (1)

[0086] According to various embodiments, an autonomous navigation (or autonomous navigation support) system (1) may refer to an overall environment that supports the autonomous navigation of a ship. The autonomous navigation (autonomous operation) is a technology that enables a ship to navigate on its own without human intervention, and ships capable of autonomous navigation are named as maritime autonomous surface ships (MASS), smart ships, unmanned ships, remote control ships, and digitalized ships. In this case, a ship capable of autonomous navigation may be implemented to automatically perform each step sequentially, including situational awareness of the external environment, decision-making, and appropriate action based on the decision. In this specification, the meaning of supporting autonomous navigation may mean performing the aforementioned steps and / or providing functions for the steps. In this case, situational awareness of the external environment is particularly important for safe autonomous navigation.

[0087]

[0088] 2. Components of the autonomous navigation system (1)

[0089] FIG. 1 is a drawing showing examples of components of an autonomous navigation (or autonomous navigation support) system (1) according to various embodiments.

[0090] Referring to FIG. 1, an autonomous navigation system (1) according to various embodiments may include a ship (100), a server (200), an external server (300), and a communication network (C).

[0091] According to various embodiments, the vessel (100) may be a vessel capable of autonomous operation. A navigation support device (200) is installed on the vessel (100) and may be implemented to support the autonomous operation of the vessel (100). As will be described later, the navigation support device (200) may be implemented to improve the accuracy of situational awareness (e.g., object recognition) by analyzing information from an information acquisition device (210) based on an early fusion method, and may be implemented to improve the accuracy of object recognition around the vessel (100) in particular under low visibility (or low light) conditions.

[0092] According to various embodiments, a vessel (100) (e.g., a first vessel (100a)) may receive information (or data, or messages) from an external source (e.g., a server (200), an external device (300), and / or a second vessel (100b)) or / or transmit information to an external source via a communication network (C) using a navigation support device (200). The navigation support device (200) of the vessel (100) may be configured to support autonomous navigation (e.g., situational awareness) based on at least a portion of the information received from an external source.

[0093] According to various embodiments, the server (200) may be a computer system implemented to provide services to support the autonomous operation of a vessel (100) on an autonomous operation system (1). For example, the server (200) may provide services for the exchange of messages between vessels (e.g., a first vessel (100a), and a second vessel (100b)). Also, for example, the server (200) may update conversion information for converting information of each of the information acquisition devices (210) equipped on the vessel (100) to one another.

[0094] According to various embodiments, the external device (300) is a variety of devices capable of communicating with the vessel (100) in addition to the vessel (100), and may include an artificial satellite, a base station, a control center, a port, a specific institution, etc., and may include a variety of devices not limited to the examples described.

[0095] According to various embodiments, the communication network (C) may mean a communication method, network, and / or system for performing communication between a vessel (100) (e.g., a first vessel (100a)) and an external (e.g., a server (200), an external device (300), and / or a second vessel (100b)).

[0096] For example, the communication network may include a wireless communication network using VHF (very high frequency), a long-distance communication network such as 3G / LTE / 5G / 6G / next-generation cellular communication network, LAN / WAN, etc., a short-distance communication network such as Bluetooth / WiFi / IrDA, a low-orbit satellite-based communication network (e.g., optical communication, RF communication), and a VDES (VHF data exchange system) communication network.

[0097] For example, the communication network may include an AIS and a maritime connection platform (MCP) for sharing information such as the location, speed, and direction of the vessel, as a communication network (or system) that supports vessel operation.

[0098]

[0099] 2.1. Components of the navigation support device (200)

[0100] FIG. 2 is a drawing showing examples of components of a navigation support device (200) according to various embodiments. Below, examples of components of a navigation support device (200) will be further described with reference to FIG. 3.

[0101] FIG. 3 is a drawing for illustrating examples of the configuration of an integrated recognition module (230a) according to various embodiments.

[0102] Referring to FIG. 2, according to various embodiments, a navigation support device (200) may include an information acquisition device (210), a control device (220), an information output device (250), and a user terminal (260), and each component may be connected via a communication interface to enable the exchange of data (or information). Meanwhile, the navigation support device (200) may include more devices, not limited to the illustrated and / or described examples. The aforementioned navigation support device (200), information acquisition device (210), control device (220), information output device (250), and user terminal (260) are machines or equipment that operate using electrical / electronic principles and may be defined as electronic devices in this specification. Meanwhile, the components included in the navigation support device (200) may not be included in the navigation support device (200) but may be provided on a vessel (100) outside the navigation support device (200). For example, an antenna (AN), a communication device (211), an electronic chart device (215), etc. may be provided externally.

[0103] According to various embodiments, the information acquisition device (210) refers to a device for acquiring detection information indicating the situation around the ship (100) and / or the situation outside the ship (100), and may be installed on at least a part of the ship (100). For example, the detection information may vary depending on the type of information acquisition device (210) and may be interpreted to include the concept of raw data acquired by the information acquisition device (210) or data processed to be interpretable by a preprocessing module (not shown).

[0104] According to various embodiments, the information acquisition device (210) may include a plurality of detection devices. For example, referring to FIG. 2, the information acquisition device (210) may include an antenna (AN), a communication device (211), a sensor device (213), an electronic chart display and information system (ECDIS) device (215), and a motion sensor (217), but may include various other types of devices for acquiring detection information, not limited to the examples described. Meanwhile, although not illustrated, as will be described later, the navigation support device (200) can improve the accuracy of object recognition in low visibility conditions by integrating and analyzing detection information acquired from various types of information acquisition devices (210) using an integrated recognition module (230a).

[0105] For example, the communication device (211) may be a device that transmits / receives information (or data, or messages) wirelessly to and from the outside via a communication network using the antenna (AN). As an example, the communication device (211) may include an AIS transceiver (211a) for transmitting information of another vessel (e.g., location, speed, direction of movement) from the AIS or for transmitting information of the vessel, and a communication circuit (211b) for transmitting / receiving information via a VHF / cellular communication network / long-range / short-range wireless communication network, a satellite communication network, and / or a VDES communication network, but may include various other types of communication devices, not limited to the examples described. The navigation support device (200) may obtain information about targets around the vessel (100) based on information (or data, or messages) received from the outside using the communication device (211).

[0106] For example, the sensor device (213) may be a device that detects physical phenomena or changes in the environment around the vessel (100) and outputs electrical or data results. For example, the sensor device (213) may include a radar (213a), a camera (213b), and a lidar (213c). The detection range and information format of each sensor device (213) may differ. For example, the detection ranges are longer in the order of radar (213a), lidar (213c), and camera (213b), so the radar (213a) may have the longest detection range. As will be described later, since the radar (213a) has the longest detection range, when the detection information of the information acquisition devices (210) is integrated, the information of other types of detection devices may be converted based on the information of the radar (213a). The meaning of the format may refer to a method of expressing information about a target. For example, a navigation support device (200) (e.g., control circuit (221)) can use a radar (213a) to acquire an echo image representing a target around the vessel (100), and a camera (213b) can use at least one image (or video) containing the target.

[0107] For example, the electronic chart device (215) may receive a digital map containing information necessary for marine and navigation. The digital map may include information about the vessel (100) and objects other than the vessel (100). The navigation support device (200) may obtain information about objects around the vessel (100) using the electronic chart device. For example, the electronic chart device (215) may include an electronic chart system (ECS) device and / or an electronic chart display and information system (ECDIS) device.

[0108] For example, the motion sensor (217) may include an inertial measurement unit (IMU) sensor, a global navigation satellite sensor (GNSS) sensor, and / or a tilt sensor. The motion sensor (217) may be used as a navigation sensor module to estimate the attitude (roll, pitch, yaw) and position of a vessel. The IMU sensor may include an accelerometer and a gyroscope to calculate the amount of attitude change in real time by measuring the linear acceleration and angular velocity of the moving object, and the GNSS sensor may provide the absolute position (latitude, longitude) and absolute heading of the moving object using satellite signals.

[0109] According to various embodiments, at least some of the information acquisition devices (210) may be implemented in a single device. For example, as an information acquisition device (210), a sensor pack including a radar (213a) and a camera (213b) may be provided.

[0110] According to various embodiments, each of the information acquisition devices (210) may be implemented to be located on the vessel (100) and oriented in different directions.

[0111] According to various embodiments, the control device (220) may mean hardware and software for controlling the overall operation of the navigation support device (200). For example, referring to FIG. 2, the control device (220) may include a control circuit (221) and a storage device (223).

[0112] For example, the control circuit (221) can control at least one other component (e.g., a hardware or software component) and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the control circuit (221) can store commands or data received from another component (e.g., a first communication circuit (503)) in volatile memory, process the commands or data stored in volatile memory, and store the resulting data in non-volatile memory. According to one embodiment, the control circuit (221) may include a main processor (not shown) (e.g., a central processing unit or an application processor) or an auxiliary processor (not shown) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the server (10) includes a main processor (not shown) and an auxiliary processor (not shown), the auxiliary processor (not shown) may be configured to use less power than the main processor (not shown) or to be specialized for a specified function. The auxiliary processor (not shown) may be implemented separately from the main processor (not shown) or as part thereof.

[0113] An auxiliary processor (not shown) may control at least some of the functions or states associated with at least one component of the server (10) (e.g., the first communication circuit (503)) on behalf of the main processor (not shown) while the main processor (not shown) is in an inactive (e.g., sleep) state, or together with the main processor (not shown) while the main processor (not shown) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (not shown) (e.g., an image signal processor or a communication processor) may be implemented as part of another functionally related component (e.g., the first communication circuit (503)). According to one embodiment, the auxiliary processor (not shown) (e.g., a neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the server (10) itself where the artificial intelligence is performed, or through a separate server (e.g., a learning server). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above. An artificial intelligence model may include multiple artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0114] Unless otherwise noted, the operation of the navigation support device (200) can be understood as an operation performed by the control circuit (221).

[0115] According to various embodiments, the storage device (223) can store software modules (230) and databases (240).

[0116] According to various embodiments, the software module (230) can control the navigation support device (200) (e.g., information acquisition device (210), control device (220), information output device (250)) to perform a specific operation (or operation) corresponding to the software module (230) when the software module (230) is executed by the control circuit (221). The software module (230) may be implemented as computer code, instructions, or an API (application process interface), etc., executable by the ship device (220) (e.g., control circuit (221)). The software module (230) may include an integrated recognition module (230a) and a display module (230b), and may include more software modules for autonomous navigation support, not limited to the examples described.

[0117] According to various embodiments, the integrated recognition module (230a) may be implemented to acquire information about a target around the vessel (100) by integrating a plurality of detection information acquired by a plurality of information acquisition devices (210). The information about the target may include information such as the location of the target, the speed of the target, the direction of movement of the target, the shape of the target, and the type of the target, and is not limited to the examples described.

[0118] According to various embodiments, the display module (230b) may control an information output device (250) (e.g., display (251)) to output information about the object obtained by the integrated recognition module (230a) so that it can be recognized by a person (e.g., captain, navigator, crew member), and / or a user terminal (260).

[0119] According to various embodiments, the database (240) may store information for supporting autonomous navigation. For example, the database (240) may store conversion information for converting between detection information. Also, for example, the database (240) may store information about targets around the vessel (100) identified by the integrated recognition module (230a).

[0120] According to various embodiments, the information output device (250) can output information about a target to the outside so that it can be recognized by a person (e.g., captain, navigator, crew) inside the vessel (100). For example, the information output device (250) includes a display (251), a speaker (253), and an optical device (255), and may further include various types of output devices that provide information recognizable by a person, such as visual, auditory, or tactile, without being limited to the examples described.

[0121] According to various embodiments, the user terminal (260) is an electronic device carried by a person (e.g., captain, navigator, crew member) on the vessel (100), and may include various electronic devices capable of outputting information, such as a smartphone, PC, laptop, electronic pad, PDA, electronic binoculars, etc. In the following description, the operation of displaying on the information output device (250) (e.g., display (251)) of the navigation support device (200) may also be understood as an operation of displaying on the user terminal (200).

[0122]

[0123] 2.1.1. Example of implementation of the integrated recognition module (230a)

[0124] FIG. 3 is a diagram showing an example of the configuration of an integrated recognition module (230a) according to various embodiments. FIG. 3 will be further explained below with reference to FIG. 4 to 7.

[0125] FIG. 4 is a drawing for illustrating an example of the operation of a conversion module (310) according to various embodiments. FIG. 5 is a drawing for illustrating an example of the operation of a conversion module (310) according to various embodiments. FIG. 6 is a drawing for illustrating an example of the operation of an AI module (320a) according to various embodiments. FIG. 7 is a drawing for illustrating an example of situational awareness of a navigation support device (200) based on an integrated recognition module (230a) according to various embodiments.

[0126] According to various embodiments, with reference to FIG. 3, the integrated recognition module (230a) may include a conversion module (310), a situation awareness module (320), a tracking module (330), and a verification module (340).

[0127] According to various embodiments, the conversion module (310) may include a plurality of conversion modules (310a, 310b) for converting the format of each of the plurality of detection information (e.g., first detection information, second detection information) obtained from a plurality of information acquisition devices (210) into the format of another detection information. For example, the format of the information may mean the type of detection information and / or the display form of the information. FIG. 5 illustrates the format of the detection information, each of the camera information (C), radar information (R), and ECDIS information (E) provided on the vessel (100). Each of the detection information includes information about the vessel (100) and the target (500) around the vessel (100), but may have different formats. For example, camera information (C) is an image captured by a camera (213b) in a specific direction of the vessel (100), and the image may be composed of multiple pixels having pixel information (e.g., two-dimensional coordinate values ​​(x, y) and color model values ​​(e.g., RGB, HSV, etc.). In this case, the two-dimensional coordinate system of the image may be a YZ or XZ plane relative to the vessel (100). Pixels at specific locations in the image may indicate a part of the vessel (100) and / or a target (500). Thus, the coordinates of the pixel area representing the target (500) from the image may be obtained as information about the target. Also, for example, radar information (R) is an image containing a circular echo image, and the echo image may be represented on a specific distance and a specific bearing angle section on the XY plane centered on the vessel (100) based on a signal detected by the radar (213a). At this time, information regarding a specific distance from the center of the vessel (100) to the target (500) and a specific bearing angle can be obtained from the image as information about the target.For example, the ECDIS (E) may be an image of an electronic chart and may include a plurality of graphic objects representing maritime targets and topographic features having two-dimensional coordinate values ​​(x, y) based on maritime information collected by the ECDIS system, and information associated with the plurality of graphic objects (e.g., type of target, speed of target). In this case, the two-dimensional coordinate system of the ECDIS may be an XY plane for the vessel (100). At least one of the plurality of graphic objects included in the ECDIS may indicate the vessel (100) and the target (500). Accordingly, the coordinates of the graphic object representing the target (500), the type of the target, etc., can be obtained from the image as information about the target. The conversion module (310) may be implemented to convert the format of information about the target (500) having a different format obtained based on each detection information into the format of a specific type of detection information. Accordingly, information about the target acquired by different information acquisition devices (210) can be fused and compared. Accuracy is enhanced by comparing based on information about the target acquired from the information acquisition device (210), rather than simply comparing the detection results of the target (e.g., whether the target is present or not). This target recognition method can be defined as target recognition (or situation recognition) by early fusion (or measurement level fusion).

[0128] According to various embodiments, with reference to FIG. 4, the conversion module (310) can convert detection information (e.g., camera information (C), LiDAR information (L), ECDIS information (E)) into other types of detection information based on a plurality of previously stored conversion information. For example, the conversion module (310) may include a conversion information generation module (410) that calculates conversion information (e.g., conversion function, conversion matrix) for conversion between each of the detection information based on information received from each of the information acquisition devices (210) installed on the vessel (100) and stores it in advance in a conversion DB (420). Each of the conversion information may include information (e.g., vector, matrix, function) for converting the coordinate system of each detection information to the coordinate system of another detection information. For example, the conversion information may be a conversion matrix (or conversion function, or conversion vector). The above transformation matrix may include information regarding a matrix (or function, or vector) for at least one of translation, rotation, scale, symmetry, or shear. In this case, the transformation information may reflect offset information to compensate for differences in the arrangement state (e.g., position and / or angle) between each detection device. When the transformation information generation module (410) calculates transformation information to convert the format of the first detection information of the first detection device into the format of the second detection information of a second detection device of a different type from the first detection device, it may apply a transformation matrix including a translation vector and / or a transformation matrix to convert the format representing the target of the first detection information (e.g., coordinate system) to the format representing the target of the second detection information (e.g., coordinate system). The transformation information generation module (410) may obtain offset information based on comparing the first information regarding a specific target identified from the first detection information to which the translation vector and / or transformation matrix are applied with the second information regarding the specific target of the second detection information.Differences in position and angle occur when each information acquisition device (210) is installed on the vessel (100), and offset information may be information to compensate for this. Consequently, the conversion information generation module (410) may store conversion information including a vector, matrix, and / or function reflecting offset information for converting the first detection information into the second detection information. At this time, the information for converting the second detection information into the first detection information is inverse conversion information, but for convenience of explanation in this specification, this may also be described as conversion information. As shown in FIG. 4, each of the plurality of conversion modules (310a, 310b, 310c) may be implemented to convert specific types of detection information based on specific conversion information, but is not limited to the illustrated example and may be implemented as a single conversion module (310) to integrate and convert various types of detection information based on the plurality of conversion information.

[0129] According to various embodiments, the conversion module (310) can convert detection information (e.g., camera information (C), lidar information (L), ECDIS information (E)) based on (or centered on) radar information (R). Converting detection information based on radar information (R) means that, in order for a specific type of detection information to be converted into another type of detection information, a specific type of detection information is converted into radar information (R), and the converted radar information (R) is converted into another type of detection information. For example, referring to FIG. 4, the conversion module (310) may include a first conversion module (310a) that uses first conversion information for conversion between radar information (R) and lidar information (L), a second conversion module (310b) that uses second conversion information for conversion between lidar information (L) and camera information (C), and a third conversion module (310c) that uses third conversion information for conversion between radar information (R) and ECDIS information (E). At this time, the conversion module (310) can convert the ECDIS information (E) into radar information (R) based on the third conversion information, convert the radar information (R) into lidar information (L) based on the first conversion information, and convert the lidar information (L) into camera information (C) based on the second conversion information in order to convert the format of the ECDIS information (E) into the format of camera information (C). Accordingly, information about a target obtained from the ECDIS information (E) can be expressed (or corresponded) to a specific pixel area on the camera information (C) (e.g., image).

[0130] Meanwhile, not limited to the examples described, the conversion module (310) may be implemented to convert ECDIS information (E) directly into LiDAR information (L) or camera information (C) based on the conversion information.

[0131] According to various embodiments, the situation awareness module (320) can obtain information about a target from the detection information. As described above, the format of the information about the target may also differ depending on the format of the detection information.

[0132] According to various embodiments, the situation awareness module (320) may be implemented to obtain information about a target by fusing a specific type of detection information obtained by the conversion module (310) and another type of detection information converted into the format of the specific type of detection information in an early fusion manner. For example, referring to FIG. 6(a), the AI ​​module (320a) may include an attention module (610) for identifying a portion of the first detection information based on additional converted second detection information, and an AI model (620) trained to obtain information about a target based on a portion of the first detection information. The information about the target may include dynamic / static information for describing the target, such as a bounding box indicating the target location, range information, size, type information, reliability information, speed, direction, relative distance, collision information, SOG, COG, etc., but is not limited to the examples described. An action of extracting a meaningful portion of the first detection information may be defined as an attention action. For example, FIG. 6(b) illustrates an example of acquiring information about a target using camera information (C) and radar information (R) according to an early fusion method. The first attention module (610a) of the AI ​​module (320a) can acquire information about a portion of the image of the camera (C) acquired as the echo image representing the target in the radar information (R) is converted into the format of camera information. The information about the portion of the image can be understood as a concept that includes not only an image in which the region (ROI area) identified based on the location information of the converted echo image in the image is cropped, but also an image in which the ROI area is visually highlighted. The vision AI model can output information about the target (e.g., a bounding box indicating the location of the target) based on the input of at least one image.

[0133] According to various embodiments, the tracking module (330) may be implemented to obtain the current movement state (e.g., direction of movement, speed) and future movement state (e.g., direction of movement, speed) of a target based on information about the target obtained by the situation awareness module (320). The information about the target may be stored in a database (240).

[0134] According to various embodiments, the verification module (340) may be implemented to verify information about a target stored in the database (240) (or verify the reliability of the target information). For example, the verification module (340) may be implemented to determine whether the information about a target stored in the database (240) is consistent based on rule information (e.g., perform a validity determination algorithm) and to delete and / or modify information that is not consistent. The rule information may be policy, specification, and / or standard information received from an external source, and may include internal operation policy information received from a ship operating entity (e.g., shipping company, maritime control center, etc.), and / or route safety guidelines, depth restriction information, Notice to Mariners (NtM), and other data sources (e.g., the latest electronic chart information server, lidar / radar observation data, message data), etc. received from an international maritime organization such as the IMO (International Maritime Organization).

[0135] In some embodiments, the verification module (340) may further include a time validity verification function to disable or mark as a target for re-verification any target information (e.g., depth markers, buoy locations) that has not been updated for a certain period of time.

[0136] Accordingly, the verification module (340) can ensure that the information on the electronic chart used by the navigation sensor and display module always meets safety navigation standards by maintaining the accuracy, reliability, and up-to-dateness of the target information stored in the database (240).

[0137] Meanwhile, the verification module (340) may be implemented to perform a matching judgment algorithm based on a language model learned based on the rule-information received from the outside as described above, but is not limited to the examples described.

[0138] According to various embodiments, the display module (230b) can control the display (251) to display information about a target obtained by the situation awareness module (320) and the tracking module (330) stored in the database (240). FIG. 7 shows an image of the display (251). Referring to FIG. 7 (a), which represents a conventional situation awareness method, when a low visibility condition (F), such as sea fog, is formed in the area (CR) of the camera (213b) installed on the ship (100), the target (O) is not properly recognized, and thus there is a problem in that information about the target (O) is not displayed on the display (251). However, referring to FIG. 7 (b), which represents a situation awareness method based on Early Fusion according to the above specification, information about the target (O) can be displayed on the display (251) as a specific area on the camera information (C) (e.g., image) is analyzed more accurately by converting other types of detection information, such as radar information. That is, the electronic device and the method of operation thereof of the present specification can be implemented based on early fusion to improve the accuracy of object recognition around the vessel (100) in low visibility (or low light) conditions.

[0139]

[0140] 3. Coordinate transformation-based target identification operation

[0141] FIG. 8 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device (200)) for coordinate transformation-based target identification according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the order illustrated and / or described. FIG. 8 will be further described below with reference to FIG. 9a and 9b.

[0142] FIG. 9a is a diagram illustrating an example of an operation for recognizing a target based on at least one conversion information according to various embodiments. FIG. 9b is a diagram illustrating offset information reflected in at least one conversion information according to various embodiments.

[0143] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can acquire an image using a camera (213b) in operation 801.

[0144] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire first position information indicating a target and a target using another marine sensor (e.g., an information acquisition device (210) of FIG. 2) in operation 803. For example, referring to FIG. 4 described above, the navigation support device (200) (e.g., a control device (220)) may acquire information from other information acquisition devices (210), such as radar information (R), lidar information (L), and ECDIS information (E), in addition to camera information (C) acquired by the camera (213b), and redundant descriptions are omitted.

[0145] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can convert a first position information into a second position information in the coordinate system of a camera based on at least one conversion information in operation 805. For example, referring to FIG. 9a, the navigation support device (200) (e.g., a control device (220)) uses the aforementioned integrated recognition module (230a) (e.g., a conversion module (310)) to convert information (e.g., a first conversion information (T)) that is pre-stored in a conversion DB (420). rl ), second transformation information (T lm ), and third transformation information (T lc ), 4th transformation information (T rc ), 5th transformation information (T xm Based on )), information of another information acquisition device (210) (e.g., radar information (R), lidar information (L), ECDIS information (E), camera information (C), and motion sensor information (M)) can be converted into information of another information acquisition device (210) having a different coordinate system. For example, each of the above pre-stored conversion information (e.g., first conversion information (T rl ), second transformation information (T lm ), third transformation information (T lc ), 4th transformation information (T rc ), and 5th transformation information (T xm )) may be a homogeneous transformation matrix including a rotation matrix and / or a translation vector. Meanwhile, in addition to ECDIS information (E), AIS information (not shown) may be transformed from LiDAR information (L) to radar information (R) based on specific transformation information. The specific transformation information is a second transformation information (T lm It can be implemented similarly to ). Also, on the other hand, the above-mentioned fifth transformation information (T xm ) is conversion information (T) for converting radar information (R) into motion sensor information (M). rm ), conversion information (T) for converting LiDAR information into motion sensor information (M)lm ), conversion information (T) for converting ECDIS information (E) into motion sensor information (M) em ) and conversion information for converting camera information into motion sensor information (T cm It may include ).

[0146] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may perform an operation of reflecting a plurality of conversion information into information of a specific information acquisition device (210) as at least part of an operation of converting to second position information of the camera's coordinate system.

[0147] In one embodiment, an electronic device (e.g., a navigation support device (200)) provides first conversion information (T as shown in Equation 1 below). rl ) and third transformation information (T lc ) radar information (R) (e.g., p in Equation 1 below) r By applying to ), the second location information (p c You can obtain ).

[0148]

[0149] In the above mathematical formula 1, K represents the camera intrinsic parameter, and the radar information (p r ) includes position information (x, y) including bearing angle and distance in a radar coordinate system, and the second position information (p c ) may include information (x,y) representing the pixel position in the camera coordinate system.

[0150] Meanwhile, in addition to the embodiment of converting the described radar information (R), information acquired by another type of information acquisition device (210) can be converted into second position information in the camera coordinate system based on the conversion relationship shown in FIG. 9a.

[0151] According to various embodiments, an electronic device (e.g., a navigation support device (200)) uses a conversion information generation module (410) to generate offset information (△T m2r , △Tm2l , △T l2r , △T l2c , △T c2r , △T m2c ) obtain (or estimate), and the offset information (△T m2r , △T m2l , △T l2r , △T l2c , △T c2r , △T m2c A plurality of conversion information (e.g., first conversion information (T)) for converting information between the aforementioned information acquisition devices (210) based on ) rl ), second transformation information (T lm ), third transformation information (T lc ), 4th transformation information (T rc ), and 5th transformation information (T xm )) can be obtained. Accordingly, multiple transformation information (e.g., first transformation information (T rl ), second transformation information (T lm ), third transformation information (T lc ), 4th transformation information (T rc ), and 5th transformation information (T xmBased on )), high-accuracy target information mapping between each information acquisition device (210) (e.g., sensors having a relative position coordinate system and an electronic chart device having an absolute position coordinate system) is possible, and based on this, sensor fusion-based target recognition performance and sensor fusion-based target display performance through a display can be improved. The operation of acquiring offset information and reflecting offset information in the conversion information can be defined as a calibration operation. For example, referring to (a) of FIG. 9b, an axis of the rolling direction (Φ) (surge axis, or x axis, or heading axis), an axis of the yawing direction (ψ) (heave axis, or z axis), and an axis of the pitching direction (θ) (sway axis, or y axis) can be defined relative to the vessel. At this time, referring to (b) of FIG. 9b, since the information acquired by each of the information acquisition devices (210) has a difference value (e.g., relative pose, relative attitude angle) for each direction (Φ, ψ, θ) and also has a difference in the center axis of each of the sensors (e.g., radar (213a), lidar (213c), camera (213b), offset transformation matrix information (△T) including angle offset information and / or position offset information between the information acquired by each of the information acquisition devices (210) m2r , △T m2l , △T l2r , △T l2c , △T c2r , △T m2c ) may occur.

[0152] According to various embodiments, the angle offset information among the offset information represents the difference in angle direction between each information acquisition device (210) in the same coordinate system (e.g., radar coordinate system), and the aforementioned transformation information (e.g., first transformation information (T rl ), second transformation information (T le ), third transformation information (T lc ), 4th transformation information (T rc), and 5th transformation information (T xm It may be reflected in the rotation matrix included in )), but is not limited to the examples described.

[0153] For example, due to the difference in the attitude (e.g., installation angle, or direction facing for sensing) installed on each of the information acquisition devices (210) having a relative coordinate system, each of the multiple pieces of information acquired by each of the information acquisition devices (210) has a difference value (e.g., relative pose) for each direction (Φ, ψ, θ), which can be defined as angle offset information.

[0154] For example, offset information (△T) between radar information (R) and lidar information (L) based on the relative pose of radar (213a) and lidar (213c). l2r ) is the difference in rolling direction (Φ) (△Φ l2R ), difference in yawing direction (ψ) (△ψ l2R ), or difference in pitching direction (θ) (△θ l2R It may include at least one of ).

[0155] Also, as another example, offset information (△T) between lidar information (L) and camera information (C) based on the relative pose of lidar (213c) and camera (213b). l2c ) is the difference in rolling direction (Φ) (△Φ l2c ), difference in yawing direction (ψ) (△ψ l2c ), or difference in pitching direction (θ) (△θ l2e It may include at least one of ).

[0156] Also, as another example, offset information (△T) between radar information (R) and camera information (C) based on the relative pose of the radar (213a) and the camera (213b). c2r ) is the difference in rolling direction (Φ) (△Φ c2r ), difference in yawing direction (ψ) (△ψ c2r ), or difference in pitching direction (θ) (△θ c2r It may include at least one of ).

[0157] Also, for example, for mapping between information acquisition devices (210) having a relative coordinate system (e.g., radar (213a), lidar (213c), camera (213b)) and information acquisition devices (210) having an absolute coordinate system (e.g., electronic chart device (215), AIS device (not shown)), a difference value (e.g., relative attitude angle) in each direction (Φ, ψ, θ) between the motion sensor (217) (or navigation sensor) that forms the basis of the information acquisition devices (210) having an absolute coordinate system and the information acquisition devices (210) (e.g., radar (213a), lidar (213c), camera (213b)) is required, and this can also be defined as angle offset information.

[0158] For example, offset information (△T) between radar information (R) and motion sensor information (M) based on the relative attitude angle of the radar (213a) and the motion sensor (217). m2r ) is the difference in rolling direction (Φ) (△Φ m2r ), difference in yawing direction (ψ) (△ψ m2r ), or difference in pitching direction (θ) (△θ m2r It may include at least one of ).

[0159] For example, offset information (△T) between radar information (R) and motion sensor information (M) based on the relative attitude angle of the lidar (213c) and the motion sensor (217). m2l ) is the difference in rolling direction (Φ) (△Φ m2l ), difference in yawing direction (ψ) (△ψ m2l ), or difference in pitching direction (θ) (△θ m2l It may include at least one of ).

[0160] According to various embodiments, the position offset information among the offset information represents the position difference in one axis direction between each information acquisition device (210) in the same coordinate system (e.g., radar coordinate system), and the aforementioned transformation information (e.g., first transformation information (T rl), second transformation information (T le ), third transformation information (T lc ), 4th transformation information (T rc ), and 5th transformation information (T xm It may be reflected in the movement vectors included in )), but is not limited to the examples described.

[0161] For example, due to the difference in the position of the central axis between the information acquisition devices (210) (e.g., radar (213a), lidar (213c), camera (213b)), each of the plurality of pieces of information acquired by each of the information acquisition devices (210) has a difference value, which can be defined as position offset information.

[0162] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can acquire at least some of a plurality of offset information in advance (or prior) by performing a direct calibration operation between at least some of a plurality of information acquisition devices (210) using a conversion information generation module (410), and can reflect this in at least some of a plurality of conversion information corresponding thereto. For example, the electronic device (e.g., a navigation support device (200)) can perform calibration between a camera (213b), a lidar (213c), and a motion sensor (217) using a conversion information generation module (410), thereby obtaining offset information (△T) between motion sensor information (M) and lidar information (L). m2l ), and / or offset information (△T) between LiDAR information (L) and camera information (C) l2c ) can be obtained in advance. Accordingly, offset information (△T) between the lidar information (L) and camera information (C) can be obtained. l2c ) is the aforementioned third transformation information (T lc It can be reflected in ).

[0163] For example, an electronic device (e.g., a navigation support device (200)) uses a conversion information generation module (410) to, as at least part of the calibration operation between the lidar (213c), camera (213b), and motion sensor (217), based on feature information of a planar object such as a structure like a marina bay and / or a checkerboard board sensed by the lidar (213c) and camera (213b) respectively, estimates the odometry of the information of the lidar (213c) and camera (213b) respectively, and then combines this with the odometry information obtained by the motion sensor (217) to obtain offset information (△T) between each coordinate system. m2l, △T l2c You can obtain ).

[0164] In addition, as another example, an electronic device (e.g., a navigation support device (200)) can calculate roll (Φ) direction offset information by using a conversion information generation module (410) to calculate the tilted angle between the camera (213b) and the horizontal plane through information of the horizon line identified in the image of the camera (213b), and calculate roll (Φ) direction offset information between the lidar and the horizontal plane by performing a calibration operation between the camera (213b) and the lidar (213c) based on this information.

[0165] Meanwhile, in the case of the radar (213a), since it is composed of two-dimensional information, it cannot directly perform calibration operations with other devices, so a separate operation may be required to obtain offset information. This will be explained in detail later.

[0166] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire the location of a target based on inputting a portion of an image corresponding to the second location information into a vision AI (e.g., the vision AI model (620a) of FIG. 6) in operation 807, and may display an execution screen including a graphic object (e.g., a bounding box) indicating the location of the target in operation 809. Since the target recognition operation of the navigation support device (200) (e.g., a control device (220)) can be performed as described above in FIG. 6, a redundant description is omitted.

[0167] Meanwhile, with reference to FIG. 8, it is described that the electronic device (e.g., navigation support device (200)) acquires information about a part of an image based on information from another marine sensor (e.g., information acquisition device (210)) converted into information in the coordinate system of a camera, and acquires information about a target by inputting the acquired information about a part of the image into a vision AI. However, the method is not limited to the described example, and the operation of acquiring information about a target may also be performed by acquiring some of the information from a specific marine sensor by converting it into information in the coordinate system of a specific marine sensor of a different type and inputting it into an AI model for analysis. For example, the electronic device (e.g., navigation support device (200)) may acquire information about a part of the radar information (R) based on information from another marine sensor (e.g., information acquisition device (210)) converted into information in the coordinate system of a radar, and acquire information about a target by inputting the acquired information about a part of the radar information (R) into an AI model.

[0168]

[0169] 3.1. Generation of Transformation Information

[0170] FIG. 10 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device (200)) for generating conversion information between a radar and other sensors according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the order illustrated and / or described. FIG. 10 will be further described below with reference to FIG. 11.

[0171] FIG. 11a is a diagram illustrating examples of offset information in yawing, pitching, and rolling directions according to various embodiments. FIG. 11b is a diagram illustrating examples of an offset information generation module (1100) for acquiring offset information of a lidar (213c) and a motion sensor (217) for a radar (213a) according to various embodiments. FIG. 11c is a diagram illustrating examples of an operation in which offset information is consequently reflected in conversion information between radar information (R) and lidar information (L) according to various embodiments.

[0172] According to various embodiments, an electronic device (e.g., a navigation support device (200)) provides first offset information (△ψ) associated with the yawing (ψ) direction of the radar (213a) and the navigation sensor (e.g., a motion sensor (217)) in operation 1001. m2r ) is obtained, and in operation 1003, second offset information (△ψ) associated with the yawing (ψ) direction of the radar (213a) and lidar is obtained. l2r ) and third offset information (△θ) associated with the pitching (θ) direction l2r You can obtain ).

[0173] As described above, due to the installation posture (e.g., installation angle, etc.) of the radar (213a) on the ship, a difference value (e.g., relative pose, and / or relative attitude angle) may occur for each of the rolling direction (Φ), yawing direction (ψ), and pitching direction (θ) between the radar information (R) and the lidar information (L), and between the radar information (R) and the absolute coordinate system (e.g., sea level) (or motion sensor (217)), as illustrated in FIG. 11a (a) to (c). At this time, as illustrated in FIG. 11a (a) to (c), first, offset information (△ψ) with respect to the absolute coordinate system (or ECDIS information (E), or motion sensor information (M)) m2r , △θ m2r , △Φ m2r After ) is calculated, as the absolute coordinate system (or ECDIS information (E), or Mohsen sensor information (M)) and the calibrated (i.e., difference value already known) LiDAR information (L) and radar information (R) are compared in the absolute coordinate system, offset information (△ψ) between LiDAR information (L) and radar information (R) is obtained. l2r , △θ l2r , △Φ l2r ) can be obtained.

[0174] Referring to FIG. 11, the aforementioned conversion information generation module (410) may include an offset information generation module (1100). The offset information generation module (1100) may include an initial offset information generation module (1110) and an offset information optimization module (1120).

[0175] According to various embodiments, the initial offset information generation module (1110) may be implemented to generate position offset information and angle offset information. The generated position offset information and angle offset information may subsequently be used as initial offset information for offset information optimization by the offset information optimization module (1120).

[0176] According to various embodiments, the initial relative position offset information generation module (1111) may be implemented to acquire position offset information among offset information.

[0177] According to various embodiments, the initial relative angle offset information generation module (1113) may be implemented to acquire angle offset information among offset information.

[0178] For example, the navigation support device (200) uses an initial relative angle offset information generation module (1113) to generate first offset information (△ψ) of the yawing direction (ψ) between radar information (R) and an absolute coordinate system (e.g., sea level) (or motion sensor (217)) based on a comparison operation between radar information (R) and ECDIS information (E). m2r ) is obtained, and based on the comparison result of radar information (R) and lidar information (L), a second offset information (△ψ) of the yawing direction (ψ) between radar information (R) and lidar information (L) is obtained. l2r ) and third offset information (△θ) of the pitching direction (θ) l2r You can obtain ).

[0179] For example, the navigation support device (200) has first offset information (△ψ) of the yawing direction (ψ) between radar information (R) and an absolute coordinate system (e.g., sea level) (or motion sensor (217)). m2rAs at least part of the operation of acquiring ), ECDIS information (E) representing a specific target (or a part of a specific target) and radar information (R) representing said specific target (or a part of a specific target) can be compared based on the same coordinate system (e.g., radar coordinate system). In other words, absolute coordinate information (e.g., (x,y)) representing a specific target (or a part of a specific target) among the ECDIS information (E) can be converted into information in the radar coordinate system (i.e., information in the relative coordinate system) (e.g., (β,r)) for comparison with the radar information (R). In this conversion process, the ECDIS information (E) representing the specific target (or a part of a specific target) is expressed based on the position of the vessel indicated by the motion sensor information (M), and the ECDIS information (E) converted into the radar coordinate system may include information about the axis of the yawing direction (ψ) of the motion sensor (217). Accordingly, based on the above comparison result, the first offset information (△ψ) of the yawing direction (ψ) between the radar information (R) and the absolute coordinate system m2r ) can be obtained. Meanwhile, the first offset information (△ψ) of the yawing direction (ψ) can be obtained. m2r An example of the operation to obtain ) will be explained in detail with reference to FIG. 12.

[0180] Also, for example, the navigation support device (200) has a second offset information (△ψ) of the yawing direction (ψ) between radar information (R) and lidar information (L). l2r ) and third offset information (△θ) of the pitching direction (θ) l2rAs at least part of the operation of acquiring ), lidar information (L) representing a specific target (or a part of a specific target) and radar information (R) representing said specific target (or a part of a specific target) can be compared based on the same coordinate system (e.g., radar coordinate system). In other words, among the lidar information (L), the coordinate information of a cloud point representing a specific target (or a part of a specific target) (e.g., (x,y,z)) can be converted into information in the radar coordinate system (i.e., information in the relative coordinate system) (e.g., (ψ,r)) for comparison with the radar information (R). At this time, during the conversion process, since the lidar information (L) representing the specific target (or a part of a specific target) is already calibrated with the motion sensor (217) as described above, the lidar information (L) converted into the radar coordinate system may include information for each axis of the motion sensor (217). Accordingly, based on the above comparison result, the second offset information (△ψ) of the yawing direction (ψ) between radar information (R) and lidar information (L) l2r ) and third offset information (△θ) of the pitching direction (θ) l2r ) can be obtained. Meanwhile, based on the comparison result between radar information (R) and lidar information (L), second offset information (△ψ) of the yawing direction (ψ) between radar information (R) and lidar information (L) can be obtained. l2r ) and third offset information (△θ) of the pitching direction (θ) l2r An example of the operation to obtain ) will be explained in detail with reference to FIG. 13.

[0181] According to various embodiments, an electronic device (e.g., a navigation support device (200)) in operation 1005 provides first offset information (△ψ m2r ), second offset information (△ψ l2r ), and third offset information (△θ l2r Based on ), a fourth offset information (△Φ) associated with the rolling (Φ) direction of the radar (213a) and the navigation sensor (e.g., motion sensor (217)) m2r) and fifth offset information (△θ) associated with the pitching (θ) direction m2r ) can be obtained. For example, as illustrated in FIG. 11b, the navigation support device (200) uses the initial offset information generation module (1110) of the offset information generation module (1100) to obtain the first offset information (△ψ) obtained. m2r ), second offset information (△ψ l2r ), and third offset information (△θ l2r By inputting ) into the pre-implemented acquisition model (1113a), fourth offset information (△Φ) associated with the rolling (Φ) direction of the radar (213a) and the navigation sensor (e.g., motion sensor (217)) is obtained. m2r ) and fifth offset information (△θ) associated with the pitching (θ) direction m2r You can obtain ).

[0182] In one embodiment, the acquisition model (1113a) is fourth offset information (△Φ) in the roll direction. m2r It may be an algorithm for calculating the value of ). For example, the acquisition model (1113a) is a fourth offset information (△Φ m2r With ) as a variable, the first offset information (△ψ m2r ), second offset information (△ψ l2r ), and third offset information (△θ l2r A fourth offset information (△Φ) that minimizes the sum of squares of the distance differences of each of the radar information (R), ECDIS information (E), and lidar information (L) in the radar coordinate system representing a specific target reflecting ) and / or maximizes the similarity of the calculated distance distribution (e.g., the number of information in the yaw direction where the distance is minimized). m2rIt may be an algorithm for finding the value of ). During the above optimization, the theory of superior dependency information may be applied to the variables, or the operation to find the optimal probability variable may be performed by configuring the variables into particles using a particle swarm optimization method, or variable-specific weights may be applied for global optimization, and / or the value of each offset information may be optimized as a range is set.

[0183] In another embodiment, the acquisition model (1113a) is the fourth offset information (△Φ m2r ) and fifth offset information (△θ) associated with the pitching (θ) direction m2r It may be a mathematical formula for comparing radar information (R) and ECDIS information (E) using ) as a variable. For example, the above acquisition model (1113a) may include a mathematical formula expressed as [Equation 2] below.

[0184]

[0185] In mathematical equation 2, I r represents the position of a unit vector in the radar coordinate system, and I r' is the first to third offset information (△ψ m2r , △ψ l2r, △θ l2r Lidar information (L), representing a unit vector based on ), indicates the identified position as it is converted into the radar coordinate system, and R r This represents my Euler matrix.

[0186]

[0187] In the Euler matrix expressed by the above mathematical formula 3, 0 is assigned to Ψ, and 4th offset information (△Φ) is assigned as a variable. m2r ) is reflected, and θ has 5th offset information (△θ) as a variable. m2r ) is reflected. At this time, the reason why 0 is reflected in the above Ψ is, I r' The first offset information (△ψ) of the yaw direction (Ψ) m2rIt may be because ) was reflected.

[0188] Accordingly, the navigation support device (200) has a fourth offset information (△Φ) reflected as the variable in [Equation 2]. m2r ) and 5th offset information (△θ m2r ) can be calculated.

[0189] Also, not limited to the examples described above, the acquisition model (1113a) is a first offset information (△ψ m2r ), second offset information (△ψ l2r ), and third offset information (△θ l2r When ) is input, fourth offset information (△Φ) associated with the rolling (Φ) direction of the radar (213a) and the navigation sensor (e.g., motion sensor (217)) m2r ) and fifth offset information (△θ) associated with the pitching (θ) direction m2r It may be an AI model implemented to output ).

[0190] According to various embodiments, an electronic device (e.g., a navigation support device (200)) provides first to fifth offset information (△ψ) in operation 1007. l2r, △θ l2r , △ψ m2r, △θ m2r, △Φ m2r Optimization can be performed based on ).

[0191] According to various embodiments, the offset information optimization module (1120) includes initial offset information (e.g., △T) comprising initial position offset information and initial angle offset information generated by the initial offset information generation module (1110). m2r, △T l2r Based on the transformation information to which ) is applied, by converting and comparing radar information (R), ECDIS information (E), and / or lidar information (L) into information on the same coordinate system (e.g., radar coordinate system), optimal offset information (e.g., △T m2r, △T l2rOptimization (1121) for identifying ) can be performed. Meanwhile, as described above, optimization (1121) for identifying optimal offset information can be performed by converting AIS information (not shown) into information on the same coordinate system (e.g., radar coordinate system) based on the conversion information and comparing them.

[0192] At this time, the above initial offset information (e.g., △T m2r, △T l2r With at least some of the variables set, the optimization (1121) can be performed.

[0193] For example, the navigation support device (200) uses an offset information optimization module (1120) to obtain angle offset information (e.g., first offset information (△ψ) m2r ), second offset information (△ψ l2r ), third offset information (△θ l2r ), 4th offset information (△Φ m2r ), and 5th offset information (△θ m2r Set at least a portion of )) as variables and the remaining offset information is reflected in the transformation information (e.g., △T m2r, △T l2r Generate ) and transformation information (e.g., △T m2r, △T l2rBased on ), radar information (R), ECDIS information (E), and lidar information (L) in a radar coordinate system representing a specific target can be generated. Subsequently, the navigation support device (200) can perform an operation to find a value of offset information set as a variable, using an offset information optimization module (1120), such that the sum of squares of the distance differences of each of the radar information (R), ECDIS information (E), and lidar information (L) in the generated radar coordinate system is minimized and / or the similarity of the calculated distance distribution is maximized (e.g., the number of information in the yaw direction where the distance is minimized is maximized). The operation to find a value of offset information set as a variable can be performed sequentially (or randomly) for each offset information, so that the optimal value of all offset information is determined as a result, but is not limited to the described example. During the above optimization, the theory of superior dependency information may be applied to the variables, or the operation to find the optimal probability variable may be performed by configuring the variables into particles using a particle swarm optimization method, or variable-specific weights may be applied for global optimization, and / or optimization may be performed as the value of each offset information is set to a range.

[0194] Meanwhile, since an operation to identify the optimal value of position offset information can be performed in the same way as the operation to identify the optimal value of the aforementioned angle offset information, a redundant explanation is omitted.

[0195] According to various embodiments, an electronic device (e.g., a navigation support device (200)) determines whether an optimization condition is satisfied in operation 1009, and if the optimization condition is satisfied (1009-Y), one or more offset information (△ψ) in operation 1011 l2r, △θ l2r , △ψ m2r, △θ m2r, △Φ m2rAt least one transformation information can be generated that reflects at least some of ). For example, as illustrated in FIG. 11c (a), the navigation support device (200) may generate a plurality of offset information (△ψ) generated by the offset information generation module (1100). l2r, △θ l2r , △ψ m2r, △θ m2r, △Φ m2r First transformation information (T) in which at least a portion of ) is reflected rl ) and / or 5th transformation information (T mr ...can generate ). Also, various types of conversion information can be generated based on the offset information described above, without being limited to the examples described. For example, as illustrated in FIG. 11c (b), the navigation support device (200) comprises a plurality of offset information (△ψ) generated by the offset information generation module (1100). l2r, △θ l2r , △ψ l2c, △θ l2c, △Φ l2c ) Fourth transformation information (T) reflecting at least a part of rc Can generate ).

[0196] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may perform an operation to acquire multiple offset information by performing operation 1001 again when the optimization condition is not satisfied (1009-N). For example, the electronic device (e.g., a navigation support device (200)) may determine whether the optimization condition is satisfied for each iteration of the aforementioned optimization operation (operation 1007), and if the optimization condition is not satisfied, may perform the optimization operation again (e.g., setting a new variable among the offset information). The optimization condition may include a condition in which the sum of the squares of the differences in distances of each radar information (R), ECDIS information (E), and lidar information (L) is minimized, or the number of information with the minimum distance is maximized. As the optimization continues to be performed, specific offset information among the offset information may be sequentially set as a variable.

[0197]

[0198] 3.1.1 Generation of Transformation Information Associated with Radar and Maritime Sensors

[0199] FIG. 12a is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation support device (200)) for generating conversion information associated with a radar and a marine sensor according to various embodiments. More and / or fewer operations may be performed without being limited to the examples shown and / or described, and operations may be performed in a different order than the order shown and / or described. FIG. 12a will be further described below with reference to FIG. 12b through FIG. 12c.

[0200] FIG. 12b is a drawing for illustrating an example of an operation to generate first bearing grid information of an information acquisition device (210) (e.g., electronic chart) according to various embodiments. FIG. 12c is a drawing for illustrating an example of an operation to acquire first offset information based on a comparison of bearing grid information according to various embodiments.

[0201] According to various embodiments, an electronic device (e.g., navigation support device (200)) in operation 1201, based on the bearing angle information of the radar, provides first offset information (△ψ) of a first direction associated with information of the radar (213a) (e.g., radar information (R)) and information of a marine sensor (e.g., electronic chart device (215), motion sensor (217)) of FIG. 2 (e.g., ECDIS information (E)). m2r You can obtain ).

[0202] For example, referring to FIG. 12b, a navigation support device (200) (e.g., a control device (220)) provides first offset information (△ψ) in a first direction m2r As at least part of the operation of acquiring ), first bearing grid information can be acquired from a detection device (e.g., electronic chart device (215) of FIG. 2, communication device (211)) based on a conversion information generation module (410). The first bearing grid information includes a first point representing a vessel identified based on information of the electronic chart device (215) (e.g., ECDIS information (E)) and a plurality of first angles (β) representing a specific target (or part of a specific target) in the yawing (ψ) direction. n,e ) Distance between multiple second points (r enc It may mean information regarding ). The first point may be acquired by a motion sensor (217) as described above. Each of the plurality of second points may mean a part of a specific object around the vessel. For example, as shown in FIG. 12b, each of the plurality of second points may be set to represent a part of the terrain around the vessel for ease of identification, but may represent various types of objects, not limited to the illustrated example. A plurality of first angles (β) set in the yawing (ψ) direction n,eThe difference between each of the above-mentioned angles can be determined according to the resolution performance of the radar (213a). At this time, the navigation support device (200) (e.g., control device (220)) may acquire first bearing grid information converted into a relative coordinate system by reflecting motion sensor information (M) in ECDIS information (E) as at least part of the operation of acquiring first bearing grid information, but is not limited to the described examples.

[0203] For example, a navigation support device (200) (e.g., a control device (220)) may acquire second bearing grid information using a radar (213a) as at least part of the operation of acquiring first offset information of a first direction. At this time, the time at which the first bearing grid information is acquired and the time at which the second bearing grid information is acquired may correspond to each other. In other words, visual synchronization between the two pieces of information may be performed. The second bearing grid information is a plurality of second angles (β) in the yaw direction (ψ) identified from the information of the radar (213a) (e.g., radar information (R)). n,r ) Distance between multiple fourth points (r radar It may mean information regarding ). The above plurality of second angles (β n,r The angle between ) is the plurality of first angles (β n,e It can correspond to the angle between ). Each of the plurality of third points can correspond to a point of a specific object around the vessel represented by the plurality of second points mentioned above.

[0204] Also, referring to FIG. 12c, for example, a navigation support device (200) (e.g., a control device (220)) acquires the aforementioned first bearing grid information (r) as at least part of the operation of acquiring first offset information in a first direction. enc ) and second bearing grid information (r radar Based on comparing ), consequently, first offset information (△ψ) for the yawing direction (ψ) m2r) can be obtained. For example, as illustrated in FIG. 11b, the navigation support device (200) can obtain first bearing grid information (r enc ) and second bearing grid information (r radar By calculating the bearing angle difference value (△β) at which the difference in distance (△r) of ) is minimized, the first offset information (△ψ) for the yawing direction (ψ) m2r You can obtain ).

[0205] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may obtain first transformation information associated with a radar and a marine sensor, with first offset information reflected in operation 1203. For example, the navigation support device (200) (e.g., a control device (220)) may adjust the value by reflecting the aforementioned offset information (e.g., a difference value) in the value of the translation vector and / or rotation transformation matrix of the first transformation information. Accordingly, the problem of inaccurate coordinate transformation accuracy due to the difference in installation positions of the radar and marine sensor installed on the ship may be resolved.

[0206]

[0207] 3.1.2. Generation of Transformation Information Associated with Radar and LiDAR

[0208] FIG. 13a is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device (200)) for generating conversion information associated with radar and lidar according to various embodiments. More and / or fewer operations may be performed without being limited to the examples illustrated and / or described, and operations may be performed in a different order than the order illustrated and / or described. FIG. 13a will be further described below with reference to FIG. 13b.

[0209] FIG. 13b is a diagram illustrating an example of an operation to obtain third offset information for a pitching direction (θ) according to various embodiments.

[0210] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can perform planar stabilization for lidar information (L) in operation 1301. For example, the navigation support device (200) (e.g., a control device (220)) can remove height values ​​(e.g., z-axis values) (or elevation values) of lidar information (L) (e.g., point cloud) obtained using lidar (213c). Accordingly, the information obtained by the lidar (213c) can be converted into information on the sea surface around the vessel.

[0211] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may correct (or reduce) distortion values ​​caused by the movement (or motion) of the vessel in the roll direction and pitch direction of the LiDAR information (L) (e.g., a point cloud) based on roll / pitch information of the vessel obtained based on a motion sensor (e.g., an IMU sensor, a GNSS sensor, a tilt sensor, etc.) as at least part of the operation for planar stabilization of LiDAR information (L), but is not limited to the described examples and various algorithms may be performed.

[0212] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire a second offset information (△ψl2r) in a first direction associated with planar stabilized lidar information (L) and radar information (R) based on bearing angle information of the radar (213a) in operation 1303. Similar to the operation 1001 described above, the navigation support device (200) (e.g., a control device (220)) may acquire a third bearing grid information including a distance (r) between a first point representing the ship position and a plurality of second points for a plurality of angles (β) for the planar stabilized lidar information (L), acquire a fourth bearing grid information by the radar (213a), and calculate a second offset information (△ψl2r) in a yawing direction that minimizes the difference value (△r) based on a comparison operation of the two bearing grid informations. The operation of calculating the second offset information above can be performed similarly to the aforementioned 1001 operation, so a redundant description is omitted.

[0213] According to various embodiments, an electronic device (e.g., a navigation support device (200)) provides, in operation 1305, a third offset information (△θ) in a second direction perpendicular to a first direction associated with lidar information (L) and radar information (R). l2r You can obtain ).

[0214] For example, an electronic device (e.g., a navigation support device (200)) provides third offset information (△θ) in a second direction. l2rAs at least part of the operation of acquiring ), by determining whether radar information (R) is acquired for an object whose height gradually decreases from a specific height at a distance of a preset distance from the periphery of the vessel as illustrated in FIG. 13b, a first angle range in which radar information (R) is acquired can be identified, and a second angle range can be identified from the corresponding lidar information (L). Based on calculating the difference between the first angle range and the second angle range, a third offset information (△θ) representing the difference angle with respect to the vertical angle direction (e.g., height (altitude) direction, pitch direction) can be identified. l2r ) can be obtained.

[0215] According to various embodiments, an electronic device (e.g., a navigation support device (200)) in operation 1307 has a second offset information (△ψl2r) and a third offset information (△θ l2r Second conversion information associated with radar and lidar that reflects ) can be obtained.

[0216]

[0217] 3.1.3. Conversion Information Calibration

[0218] FIG. 14 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device (200)) for conversion information calibration according to various embodiments. More and / or fewer operations may be performed, not limited to the examples shown and / or described, and operations may be performed in a different order than the order shown and / or described.

[0219] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can determine whether an update event has occurred in operation 1401. For example, the navigation support device (200) (e.g., a control device (220)) can determine that an update event has occurred after a preset time has elapsed and / or when the aforementioned conversion information (e.g., first to third conversion information) has been implemented and / or when it is installed on an actual vessel.

[0220] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may update at least one of a plurality of offset information in operation 1403 when an update event occurs (1401-Y), and in operation 1405, generate a plurality of conversion information different from a previously stored plurality of conversion information based on at least one of the updated plurality of offset information (△Tm2r, △Tm2l, △Tr2l, △Tl2c, △Tc2r, △Tm2c). For example, based on the operation of FIGS. 11 to 13 described above, the navigation support device (200) (e.g., a control device (220)) may calculate new first to fifth offset information and generate new first to fifth conversion information by reflecting the calculated first to fifth offset information. The newly generated first to third conversion information may be used for early fusion recognition.

[0221]

[0222] 3.2. Radar Information-Based Target Information Display Operation

[0223] 3.2.1. Multi-crop Based Target Recognition

[0224] FIG. 15 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device (200)) for multi-crop-based target recognition according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the illustrated and / or described order. FIG. 15 will be further described below with reference to FIG. 16.

[0225] FIG. 16 is a diagram illustrating an example of an operation in which multiple regions on an image captured by a camera are multi-cropped and input into a vision AI (1200) according to various embodiments.

[0226] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can identify a first region by mapping second location information onto an image (I) in operation 1501. For example, as illustrated in FIG. 16, a navigation support device (200) (e.g., a control device (220)) can identify a first region (1601a, 1601b, 1601c, 1601d, 1601e) on an image (I) based on second location information representing pixel coordinates on the image (I) obtained based on a coordinate system transformation of detection information (e.g., radar information (R), ECDIS information (E), lidar information (L)) obtained by a detection device (200) other than the camera (213b).

[0227] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may generate a plurality of second regions having different sizes based on a first region (1601a, 1601b, 1601c, 1601d, 1601e) in operation 1503. For example, the navigation support device (200) (e.g., a control device (220)) may generate a cropped image by cropping a region for a second region (1603a, 1603b, 1603c, 1603d, 1603e) that includes a feature point (e.g., a center point) included in the first region (1601a, 1601b, 1601c, 1601d, 1601e). At this time, as illustrated in FIG. 16, the navigation support device (200) (e.g., control device (220)) can obtain a plurality of cropped images (1605) of a plurality of regions having different sizes.

[0228] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire a bounding box (1607) representing a target based on inputting a plurality of second regions into an AI model (e.g., a vision AI (1600)) in operation 1505. The electronic device (e.g., a navigation support device (200)) may convert the coordinates of the bounding boxes (1607a, 1607b, 1607c) output from the AI ​​model into the coordinates of the image (I) before cropping, so that the bounding boxes (1607a, 1607b, 1607c) are displayed on the image (I). For example, an electronic device (e.g., a navigation aid device (200)) can identify the coordinates of bounding boxes (1607a, 1607b, 1607c) on image (I) based on resizing the size of a plurality of cropped images (1605) to the size of image (I). Also, for example, an electronic device (e.g., a navigation aid device (200)) can identify the coordinates of bounding boxes (1607a, 1607b, 1607c) on image (I) based on the ratio of the size of the plurality of cropped images (1605) to the size of image (I).

[0229] According to various embodiments, an electronic device (e.g., a navigation aid device (200)) performs NMS (or Soft-NMS, Weighted Boxes Fusion) to remove overlapping bounding boxes among bounding boxes (1607a, 1607b, 1607c) identified by a plurality of cropped images (1605) of different sizes. A threshold value (IoUthr) and a confidence threshold value (sthr) can be adaptively adjusted according to sea conditions (roughness, wave height, congestion).

[0230] Meanwhile, without being limited to the described and / or illustrated examples, the electronic device (e.g., navigation support device (200)) may generate an image that visually highlights the aforementioned first area (e.g., ROI area) and obtain information about a target based on inputting the generated image into an AI model. For example, the method of visually highlighting may include a method in which a line of a specific value defining the first area is displayed on the image, or the color value of the first area is set differently compared to other areas.

[0231] At this time, the AI ​​model can be pre-trained to output information about a target (e.g., output a bounding box representing the target) when a visually highlighted image is input.

[0232]

[0233] 3.2.2. Tracking Objects

[0234] FIG. 17 is a flowchart illustrating an example of a method of operation of an electronic device for tracking a target (e.g., a navigation support device (200)) according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the order illustrated and / or described. FIG. 15 will be further described below with reference to FIG. 16.

[0235] FIG. 18 is a diagram illustrating an example of an operation to track a target by setting a tracking area on an image according to various embodiments.

[0236] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can determine whether a tracking event has occurred in operation 1701. For example, the navigation support device (200) (e.g., a control device (220)) can identify that the tracking event has occurred when a specified time has elapsed since the bounding box was output, and / or when it is determined that the amount of change in a value received from a motion sensor (e.g., an IMU sensor, a GNSS sensor) is greater than a threshold.

[0237] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can, when a tracking event occurs (1701-Y), generate a tracking area including a bounding box in operation 1703 and obtain a bounding box representing a target based on inputting the tracking area into an AI model in operation 1705.

[0238] For example, as illustrated in FIG. 17 (a), an electronic device (e.g., a navigation support device (200)) can set a tracking area larger than the bounding box containing the bounding box and identify the bounding box based on the set tracking area.

[0239] Also, for example, as shown in FIG. 17 (b), an electronic device (e.g., navigation support device (200)) can move the position of the tracking area in the horizontal direction based on the heading direction of the vessel identified by a motion sensor (e.g., IMU sensor, GNSS sensor).

[0240] According to various embodiments, the size of the tracking area may be determined by information of the information acquisition device (210) associated with the bounding box based on profile information comparison described below. For example, the size of the tracking area may be set to be proportional to the speed of the radar information.

[0241]

[0242] 3.3. Object Information Display Operation Based on External Information

[0243] FIG. 19 is a flowchart illustrating an example of an operation for displaying external information-based target information of an electronic device (e.g., a navigation support device (200)) according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. FIG. 19 will be further described below with reference to FIG. 20 and FIG. 21. The operation of the navigation support device (200) can be understood as the operation of the control circuit (221).

[0244] FIG. 20 is a diagram illustrating an example of an operation to obtain instruction information indicating a target within camera information (2000) based on ECDIS information (2010) according to various embodiments. FIG. 21 is a diagram illustrating an example of an operation to obtain instruction information indicating a target within camera information (2000) based on a message (2100) received from the outside according to various embodiments.

[0245] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can obtain first information from the outside in operation 191901. For example, a navigation support device (200) (e.g., control circuit (1321) (e.g., control circuit (221) of FIG. 2)) can acquire information about a surrounding object (2003) associated with a vessel (e.g., the vessel) from the outside by using a communication device (211) and / or an electronic chart device (215) among the information acquisition devices (210). The information about the object (2003) may include detailed information about the object (2003), such as the location of the object (2003), as well as the type of the object (2003), the speed of the object (2003), the direction of movement of the object (2003), and descriptive information about the object (2003). The first information received from the outside may have high accuracy regarding the information about the object (2003) by comparing it with the image of the camera (213b). For example, the first information is compared with the image of the camera (213b). It can have relatively high accuracy by including more detailed information such as the type of object (2003).

[0246] For example, referring to FIG. 20, a navigation support device (200) (e.g., control circuit (221)) can obtain information about a target (2003) around a ship (210) among a plurality of graphic objects (2001, 2003) on an electronic chart (2010) using an electronic chart device (215) based on an integrated recognition module (230a). From the electronic chart (2010), detailed information about the type of the target (2003), as well as the location of the target (2003), can be obtained.

[0247] Also, as an example, referring to FIG. 21, a navigation support device (200) (e.g., control circuit (1321) (e.g., control circuit (220) of FIG. 2)) can obtain information about a target (2030) included in a message (2100) received from the outside (e.g., an external vessel) using a communication circuit (211) based on an integrated recognition module (1330a) (e.g., integrated recognition module (230a) of FIG. 3). The message (2100) may include location information (2110) about the target (2030) observed from the outside and content information (2120) for describing the target (2030).

[0248] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can acquire an image (2000) captured by a camera (213b) in operation 1903. Referring to FIGS. 20 and 21, the image (2000) may include a part (2001) of a vessel (210) and a target (2003).

[0249] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may obtain indication information for a specific area within the image (2000) based on at least some of the first information in operation 1905. For example, the navigation support device (200) (e.g., a control circuit (221)) may identify a specific pixel area displaying the object (2003) within the image (2000) based on information regarding the location of the object (2003) included in the first information, and obtain indication information (2005) including information for describing the object (2003) based on detailed information regarding the object (2003) included in the first information. The specific pixel area may correspond to the location of the object (2003) included in the first information.

[0250] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may control a display to show an image containing the instruction information in the specific area in 807. For example, as illustrated in FIGS. 20 to 21, the navigation support device (200) may control a display (251) to show instruction information (2005) at a location associated with an identified specific pixel area of ​​the image (2000). The instruction information (2005) may include at least some of the detailed information included in the first information (e.g., the type of object (2003), the speed of the object (2003), the direction of movement of the object (2003), and descriptive information about the object (2003). For example, if the object (2030) is a buoy, the instruction information (2005) may include descriptive information about the buoy (e.g., "stop entering the left area"). Accordingly, the navigator of the vessel (210) can recognize the target more accurately based on the instruction information (2005) displayed on the display (251).

[0251]

[0252] FIG. 22 is a flowchart for further illustrating an example of an operation to acquire first information of an electronic device (e.g., a navigation support device (200)) according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. FIG. 22 will be further described below with reference to FIG. 23. The operation of the navigation support device (200) can be understood as the operation of a control circuit (221).

[0253] FIG. 23 is a drawing for explaining an example of an operation to set a monitoring area (2300) from an electronic chart (2010) based on a conversion module (310) according to various embodiments.

[0254] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can identify a monitoring area (2300) on an electronic chart (2010) based on at least some of the camera information (C) in operation 2301. For example, referring to FIG. 23, the navigation support device (200) (e.g., a control circuit (1321) (e.g., a control circuit (221) of FIG. 2)) can establish a monitoring area (2300) on the electronic chart (2010) based on converting the coordinates of an image (2000) captured by a camera into coordinates on the electronic chart (2010) using a conversion module (1410) (e.g., a conversion module (310) of FIG. 3).

[0255] According to various embodiments, a navigation support device (200) (e.g., control circuit (1321) (e.g., control circuit (221) of FIG. 2)) may set a monitoring area (230) based on at least part of the operation of converting the coordinates of an image (2000) to coordinates on an electronic chart (2010), by converting the image (2000) into the format of radar information (R) based on first conversion information, and converting the image (2000) converted into the format of radar information (R) based on second conversion information into the format of an electronic chart (2010). As described above, each conversion information may include information for converting one type of detection information into another type of detection information. For example, the information for conversion may be a conversion matrix (or conversion function, or conversion vector). The conversion matrix may include information for a matrix (or function, or vector) for at least one of translation, rotation, scale, symmetry, or shear. At this time, the transformation matrix may include offset information to compensate for the placement state (e.g., position, angle) between detection devices corresponding to the transformation matrix (i.e., input and output relationship). That is, each of the first transformation information and the second transformation information includes a transformation matrix containing offset information, wherein the offset information included in the first transformation information is generated based on the difference between the placement state of the camera and the placement state of the electronic chart device, and the offset information included in the second transformation information may be generated based on the difference between the placement state of the camera and the placement state of the radar. In other words, the offset information may represent the difference in the placement state (e.g., position, angle) between detection devices.

[0256] For example, a control circuit (1321) (e.g., the control circuit (221) of FIG. 2) can convert a plurality of feature points (e.g., four vertices of the image (2000)) of an image (2000) into the form of radar information (R) based on the first conversion information to identify the distance and bearing angle corresponding to each of the plurality of feature points relative to the ship (210). The control circuit (221) can identify the monitoring area (2300) defined by each point by converting the distance and bearing angle corresponding to each of the plurality of feature points into a point of specific coordinates on the electronic chart (2010) based on the second conversion information.

[0257] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can obtain first information about an object included within a monitoring area (2300) identified in operation 2303. For example, when a specific graphic object is located within the monitoring area (2300), the navigation support device (200) (e.g., a control circuit (221)) can obtain the location of the specific graphic object and detailed information set on the specific graphic object (e.g., the type of target (2003), the speed of the target (2003), the direction of movement of the target (2003), and descriptive information about the target (2003) as first information.

[0258]

[0259] FIG. 24 is a flowchart for further illustrating an example of an operation to identify a specific area within an image of an electronic device (e.g., a navigation support device (200)) according to various embodiments. The operations may be performed regardless of the order of the operations shown and / or described, and more operations may be performed and / or fewer operations may be performed.

[0260] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire radar information based on at least some of the first information and the first conversion information in operation 2401. For example, the navigation support device (200) (e.g., a control circuit (221)) may convert the location of a target (2003) on an electronic chart (2010) based on the first conversion information to a specific distance and a specific bearing angle range from the center of the vessel (210) corresponding to the format of the radar information (R).

[0261] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can identify a specific area within an image based on radar information and a second transformation information in operation 2403. The navigation support device (200) (e.g., a control circuit (221)) can convert a specific distance and a specific bearing angle range from the center of a vessel (210) corresponding to the location of a target (2003) on an electronic chart (2010) based on the second transformation information into specific pixel coordinates within an image (2000) corresponding to the format of camera information (C). As described above in FIGS. 4 and 5, each transformation information may include at least one of a motion vector or a transformation matrix and offset information for compensating for the placement state (e.g., position, angle) between each detection device.

[0262] In one embodiment, a navigation support device (200) (e.g., a control circuit (221)) can identify an area corresponding to specific pixel coordinates as a specific area for the target (2003).

[0263] In another embodiment, the navigation support device (200) (e.g., control circuit (221)) may obtain a bounding box representing a target (2003) as a specific area by inputting an area corresponding to specific pixel coordinates into the AI ​​module (320a) described in FIG. 6. That is, a specific distance from the center of the vessel (210) and a specific bearing angle range corresponding to the position of the target (2003) on the electronic chart (2010) may be used for an attention action to more accurately recognize the target (2003) in the camera information (C).

[0264]

[0265] 3.4. Object Information Display Operation Based on In-Image Brightness Values

[0266] FIG. 25 is a flowchart illustrating an example of a target information display operation based on brightness values ​​within an image of an electronic device (e.g., a navigation support device (200)) according to various embodiments. The operations may be performed regardless of the order of the operations shown and / or described, and more operations may be performed and / or fewer operations may be performed. FIG. 25 will be further described below with reference to FIG. 26. The operation of the navigation support device (200) can be understood as the operation of the control circuit (221).

[0267] FIG. 26 is a drawing for explaining an example in which the brightness value of at least some part of an image (2600) is an abnormal range (or not a normal range) according to various embodiments.

[0268] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire an image (2600) captured by a camera (213b) in operation 2501 and acquire information regarding brightness values ​​of at least some area within the image (2600) in operation 2503. For example, the information regarding brightness values ​​may include at least one of information regarding brightness values ​​per pixel included in the image or contextual information associated with the brightness of the image.

[0269] In one embodiment, an electronic device (e.g., a navigation support device (200)) can monitor brightness values ​​for every pixel constituting the image (2600).

[0270] For example, an electronic device (e.g., a navigation support device (200)) can calculate a pixel-by-pixel brightness value based on the pixel-by-pixel RGB values ​​in the image (2600) and the [Equation 4] below.

[0271]

[0272]

[0273] Another example is that an electronic device (e.g., a navigation aid (200)) can identify pixel-by-pixel brightness values ​​based on converting the image (2600) into a different color model (e.g., HSV model, CIELab model).

[0274] Also, as another example, an electronic device (e.g., a navigation support device (200)) can identify pixel-by-pixel brightness values ​​based on inputting an image (2600) into a brightness value identification AI model based on Open CV, etc.

[0275] In another embodiment, an electronic device (e.g., a navigation support device (200)) can obtain context information related to brightness within an image (2600).

[0276] For example, an electronic device (e.g., a navigation support device (200)) may obtain information related to brightness within an image (2600) output from a language model based on inputting a text prompt to a language model to analyze the brightness conditions of at least one area (e.g., a partial area or the entire area) within the image (2600) together with the image (2600). For example, the information output from the language model may include text indicating the brightness conditions of at least one area within the image, such as "There is backlight in the lower left area of ​​the image" or "The entire image is dark and objects cannot be identified."

[0277] Also, as an example, an electronic device (e.g., a navigation support device (200)) can obtain information about objects affecting brightness based on inputting an image (2600) into an object identification AI model such as YoLo. For example, the objects may include objects that cause low-light conditions such as sea fog, cloud fog, etc., and objects that cause backlighting such as the sun.

[0278] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can determine whether the brightness values ​​of the pixels of an image in operation 2505 correspond to a normal range.

[0279] In one embodiment, when the information associated with the acquired brightness is a brightness value, the electronic device (e.g., navigation support device (200)) may determine that the brightness value of the image corresponds to a normal range if the brightness value of all pixels is below the threshold value, based on a comparison of a pre-stored threshold value and the brightness value per pixel. Additionally, the electronic device (e.g., navigation support device (200)) may determine that the brightness value of some pixels corresponds to an abnormal range (or is not a normal range) if the brightness value of some pixels is above the threshold value, and may identify the area of ​​pixels corresponding to the abnormal range.

[0280] In another embodiment, if the information associated with the acquired brightness is contextual information, the electronic device (e.g., navigation support device (200)) may determine that at least a portion of the image corresponds to an abnormal section when it is determined that the brightness value is high based on the contextual information. For example, the operation of determining that the brightness value is high based on the contextual information may include the operation of identifying keywords indicating high brightness values ​​from the texts included in the contextual information.

[0281] According to various embodiments, the situation awareness mode of an electronic device (e.g., a navigation support device (200)) may be set to a plurality of modes. For example, the plurality of modes may include a high visibility mode and a low visibility mode. The high visibility mode and the low visibility mode may be set according to the average value of brightness of an image captured by a camera (221b) and / or the illuminance value sensed by an illuminance sensor (not shown) of a vessel (210). The high visibility mode may be a situation awareness mode set when an average value of brightness or illuminance value greater than a specific value is identified, and the low visibility mode may be a situation awareness mode set when an average value of brightness or illuminance value less than a specific value is identified. Meanwhile, the situation awareness mode may be set according to a method other than using a camera (221b) or an illuminance sensor (not shown), not limited to the described examples, and for example, the high visibility mode may be set during the day and the low visibility mode may be set at night.

[0282] According to various embodiments, the threshold value for determining the normal range of the brightness values ​​for each pixel described above may be set differently for each of the plurality of modes. The threshold corresponding to the high visibility mode may be set higher than the threshold corresponding to the low visibility mode. However, not limited to the described examples, the thresholds for each mode may be the same or opposite.

[0283] FIG. 26 is a diagram showing examples of images (2600) including pixel areas of abnormal sections according to high visibility mode and low visibility mode. Referring to FIG. 26 (a), when set to high visibility mode, the image (2600) may include an area (2610a) having brightness values ​​of abnormal sections due to backlight (S). Accordingly, a target included in the area (2610a) may be unrecognizable. Also, referring to FIG. 26 (b), when set to low visibility mode, the image (2600) may include an area (2610b) having brightness values ​​of abnormal sections due to sea light output from a target on the sea. Accordingly, there is a high probability that a target is located in the area (2610b). Accordingly, when the abnormal section is detected for each mode, the method for recognizing the target may differ, which will be described later with reference to FIG. 27 to FIG. 30.

[0284] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may identify a target within an image (2600) based on an integrated recognition module (230a) in operation 2507 when the normal section (2505-e.) is in place, and provide information about the identified target in operation 2509. In one embodiment, the electronic device (e.g., a navigation support device (200)) may acquire information about the target (e.g., a bounding box) based on inputting the image (2600) into a vision AI model (620a), and display information about the target on the image (2600). In another embodiment, an electronic device (e.g., a navigation support device (200)) can acquire information about a target (e.g., a bounding box) based on inputting a portion of an image (2600) into a vision AI model (620a) based on information from another detection device (e.g., radar information (R)) in an early fusion manner as described in FIG. 6, and can display information about the target on the image (2600).

[0285]

[0286] FIG. 27 is a flowchart illustrating an example of operation in a first mode (e.g., high visibility mode) when the brightness value of an electronic device (e.g., navigation aid device (200)) is in an abnormal range, according to various embodiments. Operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. FIG. 27 is further described below with reference to FIG. 28. The operation of the navigation aid device (200) can be understood as the operation of the control circuit (221).

[0287] FIG. 28 is a diagram illustrating an example of an operation for recognizing an object within an area having an abnormal brightness based on images of different shutter speeds according to various embodiments.

[0288] According to various embodiments, an electronic device (e.g., navigation support device (200)) may acquire an additional image with reduced brightness in operation 2701 when an abnormal section is identified based on the result of comparing a threshold and a brightness value corresponding to a first mode (e.g., high visibility mode) (2505-No).

[0289] Referring to FIG. 28(a) in one embodiment, when an abnormal section is identified, an electronic device (e.g., navigation support device (200)) may acquire an additional image with reduced brightness using at least some of the plurality of cameras (2800a, 2800b). For example, while acquiring a first image using the first camera (2800a), if an abnormal section is identified, the electronic device (e.g., navigation support device (200)) may activate the second camera (2800b) and set the shutter speed of the second camera (2800b) to a value higher than the shutter speed of the first camera (2800a). Accordingly, the electronic device (e.g., navigation support device (200)) may acquire a second image having a lower brightness value than the first image using the second camera (2800b). Accordingly, based on the additional second image, an object in the area corresponding to the abnormal section may be visually recognized.

[0290] In another embodiment, referring to FIG. 28(b), an electronic device (e.g., navigation support device (200)) can acquire an additional image with reduced brightness by adjusting the shutter speed of the camera (2800). For example, while acquiring a first image using the camera (2800), if an abnormal section is identified, the electronic device (e.g., navigation support device (200)) can acquire a second image having a lower brightness value than the first image by increasing the shutter speed of the camera (2800). At this time, the electronic device (e.g., navigation support device (200)) can acquire the first image and the second image sequentially by acquiring images using the camera (2800) based on setting a first shutter speed at the first time and setting a second shutter speed lower than the first shutter speed at the second time. The first time and the second time may be repeated periodically. Accordingly, an object in the area corresponding to the abnormal section can be visually recognized based on the additional second image.

[0291] Although not described, an electronic device (e.g., navigation support device (200)) may acquire a second image with reduced brightness values ​​by software-wise reducing the brightness value of the first image (or the area corresponding to the abnormal section of the first image) when an abnormal section is identified while acquiring a first image using a camera (2800). The operation of reducing the brightness value may include at least one of an operation of adjusting the color value of a pixel based on a specific mathematical formula, an operation of converting to a different color model, or an operation using an AI model. Accordingly, an object in the area corresponding to the abnormal section can be visually recognized based on the additional second image.

[0292] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can identify a target within a portion of an additional image (e.g., a second image) (e.g., an area corresponding to an abnormal section) based on an integrated recognition module (1330a) (e.g., the integrated recognition module (230a) of FIG. 3) in operation 2703. In one embodiment, the electronic device (e.g., a navigation support device (200)) can obtain information (2800) (e.g., a bounding box) about the target based on inputting the additional image (e.g., the second image) into a vision AI model (620a). In another embodiment, an electronic device (e.g., a navigation support device (200)) can acquire information about a target (2800) (e.g., a bounding box) based on inputting a portion of an image (e.g., a second image) into a vision AI model (620a) based on information from another detection device (e.g., radar information (R)) in an early fusion manner, as described above in FIG. 6.

[0293] According to various embodiments, an electronic device (e.g., navigation support device (200)) may provide information about a target identified in operation 2705. For example, the electronic device (e.g., navigation support device (200)) may display an original image (2600) (e.g., a first image) containing an abnormal section on a display (251), and may display information about the target (2800) (e.g., a bounding box) on the original image (2600). Alternatively, the electronic device (e.g., navigation support device (200)) may merge an area corresponding to a normal section in the original image (e.g., a first image) with an area corresponding to an abnormal section in an additional image (e.g., a second image) to create a merged image, and display information about the target (2800) (e.g., a bounding box) while displaying the merged image on the display (251).

[0294] Meanwhile, instead of the operation of acquiring additional images with reduced brightness and providing information about the target as described in the example, the navigation support device (200) may be implemented to provide information about the target acquired based on information received from the outside as described above in “3.3 Table of Contents” on the image (2600).

[0295] For example, the navigation support device (200) can convert an area corresponding to an abnormal section into an area on an electronic chart based on conversion information, and obtain information about a target included within the converted area. The above operation can be performed similarly to the operation of setting a monitoring area (2300). The information about the acquired target can be converted into and displayed in the format of camera information (C), and since this has been explained in detail, a redundant explanation is omitted. Meanwhile, it is obvious to those skilled in the art that, in addition to the example using the described electronic chart, the operation may also be performed based on a message received from the outside.

[0296]

[0297] FIG. 29 is a flowchart illustrating an example of operation in a second mode (e.g., low visibility mode) when the brightness value of an electronic device (e.g., navigation aid device (200)) is in an abnormal range, according to various embodiments. Operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. FIG. 29 is further described below with reference to FIG. 30. The operation of the navigation aid device (200) can be understood as the operation of the control circuit (221).

[0298] FIG. 30a is a diagram illustrating an example of an operation for identifying information about a target (e.g., bounding box) using an AI module (320a) according to various embodiments. FIG. 30b is a diagram illustrating an example of an operation for preprocessing and displaying an identified abnormal section using a display module (230b) according to various embodiments.

[0299] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can identify an abnormal section (2905-No) based on the result of comparing a threshold and a brightness value corresponding to a second mode (e.g., a low visibility mode), acquire a specific area within the image based on an area having a brightness value greater than or equal to the threshold in operation 2901, and identify a target based on said specific area and an AI module (1720a) (e.g., the vision AI model (620a) of FIG. 6) in operation 2903.

[0300] In one embodiment, as illustrated in FIG. 30(a), a navigation support device (200) can acquire a portion of an image (2600) based on an area (R1) corresponding to an abnormal section within the image (2600), and acquire information about a target (e.g., bounding box) based on inputting the acquired portion of the image (2600) (3000a) into a vision AI model (1720a). For example, the portion of the image (3000a) may be cropped from the image (2600) to include the center point of the area (R1) corresponding to the abnormal section.

[0301] In another embodiment, as illustrated in FIG. 30(b), the navigation support device (200) can acquire a portion of the image (2600) (3000b) using a first attention module (1710a) (e.g., the first attention module (610a) of FIG. 6) based on an area (R1) corresponding to an abnormal section in the image (2600) and an area (R2) identified based on detection information (e.g., radar information (R)) of another detection device, and acquire information about a target (e.g., bounding box) based on inputting the acquired portion of the image (2600) (3000b) into a vision AI model (1720a) (e.g., the vision AI model (620a) of FIG. 6). For example, the above-mentioned partial area (3000a) can be cropped from the image (2600) to include both the center point of the area (R1) corresponding to the abnormal section and the center point of the area (R2).

[0302] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may provide information about a target identified in operation 2905. At this time, referring to FIG. 20, the electronic device (e.g., a navigation support device (200)) may use a display module (2000) to preprocess a portion of the area (3000a) corresponding to the abnormal section so as to be visually highlighted, and control the display of an image (2600) including the preprocessed portion of the area (3000a). The meaning of being visually highlighted may be that it is perceived differently compared to an area other than the portion of the area (3000a). For example, it may be displayed brighter. The preprocessing module (3100) within the display module (1330b) may be implemented to perform preprocessing operations such as adjusting the gamma value for the portion of the area (3000a), adjusting the color value according to a preset mathematical formula, or adjusting the color value using an AI model.

[0303] Meanwhile, instead of providing information about a target based on inputting a portion of the image corresponding to the abnormal section into the AI ​​module (320a) as described in the example, the navigation support device (200) may be implemented to provide information about a target obtained based on information received from the outside on the image (2600) as described above in "3.3 Table of Contents".

[0304] For example, the navigation support device (200) can convert an area corresponding to an abnormal section into an area on an electronic chart based on conversion information, and obtain information about a target included within the converted area. The above operation can be performed similarly to the operation of setting a monitoring area (2300). The information about the acquired target can be converted into and displayed in the format of camera information (C), and since this has been explained in detail, a redundant explanation is omitted. Meanwhile, it is obvious to those skilled in the art that, in addition to the example using the described electronic chart, the operation may also be performed based on a message received from the outside.

[0305]

[0306] 4.1 Object Location and Object Information Display Operation

[0307] FIG. 31 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device (200)) for performing a target location and target information display operation according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the order illustrated and / or described. FIG. 31 will be further described below with reference to FIG. 32.

[0308] FIG. 32 is a drawing showing an example of an image (I) (or screen) including target location and target information according to various embodiments.

[0309] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can acquire an image using a camera (213b) in operation 3101.

[0310] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire first position information indicating a target and a target using another marine sensor (e.g., an information acquisition device (210) of FIG. 2) in operation 3103. For example, referring to FIG. 4 above, the navigation support device (200) (e.g., a control device (220)) may acquire information from other information acquisition devices (210), such as radar information (R), lidar information (L), and ECDIS information (E), in addition to camera information (C) acquired by the camera (213b), and redundant descriptions are omitted.

[0311] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can convert a first position information into a second position information in the coordinate system of a camera based on at least one conversion information in operation 3105. For example, referring to FIG. 32, the navigation support device (200) (e.g., a control device (220)) can convert information of another information acquisition device (210) (e.g., radar information (R), lidar information (L), ECDIS information (E)) into information representing a specific position in the coordinate system of a camera (213b) based on conversion information (e.g., first to third conversion information) that is pre-stored in a conversion DB (420) by using a conversion module (2610) (e.g., a conversion module (310) of FIG. 3) of the aforementioned integrated recognition module (2530a) (e.g., an integrated recognition module (230a) of FIG. 3). Information in the coordinate system of the camera (213b) may refer to the coordinates (e.g., (x, y)) of a pixel on an image captured by the camera (213b). The situation awareness module (2620) of FIG. 32 may be implemented as the situation awareness module (320) of FIG. 3, and the display module (2530b) may be implemented as the display module (230b) of FIG. 3.

[0312] According to various embodiments, a navigation support device (200) (e.g., a control device (220)) can use the aforementioned integrated recognition module (230a) (e.g., a conversion module (310)) to convert information in a two-dimensional plane, such as ECDIS information (E), into information in a radar coordinate system, such as radar information (R), based on first conversion information; convert information in a radar coordinate system into information in a lidar coordinate system, such as lidar information (L), based on second conversion information; and convert information in a lidar coordinate system into information in a camera coordinate system, such as camera information (C) (e.g., an image), based on third conversion information. That is, based on the first to third conversion information, the information of each information acquisition device (210) can be sequentially converted into information in a different coordinate system based on different conversion information. In this case, in the case of the first to second conversion information, offset information based on the position difference of the information acquisition device (210) installed on the ship may be further reflected, which will be described in detail later.

[0313] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can obtain the location of a target (e.g., a bounding box) based on inputting a portion of an image corresponding to the second location information in operation 3107 into a vision AI (e.g., a vision AI model (620a) of FIG. 6).

[0314] According to various embodiments, an electronic device (e.g., navigation support device (200)) can obtain information about a target associated with the location of the target (e.g., bounding box) in operation 3109. For example, the information about the target may include various types of information to describe the target around the vessel to assist in the operation of the vessel on which the navigation support device (200) is installed.

[0315] For example, information regarding the above target may include positional / geometric information such as the distance from the vessel to the target (e.g., relative distance), the bearing to the target, latitude / longitude, and height (e.g., mast height).

[0316] Also, for example, the information regarding the above target may include motion / dynamic information regarding the target's speed (e.g., SOG), course / heading (COG), yaw rate, acceleration, predicted trajectory, predicted position, approach / deviation, etc.

[0317] For example, information regarding the above-mentioned object may include identification / classification information such as type (ship / buoy / bridge pillar / reef, etc.), detailed vessel type (merchant vessel / fishing vessel / tugboat, etc.), AIS information (MMSI, IMO, call sign, vessel name), type, color, and lighting pattern of navigation lights / beacons, and navigation status (operation / anchored / moored, etc.).

[0318] For example, the information regarding the above target may include collision risk information such as CPA / DCPA, TCPA, relative port / starboard crossing / head-on / overtaking situations (COLREGs classification), risk index / alert level, whether the safe area (vessel domain) has been violated, and recommended avoidance vectors.

[0319] In addition, for example, information regarding the above-mentioned object may include electronic chart / hydrographic linkage information such as the distance to the channel centerline, the distance to prohibited / restricted waters, and the relationship with port facilities, underwater reefs, and depth lines.

[0320] For example, the information regarding the above-mentioned object may include environmental information such as visibility around the object, wave height, wind direction / wind speed, current (flow speed / direction), sea fog, precipitation, etc.

[0321] For example, information regarding the above target may include detection reliability (score), tracking ID and tracking reliability, type of sensor used (camera / radar / lidar / AIS, etc.) and weight, timestamp and delay time, coordinate transformation / alignment error estimates, etc.

[0322] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can obtain information about a target by obtaining information about the target in which location information (e.g., a bounding box) is identified among information obtained by a detection device other than the camera (213b), which will be described in detail later.

[0323] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may display, in operation 3111, a graphic object indicating the location of a target and an execution screen containing information about the target.

[0324]

[0325] 4.1.1 Acquisition of Object Information Associated with Object Location

[0326] FIG. 33 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device (200)) for acquiring target information associated with a target location, according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the order illustrated and / or described. FIG. 33 will be further described below with reference to FIG. 34 and FIG. 35.

[0327] FIG. 34 is a diagram illustrating an example of an operation to extract information of an information acquisition device (440) based on a comparison of pattern information according to various embodiments. FIG. 35 is a diagram illustrating an example of an operation to extract information of an information acquisition device (440) based on a comparison of pattern information according to various embodiments.

[0328] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire first motion profile information associated with the position (e.g., bounding box) of an object in an image (IF) in operation 3301, and acquire second motion profile information associated with information of at least some of a plurality of marine sensors in operation 3303. For example, the motion profile information may represent the value of a specific attribute associated with the movement of the object acquired sequentially (or continuously) over time. For example, the specific attribute may include physical quantities such as the distance between the vessel and the object, position information of the specific object in a specific coordinate system (e.g., camera coordinate system, radar coordinate system), the direction of movement of the specific object, the speed of the specific object, etc. As the amount of time change of a specific attribute for a specific target identified by each of the plurality of images (IF) acquired by the camera (213b) is compared with the amount of time change of a specific attribute for each of the plurality of targets identified by another maritime sensor (e.g., another information acquisition device (3400a) (e.g., another information acquisition device (210)), information about the target acquired by another maritime sensor (e.g., another information acquisition device (210) such as radar (213a), lidar (213c), electronic chart device (215)) associated with the target captured by the camera (213b) can be acquired.

[0329] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may perform an operation of analyzing a characteristic (e.g., a physical quantity such as distance from the vessel) representing the target over time in each of the different coordinate systems as at least part of the operation of acquiring the first motion profile information and the second motion profile information.

[0330] For example, an electronic device (e.g., a navigation support device (200)) can obtain a change in the characteristics of a target over time identified in a camera coordinate system based on analyzing at least a portion of the pixel information (e.g., the center point (C1) of the bounding box) contained in a bounding box representing a specific target of each of the plurality of images (IF) using a first motion profile acquisition module (3400b) as at least part of the operation of acquiring first motion profile information. For example, a change in distance (3510a) can be calculated as the distance between the vessel and the target for each of the plurality of images (IF) is calculated based on the number (or size) of pixels contained in the bounding box of each of the plurality of images (IF). In addition, as another example, the amount of change in the direction of movement of the object is calculated based on the direction of movement of the coordinates of the feature points (e.g., center point, corner) included in each of the plurality of images (IF), and the amount of change in the speed of the object (3520b) can be calculated based on the amount of movement of the feature points (e.g., center point, corner).

[0331] Also, for example, an electronic device (e.g., a navigation support device (200)) can calculate a change amount (3510b, 3520b) of the characteristics of the aforementioned object (e.g., physical quantities such as the distance between the vessel and the object, the direction of movement of the object, the speed of the object, etc.) on a coordinate system associated with each, based on at least a portion of the information (e.g., a part of the echo signal for a specific object among the radar information) obtained from each of the information acquisition devices (210), such as a radar (213a), a lidar (213c), and an electronic chart device (215), using a second motion profile acquisition module (3400a) as at least part of the operation of acquiring second motion profile information.

[0332] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may perform an operation of analyzing a characteristic (e.g., a physical quantity such as distance from a vessel) representing the target over time in the same coordinate system as at least part of the operation of acquiring the first motion profile information and the second motion profile information.

[0333] In one embodiment, the same coordinate system may be a radar coordinate system.

[0334] For example, an electronic device (e.g., a navigation support device (200)) can convert position information (e.g., pixel coordinates) in a camera coordinate system of a part of a bounding box (e.g., center point, corner, etc.) representing a specific target of each of a plurality of images (IF) into position information (e.g., bearing angle, distance) in a radar coordinate system based on conversion information, and obtain a change amount for the aforementioned physical quantity (e.g., physical quantity such as distance from the vessel) based on the converted position information.

[0335] For example, an electronic device (e.g., a navigation support device (200)) can also convert information from another information acquisition device (440) (e.g., radar information (R), lidar information (L), ECDIS information (E)) into position information in a radar coordinate system (e.g., bearing angle, distance) based on the converted information, and obtain a change amount for the aforementioned physical quantity (e.g., physical quantity such as distance from the vessel) based on the converted position information.

[0336] Meanwhile, in addition to the embodiment of converting to the described radar coordinate system, an operation of converting to the coordinate system of various information acquisition devices (440) may be performed, and the above-described operation may be applied.

[0337] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may reflect information about a motion sensor as at least part of the operation of acquiring first motion profile information. For example, referring to FIG. 34, the navigation support device (200) may generate more accurate first motion profile information by using a movement information reflection module (3410) to reflect the movement information of the vessel acquired by the motion sensor (447) in the aforementioned first motion profile information.

[0338] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may further reflect depth information as at least part of the operation of acquiring first motion profile information. For example, referring to FIG. 34, the navigation support device (200) may generate first motion profile information having a more accurate size (or scale) by using a depth value reflection module (3420) to reflect depth information for a specific target acquired by another device (e.g., a depth sensor (not shown), a CNN-based depth estimation AI model) in the aforementioned first motion profile information.

[0339] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire marine information of at least some of a plurality of marine sensors based on a comparison of first motion profile information and second motion profile information in operation 3305. For example, the navigation support device (200) (e.g., a control device (220)) may use a motion profile information comparison module (3430) to acquire specific second motion profile information having a similarity greater than a threshold (or a difference value less than a threshold) based on the result of comparing first motion profile information and second motion profile information acquired in different coordinate systems and / or the same coordinate system as described above. The comparison may include a comparison of the shape of each motion profile. The navigation support device (200) (e.g., a control device (220)) may acquire information about the aforementioned target (e.g., distance, speed, etc.) by identifying information of an information acquisition device (440) corresponding to the specific second motion profile information and analyzing the identified information.

[0340] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can generate a plurality of first motion profiles corresponding to a plurality of types of physical quantities, and by comparing a second motion profile obtained by another information acquisition device (440) having a type of physical quantity corresponding to each of the plurality of first motion profiles, identify a specific information acquisition device (440) having the second motion profile with the highest degree of similarity, and obtain information representing the second motion profile obtained by the specific information acquisition device (440).

[0341] Alternatively, to reduce the operational burden of the comparison operation, not limited to the examples described, according to various embodiments, an electronic device (e.g., a navigation support device (200)) may select a second motion profile of a specific type of information acquisition device to be compared among a plurality of information acquisition devices (440) based on the type of physical quantity corresponding to the first motion profile, and perform a comparison operation. For example, if the first motion profile is a motion profile for speed, a motion profile corresponding to information from radar (R) and / or lidar (L) may be selected as the second motion profile to be compared. For another example, if the first motion profile is a motion profile for speed, a motion profile corresponding to information from radar (R) and / or lidar (L) may be selected as the second motion profile to be compared.

[0342] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may generate a plurality of first motion profiles corresponding to a plurality of types of physical quantities as at least part of an operation of identifying a type of physical quantity corresponding to a first motion profile, and select a first motion profile corresponding to a specific type of physical quantity having the highest accuracy. For example, the accuracy of the first motion profile may be calculated based on feature information of the motion profile. For example, the feature information may include the minimum magnitude of a value included in the motion profile, the maximum magnitude of a value, and / or the amount of change. Meanwhile, not limited to the examples described, the operation of generating the motion profile information may be generated based on the coordinate system of an information acquisition device (210) other than the camera (213b). In other words, the motion profile information may be generated based on the coordinate system of a specific information acquisition device (210).

[0343]

[0344] 4.1.2 Acquisition of Object Information Based on Coordinate System Transformation

[0345] FIG. 36 is a flowchart illustrating an example of a method of operation of an electronic device (e.g., a navigation support device (200)) for acquiring coordinate system transformation-based target information according to various embodiments. More and / or fewer operations may be performed, not limited to the examples shown and / or described, and operations may be performed in a different order than the order shown and / or described.

[0346] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may convert information regarding the location of a target in operation 3601 into location information having coordinate systems for a plurality of marine sensors, and in operation 3603, obtain first information associated with the location information among the information of at least some of the plurality of marine sensors. For example, when a bounding box is generated, the navigation support device (200) (e.g., a control device (220)) may convert the coordinates of the camera coordinate system defining the bounding box into information in the coordinate system of another type of detection device. By obtaining information of the detection device associated with the location information defined based on the information of the coordinate system of the other type of detection device, the navigation support device (200) (e.g., a control device (220)) may reduce the operational burden of generating motion profile information.

[0347] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can obtain motion profile information of the first information in operation 3605.

[0348]

[0349] 4.2. Example of Displaying Target Information

[0350] 4.2.1. Information Intensity / Magnitude Display of Detector Device

[0351] FIG. 37 is a flowchart illustrating an example of the operation method of an electronic device (e.g., navigation support device (200)) for displaying information intensity / size of a detection device as target information, according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the order illustrated and / or described. FIG. 37 will be further described below with reference to FIG. 38.

[0352] FIG. 38 is a drawing showing an example of information intensity / size of a detection device according to various embodiments.

[0353] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may display an execution screen comprising a first graphic object indicating the location of a target in operation 3701 and a second graphic object indicating the intensity and / or size of the marine information. In other words, the visual attributes (e.g., shape, size, color) of the second graphic object (E1, E2) may be determined according to the intensity and / or size of the marine information. For example, the intensity of the marine information may refer to the intensity of a signal identified by the information acquisition device (440).

[0354] In one embodiment, a navigation support device (200) (e.g., a control device (220)) acquires radar information (R) as specific second motion profile information having a similarity greater than the aforementioned threshold (or a difference value less than the threshold), and can identify the strength of an echo signal for a specific target by analyzing the radar information (R). The navigation support device (200) (e.g., a control device (220)) can display a second graphic object (E1, E2) representing the strength of the echo signal together with a first graphic object (e.g., a bounding box (B1, B2)) representing the location of the target. The second graphic object (E1, E2) representing the strength of the echo signal may have a geometric shape having a size proportional to the strength as shown in FIG. 38, but is not limited to the illustrated example and may have various shapes such as text representing the strength. Additionally, the second graphic object (E1, E2) may be displayed on the line defining the bounding box (B1, B2), but is not limited to the illustrated example and may be displayed at a location associated with the bounding box (B1, B2) (e.g., a location within a pre-set distance, an adjacent location).

[0355] Referring to FIG. 38 (a) and (b), the second graphic object (E1, E2) can be used as the reliability of the object identified by the vision AI model (620a) by verifying the intensity of the object identified by the vision AI model (620a) in a low-light environment (e.g., an environment with an illuminance value below a threshold) based on the second graphic object (E1, E2) by other marine sensor information.

[0356] In another embodiment, the navigation support device (200) (e.g., control device (220)) may obtain the density of a point cloud of lidar information (L) in addition to radar information (R) as specific second motion profile information having a similarity greater than the aforementioned threshold (or a difference value less than the threshold), and redundant descriptions thereafter are omitted.

[0357]

[0358] 4.2.2. Display of Object Distance and Direction Information

[0359] FIG. 39 is a flowchart illustrating an example of the operation of an electronic device (e.g., a navigation aid device (200)) for displaying distance and direction as target information according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the order illustrated and / or described. FIG. 39 will be further described below with reference to FIG. 40.

[0360] FIG. 40 is a drawing showing an example in which the direction and distance of a target are displayed according to various embodiments.

[0361] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may display an execution screen comprising a first graphic object indicating the position of a target in operation 3901 and at least one second graphic object indicating at least one of the speed of the target, the direction of the target, or the speed of the target obtained by a sea sensor.

[0362] For example, a navigation support device (200) (e.g., a control device (220)) acquires radar information (R) as specific second motion profile information having a similarity greater than the aforementioned threshold (or a difference value less than the threshold), and can identify at least one of the distance, direction, or speed of a specific target by analyzing the radar information (R). The navigation support device (200) (e.g., a control device (220)) can display a second graphic object (4001, 1703) representing at least one of the distance, direction, or speed together with a first graphic object (e.g., a bounding box (B)) representing the location of the target. In other words, the visual attributes (e.g., shape, size, color) of the second graphic object can be determined according to at least one of the distance, direction, or speed.

[0363] For example, a graphic object (4001) indicating direction and speed may be implemented in the shape of an arrow, whereby the direction of the arrow indicates the direction of the target and the size of the arrow indicates the magnitude of the speed. The graphic object (4001) indicating the speed may be provided at a location associated with the bounding box (B) (e.g., an adjacent location), so that the direction of the target and the magnitude of the speed indicated by the bounding box (B) can be intuitively recognized. Meanwhile, the shape of the graphic object (4001) indicating the speed may be implemented in various shapes, such as a bar shape or an elliptical shape, in addition to the arrow shape.

[0364] In addition, as another example, the graphic object (4003) representing the distance may include text representing the distance displayed on the graphic object, along with a graphic object (e.g., a line, a dotted line, a dashed line, etc.) connecting the vessel to a corresponding bounding box.

[0365] For example, a navigation support device (200) (e.g., a control device (220)) may acquire LiDAR information (L) and / or ECDIS information (E) as specific second motion profile information having a similarity greater than the aforementioned threshold (or a difference value less than the threshold), and may identify at least one of the distance, direction, or speed of a specific target based on the analysis of said information.

[0366]

[0367] 4.2.3. Display Wake Information

[0368] FIG. 41 is a flowchart illustrating an example of the operation of an electronic device (e.g., a navigation aid device (200)) for displaying distance and direction as target information according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the order illustrated and / or described. FIG. 41 will be further described below with reference to FIG. 42.

[0369] FIG. 42 is a drawing showing an example of a wake being displayed around a target according to various embodiments.

[0370] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may generate a second graphic object representing a wake of a target based on sea information in operation 4101. For example, the second graphic object may be implemented to have a shape representing the direction and / or size of the wake. The navigation support device (200) (e.g., a control device (220)) may acquire information about a target to determine the attributes of the wake based on information from an information acquisition device (440) acquired as second motion profile information (e.g., radar information (R), lidar information (L), ECDIS information (E), etc.), and may determine the shape of the second graphic object according to the determined attributes of the wake.

[0371] For example, the attributes of the wake may include the wake's starting point and the wake's direction.

[0372] For example, the starting point of the wake may be determined as a specific point within the bounding box. The specific point may be determined based on information from the other information acquisition device (440) described above. In one embodiment, the bounding box is divided into a plurality of regions, and among the divided regions, a specific region may be selected according to information acquired by the other information acquisition device (440) (e.g., direction and speed of movement of the table). The selected specific region may be determined as the starting point of the wake. In another embodiment, a location within the bounding box determined according to information acquired by the other information acquisition device (440) (e.g., direction and speed of movement of the table) based on a specific point (e.g., center point) within the bounding box may be determined as the starting point of the wake.

[0373] For example, the direction of the wake may be set opposite to the direction of the bow of the target, and the size of the wake may be set proportionally to the speed of the target, but is not limited to the examples described.

[0374] According to various embodiments, the navigation support device (200) can determine the display attributes (e.g., the starting point of the object, and the direction of the object) of a second object (W) representing a wake based on a first graphic object (e.g., a bounding box (B)) representing the location of the target according to the attributes of the determined wake (e.g., a starting point, and a direction).

[0375] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may display a screen including a first graphic object and a second graphic object indicating the location of a target in operation 4103. For example, as illustrated in FIG. 42, the navigation support device (200) (e.g., a control device (220)) may display a second graphic object (W) corresponding to the wake at a location associated with the first graphic object (e.g., a bounding box (B)).

[0376] In one embodiment, the navigation support device (200) (e.g., control device (220)) may be positioned such that the starting point of the second graphic object is placed on the center point of the bounding box (B). The starting point may represent the earliest point (e.g., end, or point where the x value is 0) among the parts of the second graphic object in the direction of the aforementioned wake (e.g., x-axis direction).

[0377] In another embodiment, the navigation support device (200) (e.g., control device (220)) may be positioned so that the starting point of the second graphic object is placed on a line defining the bounding box (B).

[0378] In addition, not limited to the examples described, the starting point of the second graphic object may be placed in a specific area within the bounding box (B) or in an adjacent area outside the bounding box (B).

[0379]

[0380] 4.3. Display of Object Information Based on Electronic Charts

[0381] 4.3.1. S-100 Information Selection-Based Target Information Display

[0382] FIG. 43 is a flowchart illustrating an example of the operation method of an electronic device (e.g., navigation support device (200)) for displaying S-100 information selection-based target information according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the illustrated and / or described order. FIG. 43 will be further described below with reference to FIG. 44.

[0383] FIG. 44 is a drawing for illustrating an example in which target location and target information are displayed according to the selection of a specific category among a plurality of categories associated with S-100 according to various embodiments.

[0384] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can acquire an image using a camera (213b) in operation 4301.

[0385] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may obtain a selection for a specific category among a plurality of categories associated with S-100 in operation 4303. The S-100 information may be marine and navigation data created according to the next-generation electronic chart common data model (S-100 Universal Hydrographic Data Model) established by the International Hydrographic Organization (IHO). The above S-100 information may include, as subcategories, S-101 information indicating buoys, obstacles, routes, etc.; S-104 information indicating time-varying water levels such as water level / tide; S-111 information indicating the direction and velocity of surface currents; S-122 information indicating the boundaries and management attributes of Marine Protected Areas (MPAs); S-123 information indicating maritime radio communication services (channels, call signs, operating hours, etc.); S-124 information indicating Navigational Warnings messages; S-129 information indicating Under Keel Clearance management parameters and guidance; S-421 (Route Exchange) information for the exchange of route plans, waypoints, constraints, etc., and may include various information not limited to the examples described.

[0386] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may provide a menu screen for selecting subcategories in succession. For example, referring to FIG. 44, when a category representing S-102 among a plurality of categories is selected as shown in FIG. 44 (a), the electronic device (e.g., a navigation support device (200)) may provide a screen including at least one subcategory included in S-102 (e.g., water depth, current direction, etc.) to receive a selection of a specific subcategory from the user.

[0387] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire first position information for a specific category in operation 4305 and convert the first position information into second position information in the camera coordinate system based on at least one conversion information in operation 4307. For example, the navigation support device (200) (e.g., a control device (220)) may acquire information for a specific category selected from an electronic chart device (445) and convert the acquired information. For example, referring to FIG. 35 (a), the navigation support device (200) (e.g., a control device (220)) may convert information representing water depth into coordinates of the camera coordinate system. For example, referring to FIG. 35 (b), the navigation support device (200) (e.g., a control device (220)) may convert information representing a buoy into coordinates of the camera coordinate system.

[0388] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may acquire the location of a target based on inputting a portion of an image corresponding to the second location information into a vision AI in operation 4309, and may acquire information about the target associated with the location of the target in operation 4311. For example, the navigation support device (200) (e.g., a control device (220)) may acquire information about a selected category (e.g., text indicating water depth, description of a buoy) from an electronic chart device (445).

[0389] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may display an execution screen containing a graphic object indicating the location of a target and information about the target in operation 4313. For example, referring to FIG. 35 (a), the navigation support device (200) (e.g., a control device (220)) may display text indicating the depth at a corresponding location on the image (I) based on the coordinates of the camera coordinate system. For example, referring to FIG. 35 (b), the navigation support device (200) (e.g., a control device (220)) may display a description of the buoy at a location associated with the bounding box, along with a bounding box indicating the location of the buoy.

[0390]

[0391] 4.3.2. Fish Farm Marking

[0392] FIG. 45 is a flowchart illustrating an example of the operation of an electronic device for indicating a fish farm (e.g., a navigation support device (200)) according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the illustrated and / or described order. FIG. 45 will be further described below with reference to FIG. 46.

[0393] FIG. 46 is a drawing for explaining an example of an operation to modify and display the location of a fish farm according to various embodiments.

[0394] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can identify second-1 position information (e.g., at least one pixel coordinate) of a dynamic object in the camera coordinate system in operation 4501. For example, the dynamic object is a type of object whose position may change over time at sea and may include buoys, aquaculture farms, marine organisms, etc.

[0395] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can identify second-second position information in the camera coordinate system of a target associated with a dynamic target in operation 4503, and determine in operation 4505 whether the difference between second-first position information and second-second position information is greater than a threshold.

[0396] For example, as illustrated in FIG. 46, if the dynamic target is a fish farm, there may be at least one buoy defining the location of the fish farm. An electronic device (e.g., a navigation support device (200)) can generate a plurality of bounding boxes by inputting an image (I) captured by a camera (213b) into a vision AI model (620b), and can identify at least one bounding box among the plurality of bounding boxes that is adjacent to the aforementioned 2-1 location information (e.g., having a distance difference of less than a threshold). The electronic device (e.g., a navigation support device (200)) can generate 2-2 location information (e.g., at least one pixel coordinate) defining the fish farm based on the at least one bounding box, and can compare the 2-1 location information with the 2-2 location information.

[0397] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may display a graphic object corresponding to the second-second position information in operation 4507 when the difference is greater than a threshold (4505-Y).

[0398]

[0399] 4.4. Display of target information based on whether the detection device identifies the target

[0400] FIG. 47 is a flowchart illustrating an example of the operation method of an electronic device (e.g., a navigation support device (200)) that displays target information depending on whether a detection device identifies a target, according to various embodiments. More and / or fewer operations may be performed, not limited to the examples illustrated and / or described, and operations may be performed in a different order than the order illustrated and / or described. FIG. 4547 will be described further below with reference to FIG. 4648.

[0401] FIG. 48 is a drawing for illustrating various object information display examples according to various embodiments.

[0402] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can acquire an image using a camera (213b) in operation 4701.

[0403] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can identify a target using another marine sensor (e.g., an information acquisition device (210) of FIG. 2) in operation 4703 and determine whether the target is identified in operation 4705.

[0404] According to various embodiments, an electronic device (e.g., a navigation support device (200)) can, when a target is identified (4703-Y), convert a first position information into a second position information in the coordinate system of a camera based on at least one conversion information (e.g., the first to third conversion information of FIG. 32) in operation 4707, input a portion of an image corresponding to the second position information into a vision AI (e.g., the vision AI model (620b) of FIG. 6) in operation 4709, and determine whether the target is identified in operation 4713.

[0405] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may display a first screen containing first information about a target acquired based on a vision AI (e.g., the vision AI model (620b) of FIG. 6) in operation 4715 when a target is identified (4713-Y). For example, as illustrated in FIG. 48 (a), the navigation support device (200) (e.g., a control device (220)) may display information about a target acquired using another information acquisition device (210) along with a bounding box.

[0406] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may display a second screen containing second information about a target acquired based on second location information in operation 4717 when the target is unidentified (4713-N). For example, as illustrated in FIG. 48 (b), the navigation support device (200) (e.g., a control device (220)) may generate a bounding box indicating the location of a target based on second location information identified based on information acquired using another information acquisition device (210) (e.g., radar information (R), lidar information (L), ECDIS information (E)), and may display the generated bounding box along with information about the target acquired using the information acquisition device (210). At this time, information about the type of detection device used to recognize the location of the target (i.e., to generate the bounding box) may be provided on the displayed bounding box.

[0407] According to various embodiments, an electronic device (e.g., a navigation support device (200)) may input an image into a vision AI in operation 4711 when the target is non-identified (4703-N), determine whether the target is identified in operation 4719, and display a third screen containing second information about the target acquired based on the vision AI (e.g., the vision AI model (620b) of FIG. 6) in operation 4721 when the target is identified (4719-Y). In other words, based on the comparison of the aforementioned motion profile information, information about other detection devices associated with a specific target may not be identified. In this case, as illustrated in FIG. 48 (b), for example, the navigation support device (200) (e.g., a control device (220)) may display an image containing only the bounding box output from the vision AI model. At this time, information about the type of other undetected marine sensor may be provided on the bounding box.

Claims

1. As a method of operating an electronic device, The operation of acquiring an image using a camera; The operation of acquiring first position information indicating a target using at least one detection device other than the camera; wherein the at least one detection device includes at least one of a radar, lidar, electronic chart device, AIS device, or motion sensor. An operation of converting the first position information into second position information of the camera's coordinate system based on at least one conversion information; An operation to acquire information about the location of a target based on inputting a portion of the image corresponding to the second location information into the vision AI; and The operation of displaying a screen including at least one graphic object indicating the location of the above object; Method of operation.

2. In Paragraph 1, Each of the above at least one transformation information comprises a homogeneous transformation matrix including a rotation matrix and / or a translation vector, Method of operation.

3. In Paragraph 2, If the other at least one detection device is the radar, the at least one conversion information includes first conversion information associated with the radar and the lidar, and third conversion information associated with the lidar and the camera. Method of operation.

4. In Paragraph 2, A homogeneous transformation matrix including a rotation matrix and / or a translation vector included in each of the above at least one transformation information is generated based on offset information generated according to the relative pose and / or relative angle between the camera and the at least one detection device. Method of operation.

5. In Paragraph 4, An operation to acquire first offset information associated with the yawing direction of the radar and the motion sensor; The operation of acquiring second offset information associated with the yawing direction and third offset information associated with the pitching direction of the radar and lidar; An operation of obtaining fourth offset information associated with the rolling direction of the radar and the motion sensor and fifth offset information associated with the pitching direction based on the first offset information, second offset information, and third offset information; An operation to perform optimization for the first offset information, the second offset information, the third offset information, the fourth offset information, and the fifth offset information; and The operation of generating at least one conversion information reflecting at least one of the optimized first offset information, the second offset information, the third offset information, the fourth offset information, or the fifth offset information; further comprising Method of operation.

6. In claim 5, the operation of generating at least one transformation information is: The operation of generating fourth conversion information associated with the radar and the camera; The first position information is converted into the second position information based on the fourth conversion information, Method of operation.

7. In claim 5, the operation of acquiring the first offset information is: An operation of acquiring first bearing grid information of the yawing direction associated with the motion sensor based on bearing angle information of the radar; An operation of acquiring second bearing grid information associated with the radar based on the radar; and The operation of obtaining the first offset information based on the first bearing grid information and the second bearing grid information; Method of operation.

8. In claim 5, the operation of acquiring the second offset information is: An operation to acquire third bearing grid information of the yawing direction associated with the lidar based on the bearing angle information of the radar; An operation of acquiring fourth bearing grid information associated with the radar based on the radar; and The operation of obtaining the second offset information based on the third bearing grid information and the fourth bearing grid information; Method of operation.

9. In claim 8, the operation of acquiring the third offset information is: An operation of acquiring first information by recognizing an object whose height changes in front of the vessel using the above radar; The operation of acquiring second information associated with the object using the above lidar; and The operation of obtaining the third offset information based on the first information and the second information; Method of operation.

10. In claim 1, the operation of acquiring information about the location of a target based on inputting a portion of the image corresponding to the second location information into the vision AI is: An operation to identify a portion included in the first position information on the above image; An operation to identify a portion of the image including the above portion; and The operation of inputting information about a part of the above image into the vision AI; Method of operation.

11. In Paragraph 9, An action of identifying a tracking area larger than a portion of the above image; and The operation of tracking the position of the target based on inputting information about the tracking area of ​​the above image into the vision AI; further comprising Method of operation.

12. As an electronic device, A control device; comprising, and said control device: Acquire an image using a camera, and First position information indicating a target is obtained using at least one detection device other than the camera, and the at least one detection device includes at least one of a radar, lidar, electronic chart device, AIS device, or motion sensor. Based on at least one transformation information, the first position information is transformed into second position information of the camera's coordinate system, and Information regarding the location of a target is obtained based on inputting a portion of the image corresponding to the second location information into the vision AI, and A screen configured to display a screen including at least one graphic object indicating the location of the above object, Electronic device.

13. In Paragraph 12, Each of the above at least one transformation information comprises a homogeneous transformation matrix including a rotation matrix and / or a translation vector, Electronic device.

14. In Paragraph 13, If the other at least one detection device is the radar, the at least one conversion information includes first conversion information associated with the radar and the lidar, and third conversion information associated with the lidar and the camera. Electronic device.

15. In Paragraph 13, A homogeneous transformation matrix including a rotation matrix and / or a translation vector included in each of the above at least one transformation information is generated based on offset information generated according to the relative pose and / or relative angle between the camera and the at least one detection device. Electronic device.

Images