Method for generating improved visibility image of ship, and apparatus thereof

Abstract

One embodiment of the present invention relates to a method in which an image having an expanded viewing angle for the rear of a container ship is generated so that an improved visibility image of the container ship is generated, the method comprising the steps of: collecting images from a first camera, a second camera and a third camera of the container ship; extracting overlapping feature points between a second image and a third image collected from the second camera and the third camera, respectively; matching the second image and the third image on the basis of the extracted feature points so as to generate a target image; and integrating a first image collected from the first camera and the generated target image so as to generate an integrated image.

Description

Method for generating images of a vessel with improved visibility and device therefor

[0001] The present invention relates to a method for generating an image, and more particularly, to a method for generating an image of a ship with significantly improved visibility and a device for implementing the method.

[0002] A container ship is a cargo vessel capable of transporting large quantities of containers loaded with goods. Container ships play a crucial role in international trade and are a type of vessel developed for efficient cargo transport. Container ships primarily serve the purpose of transporting containers safely and quickly to their destinations.

[0003] Container ships can be categorized into cellular container ships, Ro-Ro container ships, and partial container ships. Cellular container ships feature fixed cells (a lattice structure) for loading containers, allowing them to be stacked vertically in the ship's holds and decks. Most large container ships typically follow the cellular container ship architecture, which offers the advantage of fast and efficient loading and unloading.

[0004] Ro-ro container ships are designed to load and unload containers directly onto the vessel, similar to vehicles, using a roll-on / roll-off system. Ro-ro container ships allow containers to be loaded and unloaded from trailers or trucks, simplifying dockside operations.

[0005] Partial containerships are designed not only to transport containers but also to carry general cargo. Partial containerships are multi-purpose vessels designed to carry containers in some areas and general cargo or bulk cargo in other areas.

[0006] Figures 1a and 1b are schematic drawings showing an example of a conventional container ship.

[0007] Fig. 1a is an example of the starboard side of a container ship. The container ship of Fig. 1a is considered a cellular container ship in which the wheelhouse (10) is arranged closer to the bow than the stern and only containers are loaded. The container ship of Fig. 1a has a bow container (200) loaded forward of the wheelhouse (10). Hereinafter, for convenience of explanation, the bow containers (200) are considered to be containers loaded between the bow of the container ship and the wheelhouse (10). As illustrated in Fig. 1a, in order to satisfy the visibility rule of container ships, conventional container ships tend to have lower loading heights of containers as they get closer to the bow of the container ship, which results in a tendency for the loading capacity to be recognized as low.

[0008] Fig. 1b is a drawing showing a container ship in plan view. Referring to Fig. 1b, if the forward container (200) is loaded high, a blind spot (blind spot) is created at a location beyond the viewing angle α in the wheelhouse (10), and there is a problem in that other ships or floating objects in the blind spot cannot be discovered or are discovered late.

[0009] When operating a vessel, visual observation of the outside world from within the wheelhouse is essential for safe navigation. The area outside the wheelhouse, specifically the bow, is often occupied by ship structures, deck cargo, and other equipment. These structures can sometimes limit the forward and side visibility from the wheelhouse.

[0010] Areas with limited visibility like this are called "blind sectors." These blind sectors can pose significant hazards, especially when navigating, anchoring, or berthing in narrow waters, and the International Maritime Organization (IMO) has regulations regarding them.

[0011] Previous attempts have been made to secure visibility by designing wheelhouses higher or adjusting the layout of structures in blind sections. However, structural constraints in ship design and loading efficiency have made it difficult to completely eliminate blind sections.

[0012] The background technology of the present invention is disclosed in Korean Patent Publication No. 10-1130658 (registered on March 20, 2012, title of invention: Liquefied natural gas storage tank carrier).

[0013] The technical problem to be solved by the present invention is to provide a method for generating an image of a ship with improved visibility and a device for implementing the method.

[0014] According to one embodiment of the present invention for solving the above technical problem, a method for generating an image of a ship with improved visibility comprises the steps of: collecting images from a first camera, a second camera, and a third camera of the ship; extracting overlapping feature points between second and third images collected from the second and third cameras; generating a target image by aligning the second and third images based on the extracted feature points; and generating an integrated image by integrating the first image collected from the first camera and the generated target image.

[0015] According to another embodiment of the present invention for solving the above technical problem, a device for generating an image of a ship with improved visibility comprises: a memory in which at least one program is stored; and a processor for performing a calculation by executing the at least one program, wherein the processor collects images from a first camera, a second camera, and a third camera of a container ship, extracts overlapping feature points between second images and third images collected from the second camera and the third camera, and generates a target image by matching the second images and the third images based on the extracted feature points, and generates an integrated image by integrating the first image collected from the first camera and the generated target image.

[0016] According to one embodiment of the present invention, a method for generating an image is provided, comprising: collecting images from a first camera, a second camera, and a third camera of a ship; extracting overlapping feature points between a second image and a third image collected from the second camera and the third camera; generating a target image by matching the second image and the third image based on the extracted feature points; generating an integrated image by integrating a first image collected from the first camera and the generated target image; and generating an image in which an obstacle area defined in the first image is replaced with a corresponding area in the target image.

[0017] According to one embodiment of the present invention, an image generating device is provided, comprising: a memory storing at least one program; a processor executing the at least one program to perform an association, wherein the processor collects images from a first camera, a second camera, and a third camera of a ship, extracts overlapping feature points between second images and third images collected from the second camera and the third camera, matches the second images and the third images based on the extracted feature points to generate a target image, integrates the first image collected from the first camera and the generated target image to generate an integrated image, and replaces an obstacle area defined in the first image with a corresponding area in the target image in the integrated image.

[0018] A method according to one embodiment of the present invention for solving the above technical problem includes the steps of collecting images from a first camera, a second camera, and a third camera of a ship; extracting overlapping feature points between second and third images collected from the second and third cameras; generating a target image by aligning the second and third images based on the extracted feature points; and generating an integrated image by integrating the first image collected from the first camera and the generated target image.

[0019] According to another embodiment of the present invention for solving the above technical problem, a device comprises: a memory in which at least one program is stored; and a processor for performing a calculation by executing the at least one program, wherein the processor collects images from a first camera, a second camera, and a third camera of a ship, extracts overlapping feature points between second images and third images collected from the second camera and the third camera, and generates a target image by aligning the second images and the third images based on the extracted feature points, and generates an integrated image by integrating the first image collected from the first camera and the generated target image.

[0020] In addition, according to one embodiment of the present invention, a method for generating an image is provided, including: a step of collecting images from a first camera, a second camera, and a third camera of a ship; a step of recognizing a position of a user within a wheelhouse of the ship; a step of assigning relative weights to the second camera and the third camera according to the recognized position of the user; a step of extracting overlapping feature points between a second image and a third image collected from the second camera and the third camera; a step of generating a target image by matching the second image and the third image based on the extracted feature points; and a step of generating an integrated image by integrating the first image collected from the first camera and the generated target image; wherein the integrated image is provided with an image generation method in which an obstacle area defined in the first image is replaced with a corresponding area in the target image.

[0021] In addition, according to one embodiment of the present invention, there is provided an image generating device including a memory having at least one program stored therein; a processor that performs an association by executing the at least one program, wherein the processor collects images from a first camera, a second camera, and a third camera of a ship, recognizes a position of a user within a wheelhouse of the ship, assigns relative weights to the second camera and the third camera according to the recognized position of the user, extracts overlapping feature points between the second and third images collected from the second and third cameras, matches the second and third images based on the extracted feature points to generate a target image, and generates an integrated image by integrating the first image collected from the first camera and the generated target image, and replaces an obstacle area defined in the first image with a corresponding area in the target image in the integrated image.

[0022] According to one embodiment of the present invention for solving the above technical problem, a method for generating a ship image having improved visibility of a blind spot of a shield in front of a wheelhouse comprises the steps of: collecting images from a first camera, a second camera, and a third camera of a ship; extracting overlapping feature points between second and third images collected from the second and third cameras; generating a target image by aligning the second and third images based on the extracted feature points; and generating an integrated image by integrating the first image collected from the first camera and the generated target image.

[0023] According to another embodiment of the present invention for solving the above technical problem, a device is provided for generating a ship image with improved visibility of a blind spot of a shield in front of a wheelhouse, the device comprising: a memory storing at least one program; and a processor performing a calculation by executing the at least one program, wherein the processor collects images from a first camera, a second camera, and a third camera of the ship, extracts overlapping feature points between the second images and the third images collected from the second camera and the third camera, and generates a target image by aligning the second images and the third images based on the extracted feature points, and generates an integrated image by integrating the first image collected from the first camera and the generated target image.

[0024] According to one embodiment of the present invention for solving the above technical problem, a method is provided for creating an image of a ship with improved visibility by creating an image with an expanded field of view toward the rear of the ship, the method comprising: collecting images from a first camera, a second camera, a third camera, and a fourth camera of the ship; extracting overlapping feature points between second images to fourth images collected from the second camera to the fourth camera; generating a target image by aligning the second images to the fourth images based on the extracted feature points; and generating an integrated image by integrating the first image collected from the first camera and the generated target image.

[0025] According to another embodiment of the present invention for solving the above technical problem, a device is provided, which generates an image by expanding the field of view toward the rear of a ship, thereby generating a ship image with improved visibility, the device comprising: a memory storing at least one program; and a processor performing a calculation by executing the at least one program, wherein the processor collects images from a first camera, a second camera, a third camera, and a fourth camera of the ship, extracts overlapping feature points between second images to fourth images collected from the second camera to the fourth camera, and generates a target image by aligning the second images to the fourth images based on the extracted feature points, and generates an integrated image by integrating the first image collected from the first camera and the generated target image.

[0026] According to one embodiment of the present invention for solving the above technical problem, a method for generating a monitoring image of a ship based on information collected from a plurality of sensors of the ship includes the steps of: collecting first type sensor information of a first type sensor installed in a first area of ​​the ship regardless of the type of the ship; collecting second type sensor information of a second type sensor installed in a second area of ​​the ship determined based on the type of the ship; identifying the type of the ship using the collected second type sensor information; and creating an integrated image including monitoring information of the identified type of ship based on the collected first type sensor information and the collected second type sensor information.

[0027] According to another embodiment of the present invention for solving the above technical problem, a device for generating a monitoring image of a ship based on information collected from a plurality of sensors of the ship, the device includes: a communication unit; a memory in which at least one program is stored; and a processor for performing a calculation by executing the at least one program, wherein the processor collects first type sensor information of a first type sensor installed in a first area of ​​the ship regardless of the type of the ship, collects second type sensor information of a second type sensor installed in a second area of ​​the ship determined based on the type of the ship, identifies the type of the ship based on the collected second type sensor information, and generates an integrated image including monitoring information of the identified type of ship based on the collected first type sensor information and the collected second type sensor information.

[0028] According to one embodiment of the present invention for solving the above technical problem, a method is provided for creating an image of a container ship with improved visibility by creating an image with an expanded field of view toward the rear of the container ship, the method comprising: collecting images from a first camera, a second camera, and a third camera of the container ship; extracting overlapping feature points between second and third images collected from the second and third cameras; generating a target image by aligning the second and third images based on the extracted feature points; and generating an integrated image by integrating the first image collected from the first camera and the generated target image.

[0029] According to another embodiment of the present invention for solving the above technical problem, a device is provided, which generates an image of a container ship with improved visibility by generating an image with an expanded field of view toward the rear of the container ship, the device comprising: a memory in which at least one program is stored; and a processor which performs a calculation by executing the at least one program, wherein the processor collects images from a first camera, a second camera, and a third camera of the container ship, extracts overlapping feature points between the second images and the third images collected from the second camera and the third camera, and generates a target image by matching the second images and the third images based on the extracted feature points, and generates an integrated image by integrating the first image collected from the first camera and the generated target image.

[0030] According to one embodiment of the present invention, a vessel including the image generating device is provided.

[0031] One embodiment of the present invention can provide a computer-readable recording medium storing a program for executing the above method.

[0032] According to the present invention, it is possible to manufacture a container ship capable of loading more containers in the bow section than a conventional container ship.

[0033] In addition, according to the present invention, an image with an expanded field of view for the front of a container ship can be created, thereby helping a navigator operating the ship to operate safely.

[0034] In addition, the image generation method according to one embodiment of the present invention has the effect of eliminating a blind section formed by a structure located in front of the wheelhouse by generating an integrated image in which an obstacle area defined in a first image acquired by a first camera placed adjacent to the wheelhouse of a ship is replaced with an area corresponding to the obstacle area in a target area generated by matching the second and third images acquired by the second and third cameras, respectively.

[0035] In addition, according to the present invention, when a user looks forward or backward from inside the wheelhouse, visibility is improved by eliminating blind spots formed by structures.

[0036] Figures 1a and 1b are schematic drawings showing an example of a conventional container ship.

[0037] FIG. 2a and FIG. 2b are schematic drawings showing the structure of an improved container ship that improves the structure of a conventional container ship in order to implement a method according to the present invention.

[0038] Figure 3 is a drawing exemplarily showing an improved structure of a container ship according to the present invention.

[0039] FIG. 4 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0040] FIG. 5 is a drawing for explaining a sub-module included in the processor of FIG. 4.

[0041] Figure 6 is a flowchart showing the process of generating an integrated image by an integrated image generation device, by unit process.

[0042] Figure 7 is a drawing showing an example of a first image generated by a first camera.

[0043] Figure 8 is a drawing showing an example of a second image generated by a second camera.

[0044] Figure 9 is a drawing showing an example of a third image generated by a third camera.

[0045] Figure 10 is a drawing showing an example of a fourth image generated by a fourth camera.

[0046] Figure 11 is a diagram illustrating an example of a fifth image generated by a fifth camera.

[0047] Figure 12 is a diagram schematically illustrating an example of a first image that can be generated by a first camera when the field of view of the first camera is secured up to the second front field of view.

[0048] Figure 13 is a drawing showing an example of the results of comparing a conventional first image and an integrated image according to the present invention by implementing them as two-dimensional images.

[0049] FIG. 14 is a flowchart illustrating an example of a method for generating an integrated image for a container ship according to the present invention.

[0050] FIG. 15 is a schematic drawing of a vessel including an image generating device according to one embodiment of the present invention.

[0051] Fig. 16 is a drawing partially showing the interior of the wheelhouse corresponding to part A of Fig. 15.

[0052] Figure 17 is an enlarged view of part B of Figure 16.

[0053] FIG. 18 is a drawing illustrating an image generating device according to one embodiment of the present invention.

[0054] FIG. 19 is a diagram illustrating a processor according to one embodiment of the present invention.

[0055] Figure 20 is a flowchart showing the process of generating an integrated image by an image generating device according to one embodiment of the present invention, by unit process.

[0056] Figure 21 is a flowchart illustrating steps for setting camera combinations and weights according to the user's location.

[0057] Figure 22 is a flowchart illustrating the steps for secondary correction of an integrated image.

[0058] FIG. 23 is a drawing showing a state in which an integrated image corresponding to an obstacle area is projected onto a window in a wheelhouse of a ship by a projection unit according to one embodiment of the present invention.

[0059] Figure 24 is a flowchart illustrating another embodiment of an image generation method for generating an integrated image according to the present invention.

[0060] FIG. 25 is a schematic drawing of a vessel including an image generating device according to one embodiment of the present invention.

[0061] Figure 26 is a drawing partially showing the interior of the wheelhouse corresponding to section A of Figure 25.

[0062] Figure 27 is an enlarged view of part B of Figure 26.

[0063] FIG. 28 is a drawing illustrating an image generating device according to one embodiment of the present invention.

[0064] FIG. 29 is a diagram illustrating a processor according to one embodiment of the present invention.

[0065] Figure 30 is a flowchart showing the process of generating an integrated image by an image generating device according to one embodiment of the present invention, by unit process.

[0066] Figure 31 is a flowchart illustrating steps for setting camera combinations and weights according to the user's location.

[0067] Figure 32 is a flowchart illustrating the steps for secondary correction of an integrated image.

[0068] FIG. 33 is a drawing showing a state in which an integrated image corresponding to an obstacle area is projected onto a window in a wheelhouse of a ship by a display unit according to one embodiment of the present invention.

[0069] Figure 34 is a flowchart illustrating another embodiment of an image generation method for generating an integrated image according to the present invention.

[0070] Figures 35 and 36 are schematic drawings showing a ship to which a conventional OCCS is applied.

[0071] Figure 37 is a diagram schematically showing a structure in which the structure of a ship to which a conventional OCCS is applied is improved in order to implement a method according to the present invention.

[0072] Figure 38 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0073] Figure 39 is a drawing for explaining a sub-module included in the processor of Figure 38.

[0074] Figure 40 is a flowchart showing the process of generating an integrated image by an integrated image generation device, by unit process.

[0075] Figure 41 is a drawing showing an example of an obstacle area being designated in the first image.

[0076] Figure 42 is a drawing showing a second image generated by a second camera as a three-dimensional image.

[0077] Figure 43 is a drawing showing a third image generated by a third camera as a three-dimensional image.

[0078] Figure 44 is a drawing showing an integrated image generated by an integrated image generation device as a three-dimensional image.

[0079] Figure 45 is a drawing showing an example of the results of comparing a conventional first image and an integrated image according to the present invention by implementing them as two-dimensional images.

[0080] Figure 46 is a drawing showing an example of a wind-powered vessel.

[0081] Figure 47 is a drawing showing a typical wind-powered ship by configuration.

[0082] Figures 48a and 48b are drawings for explaining the visibility of a wind-powered vessel.

[0083] Figure 49 is a diagram schematically showing a structure in which the structure of a conventional wind-powered ship is improved in order to implement a method according to the present invention.

[0084] Figure 50 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0085] Figure 51 is a drawing for explaining a sub-module included in the processor of Figure 50.

[0086] Figure 52 is a drawing showing the first image generated by the first camera as a three-dimensional image.

[0087] Figure 53 is a drawing showing a second image generated by a second camera as a three-dimensional image.

[0088] Figure 54 is a drawing showing a third image generated by a third camera as a three-dimensional image.

[0089] Figure 55 is a drawing showing the fourth image generated by the fourth camera as a three-dimensional image.

[0090] Figure 56 is a drawing showing an example of the results of comparing a conventional first image and an integrated image according to the present invention by implementing them as two-dimensional images.

[0091] Figure 57 is a drawing showing a wind-powered ship by configuration, as an example, with a funnel installed at the stern.

[0092] Figures 58a and 58b are drawings for explaining the visibility of a vessel equipped with a funnel.

[0093] Figure 59 is a drawing showing a plurality of cameras arranged on the stern side to implement a method according to the present invention.

[0094] Figure 60 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0095] Figure 61 is a drawing for explaining a sub-module included in the processor of Figure 4.

[0096] Figure 62 is a flowchart showing the process of generating an integrated image by an integrated image generation device, by unit process.

[0097] Figure 63 is a drawing showing an example of a first image generated by a first camera.

[0098] Figure 64 is a drawing showing an example of a second image generated by a second camera.

[0099] Figure 65 is a drawing showing an example of a third image generated by a third camera.

[0100] Figure 66 is a drawing showing an example of a fourth image generated by a fourth camera.

[0101] Figure 67 is a drawing exemplarily showing an integrated image according to the present invention.

[0102] Figure 68 is a flowchart illustrating an example of a method for generating an integrated image of a ship according to the present invention.

[0103] Figure 69 is a drawing for explaining a bridge wing installed on a conventional ship.

[0104] Figure 70 is a diagram schematically showing the structure of an improved ship that improves the structure of a conventional ship to implement a method according to the present invention.

[0105] Figure 71 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0106] Figure 72 is a drawing for explaining a sub-module included in the processor of Figure 71.

[0107] Figure 73 is a flowchart showing the process of generating an integrated image by an integrated image generation device, by unit process.

[0108] Figure 74 is a drawing for explaining the first type sensor and the first area.

[0109] Figure 75 is a schematic drawing showing an example of a first type sensor and a second type sensor installed on an OCCS-applied ship.

[0110] Figure 76 is a schematic drawing showing an example of a first type sensor and a second type sensor installed on a container ship.

[0111] Figure 77 is a schematic drawing showing an example of a first type sensor and a second type sensor installed on a wind-powered vessel.

[0112] Figure 78 is a drawing showing an example of an integrated image generated by an integrated image generation device.

[0113] Figure 79 is a flowchart illustrating an example of a method according to the present invention.

[0114] Figures 80a and 80b are schematic drawings showing an example of a conventional container ship.

[0115] FIGS. 81a and 81b are schematic drawings showing the structure of an improved container ship that improves the structure of a conventional container ship in order to implement a method according to the present invention.

[0116] Figure 82 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0117] Figure 83 is a drawing for explaining a sub-module included in the processor of Figure 79.

[0118] Figure 84 is a flowchart showing the process of generating an integrated image by an integrated image generation device, by unit process.

[0119] Figure 85 is a drawing showing an example of the rear view of a conventional container ship.

[0120] FIG. 86 is a diagram illustrating second and third images generated by the second and third cameras, respectively.

[0121] Figures 87a and 87b are diagrams schematically showing the results of comparing a first image generated by a first camera and an integrated image generated in a container ship according to the present invention.

[0122] Figure 88 is a diagram schematically showing the results of comparing the loading capacity of containers of a conventional container ship and a container ship according to the present invention.

[0123] According to one embodiment of the present invention for solving the above technical problem, a method for generating an image of a ship with improved visibility comprises the steps of: collecting images from a first camera, a second camera, and a third camera of the ship; extracting overlapping feature points between second and third images collected from the second and third cameras; generating a target image by aligning the second and third images based on the extracted feature points; and generating an integrated image by integrating the first image collected from the first camera and the generated target image.

[0124] In the above method, the first camera may be a camera installed adjacent to the wheelhouse of the container ship.

[0125] In the above method, the second camera and the third camera can be installed so as to face forward on the bow of the container ship.

[0126] In the above method, the second camera and the third camera may be installed symmetrically around the ship center line of the container ship.

[0127] In the above method, the step of collecting the image may further collect images from the fourth camera and the fifth camera of the container ship, the step of extracting the feature points may extract overlapping feature points between the fourth and fifth images collected from the fourth camera and the fifth camera, and the step of generating the target image may generate the target image by aligning the second to fifth images based on the extracted feature points.

[0128] In the above method, the fourth camera and the fifth camera may be installed further rearward than the second camera and the third camera.

[0129] In the above method, the container ship may be a cellular container ship in which the wheelhouse is arranged closer to the bow than to the stern.

[0130] In the above method, the second camera and the third camera can be installed at a position exceeding a preset first forward viewing angle of the wheelhouse of the container ship.

[0131] In the above method, the area to be made transparent in the first image may include an area exceeding the first forward viewing angle based on the wheelhouse.

[0132] In the above method, the step of generating the integrated image may, in the process of generating the integrated image, process transparency processing for at least some of the objects displayed in the first image.

[0133] According to another embodiment of the present invention for solving the above technical problem, a device for generating an image of a ship with improved visibility comprises: a memory in which at least one program is stored; and a processor for performing a calculation by executing the at least one program, wherein the processor collects images from a first camera, a second camera, and a third camera of a container ship, extracts overlapping feature points between second images and third images collected from the second camera and the third camera, and generates a target image by matching the second images and the third images based on the extracted feature points, and generates an integrated image by integrating the first image collected from the first camera and the generated target image.

[0134] According to one embodiment of the present invention, a method for generating an image is provided, comprising: collecting images from a first camera, a second camera, and a third camera of a ship; extracting overlapping feature points between a second image and a third image collected from the second camera and the third camera; generating a target image by matching the second image and the third image based on the extracted feature points; generating an integrated image by integrating a first image collected from the first camera and the generated target image; and generating an image in which an obstacle area defined in the first image is replaced with a corresponding area in the target image.

[0135] Additionally, the first camera may be installed adjacent to the wheelhouse of the ship.

[0136] Additionally, the second camera and the third camera may be installed in a plurality of structures positioned adjacent to the bow of the ship.

[0137] Additionally, an area corresponding to a plurality of the structures in the first image may be defined as an obstacle area.

[0138] In addition, the step of generating the integrated image may, in the process of generating the integrated image, make at least some of the structures displayed in the obstacle area in the first image semi-transparent.

[0139] Additionally, the step of generating the integrated image may include a step of defining the obstacle area in the first image; and a step of replacing the obstacle area with the target image.

[0140] According to one embodiment of the present invention, an image generating device is provided, comprising: a memory storing at least one program; a processor executing the at least one program to perform an association, wherein the processor collects images from a first camera, a second camera, and a third camera of a ship, extracts overlapping feature points between second images and third images collected from the second camera and the third camera, matches the second images and the third images based on the extracted feature points to generate a target image, integrates the first image collected from the first camera and the generated target image to generate an integrated image, and replaces an obstacle area defined in the first image with a corresponding area in the target image in the integrated image.

[0141] In addition, the invention may further include a projection unit that receives the integrated image from the processor and projects it onto a window located in the wheelhouse of the ship.

[0142] Additionally, the projection unit can project onto the window only in an area corresponding to the obstacle area.

[0143] In addition, the above obstacle areas may be provided in multiple numbers, and the above projection units may be provided in multiple numbers to correspond to the multiple obstacle areas.

[0144] In addition, the system may further include a display unit that receives the integrated image from the processor and provides visual information to the user.

[0145] According to one embodiment of the present invention, a vessel including the image generating device is provided.

[0146] According to one embodiment of the present invention, there is provided a method for controlling a ship, comprising: collecting images from a first camera, a second camera, and a third camera of a ship; recognizing a user's position within a wheelhouse of the ship; assigning relative weights to the second camera and the third camera based on the recognized user's position; extracting overlapping feature points between the second and third images collected from the second and third cameras; generating a target image by matching the second and third images based on the extracted feature points; and

[0147] A method for generating an image includes a step of generating an integrated image by integrating a first image collected from the first camera and the generated target image, wherein the integrated image is provided in which an obstacle area defined in the first image is replaced with a corresponding area in the target image.

[0148] Additionally, the first camera, the second camera, and the third camera may be installed adjacent to the wheelhouse.

[0149] Additionally, the first camera may be placed inside the wheelhouse, and the second camera and the third camera may be placed outside the wheelhouse.

[0150] Additionally, the obstacle area defined in the first image may be an outer area of ​​a window placed in the wheelhouse.

[0151] In addition, in the step of generating the integrated image, during the process of generating the integrated image, at least a portion of the obstacle area in the first image can be made transparent.

[0152] Additionally, the step of generating the integrated image may include a step of defining the obstacle area in the first image; and a step of replacing the obstacle area with the target image.

[0153] According to one embodiment of the present invention, a computer-readable recording medium storing a program for executing the image generation method is provided.

[0154] According to one embodiment of the present invention, there is provided an image generating device, comprising: a memory storing at least one program; a processor executing the at least one program to perform an association, wherein the processor collects images from a first camera, a second camera, and a third camera of a ship, recognizes a position of a user within a wheelhouse of the ship, assigns relative weights to the second camera and the third camera according to the recognized position of the user, extracts overlapping feature points between a second image and a third image collected from the second camera and the third camera, matches the second image and the third image based on the extracted feature points to generate a target image, generates an integrated image by integrating a first image collected from the first camera and the generated target image, and replaces an obstacle area defined in the first image with a corresponding area in the target image in the integrated image.

[0155] In addition, the method may further include a display unit that receives a portion corresponding to the obstacle area from the integrated image from the processor and provides visual information to the user.

[0156] Additionally, the display unit may be placed on the side of a window in the wheelhouse.

[0157] In addition, the above windows may be provided in multiple numbers, the multiple windows may be spaced apart from each other at a preset interval, and the display unit may be placed between the multiple windows.

[0158] In addition, the system may further include a screen output unit that receives the integrated image from the processor and provides visual information to the user.

[0159] A method according to one embodiment of the present invention for solving the above technical problem includes the steps of collecting images from a first camera, a second camera, and a third camera of a ship; extracting overlapping feature points between second and third images collected from the second and third cameras; generating a target image by aligning the second and third images based on the extracted feature points; and generating an integrated image by integrating the first image collected from the first camera and the generated target image.

[0160] In the above method, the first camera may be a camera installed adjacent to the wheelhouse of the ship.

[0161] In the above method, the second camera and the third camera may be installed in an LCO2 tank adjacent to the bow of the ship.

[0162] In the above method, the second camera and the third camera may be installed in one LCO2 tank adjacent to the bow of the ship.

[0163] In the above method, the second camera and the third camera can be installed symmetrically with respect to the LCO2 tank.

[0164] In the above method, the vessel may be a vessel to which an OCCS (Onboard Carbon Capture System) is applied.

[0165] In the above method, the step of generating the integrated image may, in the process of generating the integrated image, process transparency processing for at least some of the objects displayed in the first image.

[0166] In the above method, the step of generating the integrated image may include the step of defining an obstacle area in the first image; and the step of generating the integrated image by replacing the defined obstacle area with the target image.

[0167] In the above method, the step of collecting the image may include calculating parameters and distortion coefficients of the camera, calculating correction values ​​based on the calculated parameters and distortion coefficients, and the step of extracting the feature points may include extracting the feature points based on the calculated correction values.

[0168] According to another embodiment of the present invention for solving the above technical problem, a device comprises: a memory in which at least one program is stored; and a processor for performing a calculation by executing the at least one program, wherein the processor collects images from a first camera, a second camera, and a third camera of a ship, extracts overlapping feature points between second images and third images collected from the second camera and the third camera, and generates a target image by aligning the second images and the third images based on the extracted feature points, and generates an integrated image by integrating the first image collected from the first camera and the generated target image.

[0169] In the above device, the first camera may be a camera installed adjacent to the wheelhouse of the ship.

[0170] In the above device, the second camera and the third camera may be installed in an LCO2 tank adjacent to the bow of the ship.

[0171] In the above device, the second camera and the third camera may be installed in one LCO2 tank adjacent to the bow of the ship.

[0172] In the above device, the second camera and the third camera can be installed symmetrically with respect to the LCO2 tank.

[0173] In the above device, the vessel may be a vessel to which an OCCS (Onboard Carbon Capture System) is applied.

[0174] In the above device, the processor can, in the process of generating the integrated image, make transparent at least some of the objects displayed in the first image.

[0175] In the above device, the processor can define an obstacle area in the first image and replace the defined obstacle area with the target image to generate the integrated image.

[0176] In the above device, the processor can calculate parameters and distortion coefficients of the camera, calculate correction values ​​based on the calculated parameters and distortion coefficients, and extract the feature points based on the calculated correction values.

[0177] According to one embodiment of the present invention for solving the above technical problem, a method for generating a ship image having improved visibility of a blind spot of a shield in front of a wheelhouse comprises the steps of: collecting images from a first camera, a second camera, and a third camera of a ship; extracting overlapping feature points between second and third images collected from the second and third cameras; generating a target image by aligning the second and third images based on the extracted feature points; and generating an integrated image by integrating the first image collected from the first camera and the generated target image.

[0178] In the above method, the first camera may be a camera installed adjacent to the wheelhouse of the ship.

[0179] In the above method, the second camera and the third camera can be installed on the bridge wing of the ship.

[0180] In the above method, the step of collecting the image may further collect the image from the fourth camera of the ship, the step of extracting the feature points may extract overlapping feature points between the second image to the fourth image, and the step of generating the target image may generate the target image by aligning the second image to the fourth image.

[0181] In the above method, the fourth camera may be installed on the bow of the ship.

[0182] In the above method, the second camera and the third camera may be installed symmetrically to each other on the bridge wing of the ship based on the same distance from the first camera.

[0183] In the above method, the ship may be a ship in which at least one wind auxiliary device is installed protruding from the deck of the ship.

[0184] In the above method, the step of generating the integrated image may, in the process of generating the integrated image, process transparency processing for at least some of the objects displayed in the first image.

[0185] In the above method, the step of generating the integrated image may include the step of defining an obstacle area in the first image; and the step of generating the integrated image by replacing the defined obstacle area with the target image.

[0186] According to another embodiment of the present invention for solving the above technical problem, a device is provided for generating a ship image with improved visibility of a blind spot of a shield in front of a wheelhouse, the device comprising: a memory storing at least one program; and a processor performing a calculation by executing the at least one program, wherein the processor collects images from a first camera, a second camera, and a third camera of the ship, extracts overlapping feature points between the second images and the third images collected from the second camera and the third camera, and generates a target image by aligning the second images and the third images based on the extracted feature points, and generates an integrated image by integrating the first image collected from the first camera and the generated target image.

[0187] According to one embodiment of the present invention for solving the above technical problem, a method is provided for creating an image of a ship with improved visibility by creating an image with an expanded field of view toward the rear of the ship, the method comprising: collecting images from a first camera, a second camera, a third camera, and a fourth camera of the ship; extracting overlapping feature points between second images to fourth images collected from the second camera to the fourth camera; generating a target image by aligning the second images to the fourth images based on the extracted feature points; and generating an integrated image by integrating the first image collected from the first camera and the generated target image.

[0188] In the above method, the first camera may be a camera installed adjacent to the wheelhouse of the ship.

[0189] In the above method, the second camera and the third camera can be installed on the bridge wing of the ship.

[0190] In the above method, the first to fourth cameras can be installed to photograph the direction of the stern of the ship.

[0191] In the above method, the fourth camera may be installed at the stern of the ship.

[0192] In the above method, the second camera and the third camera may be installed symmetrically to each other on the bridge wing of the ship based on the same distance from the first camera.

[0193] In the above method, the ship may be a ship equipped with an obstacle installed at the rear of the wheelhouse to block the rear view.

[0194] In the above method, the step of generating the integrated image may, in the process of generating the integrated image, process transparency processing for at least some of the objects displayed in the first image.

[0195] In the above method, the step of generating the integrated image may include the step of defining an obstacle area in the first image; and the step of generating the integrated image by replacing the defined obstacle area with the target image.

[0196] According to another embodiment of the present invention for solving the above technical problem, a device is provided, which generates an image by expanding the field of view toward the rear of a ship, thereby generating a ship image with improved visibility, the device comprising: a memory storing at least one program; and a processor performing a calculation by executing the at least one program, wherein the processor collects images from a first camera, a second camera, a third camera, and a fourth camera of the ship, extracts overlapping feature points between second images to fourth images collected from the second camera to the fourth camera, and generates a target image by aligning the second images to the fourth images based on the extracted feature points, and generates an integrated image by integrating the first image collected from the first camera and the generated target image.

[0197] According to one embodiment of the present invention for solving the above technical problem, a method for generating a monitoring image of a ship based on information collected from a plurality of sensors of the ship includes the steps of: collecting first type sensor information of a first type sensor installed in a first area of ​​the ship regardless of the type of the ship; collecting second type sensor information of a second type sensor installed in a second area of ​​the ship determined based on the type of the ship; identifying the type of the ship using the collected second type sensor information; and creating an integrated image including monitoring information of the identified type of ship based on the collected first type sensor information and the collected second type sensor information.

[0198] In the above method, the first type sensor may be one of a camera or a lidar.

[0199] In the above method, the first area is an area adjacent to the wheelhouse, and the wheelhouse may have a bridge wing omitted and a height of a bridge tower including the wheelhouse may be lower than a first height of the highest installation on the ship.

[0200] In the above method, the step of generating the integrated image may include analyzing the image included in the collected first type sensor information and the second type sensor information to recognize an object, and adding visual marks to objects of a preset type among the recognized objects to generate the integrated image.

[0201] In the above method, the visual indicator may be a bounding box that is displayed in different colors depending on the type of the object.

[0202] In the above method, the first type sensor and the second type sensor can be installed symmetrically around the bow and stern of the ship.

[0203] In the above method, the method further includes a step of collecting third type sensor information of a third type sensor that is capable of movement through a user's input and includes a camera, and the step of identifying the type of ship may identify the type of ship with the collected second type sensor information and the third type sensor information, and the step of generating the integrated image may generate an integrated image including monitoring information of the identified type of ship based on the collected first type sensor information or the collected third type sensor information.

[0204] In the above method, the third type sensor may be a drone capable of flight.

[0205] In the above method, the type of the identified ship may be one of a container ship, a wind-powered ship, and a ship to which an OCCS (Onboard Carbon Capture System) is applied.

[0206] According to another embodiment of the present invention for solving the above technical problem, a device for generating a monitoring image of a ship based on information collected from a plurality of sensors of the ship, the device includes: a communication unit; a memory in which at least one program is stored; and a processor for performing a calculation by executing the at least one program, wherein the processor collects first type sensor information of a first type sensor installed in a first area of ​​the ship regardless of the type of the ship, collects second type sensor information of a second type sensor installed in a second area of ​​the ship determined based on the type of the ship, identifies the type of the ship based on the collected second type sensor information, and generates an integrated image including monitoring information of the identified type of ship based on the collected first type sensor information and the collected second type sensor information.

[0207] According to one embodiment of the present invention for solving the above technical problem, a method is provided for creating an image of a container ship with improved visibility by creating an image with an expanded field of view toward the rear of the container ship, the method comprising: collecting images from a first camera, a second camera, and a third camera of the container ship; extracting overlapping feature points between second and third images collected from the second and third cameras; generating a target image by aligning the second and third images based on the extracted feature points; and generating an integrated image by integrating the first image collected from the first camera and the generated target image.

[0208] In the above method, the first camera may be a camera installed adjacent to the wheelhouse of the container ship.

[0209] In the above method, the second camera and the third camera may be installed on the bow of the container ship so as to face the rear of the container ship.

[0210] In the above method, the second camera and the third camera may be installed symmetrically around the ship center line of the container ship.

[0211] In the above method, the step of collecting the image may further collect images from the fourth camera and the fifth camera of the container ship, the step of extracting the feature points may extract overlapping feature points between the fourth image and the fifth image collected from the fourth camera and the fifth camera, respectively, and the step of generating the target image may generate the target image by aligning the second image to the fifth image based on the extracted feature points.

[0212] In the above method, the fourth camera and the fifth camera may be installed further rearward than the second camera and the third camera.

[0213] In the above method, the container ship may be a cellular container ship in which a wheelhouse including a bridge wing is arranged close to the bow, and a funnel is arranged behind the wheelhouse and the loaded container group.

[0214] In the above method, the container ship includes a first loading area, a second loading area, and a third loading area for sequentially loading a plurality of containers, the first loading area is located in the bow of the container ship, the second loading area is located between the wheelhouse and the funnel of the container ship, the third loading area is located at the rear of the funnel of the container ship, and the integrated image may be an image having a visual effect added to the loaded container ship in the second loading area.

[0215] In the above method, the step of generating the integrated image may, in the process of generating the integrated image, process transparency processing for at least some of the objects displayed in the first image.

[0216] According to another embodiment of the present invention for solving the above technical problem, a device is provided, which generates an image of a container ship with improved visibility by generating an image with an expanded field of view toward the rear of the container ship, the device comprising: a memory in which at least one program is stored; and a processor which performs a calculation by executing the at least one program, wherein the processor collects images from a first camera, a second camera, and a third camera of the container ship, extracts overlapping feature points between the second images and the third images collected from the second camera and the third camera, and generates a target image by matching the second images and the third images based on the extracted feature points, and generates an integrated image by integrating the first image collected from the first camera and the generated target image.

[0217] One embodiment of the present invention can provide a computer-readable recording medium storing a program for executing the above method.

[0218] The present invention is capable of various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, as well as the methods for achieving them, will become clearer with reference to the embodiments described in detail below, along with the drawings. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various forms.

[0219] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same drawing reference numerals, and redundant descriptions thereof will be omitted.

[0220] In the following examples, the terms first, second, etc. are not used in a limiting sense, but are used for the purpose of distinguishing one component from another.

[0221] In the examples below, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0222] In the following examples, terms such as “include” or “have” mean that a feature or component described in the specification is present, and do not preclude the possibility that one or more other features or components may be added.

[0223] In some embodiments, where the implementation is otherwise feasible, a particular process sequence may be performed in a different order than described. For example, two processes described in succession may be performed substantially simultaneously, or in a reverse order from the described order.

[0224] Hereinafter, the term “ship image” is a word that comprehensively refers to an image generated through a camera installed on a ship. As an example, the ship image may be any one of the first image, second image, third image, fourth image, and integrated image described below.

[0225] FIG. 2a and FIG. 2b are schematic drawings showing the structure of an improved container ship that improves the structure of a conventional container ship in order to implement a method according to the present invention.

[0226] Fig. 2a schematically illustrates that a container ship according to the present invention can load more containers than a conventional container ship. In Fig. 1a, the forward container (200) had to be loaded lower as it approached the forward portion, but according to the present invention, even if the number of containers loaded forward in the wheelhouse (10) further increases, the visibility in the wheelhouse (10) is not obstructed. In other words, according to the present invention, when loading containers in the forward portion as illustrated in Figs. 2a and 2b, there is no need to load them lower as it approaches the forward portion.

[0227] FIG. 2b is a plan view showing the structure of a container ship according to the present invention. The additionally loadable containers (200') shaded in FIG. 2 represent containers that can be loaded in greater numbers in the container ship according to the present invention compared to conventional container ships. In the container ship according to the present invention, a first camera may be installed inside the wheelhouse (10) and at a location adjacent to the wheelhouse (10), and a second camera (CA2) and a third camera (CA3) may be installed on the left and right sides of the bow of the container ship, respectively. The second camera (CA2) and the third camera (CA3) are installed to supplement the first camera of the wheelhouse (10) whose field of view is reduced due to the additionally loaded containers (200'), and the specific operations of the second camera (CA2) and the third camera (CA3) will be described in FIG. 3.

[0228] Figure 3 is a drawing exemplarily showing an improved structure of a container ship according to the present invention.

[0229] More specifically, Fig. 3 focuses on the bow portion of a container ship (1') according to the present invention. The container ship (1') according to the present invention may be a cellular container ship including at least three cameras and having a wheelhouse (10) positioned closer to the bow than the stern. However, since the method according to the present invention is not limited by the type of container ship, it may also be used for other types of container ships including at least three cameras, depending on the embodiment.

[0230] The container ship (1') according to the present invention has a differentiated feature in that it has several additional cameras that were not installed in conventional container ships, and further includes a device that creates an integrated image by aligning images taken by several cameras, thereby ensuring improved visibility compared to conventional container ships and enabling more containers to be loaded on the bow. The first camera (CA1) to the fifth camera (CA5) illustrated in Fig. 3 are considered devices that can not only take pictures of objects to create images, but also have a built-in communication module that can transmit the created images to an external device via wired / wireless communication.

[0231] The first camera (CA1) may be installed inside the wheelhouse (10) or adjacent to the wheelhouse (10). The first camera (CA1) may capture a bow direction from the wheelhouse (10) to generate a first image. The second camera (CA2) and the third camera (CA3) may be installed forward on the bow of the container ship (1'), respectively. The second camera (CA2) and the third camera (CA3) may be installed symmetrically around the ship center line based on the same distance from the first camera (CA1). Referring to FIG. 3, the distance between the first camera (CA1) and the second camera (CA2) may be the same as the distance between the first camera (CA1) and the third camera (CA3).

[0232] As an example, the second camera (CA2) and the third camera (CA3) may be installed so as to be outside the first forward viewing angle, '2α', of the first camera (CA1). The first camera (CA1) cannot photograph objects outside the first forward viewing angle, and the area that is blocked by containers loaded on the bow and cannot be photographed by the first camera (CA1) is referred to as the 'blind zone of the bow container'.

[0233] Here, α, which is half the size of the first forward viewing angle, may be 10 degrees. That is, while satisfying a range exceeding the left and right viewing angles of 10 degrees of the first camera (CA1), it may be installed symmetrically on the left and right sides of the bow of the container ship as illustrated in FIG. 3. Hereinafter, the positions where the second camera (CA2) and the third camera (CA3) are installed in the container ship (1') will be referred to as the second position and the third position, respectively.

[0234] The second camera (CA2) can capture a portion of the blind zone of the bow container based on the second position of the second camera (CA2) to generate a second image. In addition, the third camera (CA3) can capture another portion of the blind zone of the bow container based on the third position of the third camera (CA3) to generate a third image. The device according to the present invention can generate an integrated image through a process of aligning the second image and the third image to generate a target image and integrating the target image into the first image. The integrated image generated as described above can display the first object (ob1) that is blocked by the bow container and thus not visible in the first image, and a specific process of generating the integrated image will be described later with reference to FIG. 5.

[0235] If the first object (ob1) is far away from the second camera (CA2) and the third camera (CA3) or is at a different angle from the installation direction of the second camera (CA2) and the third camera (CA3), the first object (ob1) may not be sufficiently identified by the second camera (CA2) and the third camera (CA3). In this case, the container ship (1') according to the present invention can additionally collect the fourth image and the fifth image through the fourth camera (CA4) and the fifth camera (CA5), respectively, and reflect the information included in the fourth image and the fifth image in the process of generating the integrated image.

[0236] As illustrated in FIG. 3, the fourth camera (CA4) may be installed at a fourth position to capture a direction rotated by β more than the second camera (CA2) with respect to the bow line, and the fifth camera (CA5) may be installed at a fifth position to capture a direction rotated by β more than the third camera (CA3) with respect to the bow line. The fourth camera (CA4) and the fifth camera (CA5) may be installed further in front of the first camera (CA1) and further behind the second camera (CA2) and the third camera (CA3). The fourth camera (CA4) and the fifth camera (CA5) may be installed symmetrically with respect to the bow line, like the second camera (CA2) and the third camera (CA3). In the present invention, 2α is referred to as a first forward viewing angle of the first camera (CA1), and 2β is referred to as a second forward viewing angle of the first camera (CA1). That is, if there are no containers loaded on the bow, the first camera (CA1) can be interpreted as having a maximum field of view of (α+β) degrees to the left and right, respectively. As illustrated in Fig. 3, the fourth camera (CA4) and the fifth camera (CA5) further supplement the second and third images captured by the second camera (CA2) and the third camera (CA3), thereby ultimately generating a more complete integrated image.

[0237] The method according to the present invention can generate an integrated image with improved visibility by using images collected by the first camera (CA1), the second camera (CA2), the third camera (CA3), the fourth camera (CA4), and the fifth camera (CA5) described in FIG. 3. Hereinafter, the device for generating the integrated image will be referred to as an integrated image generating device (400). The integrated image generated by the integrated image generating device (400) will be described in detail with reference to FIGS. 4 to 12.

[0238] FIG. 4 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0239] The integrated image generation device (400) according to the present invention generates an integrated image and transmits it to an electronic device used by a captain or navigator in the wheelhouse (10) so that the integrated image can be played back on the electronic device. The captain or navigator can easily obtain information necessary for navigating the vessel through the integrated image output through the electronic device. Referring to FIG. 4, it can be seen that the integrated image generation device (400) includes a communication unit (410), a processor (430), and a memory (450).

[0240] The communication unit (410) may include one or more components that enable wired / wireless communication with external devices. For example, the communication unit (410) may include at least one piece of hardware necessary to implement short-range communication, such as Wi-Fi or Bluetooth, in a network provided by a communication network, or to implement various communications, including the Internet, when a LAN cable is connected.

[0241] The memory (450) is hardware that stores various data processed within the integrated image generation device (400), and can store a program for processing and controlling the processor (430). The memory (450) may include a random access memory (RAM) such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM, a Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

[0242] The processor (430) can control the overall operation of the integrated image generation device (400). For example, the processor (430) can control the operation of the input unit (not shown), display (not shown), communication unit (410), memory (450), etc. included in the integrated image generation device (400) by executing programs stored in the memory (450).

[0243] As an example, the processor (430) may collect images from the first camera, the second camera, and the third camera of the container ship, extract overlapping feature points between the second and third images collected from the second and third cameras, match the second and third images based on the extracted feature points to generate a target image, and integrate the first image collected from the first camera and the target image to generate an integrated image. The specific process of the processor (430) will be described later with reference to FIGS. 5 to 12.

[0244] When the integrated image generation device (400) is implemented as a physical device, the processor (430) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0245] In addition, in the present invention, when the integrated image generation device (400) is implemented in the form of an application (program) that runs on an integrated data processing device such as a server, the processor (430) and memory (450) included in the integrated image generation device (400) may be implemented in the form of a virtual machine that implements hardware such as DSPs, microcontrollers, RAM, ROM, HDD, etc. as software (command script).

[0246] FIG. 5 is a drawing for explaining a sub-module included in the processor of FIG. 4.

[0247] Below, the explanation will be given with reference to Fig. 3.

[0248] Referring to FIG. 5, it can be seen that the processor (430) includes a first image processing unit (431), a second image processing unit (433), a third image processing unit (435), and an integrated image generation unit (437). The first image processing unit (431), the second image processing unit (433), the third image processing unit (435), and the integrated image generation unit (437) illustrated in FIG. 5 are modules that are logically and conceptually separated in order to explain the process performed by the processor (430) in the process of implementing the method according to the present invention. Therefore, although four sub-modules are illustrated in FIG. 5, the processor (430) may include fewer than four or more than four sub-modules depending on the embodiment. In addition, the first image processing unit (431), the second image processing unit (433), the third image processing unit (435), and the integrated image generation unit (437) of FIG. 5 are sub-modules of the processor (430), and thus, like the processor (430), they can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0249] When images are collected from the first camera (CA1), the second camera (CA2), the third camera (CA3), the fourth camera (CA4), and the fifth camera (CA5) of the container ship (1'), the first image processing unit (431) can extract overlapping feature points between the second images to the fifth images collected from the second camera (CA2) to the fifth camera (CA5).

[0250] The second image processing unit (433) can generate a target image by matching the second image and the third image (or the second image to the fifth image) based on the feature points extracted by the first image processing unit (431). The second image processing unit (433) is equipped with at least one image registration algorithm and sufficient hardware and resources to execute the algorithm in order to match images generated from different cameras. Here, the target image refers to data that processes the appearance information of the first target (ob1) that is shielded by a container loaded high in the bow when observing the front from the wheelhouse (10) into an image, and can be generated by matching the second image and the third image, or by matching the second image to the fifth image.

[0251] The third image processing unit (435) can use a preset object recognition algorithm to identify an obstacle area in the first image. Here, the obstacle area may be an area occupied by containers (200') that were not loaded in a conventional container ship in the first image but are additionally loaded in the bow section of a container ship (1') according to the present invention.

[0252] The integrated image generation unit (437) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the target image generated by the second image processing unit (433). At this time, the first image and the target image can be integrated by the third image processing unit (435) based on a specific obstacle area.

[0253] Figure 6 is a flowchart showing the process of generating an integrated image by an integrated image generation device, by unit process.

[0254] Since Fig. 6 can be implemented by the integrated image generation device (400) described in Fig. 4, descriptions overlapping with those described in Figs. 4 and 5 will be omitted, and below, with reference to Figs. 4 and 5, the integrated image generation device (400) and the process of generating an integrated image by the sub-module included in the integrated image generation device (400) will be described for each unit process.

[0255] The integrated image generation device (400) can acquire images from each camera and store the data in memory (450) (S605). Here, each camera refers to the first camera (CA1) to the fifth camera (CA5) described above.

[0256] The integrated image generation device (400) can calculate internal parameters and distortion coefficients of each camera (S610). In addition, the integrated image generation device (400) can additionally calculate external parameters between cameras, wherein the external parameters may be rotation and translation vectors (S615).

[0257] The integrated image generation device (400) can define an obstacle area in the first image collected from the first camera (CA1) (S620). Typically, the obstacle area is the area occupied by containers (200') additionally loaded on the bow when the image is captured by the adjacent first camera (CA1) of the wheelhouse (10). In the first image, the obstacle area can be classified as a separate layer from the remaining areas of the first image and managed separately.

[0258] The integrated image generation device (400) can first correct the misalignment of each individual camera (S625). At this time, the integrated image generation device (400) can identify and correct the rotation angle of each camera through information linkage with the sensor of the container ship (1'). Subsequently, the integrated image generation device (400) can secondarily correct the misalignment of each individual camera (S630). At this time, the integrated image generation device (400) can correct the roll misalignment based on the sea level, and can correct the pitch misalignment through the visual correlation between the unique points of the structure of the container ship (1') and the sea level.

[0259] The integrated image generation device (400) can extract common feature points from the second to fifth images captured by the second camera (CA2) to the fifth camera (CA5) (S635). After synchronizing the viewpoints of the second to fifth images, the integrated image generation device (400) can set at least a portion of an object that is continuously observed as a feature point and extract it. For example, if the second and third images commonly include a buoy in front of the container ship (1') or another vessel in the blind spot of the additionally loaded containers (200'), the feature points between the two images can be extracted from the buoy or the other vessel.

[0260] The integrated image generation device (400) can align the second and third images (or, depending on the embodiment, the second to fifth images) based on the matching of extracted feature points (S640). The integrated image generation device (400) can project the aligned "target image" onto the rear camera view (S645).

[0261] The integrated image generation device (400) can generate an integrated image by replacing the obstacle area in the first image of the first camera (CA1) with the target image (aligned image) generated in the previous step (S650). The integrated image generation device (400) can transparently synthesize only the obstacle area in the integrated image (S655) and control the output of the integrated image through an electronic device in the wheelhouse (10) (S660).

[0262] Figure 77 is a drawing showing an example of a first image generated by a first camera.

[0263] The first image (7710) of FIG. 77 is a three-dimensional image of one frame of the first image generated by the first camera inside the wheelhouse (10) photographing the bow direction, and the second image (730) of FIG. 77 shows a schematic diagram of the first image. As illustrated in FIG. 7, the first image includes additionally loaded containers (200'). That is, referring to FIG. 7, it can be seen that the blind spot (blind spot) behind the additionally loaded containers (200') is not secured due to the additionally loaded containers (200'). The integrated image generation device (400) can designate the area occupied by the additionally loaded containers (200') in the first image as an obstacle area, as described in FIG. 5. The integrated image generation device (400) stores a unique object recognition algorithm to selectively recognize additionally loaded containers (200') in the first image and designate them as obstacle areas.

[0264] Figure 8 is a drawing showing an example of a second image generated by a second camera.

[0265] The third image (810) of FIG. 8 is a three-dimensional image of one frame of the second image generated by the second camera (CA2) photographing the front, and the fourth image (830) of FIG. 8 shows a schematic diagram of the third image (810). More specifically, the third image (810) and the fourth image (830) of FIG. 8 are exemplary second images captured by the second camera (CA2) installed on the left side of the bow of the container ship (1'). Referring to FIG. 3, it can be seen that the second image includes some common features (sea surface, sky, containers, etc. in the front) with the third image generated by the third camera (CA3).

[0266] Figure 9 is a drawing showing an example of a third image generated by a third camera.

[0267] The fifth image (910) of FIG. 9 is a three-dimensional image of one frame of a third image generated by the third camera (CA3) photographing the front, and the sixth image (930) of FIG. 9 shows a schematic diagram of the fifth image (910). More specifically, the fifth image (910) and the sixth image (930) of FIG. 9 are exemplary third images captured by the third camera (CA3) installed on the right side of the bow of the container ship (1'). Referring to FIG. 3, it can be seen that the third image includes some common features (sea surface, sky, containers, etc. in the front) with the second image generated by the second camera (CA2).

[0268] Figure 10 is a drawing showing an example of a fourth image generated by a fourth camera.

[0269] The seventh image (1010) of FIG. 10 is a three-dimensional image of one frame of the fourth image generated by the fourth camera (CA4), and the eighth image (1030) of FIG. 10 is a schematic diagram of the seventh image (1010). More specifically, the seventh image (1010) and the eighth image (1030) of FIG. 10 are exemplary images of the fourth image generated by the fourth camera (CA4) installed at the fourth position of the container ship (1'). Referring to FIG. 3, it can be seen that the fourth image includes some common features (sea surface in front, objects floating on the sea surface, additionally loaded containers (200'), other containers, etc.) with the first image generated by the first camera (CA1) and the second image generated by the second camera (CA2).

[0270] Figure 11 is a diagram illustrating an example of a fifth image generated by a fifth camera.

[0271] The ninth image (1110) of FIG. 11 is a three-dimensional image of one frame of the fifth image generated by the fifth camera (CA5), and the tenth image (1130) of FIG. 11 is a schematic diagram of the ninth image (1110). More specifically, the ninth image (1110) and the tenth image (1130) of FIG. 11 are exemplary images of the fifth image generated by the fifth camera (CA5) installed at the fifth position of the container ship (1'). Referring to FIG. 3, it can be seen that the fifth image includes some common features (sea surface in front, objects floating on the sea surface, additionally loaded containers (200'), other containers, etc.) with the first image generated by the first camera (CA1) and the third image generated by the third camera (CA3). In particular, the fifth image of FIG. 11 includes a first object (ob1) that was not observed in the first to fourth images, and due to the first object (ob1) included in the fifth image, the integrated image generated by the integrated image generating device (400) may include the first object (ob1).

[0272] In addition, since the additionally loaded containers (200') are displayed on both the left and right sides of the bow in the fourth and fifth images shown in FIGS. 10 and 11, the additionally loaded containers (200') displayed on the left and right sides of the bow in the fourth and fifth images can be important references in the process of the integrated image generation device (400) extracting feature points of the first to fifth images and setting an obstacle area of ​​the first image.

[0273] Figure 12 is a diagram schematically illustrating an example of a first image that can be generated by a first camera when the field of view of the first camera is secured up to the second front field of view.

[0274] More specifically, FIG. 12 is a drawing exemplarily showing that a first object (ob1) can be displayed in a first image when a container is not loaded on a container ship (1') and the first camera (CA1) is secured up to a second forward viewing angle. Referring to FIG. 3, when a container is loaded on the bow, the viewing angle of the first camera is only α degrees in each of the left and right, but when a container is not loaded on the bow, the viewing angle of the first camera is expanded to (α+β) degrees in each of the left and right, so that the first object (ob1) can be observed in the first image.

[0275] Figure 13 is a drawing showing an example of the results of comparing a conventional first image and an integrated image according to the present invention by implementing them as two-dimensional images.

[0276] Fig. 13 (a) is an example of a first image generated by a first camera when containers are uniformly loaded at the same height on the bow of a conventional container ship. According to Fig. 13 (a), the first camera (CA1) can secure a field of view only as much as the first forward viewing angle α degree in the left and right direction due to the additionally loaded containers (200'), and the first target (ob1) cannot be observed.

[0277] On the other hand, (b) of Fig. 13 exemplarily shows an integrated image generated by the integrated image generating device (400) when containers are uniformly loaded at the same height on the bow of the container ship (1') according to the present invention. In (b) of Fig. 13 , the area occupied by the additionally loaded containers (200') is designated as an obstacle area and then made transparent, and the images generated by the second camera (CA2) to the fifth camera (CA5) are aligned and integrated into the first image, so that regardless of the presence of the additionally loaded containers (200'), it is considered that a field of view is secured up to the second forward viewing angle (β) in addition to the first forward viewing angle (α) in the left and right directions, and the first object (ob1) can be included and displayed in the integrated image by the containers that have been made transparent.

[0278] As described above, the first camera is installed inside the wheelhouse (10) of the container ship according to the present invention and at a location adjacent to the wheelhouse (10), and the second camera (CA2) and the third camera (CA3) are installed on the left and right sides of the bow of the container ship, respectively. Therefore, in one embodiment, the area to be made transparent in the first image may include an area exceeding the first forward viewing angle (α) based on the wheelhouse (10) of FIG. 2, as illustrated in FIG. 13.

[0279] FIG. 14 is a flowchart illustrating an example of a method for generating an integrated image for a container ship according to the present invention.

[0280] The method according to FIG. 14 can be implemented by the integrated image generation device (400), processor (430) and sub-modules included in the processor (430) described in FIG. 4, and therefore, the method will be described below with reference to FIGS. 4 to 13, and any description that overlaps with the content already described will be omitted.

[0281] The integrated image generation device (400) can collect first, second, and third images from the first camera (CA1), the second camera (CA2), and the third camera (CA3) of the container ship (1') (S1410). In step S1410, fourth and fifth images by the fourth camera (CA4) and the fifth camera (CA5) may be added.

[0282] The integrated image generation device (400) can extract overlapping feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3) (S1430). In step S1430, feature points may also be extracted from the fourth and fifth images.

[0283] The integrated image generation device (400) can generate a target image by aligning the second and third images based on the extracted feature points (S1450). Depending on the embodiment, the fourth and fifth images may also be considered when generating the target image.

[0284] The integrated image generation device (400) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the generated target image (S1470).

[0285] FIG. 15 is a schematic diagram of a vessel including an image generating device according to one embodiment of the present invention. FIG. 16 is a diagram partially illustrating the interior of a wheelhouse corresponding to portion A of FIG. 15. FIG. 17 is an enlarged diagram of portion B of FIG. 16. FIG. 23 is a diagram illustrating a state in which an integrated image corresponding to an obstacle area is projected onto a window in a wheelhouse of a vessel by a projection unit according to one embodiment of the present invention.

[0286] Referring to FIGS. 15 to 17, a wheelhouse (2011) may be located at the stern of a ship (2010). The wheelhouse (2011) is located relatively rearward based on the longitudinal direction of the ship (2010), and cargo, fuel tanks, etc. may be placed forward of the wheelhouse (2011).

[0287] In the present invention, a vessel (2010) may be equipped with an Onboard Carbon Capture System (OCCS). Although not shown in the drawing, a tank for storing liquid carbon dioxide (LCO2) may be installed on the deck forward of the wheelhouse (2011).

[0288] Referring to Fig. 15, at least one structure (OB1, OB2, OB3, OB4) may be placed on the deck adjacent to the bow, forward of the wheelhouse (2011). The structures may include a crane, a funnel, an exhaust duct, etc.

[0289] The structures (OB1, OB2, OB3, OB4) illustrated in FIGS. 15 and 17 are exemplary structures that extend in the height direction (up-down direction based on FIG. 15) on the deck and, when looking outside from inside the wheelhouse (2011), may be structures that block the user's view and form a blind sector.

[0290] A vessel (2010) according to the present invention may include a total of six cameras. The first camera (CA1) may be installed adjacent to the wheelhouse (2011), and specifically, may be installed inside the wheelhouse (2011). The second camera (CA2), the third camera (CA3), the fourth camera (CA4), and the fifth camera (CA5) may be installed in each structure (OB1, OB2, OB3, OB4) located at the front exterior of the wheelhouse (2011), respectively.

[0291] Referring to FIG. 15, the second camera (CA2), the third camera (CA3), the fourth camera (CA4), and the fifth camera (CA5) are positioned in front of the corresponding structures (OB1, OB2, OB3, OB4) and can obtain front images of the respective structures (OB1, OB2, OB3, OB4).

[0292] The sixth camera (CA6) is located at the bow of the ship (2010), and may be located relatively forward (on the right side of Fig. 15) relative to the wheelhouse (2011) and each structure (OB1, OB2, OB3, OB4).

[0293] The image acquired from the 6th camera (CA6) can be provided as visual information to users (U1, U2, U3) located inside the wheelhouse (2011) through the image generation device (2400) described later, specifically the display unit (2450).

[0294] Referring to FIG. 15, FIG. 17, and FIG. 23, in the present invention, there are four structures forming a blind section, but this is not limited to the number of structures, and various modifications are possible, such as forming five or more structures.

[0295] At this time, a camera corresponding to each structure (OB1, OB2, OB3, OB4) is installed, and an image (data) can be obtained by taking a video of the front area of ​​each structure (OB1, OB2, OB3, OB4).

[0296] An image generation method according to one embodiment of the present invention can generate an integrated image with improved visibility by using images collected by the first camera (CA1), the second camera (CA2), the third camera (CA3), the fourth camera (CA4), and the fifth camera (CA5) illustrated in FIG. 15.

[0297] Hereinafter, the integrated image generated by the device generating the integrated image (hereinafter referred to as the 'image generating device (400)') will be described in detail through FIGS. 18 to 23.

[0298] Referring to FIG. 18, FIG. 19, and FIG. 23, an image generation device (2400) according to one embodiment of the present invention can generate an integrated image and transmit it to a captain or navigator (hereinafter referred to as a “user”) in the wheelhouse (2011).

[0299] Referring to FIG. 23, a portion of the integrated image can be projected onto a window (W) located in the wheelhouse (2011) of a ship (2010) through a projection unit (2440) provided in an image generating device (2400) according to one embodiment of the present invention.

[0300] Referring to Fig. 23, the image generation device (2400) may be provided with a display unit (2450) that can be transmitted to the user. The display unit (2450) may be a display panel that can receive an electrical signal from the outside and provide visual information to the user.

[0301] Referring to FIG. 16, a user may be located at various locations within the wheelhouse (2011) of a ship (2010). Specifically, a user (U1) may be located on the left, a user (U2) may be located in the center, and a user (U3) may be located on the right.

[0302] As an optional embodiment, the integrated image generated by the image generating device (2400) can be played back through an electronic device (not shown in the drawing) used by the captain or navigator in the wheelhouse (2011).

[0303] The user can easily obtain information necessary for operating the ship (2010) through the integrated image output through the projector (2440).

[0304] Specifically, a part of the integrated image corresponding to the blind area where the view is obstructed by structures located in front of the wheelhouse (2011), that is, the obstacle area, is projected onto the window through the projector (2440), thereby enabling the user to receive an image of the area in front of the structure, and there is an effect of eliminating the blind area.

[0305] Referring to FIG. 18, the image generation device (2400) may include a communication unit (2410), a processor (2420), a memory (2430), a projection unit (2440), and a display unit (2450).

[0306] The communication unit (2410) may include one or more components that enable wired / wireless communication with external devices. For example, the communication unit (410) may include at least one piece of hardware necessary to implement short-range communication, such as Wi-Fi or Bluetooth, in a network provided by a communication network, or to implement various communications, including the Internet, when a LAN cable is connected.

[0307] The memory (2430) is hardware that stores various data processed within the image generation device (2400), and can store a program for processing and controlling the processor (2420). The memory (2430) may include a random access memory (RAM) such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM, a Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

[0308] The processor (2420) can control the overall operation of the image generation device (2400). For example, the processor (2420) can control the operation of the input unit (not shown), the projection unit (2440), the display unit (2450), the communication unit (2410), the memory (2430), etc. included in the image generation device (2400) by executing programs stored in the memory (2430).

[0309] As an example, the processor (2420) may collect and acquire images from a first camera (CA1), a second camera (CA2), and a third camera (CA3) of a ship (2010), extract overlapping feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3), match the second and third images based on the extracted feature points to generate a target image, and integrate the first image collected from the first camera (CA1) and the generated target image to generate an integrated image.

[0310] As an optional embodiment, feature points may be extracted not only between the second image and the third image collected from the second camera (CA2) and the third camera (CA3), but also between the fourth image and the fifth image collected from the fourth camera (CA4) and the fifth camera (CA5), and then the fourth image and the fifth image may be aligned based on the extracted feature points to generate a target image.

[0311] That is, extracting feature points and creating a target image by matching the extracted feature points can be performed using images from the second camera (CA2) to the fifth camera (CA5) excluding the image acquired from the first camera (CA1).

[0312] The specific process of the processor (2420) will be described later with reference to FIGS. 19 to 22.

[0313] When the image generation device (2400) is implemented as a physical device, the processor (2420) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0314] In addition, when the image generation device (2400) according to one embodiment of the present invention is implemented in the form of an application (program) that runs on an integrated data processing device such as a server, the processor (2420) and memory (2430) included in the image generation device (2400) may be implemented in the form of a virtual machine that implements hardware such as DSPs, microcontrollers, RAM, ROM, HDD, etc. as software (command script).

[0315] Referring to FIG. 18 and FIG. 23, the projection unit (2440) receives power from the outside and projects an integrated image generated by the processor (2420) to the outside, and can project a portion corresponding to an obstacle area in the integrated image onto a window (W).

[0316] Since the integrated image processed as semi-transparent is projected onto the window (W), when a user located in the wheelhouse (2011) looks at the window (W), an image of the front area of ​​the structures (OB1, OB2, OB3, OB4) aligned by the integrated image is shown in the obstacle area that is actually covered by the structures (OB1, OB2, OB3, OB4) and forms a blind area, thereby having the effect of eliminating the blind area.

[0317] A plurality of projection units (2440) may be provided, and specifically, a plurality of projection units (2440A, 2440B, 2440C, 2440D) may be provided to correspond to a plurality of obstacle areas (A1, A2, A3, A4) formed by a plurality of structures (OB1, OB2, OB3, OB4) forming a blind section.

[0318] In the present invention, a plurality of projection units (2440) are provided, but this is not a limitation, and an integrated image can be projected entirely from a single projection unit (2440) toward a window (W). In this case, various modifications are possible, such as projecting the entire integrated image area, in which an obstacle area is synthesized to be semi-transparent, within the first image acquired from the first camera (CA1) onto the window (W).

[0319] Referring to FIG. 17 and FIG. 23, the display unit (2450) is installed in the wheelhouse (2011) and can receive information about the integrated image from the processor (2420) and provide it as visual information to the outside.

[0320] The display unit (2450) may be equipped with a display, and the entire integrated image area in which the obstacle area is synthesized translucently in the first image acquired from the first camera (CA1) may be displayed.

[0321] As a result, the user can receive an image with the blind section removed through the display unit (2450).

[0322] FIG. 19 is a diagram illustrating a processor (2420) according to one embodiment of the present invention.

[0323] Referring to FIG. 19, the processor (2420) may include a first image processing unit (2421), a second image processing unit (2423), a third image processing unit (2425), and an integrated image generation unit (2427).

[0324] The first image processing unit (2421), the second image processing unit (2423), the third image processing unit (2425), and the integrated image generation unit (2427) are modules that are logically and conceptually separated to explain the process performed by the processor (2420) in the process of implementing the method according to the present invention. Therefore, although four sub-modules are illustrated in FIG. 19, the processor (2420) may include fewer or more sub-modules than four, depending on the embodiment.

[0325] In addition, the first image processing unit (2421), the second image processing unit (2423), the third image processing unit (2425), and the integrated image generation unit (2427) of FIG. 19 are sub-modules of the processor (2420), and thus, like the processor (2420), they can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0326] When images are collected from the first camera (CA1), the second camera (CA2), and the third camera (CA3) of the ship (2010), the first image processing unit (2421) can extract overlapping feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3).

[0327] As mentioned above, for the convenience of explanation, the extraction of feature points between the second camera (CA2) and the third camera (CA3) is described, and it is of course possible to extract feature points between the third camera (CA3) and the fourth camera (CA4), and between the fourth camera (CA4) and the fifth camera (CA5).

[0328] The second image processing unit (2423) can generate a target image by matching the second and third images based on the feature points extracted by the first image processing unit (2421). The second image processing unit (2423) stores at least one image registration algorithm and sufficient hardware and resources to execute the algorithm in order to match images generated from different cameras.

[0329] Here, the target image refers to data processed into an image of information about a structure located in front of the wheelhouse (2011), specifically, the front of the first structure (OB1) and the second structure (OB2), when observed from the wheelhouse (2011), and can be generated by aligning the second image and the third image.

[0330] The third image processing unit (2425) can use a preset object recognition algorithm to identify an obstacle area in the first image. In this specification, the term "obstacle area" refers to an area (A1, A2, A3, A4) that is blocked by a structure located in front of the wheelhouse (2011) among the forward areas where the sea surface is visible when looking forward from the wheelhouse (2011) of the ship (2010).

[0331] Referring to FIG. 17 and FIG. 23, since there are four structures in the present invention, the corresponding shielded areas (A1, A2, A3, A4) are four, but this is not limited to the number and may be less than or more than four.

[0332] The integrated image generation unit (2427) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the target image generated by the second image processing unit (2423) and integrating the target area generated by the second image processing unit (2423). At this time, the first image and the target image can be integrated by the third image processing unit (2425) based on a specific obstacle area.

[0333] Figure 20 is a flowchart illustrating, by unit process, the process of generating an integrated image by an image generation device (2400) according to one embodiment of the present invention. Figure 21 is a flowchart illustrating the steps of setting camera combinations and weights according to the user's position. Figure 22 is a flowchart illustrating the steps of performing secondary correction on an integrated image.

[0334] Since FIGS. 20 to 22 can be implemented by the image generation device (2400) described in FIG. 18, any description overlapping with the description in FIGS. 18 and 19 will be omitted and described in detail below.

[0335] The image generation device (2400) can combine cameras and set weights according to the user's location (S2605). Here, the cameras to be combined may be the second camera (CA2), the third camera (CA3), the fourth camera (CA4), and the fifth camera (CA5), excluding the first camera (CA1) located inside the wheelhouse (2011).

[0336] For convenience of explanation, the case of the second camera (CA2) and the third camera (CA3) will be explained as examples.

[0337] Referring to FIG. 21, in the step of combining cameras and setting weights (S2605), a distance matrix can be generated by first calculating the distance from the position of each camera, specifically the second camera (CA2) and the third camera (CA3), to the observation position of the wheelhouse (2011) (S26051). Next, the relative distance for each camera from the reference point can be measured, and then a distortion coefficient can be calculated (S26053).

[0338] Referring to FIGS. 15 to 17, when the user (U1) is located on the left side with respect to the center of the wheelhouse (2011), a weight may be given to the second camera (CA2) located closer to the user than to the third camera (CA3) and utilized when generating a matched image (S26055).

[0339] The image generation device (2400) can acquire images from each camera and store the data in a memory (2430) (S2610). Here, each camera refers to the first camera (CA1) to the fifth camera (CA5) described above.

[0340] The image generation device (2400) can synchronize the time between images from each camera (S2615). Images can be acquired at a preset frame rate (e.g., 30 fps) for each camera.

[0341] When synchronizing time between images, the maximum frame time difference can be compared with the allowable time. Specifically, if the allowable time is 50ms, any difference less than the allowable time is considered the same and the original frame can be returned.

[0342] Conversely, if the allowed time is exceeded, slow frames can be removed one by one and compared again, with the allowed time gradually increased. If the allowed time is still exceeded despite the increased time, previous frames that fall below the allowed time can ultimately be used.

[0343] The image generation device (2400) can calculate internal parameters and distortion coefficients of each camera (S2620). In addition, the image generation device (2400) can additionally calculate external parameters between the cameras, wherein the external parameters may be rotation and translation vectors (S2625).

[0344] The image generation device (2400) can define an obstacle area in a first image collected from a first camera (CA1) (S2630). In the present invention, the 'obstacle area' may be an area occupied by structures (OB1, OB2, OB3, OB4) that are located in front of the wheelhouse (2011) and form a blind section when an image is captured by the first camera (CA1) installed adjacent to, specifically inside, the wheelhouse (2011). The obstacle area in the first image may be classified and managed as a separate layer from the remaining areas of the first image.

[0345] In the present invention, there are a total of four obstacle areas (A1, A2, A3, A4), but this is not limited to them and may increase depending on the number of structures located in front of the wheelhouse (2011).

[0346] The image generation device (2400) can first correct the misalignment of each individual camera (S2635). At this time, the image generation device (2400) can identify and correct the rotation angle of each camera through information linkage with the ship's (2010) sensor. Subsequently, the image generation device (2400) can secondarily correct the misalignment of each individual camera (S2640). In step S2640, the image generation device (2400) can correct the roll misalignment based on the sea level, and can correct the pitch misalignment through the visual correlation between the singularity of the ship's (2010) structure and the sea level.

[0347] The image generation device (2400) can perform image batch preprocessing for alignment (S2645). At this time, the RGB values ​​of the images acquired from the second camera (CA2) and the third camera (CA3) can be normalized to the range of 0 to 1. Furthermore, the resolution can be adjusted by resizing the images. Furthermore, histogram equalization can be used to uniformize the brightness distribution between images under different lighting conditions. In other words, there is an effect of enhancing feature point extraction performance by enhancing contrast.

[0348] In step S2645, a minimum of four and a maximum of six frames captured by each camera can be organized into a single batch, and the images can be converted into tensors and then processed for operation on a graphics processing unit (GPU).

[0349] The image generation device (2400) can extract feature points from a minimum of four and a maximum of six frames based on deep learning (S2650). In step S2650, common feature points can be extracted from images captured by the second camera (CA2) and the third camera (CA3).

[0350] The image generation device (2400) can align the second image and the third image based on matching common feature points extracted in step S2650.

[0351] As an optional embodiment, a common feature point may be extracted from images captured by two cameras positioned closely among the second camera (CA2) to the fifth camera (CA5), and a matched “target image” may be generated based on matching the common feature points.

[0352] As an optional embodiment, a common feature point may be extracted from an image captured by one of the second camera (CA2) to the fifth camera (CA5) and an image captured by the first camera (CA1), and a matched target image may be generated based on matching the common feature points.

[0353] The image generation device (2400) can align the first image with other images based on common feature point matching (S2655). In step S2655, two bidirectional closest feature points can be selected and then matched. Furthermore, outliers can be removed through the distance ratio between the two feature points, and images from different viewpoints can be aligned through a homography calculation.

[0354] The image generation device (2400) can perform a primary correction on the “integrated image” generated in step S2655 (S2660). In step S2660, spatial distortion can be precisely corrected based on the geometric relationship between each viewpoint through multi-view geometry-based area correction.

[0355] Referring to FIGS. 20 and 22, the image generation device (2400) can perform secondary correction on the integrated image (S2665). First, weather information can be collected from the Automatic Identification System (AIS) (S26651).

[0356] Afterwards, it can be classified (S26653) based on sunlight intensity, rainfall status, fog status, and time.

[0357] Depending on the sunlight intensity, visibility can be secured with a transparency of about 70% for sunlight stronger than a preset standard, and visibility can be secured with a transparency of about 30% to 40% for relatively weak sunlight, such as cloudy weather or weak sunlight (S26655a).

[0358] Depending on the precipitation conditions, such as snow or rain, screen distortion caused by water droplets or snow crystals can be corrected and the clarity can be adjusted (S26655b). Depending on the foggy conditions, contrast can be enhanced to secure visibility (S26655c). Depending on the time of day, visibility can be secured by dividing the day into preset criteria and dynamically adjusting transparency for each time zone (S26655d). As an optional embodiment, at night, areas that are relatively darker than the preset brightness standard can be brightened through gamma adjustment. A corrected image can be generated through steps S26655a to S26655d (S26657).

[0359] As an optional embodiment, in the case of night, areas that are relatively darker than the preset brightness standard can be brightened by gamma adjustment.

[0360] As an optional embodiment, when analyzing the properties of images acquired from each camera, they can also be classified based on brightness, contrast, color, and blur.

[0361] Specifically, for clear days with relatively high brightness, high contrast, and blue color tones, contrast can be adjusted for greater clarity. Conversely, for cloudy days with relatively low brightness, low contrast, gray color tones, and low contrast and clarity, brightness and contrast can be increased.

[0362] Referring to Figure 20, the secondary correction of the integrated image is performed after the primary correction described above, but is not limited thereto, and various modifications are possible, such as being performed as preprocessing before synchronizing the time between images acquired from each camera (S2615).

[0363] The image generation device (2400) can replace the obstacle area in the first image of the first camera (CA1) with a matched image (S2670). The image generation device (2400) can synthesize (or process) the area corresponding to the obstacle area in the integrated image to be semitransparent (S2675), and output the integrated image through the projector (2440), the display unit (2450), and electronic devices in the wheelhouse (2011) (S2680).

[0364] Referring to FIG. 23, an integrated image can be output through a projection unit (2440) and a display unit (2450), and a detailed description thereof is omitted to the extent of overlap as described above.

[0365] The image generation device (2400) can detect the positions of users (U1, U2, U3) in real time after generating the final image, which is an integrated image, and generate a new integrated image as the user's position changes.

[0366] First, the user's location can be detected. The image generation device (2400) accumulates and calculates the amount of change in the user's location over a preset period of time, and if the total amount of change in location exceeds a preset standard (threshold), the camera that acquires the image can be reselected to generate multiple integrated images.

[0367] When a camera other than the first camera (CA1), for example, a camera to be aligned among the second camera (CA2) to the fifth camera (CA5), is selected, a new integrated image (hereinafter referred to as a “second integrated image”) can be generated by going through steps S2605 to S2680.

[0368] Once the secondary integrated image is generated, it can be gradually reflected by blending it with the primary integrated image, which is the previous integrated image. Specifically, as frames change over time, the secondary integrated image is given a relatively greater weight than the primary integrated image, so that it can ultimately be replaced by the secondary integrated image over time.

[0369] FIG. 24 is a flowchart illustrating another embodiment of an image generation method for generating an integrated image according to the present invention.

[0370] Since the image generation method according to Fig. 24 can be implemented by the processor and sub-modules included in the processor described in Fig. 18, any description that overlaps with what has already been described will be omitted.

[0371] The image generation device can collect images from the ship's first camera (CA1), second camera (CA2), and third camera (CA3) (S3010). As mentioned above, it is also possible to collect images from the fourth camera and fifth camera.

[0372] The image generation device can extract overlapping feature points between the second image and the third image collected from the second camera (CA2) and the third camera (CA3) (S3030).

[0373] The image generation device can generate a target image by aligning the second image and the third image based on the extracted feature points (S3050).

[0374] The image generation device can generate an integrated image by integrating the first image collected from the first camera (CA1) and the generated target image (S3070). In the present invention, the integrated image may also be referred to as a ship image.

[0375] FIG. 25 is a schematic diagram of a vessel including an image generating device according to one embodiment of the present invention. FIG. 26 is a diagram partially illustrating the interior of a wheelhouse corresponding to portion A of FIG. 25. FIG. 27 is an enlarged diagram of portion B of FIG. 26. FIG. 33 is a diagram illustrating a state in which an integrated image corresponding to an obstacle area is projected onto a window in a wheelhouse of a vessel by a display unit according to one embodiment of the present invention.

[0376] Referring to FIGS. 25 to 27, a wheelhouse (2011) may be located at the stern of a ship (2010). The wheelhouse (2011) is located relatively rearward based on the longitudinal direction of the ship (2010), and cargo, fuel tanks, etc. may be placed forward of the wheelhouse (2011).

[0377] In the present invention, a vessel (2010) may be equipped with an Onboard Carbon Capture System (OCCS). Although not shown in the drawing, a tank for storing liquid carbon dioxide (LCO2) may be installed on the deck forward of the wheelhouse (2011).

[0378] Referring to Fig. 26, when looking forward from inside the wheelhouse (2011), a window frame (WF) may be arranged between windows (W) installed inside the wheelhouse (2011). The window frame (WF) may include a pair of horizontal frames (drawing symbol not set) arranged in a horizontal direction in parallel, and a vertical frame (drawing symbol not set) connecting the pair of horizontal frames. The horizontal frame and the vertical frame may be formed as one piece.

[0379] In this specification, a window frame (WF) placed between windows (W) corresponds to a vertical frame.

[0380] The window (W) of the wheelhouse (2011) can only extend to a preset width along the width direction of the wheelhouse (2011) due to rigidity issues. A plurality of windows (W) are provided, and a plurality of windows (W) can be connected to a window frame (WF).

[0381] A window frame (WF) may be arranged between a plurality of windows (W) along the width direction (X-axis direction based on Fig. 2) of the wheelhouse (2011), and the plurality of windows (W) may be spaced apart from each other by the width of the window frame (WF).

[0382] Referring to FIGS. 26 and 27, a portion of the forward area viewed from inside the wheelhouse (2011) may be obscured due to a window frame (WF) positioned between a plurality of windows (W) arranged close to each other.

[0383] That is, when looking outside from inside the wheelhouse (2011), a blind sector may be formed, blocking the view of the users (U1, U2, U3) by an area corresponding to the window frame (WF). This causes a problem in that the user cannot recognize objects (OB1, OB2) located in the blind sector.

[0384] A vessel (2010) according to the present invention may include at least two cameras. The first camera (CA1) may be installed adjacent to the wheelhouse (2011), and specifically, may be installed inside the wheelhouse (2011).

[0385] The first camera (CA1) is positioned inside the wheelhouse (2011) and can acquire a first image in the direction of the window (W), i.e., the front, from a preset position. This first image can be aligned with the second and third images acquired from cameras other than the first camera (CA1), specifically the second camera (CA2) and the third camera (CA3), which will be described later, and can be used as a reference image.

[0386] In addition to acquiring a first image facing forward from inside the wheelhouse (2011), the first camera (CA1) can acquire images of users (U1, U2, U3) located inside the wheelhouse (2011). As an optional embodiment, a plurality of first cameras (CA1) can be provided, and can be spaced apart at preset intervals along the width direction (X-axis direction based on FIG. 26) of the wheelhouse (2011).

[0387] By having multiple first cameras (CA1) obtain images of the interior of the wheelhouse (2011) or images of the user from different locations, it is possible to accurately recognize and identify the location of the user within the wheelhouse (2011).

[0388] Multiple first cameras (CA1) each have independent fields of view and can capture images of users in the wheelhouse (2011) from different angles and distances.

[0389] Specifically, when a user is detected by multiple first cameras (CA1), the different first camera (CA1) coordinate systems can be normalized to a unified reference coordinate system within the wheelhouse (2011) based on the user's location information acquired from each first camera (CA1).

[0390] This coordinate normalization process has the effect of improving the accuracy of the integrated image, which will be described later, based on the user's position by converting the pixel coordinates in the image of each first camera (CA1) into real space coordinates based on the position of the common point observed by the user, and correcting the viewpoint difference and position deviation between cameras.

[0391] Referring to FIG. 27, the second camera (CA2), the third camera (CA3), and the fourth camera (CA4) are positioned in front of the corresponding window frame (WF), and can acquire a front image of each window frame (WF).

[0392] In the present invention, one camera is placed in one window frame (WF), but this is not limited to this, and various modifications are possible, such as one camera being placed every two or three along the width direction of the wheelhouse (2011).

[0393] In addition, within the technical concept of being able to acquire the forward area of ​​the wheelhouse (2011), it can be positioned in a preset position on one side of the forward side of the wheelhouse (2011) rather than being coupled to the window frame (WF).

[0394] Below, the explanation will be based on the placement of each camera in the window frame (WF).

[0395] Referring to FIG. 27 and FIG. 33, for convenience of explanation, three window frames (WF) forming a blind section are shown, and the cameras are also shown as the second camera (CA2) to the fourth camera (CA4), but various modifications are possible, such as forming three or more.

[0396] At this time, a camera corresponding to each of a plurality of window frames (WF) is installed, and an image (data) can be obtained by capturing an image of the front area of ​​each window frame (WF).

[0397] An image generation method according to one embodiment of the present invention can generate an integrated image with improved visibility by using images collected by the first camera (CA1) illustrated in FIG. 25, the second camera (CA2), the third camera (CA3), and the fourth camera (CA4) illustrated in FIG. 27.

[0398] Hereinafter, the integrated image generated by the device generating the integrated image (hereinafter referred to as the “image generating device (3400)”) will be described in detail with reference to FIGS. 28 to 33.

[0399] Referring to FIG. 28, FIG. 29, and FIG. 33, an image generating device (3400) according to one embodiment of the present invention can generate an integrated image and transmit it to a captain or navigator (hereinafter referred to as a “user”) in the wheelhouse (2011).

[0400] Referring to FIG. 33, a portion of the integrated image can be displayed through a display unit (3440) provided in an image generation device (3400) according to one embodiment of the present invention and provided to a user as visual information.

[0401] Referring to Fig. 28, an image of the front of the wheelhouse (2011) can be transmitted to the user through a screen output unit (3450) provided in the image generation device (3400). The screen output unit (3450) may be a display panel that can receive an electrical signal from the outside and provide visual information to the user.

[0402] Referring to FIG. 26, a user may be located at various locations within the wheelhouse (2011) of a ship (2010). Specifically, a user (U1) may be located on the left, a user (U2) may be located in the center, and a user (U3) may be located on the right.

[0403] As an optional embodiment, the integrated image generated by the image generating device (3400) can be played back through an electronic device (not shown in the drawing) used by the captain or navigator in the wheelhouse (2011).

[0404] The user can easily obtain information necessary for operating the ship (2010) through the integrated image output through the display unit (3440).

[0405] Specifically, a part of the integrated image corresponding to the blind section where the view is obstructed by window frames (WF), which are structures located inside the wheelhouse (2011), i.e., the obstacle area, is displayed through the display unit (3440), thereby enabling the user to recognize the image in front of the wheelhouse (2011), in particular, objects (OB1, OB2) such as external vessels (2010) on the sea surface, and thereby eliminate the blind section.

[0406] Referring to FIG. 28, the image generation device (3400) may include a communication unit (3410), a processor (3420), a memory (3430), a display unit (3440), and a screen output unit (3450).

[0407] The communication unit (3410) may include one or more components that enable wired / wireless communication with external devices. For example, the communication unit (3410) may include at least one piece of hardware necessary to implement short-range communication, such as Wi-Fi or Bluetooth, in a network provided by a communication network, or to implement various communications, including the Internet, when a LAN cable is connected.

[0408] The memory (3430) is hardware that stores various data processed within the image generation device (3400), and can store a program for processing and controlling the processor (3420). The memory (3430) may include a random access memory (RAM) such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM, a Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

[0409] The processor (3420) can control the overall operation of the image generation device (3400). For example, the processor (3420) can control the operation of the input unit (not shown), display unit (3440), screen output unit (3450), communication unit (3410), memory (3430), etc. included in the image generation device (3400) by executing programs stored in the memory (3430).

[0410] In this specification, when aligning images of the front of the wheelhouse (2011), images acquired from at least two cameras among the second camera (CA2) to the fourth camera (CA4) are used, and for convenience of explanation, only the second camera (CA2) and the third camera (CA3) are described.

[0411] As an example, the processor (3420) may collect and acquire images from a first camera (CA1), a second camera (CA2), and a third camera (CA3) of a ship (2010), extract overlapping feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3), match the second and third images based on the extracted feature points to generate a target image, and integrate the first image collected from the first camera (CA1) and the generated target image to generate an integrated image.

[0412] As an optional embodiment, it is possible to extract feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3), as well as between the fourth image collected from the fourth camera (CA4) and images acquired from other cameras, and then generate a target image by matching the fourth image and other images based on the extracted feature points.

[0413] That is, extracting feature points and creating a target image by matching the extracted feature points can be performed using images from the second camera (CA2) to the fourth camera (CA4) excluding the image acquired from the first camera (CA1).

[0414] The specific process of the processor (3420) will be described later with reference to FIGS. 29 to 32.

[0415] When the image generation device (3400) is implemented as a physical device, the processor (3420) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0416] In addition, when the image generation device (3400) according to one embodiment of the present invention is implemented in the form of an application (program) that runs on an integrated data processing device such as a server, the processor (3420) and memory (3430) included in the image generation device (3400) may be implemented in the form of a virtual machine that implements hardware such as DSPs, microcontrollers, RAM, ROM, HDD, etc. as software (command script).

[0417] Referring to FIG. 28 and FIG. 33, the display unit (3440) receives power from the outside and projects an integrated image generated by the processor (3420) to the outside, thereby displaying a portion corresponding to an obstacle area in the integrated image and transmitting it as visual information to the outside.

[0418] The display unit (3440) may be equipped with a display so that an image received from the processor (3420) can be displayed externally. The display unit (3440) may be placed on the side of a window (W) in the wheelhouse (2011). Specifically, it may be placed between a plurality of windows (W) that are placed close to each other and spaced apart by a preset interval.

[0419] Referring to FIG. 27, each side of a plurality of facing windows (W) is coupled to a window frame (WF), and a display unit (3440) can be positioned and fixed on the window frame (WF).

[0420] The user can view the forward area of ​​the wheelhouse (2011) through the window (W). However, a window frame (WF) is positioned between a plurality of windows (W) that are spaced apart from each other, and this window frame (WF) may block part of the forward area of ​​the wheelhouse (2011) and form a blind area.

[0421] Referring to FIG. 27 and FIG. 33, the display unit (3440) is fixed in position on the window frame (WF), and an integrated image, specifically an image corresponding to an obstacle area, is displayed, so that the user can secure a view of the front area of ​​the wheelhouse (2011) in the area obscured by the window frame (WF).

[0422] As will be explained in detail later, in the process of generating an integrated image, at least a portion of the obstacle area defined in the first image is made transparent and reflected in the integrated image, so that when a user located within the wheelhouse (2011) looks at the window (W), an image of the front area of ​​the wheelhouse (2011) is displayed in the obstacle area that is covered by the window frame (WF) and forms a blind area, thereby eliminating the blind area.

[0423] A plurality of display units (3440) may be provided, and specifically, a plurality of display units may be provided to correspond to a plurality of window frames (WF) forming a blind section.

[0424] Referring to FIG. 28, the screen output unit (3450) is installed in the wheelhouse (2011) and can receive information about the integrated image from the processor (3420) and provide it as visual information to the outside.

[0425] The screen output unit (3450) may be equipped with a display, and the entire area of ​​the integrated image in which the obstacle area is transparently synthesized within the first image acquired from the first camera (CA1) may be displayed.

[0426] As a result, the user can receive an image with the blind section removed through the screen output unit (3450).

[0427] FIG. 29 is a diagram illustrating a processor (3420) according to one embodiment of the present invention.

[0428] Referring to FIG. 29, the processor (3420) may include a first image processing unit (3421), a second image processing unit (3423), a third image processing unit (3425), and an integrated image generation unit (3427).

[0429] The first image processing unit (3421), the second image processing unit (3423), the third image processing unit (3425), and the integrated image generation unit (3427) are modules that are logically and conceptually separated to explain the process performed by the processor (3420) in the process of implementing the method according to the present invention. Therefore, although four sub-modules are illustrated in FIG. 29, the processor (3420) may include fewer or more sub-modules than four, depending on the embodiment.

[0430] In addition, the first image processing unit (3421), the second image processing unit (3423), the third image processing unit (3425), and the integrated image generation unit (3427) of FIG. 29 are sub-modules of the processor (3420), and thus, like the processor (3420), they can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0431] When images are collected from the first camera (CA1), the second camera (CA2), and the third camera (CA3) of the ship (2010), the first image processing unit (3421) can extract overlapping feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3).

[0432] As mentioned above, for the convenience of explanation, the extraction of feature points between the second camera (CA2) and the third camera (CA3) is described, and it is of course possible to extract feature points between the third camera (CA3) and the fourth camera (CA4).

[0433] The second image processing unit (3423) can generate a target image by matching the second and third images based on the feature points extracted by the first image processing unit (3421). The second image processing unit (3423) stores at least one image registration algorithm and sufficient hardware and resources to execute the algorithm in order to match images generated from different cameras.

[0434] Here, the target image refers to data processed into an image of information about the front of the wheelhouse (2011) when observed from the wheelhouse (2011), and can be created by aligning the second image and the third image.

[0435] The third image processing unit (3425) can use a preset object recognition algorithm to identify an obstacle area in the first image. In this specification, the term "obstacle area" refers to an area in the front of the wheelhouse (2011) of the ship (2010) where the sea surface is visible, and is located inside the wheelhouse (2011), specifically, an area that is shielded by a window frame (WF).

[0436] Referring to FIG. 27 and FIG. 33, since there are three window frames (WF) in the internal area of ​​the partially illustrated wheelhouse (2011), there are three corresponding shielded areas, but this is not limited to the number of shielded areas corresponding to the window frames (WF) and shielded areas can be formed.

[0437] The integrated image generation unit (3427) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the target image generated by the second image processing unit (3423) and integrating the target area generated by the second image processing unit (3423). At this time, the first image and the target image can be integrated by the third image processing unit (3425) based on a specific obstacle area.

[0438] Figure 30 is a flowchart illustrating, by unit process, the process of generating an integrated image using an image generation device according to one embodiment of the present invention. Figure 31 is a flowchart illustrating the steps of setting camera combinations and weights according to the user's position. Figure 32 is a flowchart illustrating the steps of performing secondary correction on an integrated image.

[0439] Since Figs. 30 to 32 can be implemented by the image generation device (3400) described in Fig. 28, any description overlapping with the description in Figs. 28 and 29 will be omitted and described in detail below.

[0440] The image generation device (3400) can combine cameras and set weights according to the user's location (S3605). Here, the cameras to be combined may be the second camera (CA2), the third camera (CA3), and the fourth camera (CA4), excluding the first camera (CA1) located inside the wheelhouse (2011).

[0441] For convenience of explanation, the case of the second camera (CA2) and the third camera (CA3) will be explained as examples.

[0442] Referring to FIG. 31, in the step of combining cameras and setting weights (S3605), the areas within the wheelhouse (2011) can first be separated and then coordinated (S36051). That is, the area within the wheelhouse (2011) of the ship (2010) can be separated into multiple areas along the width direction of the wheelhouse (2011).

[0443] Referring to Fig. 26, the wheelhouse (2011) can be divided into multiple zones (Z1, Z2, Z3, Z4, Z5) from the leftmost zone to the rightmost zone along the width direction (X-axis direction based on Fig. 26).

[0444] Referring to Fig. 26, since it illustrates a portion of the interior area of ​​the wheelhouse (2011), it is obvious that multiple zones can be set based on the width direction of the entire area within the wheelhouse (2011) of the ship (2010).

[0445] Next, using an object detection model, users within the wheelhouse (2011) can be detected (S36053). The first camera (CA1) detects users (U1, U2, U3) within the wheelhouse (2011), and multiple first cameras (CA1) may be provided.

[0446] Multiple first cameras (CA1) can detect a single object (user) at different locations. When there are multiple users to be detected, the first camera (CA1) can select and detect the user closest to the window (W). Using the multiple first cameras (CA1) that have detected the user, the coordinates between the multiple first cameras (CA1) can be normalized.

[0447] Next, a zone within the wheelhouse (2011) can be defined (S36055). That is, for the multiple zones identified above, a range can be determined within a range of 0 to 1 based on normalized coordinates. Specifically, 0 may be the coordinate at the leftmost position (based on Figure 26) within the wheelhouse (2011), and 1 may be the coordinate at the rightmost position (based on Figure 26) within the wheelhouse (2011).

[0448] Next, the coordinates within the image acquired from the first camera (CA1) can be converted (S36057). Specifically, when a user is detected from multiple first cameras (CA1), the pixel coordinates within the image of the first camera (CA1) are converted into coordinates in real space, and the viewpoint difference and positional deviation between the multiple first cameras (CA1) are corrected, thereby improving the accuracy of the integrated image described later based on the user's location.

[0449] Referring to Fig. 31, camera combinations and weights can be calculated based on the user's location (S36059). In the case of detecting a user with multiple first cameras (CA1), if the user (U1) is located relatively to the left of the center of the wheelhouse (2011) within the wheelhouse (2011) of the ship (2010), the final location (coordinates) of the user within the wheelhouse (2011) can be determined by assigning a weight to the left first camera (CA1) positioned closer to the user (U1) relative to the center of the wheelhouse (2011).

[0450] In addition, when the user (U1) is located on the left side with respect to the center of the wheelhouse (2011), a weight can be given to the second camera (CA2) located closer to the user than to the third camera (CA3) located on the right side, and this can be utilized when creating a matched image.

[0451] The image generation device (3400) can acquire images from each camera and store the data in memory (3430) (S3610). Here, each camera refers to the first camera (CA1) to the fifth camera described above.

[0452] The image generation device (3400) can synchronize the time between images from each camera (S615). Images can be acquired at a preset frame rate (e.g., 30 fps) for each camera.

[0453] When synchronizing time between images, the maximum frame time difference can be compared with the allowable time. Specifically, if the allowable time is 50ms, any difference less than the allowable time is considered the same and the original frame can be returned.

[0454] Conversely, if the allowed time is exceeded, slow frames can be removed one by one and compared again, with the allowed time gradually increased. If the allowed time is still exceeded despite the increased time, previous frames that fall below the allowed time can ultimately be used.

[0455] The image generation device (3400) can calculate internal parameters and distortion coefficients of each camera (S3620). In addition, the image generation device (3400) can additionally calculate external parameters between the cameras, wherein the external parameters may be rotation and translation vectors (S3625).

[0456] The image generation device (3400) can define an obstacle area in the first image collected from the first camera (CA1) (S3630). In the present invention, the "obstacle area" may be an area occupied by a window frame (WF), which is a structure that forms a blind area and is located inside the wheelhouse (2011), when an image is captured by the first camera (CA1) installed adjacent to, or specifically inside, the wheelhouse (2011).

[0457] Referring to Fig. 27, the area occupied by the window frame (WF) forming the obstacle area may be an outer area of ​​the window (W), specifically, a side area.

[0458] In the first image, the obstacle area can be classified and managed as a separate layer from the rest of the first image.

[0459] Referring to FIG. 27, there are obstacle areas located in a total of three window frames (WF), but this is not limited thereto and may increase depending on the total number of window frames (WF) located inside the wheelhouse (2011).

[0460] The image generation device (3400) can first correct the misalignment of each individual camera (S3635). At this time, the image generation device (3400) can identify and correct the rotation angle of each camera through information linkage with the ship (2010) sensor. Subsequently, the image generation device (3400) can secondarily correct the misalignment of each individual camera (S3640). In step S3640, the image generation device (3400) can correct the roll misalignment based on the sea level, and can correct the pitch misalignment through the visual correlation between the singularity of the ship (2010) structure and the sea level.

[0461] The image generation device (3400) can perform image batch preprocessing for alignment (S3645). At this time, the RGB values ​​of the images acquired from the second camera (CA2) and the third camera (CA3) can be normalized to the range of 0 to 1. Furthermore, the resolution can be adjusted by resizing the images. Furthermore, histogram equalization can be used to uniformize the brightness distribution between images under different lighting conditions. In other words, there is an effect of enhancing the performance of feature extraction by enhancing contrast.

[0462] In step S3645, a minimum of four and a maximum of six frames captured by each camera can be organized into a single batch, and the images can be converted into tensors and then processed for operation on a graphics processing unit (GPU).

[0463] The image generation device (3400) can extract feature points from a minimum of four and a maximum of six frames based on deep learning (S3650). In step S650, common feature points can be extracted from images captured by the second camera (CA2) and the third camera (CA3).

[0464] The image generation device (3400) can align the second image and the third image based on matching common feature points extracted in step S3650.

[0465] As an optional embodiment, common feature points may be extracted from images captured by two cameras positioned closely among the second camera (CA2) to the fourth camera (CA4), and a matched “target image” may be generated based on matching the common feature points.

[0466] As an optional embodiment, a common feature point may be extracted from an image captured by one of the second camera (CA2) to the fourth camera (CA4) and an image captured by the first camera (CA1), and a matched target image may be generated based on matching the common feature points.

[0467] The image generation device (3400) can align the first image with other images based on common feature point matching (S3655). In step S3655, two bidirectional closest feature points can be selected and then matched. Furthermore, outliers can be removed through the distance ratio between the two feature points, and images from different viewpoints can be aligned through a homography calculation.

[0468] The image generation device (3400) can perform a primary correction on the "integrated image" generated in step S3655 (S3660). In step S3660, spatial distortion can be precisely corrected based on the geometric relationship between each viewpoint through multi-view geometry-based area correction.

[0469] Referring to FIGS. 30 and 32, the image generation device (3400) can perform secondary correction on the integrated image (S3665). First, weather information can be collected from the Automatic Identification System (AIS) (S36651).

[0470] Afterwards, it can be classified (S36653) based on sunlight intensity, rainfall status, fog status, and time.

[0471] Depending on the sunlight intensity, visibility can be secured with a transparency of about 70% for sunlight stronger than a preset standard, and visibility can be secured with a transparency of about 30% to 40% for relatively weak sunlight, such as cloudy weather or weak sunlight (S36655a).

[0472] Depending on the precipitation conditions, such as snow or rain, screen distortion caused by water droplets or snow crystals can be corrected and the clarity adjusted (S36655b). Depending on the foggy conditions, contrast can be enhanced to ensure visibility (S36655c). Depending on the time of day, visibility can be secured by dividing the day into preset time periods and dynamically adjusting transparency for each time period (S36655d). A corrected image can be generated through steps S36655a to S36655d (S36657).

[0473] As an optional embodiment, in the case of night, areas that are relatively darker than the preset brightness standard can be brightened by gamma adjustment.

[0474] As an optional embodiment, when analyzing the properties of images acquired from each camera, they can also be classified based on brightness, contrast, color, and blur.

[0475] Specifically, for clear days with relatively high brightness, high contrast, and blue color tones, contrast can be adjusted for greater clarity. Conversely, for cloudy days with relatively low brightness, low contrast, gray color tones, and low contrast and clarity, brightness and contrast can be increased.

[0476] Referring to Figure 30, the secondary correction of the integrated image is performed after the primary correction described above, but is not limited thereto, and various modifications are possible, such as being performed as preprocessing before synchronizing the time between images acquired from each camera (S615).

[0477] The image generation device (3400) can replace the obstacle area in the first image of the first camera (CA1) with a matched image (target image) (S3670). The image generation device (3400) can transparently synthesize (or process) the area corresponding to the obstacle area in the integrated image (S3675), and output the integrated image through the display unit (3440), screen output unit (3450), and electronic devices in the wheelhouse (2011) (S3680).

[0478] As an optional embodiment, the transparency of the area corresponding to the obstacle area may be adjusted to display it semi-transparently.

[0479] Referring to FIG. 33, an integrated image can be output through a display unit (3440). That is, an integrated image, specifically an area corresponding to an obstacle area in the first image, can be displayed through a display unit (3440) that is fixedly positioned on a window frame (WF) located between a plurality of windows (W) that are positioned closely and spaced apart from each other.

[0480] This has the effect of allowing the user to see an image of the area in front of the wheelhouse (2011), which actually forms a blind section by the window frame (WF), and allowing the user to see objects (OB1, OB2) that are hidden by the obstacle area from inside the wheelhouse (2011).

[0481] The image generation device (3400) can detect the positions of users (U1, U2, U3) in real time after generating the final image, which is an integrated image, and generate a new integrated image as the user's position changes.

[0482] First, the user's location can be detected. The image generation device (3400) accumulates and calculates the change in the user's location over a preset period of time, and if the total change in location exceeds a preset standard (threshold), the camera that acquires the image can be reselected to generate multiple integrated images.

[0483] When a camera other than the first camera (CA1), for example, a camera to be aligned among the second camera (CA2) to the fourth camera (CA4), is selected, a new integrated image (hereinafter referred to as a “second integrated image”) can be generated by going through steps S3605 to S3680.

[0484] Once the secondary integrated image is generated, it can be gradually reflected by blending it with the primary integrated image, which is the previous integrated image. Specifically, as frames change over time, the secondary integrated image is given a relatively greater weight than the primary integrated image, so that it can ultimately be replaced by the secondary integrated image over time.

[0485] Figure 34 is a flowchart illustrating another embodiment of an image generation method for generating an integrated image according to the present invention.

[0486] Since the image generation method according to Fig. 34 can be implemented by the processor and sub-modules included in the processor described in Fig. 28, any description that overlaps with what has already been described will be omitted.

[0487] The image generation device can collect images from the ship's first camera (CA1), second camera (CA2), and third camera (CA3) (S4010). As mentioned above, it is also possible to collect images from the fourth camera (CA4).

[0488] The image generation device can extract overlapping feature points between the second image and the third image collected from the second camera (CA2) and the third camera (CA3) (S4030).

[0489] The image generation device can generate a target image by aligning the second image and the third image based on the extracted feature points (S4050).

[0490] The image generation device can generate an integrated image by integrating the first image collected from the first camera (CA1) and the generated target image (S4070). In the present invention, the integrated image may also be referred to as a ship image.

[0491] A ship's Onboard Carbon Capture System (OCCS) captures carbon dioxide (CO2) directly from the ship's exhaust gas. OCCS was developed as part of efforts to reduce the environmental impact of the shipping industry, particularly in response to international regulations aimed at reducing greenhouse gas emissions from ships.

[0492] Ships equipped with OCCS can significantly reduce carbon emissions by capturing CO2 from exhaust gas and blocking it before it is released into the atmosphere. Furthermore, ships equipped with OCCS can store the captured CO2 in a liquid form onboard, and the liquefied carbon dioxide can be recycled in industrial processes or appropriately disposed of after arriving at port. Furthermore, ships equipped with OCCS can further reduce CO2 emissions compared to non-equipped ships, making it easier to comply with International Maritime Organization (IMO) environmental regulations. Thus, while OCCS technology is still in its early stages, it can play a key role in transforming the shipping industry toward more environmentally friendly operations.

[0493] Meanwhile, ships with conventional OCCS have limited height and size of LCO2 tanks (liquefied carbon dioxide tanks) on the bow side to secure a wide field of view based on the steering line.

[0494] Figures 35 and 36 are schematic drawings showing a ship to which a conventional OCCS is applied.

[0495] Fig. 35a is an exemplary view of the starboard side of a ship with OCCS applied, wherein the ship with OCCS applied includes a stern side LCO2 tank (4301) and a bow side LCO2 tank (4302) installed adjacent to the wheelhouse (4010). Fig. 35b is a plan view of a ship with OCCS applied, wherein, referring to Fig. 35b, the ship with OCCS applied includes two bow side LCO2 tanks (4302, 4303).

[0496] The vessel to which the OCCS illustrated in Fig. 36 is applied includes an integrated LCO2 tank (4304) in which the bow-side LCO2 tanks (4302, 4303) of Fig. 35 are integrated into one. As illustrated in Fig. 36, when the integrated LCO2 tank (4304) is placed on the bow side of the vessel, a blind spot (blind spot) is created, which causes a problem in that people or other objects behind the integrated LCO2 tank (4304) cannot be recognized in the wheelhouse (4010).

[0497] Figure 37 is a diagram schematically showing a structure in which the structure of a ship to which a conventional OCCS is applied is improved in order to implement a method according to the present invention.

[0498] The OCCS-applied vessel (4001) according to the present invention may include a total of four cameras. The first camera (CA1) may be installed adjacent to the wheelhouse (4010). The second camera (CA2) and the third camera (CA3) may be installed in the integrated LCO2 tank (4304) on the bow side, respectively. The second camera (CA2) and the third camera (CA3) may capture the first target (ob1) and the second target (ob2) in the blind spot of the integrated LCO2 tank (4304). The fourth camera (CA4) may be installed at the bow end of the OCCS-applied vessel (4001). The fourth camera (CA4) may capture the second target (ob2).

[0499] As an example, the second camera (CA2) and the third camera (CA3) may be installed in the LCO2 tank closest to the bow of the OCCS-applied vessel (4001).

[0500] As another example, the second camera (CA2) and the third camera (CA3) may be installed in one LCO2 tank adjacent to the bow of the OCCS-applied vessel (4001). In addition, the second camera (CA2) and the third camera (CA3) may be installed symmetrically with respect to one LCO2 tank.

[0501] The method according to the present invention can generate an integrated image with improved visibility by using images collected by the first camera (CA1), the second camera (CA2), the third camera (CA3), and the fourth camera (CA4) described in FIG. 37. Hereinafter, the device for generating the integrated image will be referred to as an integrated image generating device (4400). The integrated image generated by the integrated image generating device (4400) will be described in detail with reference to FIGS. 38 to 45.

[0502] Figure 38 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0503] The integrated image generation device (4400) according to the present invention generates an integrated image so that it can be played back through an electronic device used by a captain or navigator in the wheelhouse (4010). The captain or navigator can easily obtain information necessary for navigating a vessel through the integrated image output through the electronic device. Referring to FIG. 38, it can be seen that the integrated image generation device (4400) includes a communication unit (4410), a processor (4430), and a memory (4450).

[0504] The communication unit (4410) may include one or more components that enable wired / wireless communication with external devices. For example, the communication unit (4410) may include at least one piece of hardware necessary to implement short-range communication, such as Wi-Fi or Bluetooth, in a network provided by a communication network, or to implement various communications, including the Internet, when a LAN cable is connected.

[0505] The memory (4450) is hardware that stores various data processed within the integrated image generation device (4400), and can store a program for processing and controlling the processor (4430). The memory (4450) may include a random access memory (RAM) such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM, a Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

[0506] The processor (4430) can control the overall operation of the integrated image generation device (4400). For example, the processor (4430) can control the operation of the input unit (not shown), display (not shown), communication unit (4410), memory (4450), etc. included in the integrated image generation device (4400) by executing programs stored in the memory (4450).

[0507] As an example, the processor (4430) may collect images from the first camera, the second camera, and the third camera of the ship, extract overlapping feature points between the second and third images collected from the second and third cameras, match the second and third images based on the extracted feature points to generate a target image, and integrate the first image collected from the first camera and the generated target image to generate an integrated image. The specific process of the processor (4430) will be described later with reference to FIGS. 39 to 45.

[0508] When the integrated image generation device (4400) is implemented as a physical device, the processor (4430) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0509] In addition, in the present invention, when the integrated image generation device (4400) is implemented in the form of an application (program) that runs on an integrated data processing device such as a server, the processor (4430) and memory (4450) included in the integrated image generation device (4400) may be implemented in the form of a virtual machine that implements hardware such as DSPs, microcontrollers, RAM, ROM, HDD, etc. as software (command script).

[0510] Figure 39 is a drawing for explaining a sub-module included in the processor of Figure 38.

[0511] Below, the explanation will be given with reference to Fig. 37.

[0512] Referring to FIG. 39, it can be seen that the processor (4430) includes a first image processing unit (4431), a second image processing unit (4433), a third image processing unit (4435), and an integrated image generation unit (4437). The first image processing unit (4431), the second image processing unit (4433), the third image processing unit (4435), and the integrated image generation unit (4437) illustrated in FIG. 39 are modules that are logically and conceptually separated in order to explain the process performed by the processor (4430) in the process of implementing the method according to the present invention. Therefore, although four sub-modules are illustrated in FIG. 39, the processor (4430) may include fewer than four or more than four sub-modules depending on the embodiment. In addition, the first image processing unit (4431), the second image processing unit (4433), the third image processing unit (4435), and the integrated image generation unit (4437) of FIG. 39 are sub-modules of the processor (4430), and thus, like the processor (4430), they can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0513] When images are collected from the first camera (CA1), the second camera (CA2), and the third camera (CA3) of the OCCS-applied ship (4001), the first image processing unit (4431) can extract overlapping feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3).

[0514] The second image processing unit (4433) can generate a target image by matching the second image and the third image based on the feature points extracted by the first image processing unit (4431). The second image processing unit (4433) stores at least one image registration algorithm and sufficient hardware and resources to execute the algorithm in order to match images generated from different cameras. Here, the target image refers to data that processes the appearance information of the first target (ob1) or the second target (ob2) shielded by the integrated LCO2 tank (4304) into an image when observed from the wheelhouse (4010), and can be generated by matching the second image and the third image.

[0515] The third image processing unit (4435) can identify an obstacle area in the first image using a preset object recognition algorithm. Here, the obstacle area may be the area occupied by the integrated LCO2 tank (4304) based on the wheelhouse (4010) of the OCCS-applied vessel (4001) in the first image. The obstacle area will be described later with reference to FIGS. 41 and 44.

[0516] The integrated image generation unit (4437) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the target image generated by the second image processing unit (4433). At this time, the first image and the target image can be integrated by the third image processing unit (4435) based on a specific obstacle area.

[0517] Figure 40 is a flowchart showing the process of generating an integrated image by an integrated image generation device, by unit process.

[0518] Since Fig. 40 can be implemented by the integrated image generation device (4400) described in Fig. 4, the description overlapping with the description in Fig. 38 and Fig. 39 will be omitted, and the following description will be made with reference to Fig. 38 and Fig. 39.

[0519] The integrated image generation device (4400) can acquire images from each camera and store the data in memory (4450) (S4605). Here, each camera refers to the first camera (CA1) to the fourth camera (CA4) described above.

[0520] The integrated image generation device (4400) can calculate internal parameters and distortion coefficients of each camera (S4610). In addition, the integrated image generation device (4400) can additionally calculate external parameters between cameras, wherein the external parameters may be rotation and translation vectors (S4615).

[0521] The integrated image generation device (4400) can define an obstacle area in the first image collected from the first camera (CA) (S4620). Typically, the obstacle area is the area occupied by the integrated LCO2 tank (4304) when the image is captured by the adjacent first camera (CA1) of the wheelhouse (4010). In the first image, the obstacle area is classified and managed as a separate layer from the remaining areas of the first image.

[0522] The integrated image generation device (4400) can first correct the misalignment of each individual camera (S4625). At this time, the integrated image generation device (4400) can identify and correct the rotation angle of each camera through information linkage with the ship sensor. Subsequently, the integrated image generation device (4400) can secondarily correct the misalignment of each individual camera (S4630). In step S4630, the integrated image generation device (4400) can correct the roll misalignment based on the sea level, and can correct the pitch misalignment through the visual correlation between the unique points of the ship structure and the sea level.

[0523] The integrated image generation device (4400) can extract common feature points from images captured by the second camera (CA2) and the third camera (CA3) (S4635). The integrated image generation device (4400) can align the second and third images based on the matching of the common feature points extracted in step S4635 (S4640). The integrated image generation device (4400) can project the aligned "target image" onto the rear camera view (S4645).

[0524] The integrated image generation device (4400) can replace the obstacle area in the first image of the first camera (CA1) with the aligned image from step S4640 (S4650). The integrated image generation device (4400) can transparently synthesize only the obstacle area in the integrated image generated from step S4650 (S4655) and output the integrated image through an electronic device in the wheelhouse (4010) (S4660).

[0525] Figure 41 is a drawing showing an example of an obstacle area being designated in the first image.

[0526] FIG. 41 is a three-dimensional image of the first image taken in the direction of the bow from the wheelhouse (4010). Referring to FIG. 41, it can be seen that the blind spot (blind spot) behind the integrated LCO2 tank (4304) is not secured at all due to the integrated LCO2 tank (4304). The integrated image generation device (4400) can designate the area where the integrated LCO2 tank (4304) is located in the first image as an obstacle area according to step S4620 of FIG. 40.

[0527] Figure 42 is a drawing showing a second image generated by a second camera as a three-dimensional image.

[0528] More specifically, FIG. 42 is an example of a second image (4810) taken by a second camera (CA2) installed in an integrated LCO2 tank (4304) toward the player side, and referring to FIG. 37, it can be seen that it includes some common features with a third image (910) generated by a third camera (CA3).

[0529] Figure 43 is a drawing showing a third image generated by a third camera as a three-dimensional image.

[0530] More specifically, FIG. 43 is an example of a third image (4910) taken by a third camera (CA3) installed in an integrated LCO2 tank (4304) toward the player side, and referring to FIG. 37, it can be seen that it includes some common features with the second image (810) generated by the second camera (CA2) of FIG. 42. The integrated image generation device (4400) can generate a target image by matching the second image (4810) and the third image (4910) based on the features, as illustrated in FIGS. 42 and 43.

[0531] Figure 44 is a drawing showing an integrated image generated by an integrated image generation device as a three-dimensional image.

[0532] Referring to FIG. 44, in the integrated image (5010), the integrated LCO2 tank (4304) designated as an obstacle area is made transparent, and the target image is replaced behind it, so that objects in the blind spot behind the integrated LCO2 tank (4304), which would not have been observed if originally filmed by the first camera (CA1) of the wheelhouse (4010), can be included and displayed in the integrated image (1010).

[0533] Figure 45 is a drawing showing an example of the results of comparing a conventional first image and an integrated image according to the present invention by implementing them as two-dimensional images.

[0534] The left drawing of Fig. 45 is an example of a first image generated by a first camera in a conventional OCCS-applied ship. In the left drawing of Fig. 45, the bow end of the OCCS-applied ship is completely shielded due to the height and size of the integrated LCO2 tank (4304), so nothing is observed.

[0535] Meanwhile, the right drawing of FIG. 45 is an exemplary integrated image according to the present invention. In the right drawing of FIG. 45, it can be seen that not only the first target (ob1) located at the bow end of the OCCS-applied ship (4001) but also the second target (ob2), another ship located at the front of the OCCS-applied ship (4001), is observed regardless of the height and size of the integrated LCO2 tank (4304).

[0536] Since the method according to the present invention can be implemented by the processor (4430) and sub-modules included in the processor (4430) described in FIG. 38, the method will be described below with reference to FIGS. 37 to 45, and any description that overlaps with the content already described will be omitted.

[0537] The integrated image generation device (4400) can collect images from the first camera (CA1), the second camera (CA2), and the third camera (CA3) of the ship.

[0538] The integrated image generation device (4400) can extract overlapping feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3).

[0539] The integrated image generation device (4400) can generate a target image by aligning the second image and the third image based on the extracted feature points.

[0540] The integrated image generation device (4400) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the generated target image. In the present invention, the integrated image may also be referred to as a ship image.

[0541] A rotor sail is a wind-assisted propulsion device installed on a ship's deck. It utilizes wind to generate propulsion. A rotor sail is a cylindrical structure that generates additional propulsion, reducing fuel consumption and carbon emissions. Rotor sails, which rotate via an electric motor, encounter wind blowing around the ship. This pressure difference creates a pressure difference around the sail, which, in turn, generates forward propulsion through the Magnus effect. Hereinafter, ships equipped with rotor sails will be referred to as "wind-powered ships."

[0542] Figure 46 is a drawing showing an example of a wind-powered vessel.

[0543] The first image (5110) of FIG. 46 is a drawing illustrating a 2D image of a wind-powered vessel with a rotor sail installed. More specifically, the first image (5110) is a 2D image of the right side of the wind-powered vessel. Referring to the first image (5110), it can be seen that two of the three rotor sails are installed close to the bow of the wind-powered vessel, and the remaining one is installed close to the stern of the wind-powered vessel. In an actual wind-powered vessel, three rotor sails are installed on each of the left and right sides of the wind-powered vessel and operate.

[0544] Figure 47 is a drawing showing a typical wind-powered ship by configuration.

[0545] The wind-powered vessel (5001) of Fig. 47 may include a wheelhouse (5010), a port side (5011) of a bridge wing, a starboard side (5012) of a bridge wing, a funnel (5020), and at least one rotor sail (5100). The wheelhouse (5010) is a space for managing and supervising the operation of the wind-powered vessel (5001), and various control panels are installed therein to control the safe and efficient navigation of the wind-powered vessel (5001) while the captain and navigator reside there. The port side (5011) of the bridge wing and the starboard side (5012) of the bridge wing refer to protruding spaces at both ends of the wheelhouse (5010), and crew members can be temporarily stationed there as needed to assist the operation and manipulation of the wind-powered vessel (5001) performed in the wheelhouse (5010).

[0546] The chimney (5020) refers to a funnel-shaped structure that safely discharges exhaust gas generated from the engine of a wind-powered vessel (5001) to the outside. Since the wind-powered vessel (5001) is not a type that completely omits the engine, the chimney (5020) is essential to efficiently process exhaust gas generated when the auxiliary engine or auxiliary equipment operates, and if the position and height of the chimney (5020) are appropriately designed to properly discharge exhaust gas through the chimney (5020), the efficiency of the rotor sail (5100) may not be reduced.

[0547] A rotor sail (5100) is a cylindrical structure installed on the deck of a wind-powered vessel (5001). It rotates based on an electric motor to induce the Magnus effect and generate propulsion force for the wind-powered vessel (5001). At least one rotor sail (5100) is installed, and in FIG. 47, four rotor sails (5100) are installed in a direction crossing the bow and stern of the hull, but the number and arrangement characteristics of the rotor sails (5100) may vary depending on the embodiment.

[0548] Meanwhile, the wind-powered vessel described in FIGS. 46 and 47 has a rotor sail installed high on the deck, so when monitoring the bow direction from the wheelhouse, visibility is reduced by the rotor sail, and objects in the blind spot of the rotor sail cannot be seen, or other ships or other floating objects floating on the sea and coming close to the bow cannot be seen.

[0549] Figures 48a and 48b are drawings for explaining the visibility of a wind-powered vessel.

[0550] Fig. 48a is a drawing illustrating an example of a phenomenon in which the first target (ob1) and the second target (ob2) are shielded by the rotor sail and thus cannot be observed when observing the bow direction from the wheelhouse (5010) of a wind-powered vessel. Fig. 48b is a drawing illustrating the phenomenon in which the first target (ob1) and the second target (ob2) described in Fig. 48a are not observed from a different direction.

[0551] Figure 49 is a diagram schematically showing the structure of an improved wind-powered vessel that improves the structure of a conventional wind-powered vessel in order to implement a method according to the present invention.

[0552] The wind-powered vessel (5001') according to the present invention may include a total of four cameras. The wind-powered vessel (5001') according to the present invention is the same as the conventional wind-powered vessel (5001) in that at least one rotor sail (5100) is protruding and installed on the deck. However, the wind-powered vessel (5001') according to the present invention has a differentiated feature in that it additionally installs several cameras that were not installed in the conventional wind-powered vessel (5001) and further includes a device that creates an integrated image by aligning images captured by several cameras, thereby ensuring improved visibility compared to the conventional wind-powered vessel (5001). The first camera (CA1) to the fourth camera (CA4) illustrated in FIG. 49 are considered devices that can not only capture an object to create an image, but also have a built-in communication module that can transmit the created image to an external device via wired / wireless communication.

[0553] The first camera (CA1) may be installed inside the wheelhouse (5010) or adjacent to the wheelhouse (5010). The first camera (CA1) may capture a bow direction from the wheelhouse (5010) to generate a first image. The second camera (CA2) and the third camera (CA3) may be installed on the port and starboard sides of the bridge wing of the wind-powered vessel (5001'), respectively. The second camera (CA2) and the third camera (CA3) may be installed symmetrically on the bridge wing of the wind-powered vessel (5001') based on the same distance from the first camera (CA1). Referring to FIG. 49, the distance between the first camera (CA1) and the second camera (CA2) may be the same as the distance between the first camera (CA1) and the third camera (CA3).

[0554] The second camera (CA2) and the third camera (CA3) can capture the first target (ob1) and the second target (ob2) in the blind spot of the rotor sail (5100) to generate the second image and the third image, respectively. In FIG. 49, two rotor sails (5100) are installed in front of the first camera (CA1), so that the first camera (CA1) alone cannot capture the first target (ob1) shielded by the rotor sail (5100), but the second camera (CA2) and the third camera (CA3) are installed on the port and starboard sides of the bridge wing, respectively, to generate the second image and the third image including the first target (ob1).

[0555] If the second object (ob2) is located at a certain distance from the player and cannot be sufficiently identified by the second camera (CA2) and the third camera (CA3), the second object (ob2) may be captured by the fourth camera (CA4). The fourth camera (CA4) may be installed at the very tip of the bow of the wind-powered vessel (5001'). The fourth camera (CA4) may capture the second object (ob2) to generate a fourth image.

[0556] The method according to the present invention can generate an integrated image with improved visibility by using the images collected by the first camera (CA1), the second camera (CA2), the third camera (CA3), and the fourth camera (CA4) described in FIG. 49. Hereinafter, the device for generating the integrated image will be referred to as an integrated image generating device (5500). The integrated image generated by the integrated image generating device (5500) will be described in detail with reference to FIGS. 50 to 56.

[0557] Figure 50 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0558] The integrated image generation device (5500) according to the present invention generates an integrated image so that it can be played back through an electronic device used by a captain or navigator in the wheelhouse (5010). The captain or navigator can easily obtain information necessary for navigating a vessel through the integrated image output through the electronic device. Referring to FIG. 50, it can be seen that the integrated image generation device (5500) includes a communication unit (5510), a processor (5530), and a memory (5550).

[0559] The communication unit (5510) may include one or more components that enable wired / wireless communication with external devices. For example, the communication unit (5510) may include at least one piece of hardware necessary to implement short-range communication, such as Wi-Fi or Bluetooth, in a network provided by a communication network, or to implement various communications, including the Internet, when a LAN cable is connected.

[0560] The memory (5550) is a hardware that stores various data processed within the integrated image generation device (5500), and can store a program for processing and controlling the processor (5530). The memory (5550) may include a random access memory (RAM) such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM, a Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

[0561] The processor (5530) can control the overall operation of the integrated image generation device (5500). For example, the processor (5530) can control the operation of the input unit (not shown), display (not shown), communication unit (5510), memory (5550), etc. included in the integrated image generation device (5500) by executing programs stored in the memory (5550).

[0562] As an example, the processor (5530) may collect images from the first camera, the second camera, and the third camera of the ship, extract overlapping feature points between the second and third images collected from the second and third cameras, match the second and third images based on the extracted feature points to generate a target image, and integrate the first image collected from the first camera and the generated target image to generate an integrated image. The specific process of the processor (5530) will be described later with reference to FIGS. 51 to 56.

[0563] When the integrated image generation device (5500) is implemented as a physical device, the processor (5530) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0564] In addition, in the present invention, when the integrated image generation device (5500) is implemented in the form of an application (program) that runs on an integrated data processing device such as a server, the processor (5530) and memory (5550) included in the integrated image generation device (5500) may be implemented in the form of a virtual machine that implements hardware such as DSPs, microcontrollers, RAM, ROM, HDD, etc. as software (command script).

[0565] Figure 51 is a drawing for explaining a sub-module included in the processor of Figure 50.

[0566] Below, the explanation will be given with reference to Fig. 49.

[0567] Referring to FIG. 51, it can be seen that the processor (5530) includes a first image processing unit (5531), a second image processing unit (5533), a third image processing unit (5535), and an integrated image generation unit (5537). The first image processing unit (5531), the second image processing unit (5533), the third image processing unit (5535), and the integrated image generation unit (5537) illustrated in FIG. 51 are modules that are logically and conceptually separated in order to explain the process performed by the processor (5530) in the process of implementing the method according to the present invention. Therefore, although four sub-modules are illustrated in FIG. 51, the processor (5530) may include fewer than four or more than four sub-modules depending on the embodiment. In addition, the first image processing unit (5531), the second image processing unit (5533), the third image processing unit (5535), and the integrated image generation unit (5537) of FIG. 51 are sub-modules of the processor (5530), and thus, like the processor (5530), they can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0568] When images are collected from the first camera (CA1), the second camera (CA2), and the third camera (CA3) of the wind-powered vessel (5001'), the first image processing unit (5531) can extract overlapping feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3).

[0569] The second image processing unit (5533) can generate a target image by matching the second image and the third image based on the feature points extracted by the first image processing unit (5531). The second image processing unit (5533) is equipped with at least one image registration algorithm and sufficient hardware and resources to execute the algorithm in order to match images generated from different cameras. Here, the target image refers to data that processes the appearance information of the first target (ob1) or the second target (ob2) shielded by the rotor sail (5100) into an image when observed from the wheelhouse (5010), and can be generated by matching the second image and the third image.

[0570] The third image processing unit (5535) can use a preset object recognition algorithm to identify an obstacle area in the first image. Here, the obstacle area may be the area occupied by the first and second closest rotor sails (5100) from the wheelhouse (5010) in the first image. Depending on the embodiment, the obstacle area may be wider or narrower than a preset value. The obstacle area will be described later with reference to FIGS. 52 and 56.

[0571] The integrated image generation unit (5537) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the target image generated by the second image processing unit (5533). At this time, the first image and the target image can be integrated by the third image processing unit (5535) based on a specific obstacle area.

[0572] Hereinafter, based on FIGS. 50 and 51, the process of generating an integrated image by the integrated image generation device (5500) and the sub-modules included in the integrated image generation device (5500) will be described for each unit process.

[0573] The integrated image generation device (5500) can acquire images from each camera and store the data in a memory (5550). Here, each camera refers to the first camera (CA1) to the fourth camera (CA4) described above.

[0574] The integrated image generation device (5500) can calculate internal parameters and distortion coefficients of each camera. In addition, the integrated image generation device (5500) can additionally calculate external parameters between cameras, wherein the external parameters may be rotation and translation vectors.

[0575] The integrated image generation device (5500) can define an obstacle area in the first image collected from the first camera (CA1). Typically, the obstacle area is the area occupied by the rotor sail (5100) seen in the forward direction when the image is captured by the adjacent first camera (CA1) of the wheelhouse (5010). The obstacle area in the first image can be classified as a separate layer from the remaining areas of the first image and managed separately.

[0576] The integrated image generation device (5500) can first correct the misalignment of each individual camera. At this time, the integrated image generation device (5500) can identify and correct the rotation angle of each camera through information linkage with the sensor of the wind-powered vessel (5001'). Subsequently, the integrated image generation device (5500) can secondarily correct the misalignment of each individual camera. At this time, the integrated image generation device (5500) can correct the roll misalignment based on the sea level, and can correct the pitch misalignment through the visual correlation between the singularity of the structure of the wind-powered vessel (5001') and the sea level.

[0577] The integrated image generation device (5500) can extract common feature points from the second and third images captured by the second camera (CA2) and the third camera (CA3). The integrated image generation device (5500) can align the second and third images based on the matching of the extracted feature points. The integrated image generation device (5500) can project the aligned "target image" onto the rear camera view.

[0578] The integrated image generation device (5500) can generate an integrated image by replacing the obstacle area in the first image of the first camera (CA1) with the target image (aligned image) generated in the previous step. The integrated image generation device (5500) can transparently synthesize only the obstacle area in the integrated image and control the output of the integrated image through electronic devices in the wheelhouse (5010).

[0579] Figure 52 is a drawing showing the first image generated by the first camera as a three-dimensional image.

[0580] FIG. 52 illustrates a third image (5710) that shows the result of looking in the direction of the player from inside the wheelhouse (5010) and a fourth image (5730) that exemplarily shows the first image captured by the first camera (CA1).

[0581] The third image (5710) is an image created assuming that a crew member inside the wheelhouse (5010) is observing the direction of the bow. The third image (5710) also depicts the glass window, window frame, and chairs inside the wheelhouse (5010).

[0582] The fourth image (5730) is a three-dimensional image of the first image, and referring to FIG. 52, it can be seen that the blind spot (blind spot) behind the rotor sail is not secured at all due to the rotor sail (5100). As described in FIG. 51, the integrated image generation device (5500) can designate the area where the rotor sail is located in the first image as an obstacle area (5070). As described above, the integrated image generation device (5500) stores a unique object recognition algorithm in order to selectively recognize the rotor sail in the image and designate it as an obstacle area (5070).

[0583] Figure 53 is a drawing showing a second image generated by a second camera as a three-dimensional image.

[0584] More specifically, FIG. 53 is an example of a second image (5810) taken by a second camera (CA2) installed on the port side of the bridge wing toward the bow side. Referring to FIG. 49, it can be seen that the second image (5810) includes some common features (e.g., information about the first target (ob1) or the second target (ob2)) with the third image (5910) generated by the third camera (CA3).

[0585] Figure 54 is a drawing showing a third image generated by a third camera as a three-dimensional image.

[0586] More specifically, FIG. 54 is an example of a third image (5910) taken by a third camera (CA3) installed on the starboard side of the bridge wing toward the bow side. Referring to FIG. 49, it can be seen that the third image (5910) includes some common features with the second image (810) generated by the second camera (CA2) of FIG. 53. The integrated image generation device (5500) can generate a target image by matching the second image (810) and the third image (910) based on the features, as illustrated in FIGS. 53 and 54.

[0587] Figure 55 is a drawing showing the fourth image generated by the fourth camera as a three-dimensional image.

[0588] More specifically, FIG. 55 is an example of a fourth image (6010) captured by the fourth camera (CA4) installed at the very tip of the bow of a wind-powered vessel (5001'). Referring to FIG. 49, it can be seen that the fourth image (6010) includes some common features with the second image (5810) generated by the second camera (CA2) in FIG. 53 and the third image (5910) generated by the third camera (CA3) in FIG. 54. Since the fourth camera (CA4) is installed at the very tip of the bow, the common features in the fourth image (6010) with the second image (5810) and the third image (5910) are mainly located at the sea surface. The integrated image generation device (5500) may generate a target image by aligning the second image (5810) and the third image (5910) based on feature points, as illustrated in FIGS. 53 and 54, and may also generate a target image by aligning the second image (5810) to the fourth image (6010), depending on the embodiment.

[0589] Figure 56 is a drawing showing an example of the results of comparing a conventional first image and an integrated image according to the present invention by implementing them as two-dimensional images.

[0590] The fifth image (6110) of FIG. 56 is an example of a first image generated by a first camera in a conventional wind-powered vessel (5001). In the fifth image (6110), the bow end of the wind-powered vessel (5001) is completely shielded due to the height and size of the rotor sail (5100), so nothing is observed.

[0591] Meanwhile, the sixth image (6130) of FIG. 56 is an exemplary integrated image according to the present invention, and in the sixth image (6130), it can be seen that not only the first object (ob1) located behind the rotor sail (5100) regardless of the height and size of the rotor sail (5100), but also another vessel and the second object (ob2) located in front of the wind-powered vessel (5001') are observed. In the sixth image (6130), the area where the rotor sail (5100) is located is classified as an "obstacle area", and then the obstacle area is replaced with the target image by the integrated image generating device (5500), and only the layer constituting the obstacle area is transparentized, so that as a result, objects located in the blind spot of the rotor sail (5100) can also be displayed in the integrated image. In the sixth image (6130), the transparent rotor sail (5100') is marked with a different number from the rotor sail (5100) in the fifth image (6110).

[0592] Since the method according to the present invention can be implemented by the integrated image generation device (5500), processor (5530) and sub-modules included in the processor (5530) described in FIG. 47, the following description will be given with reference to FIGS. 47 to 56, and any description that overlaps with the content already described will be omitted.

[0593] The integrated image generation device (5500) can collect first, second, and third images from the ship's first camera (CA1), second camera (CA2), and third camera (CA3). Depending on the embodiment, the integrated image generation device (5500) may additionally collect a fourth image from a fourth camera (CA4).

[0594] The integrated image generation device (5500) can extract overlapping feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3).

[0595] The integrated image generation device (5500) can generate a target image by aligning the second and third images based on the extracted feature points. In some embodiments, the integrated image generation device (5500) can also generate a target image by aligning the second to fourth images.

[0596] The integrated image generation device (5500) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the generated target image.

[0597] Figure 57 is a drawing showing a wind-powered ship by configuration, as an example, with a funnel installed at the stern.

[0598] The vessel (6001) of Fig. 57 may include a wheelhouse (6010), a port side (6011) of a bridge wing, a starboard side (6012) of a bridge wing, a funnel (6020), and at least one rotor sail (6100). The wheelhouse (6010) is a space for managing and supervising the operation of the vessel (6001), and various control panels are installed therein to control the vessel (6001) to navigate safely and efficiently while the captain and navigator reside there. The port side (6011) of the bridge wing and the starboard side (6012) of the bridge wing refer to protruding spaces at both ends of the wheelhouse (6010), and crew members can be temporarily stationed there as needed to assist the operation and manipulation of the vessel (6001) performed in the wheelhouse (10).

[0599] A chimney (6020) refers to a funnel-shaped structure that safely discharges exhaust gases generated from the engine of a ship (6001) to the outside.

[0600] A rotor sail (6100) is a cylindrical structure installed on the deck of a wind-powered vessel (6001) when the vessel is a wind-powered vessel. It rotates based on an electric motor to induce the Magnus effect and generate propulsive force for the wind-powered vessel. At least one rotor sail (6100) is installed on the wind-powered vessel, and in FIG. 57, four rotor sails (6100) are installed in a direction crossing the bow and stern of the hull, but the number and arrangement characteristics of the rotor sails (6100) may vary depending on the embodiment.

[0601] Figures 58a and 58b are drawings for explaining the visibility of a vessel equipped with a funnel.

[0602] Figure 58a is a drawing that exemplifies a phenomenon in which, when a crew member of a ship (6001) observes the stern direction from the port side (6011) of the bridge wing, the first object (ob1) is observed, but the second object (ob2) is not observed because it is blocked by the funnel (6020).

[0603] In addition, FIG. 58b is a drawing exemplifying a phenomenon in which, when the crew of a ship (6001) observes the stern direction from the starboard side (6012) of the bridge wing, the second object (ob2) is observed, but the first object (ob1) is not observed because it is blocked by the funnel (6020).

[0604] Although not shown in FIGS. 58a and 58b, most of the view in the wheelhouse (6010) is blocked by the funnel (6020), so that neither the first target (ob1) nor the second target (ob2) are observed. Consequently, due to the height and size of the funnel (6020), the ship (6001) has a blind spot in the view based on the wheelhouse (6010) and the port (6011) and starboard (6012) sides of the bridge wing, and there is a problem in that the first target (ob1) and the second target (ob2) cannot be observed simultaneously.

[0605] Figure 59 is a diagram schematically showing the structure of an improved ship that improves the structure of a conventional ship to implement a method according to the present invention.

[0606] More specifically, Fig. 59 focuses on the stern portion of a vessel (6001') according to the present invention. The vessel (6001') according to the present invention may include a total of four cameras. The vessel (6001') according to the present invention illustrated in Fig. 59 is depicted as a wind-powered vessel to correspond to the conventional vessel (6001) described in Fig. 57. However, it will be understood by those skilled in the art that the present invention can be applied regardless of the type or structure of the vessel, as long as the vessel has a funnel installed at the rear of the wheelhouse and thus has a limited view of the rear of the wheelhouse.

[0607] A vessel (6001') according to the present invention is identical to a conventional vessel (6001) in that at least one rotor sail (6100) is protruding and installed on the deck, but has a differentiated feature in that it additionally installs several cameras that were not installed in the conventional vessel (6001) and further includes a device that generates an integrated image by aligning images captured by several cameras, thereby ensuring improved visibility compared to the conventional vessel (6001). The first camera (CA1) to the fourth camera (CA4) illustrated in FIG. 59 are considered devices that can not only capture an object to generate an image, but also have a built-in communication module that can transmit the generated image to an external device via wired / wireless communication.

[0608] The first camera (CA1) may be installed inside the wheelhouse (6010) or adjacent to the wheelhouse (6010). The first camera (CA1) may capture a stern direction from the wheelhouse (6010) to generate a first image. The second camera (CA2) and the third camera (CA3) may be installed on the port (6011) and starboard (6012) sides of the bridge wing of the ship (6001') according to the present invention, respectively. The second camera (CA2) and the third camera (CA3) may be installed symmetrically on the bridge wing of the ship (6001') according to the present invention based on the same distance from the first camera (CA1). Referring to FIG. 59, the distance between the first camera (CA1) and the second camera (CA2) may be the same as the distance between the first camera (CA1) and the third camera (CA3).

[0609] The second camera (CA2) can capture a first object (ob1) that is outside the blind spot of the chimney (6020) based on the position of the second camera (CA2) to generate a second image. The third camera (CA3) can capture a second object (ob2) that is outside the blind spot of the chimney (6020) based on the position of the third camera (CA3) to generate a third image. In FIG. 59, a chimney (6020) is installed close to the front of the first camera (CA1), so that the first camera (CA1) alone cannot capture the first target (ob1) and the second target (ob2) shielded by the chimney (6020), but the second camera (CA2) and the third camera (CA3) are installed on the port side (6011) and starboard side (6012) of the bridge wing, respectively, to capture the stern direction of the ship (6001') according to the present invention, so that a second image including the first target (ob1) and a third image including the second target (ob2) can be generated, respectively.

[0610] If the first object (ob1) is located a certain distance from the stern of the ship (6001') according to the present invention, it may not be sufficiently identified by the second camera (CA2), and in this case, the first object (ob1) may be supplementarily photographed by the fourth camera (CA4). As illustrated in FIG. 59, the fourth camera (CA4) may be installed at the very end of the stern of the ship (6001') according to the present invention, or on the funnel (6020) facing the stern. The fourth camera (CA4) may photograph the first object (ob1) to generate a fourth image, and according to an embodiment, if the second object (ob2) is located close to the center of the stern, the fourth camera (CA4) may generate a fourth image in which not only the first object (ob1) but also the second object (ob2) is photographed together.

[0611] The method according to the present invention can generate an integrated image with improved visibility by using the images collected by the first camera (CA1), the second camera (CA2), the third camera (CA3), and the fourth camera (CA4) described in FIG. 59. Hereinafter, the device for generating the integrated image will be referred to as an integrated image generating device (6400). The integrated image generated by the integrated image generating device (6400) will be described in detail with reference to FIGS. 60 to 68.

[0612] Figure 60 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0613] The integrated image generation device (6400) according to the present invention generates an integrated image and transmits it to an electronic device used by a captain or navigator in the wheelhouse (6010) so that the integrated image can be played back on the electronic device. The captain or navigator can easily obtain information necessary for navigating the vessel through the integrated image output through the electronic device. Referring to FIG. 60, it can be seen that the integrated image generation device (6400) includes a communication unit (6410), a processor (6430), and a memory (6450).

[0614] The communication unit (6410) may include one or more components that enable wired / wireless communication with external devices. For example, the communication unit (6410) may include at least one piece of hardware necessary to implement short-range communication, such as Wi-Fi or Bluetooth, in a network provided by a communication network, or to implement various communications, including the Internet, when a LAN cable is connected.

[0615] The memory (6450) is a hardware that stores various data processed within the integrated image generation device (6400), and can store a program for processing and controlling the processor (6430). The memory (6450) may include a random access memory (RAM) such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM, a Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

[0616] The processor (6430) can control the overall operation of the integrated image generation device (6400). For example, the processor (6430) can control the operation of the input unit (not shown), display (not shown), communication unit (6410), memory (6450), etc. included in the integrated image generation device (6400) by executing programs stored in the memory (6450).

[0617] As an example, the processor (6430) may collect images from the first camera, the second camera, the third camera, and the fourth camera of the ship, extract overlapping feature points between the second to fourth images collected from the second to fourth cameras, match the second to fourth images based on the extracted feature points to generate a target image, and integrate the first image collected from the first camera and the generated target image to generate an integrated image. The specific process of the processor (6430) will be described later with reference to FIGS. 61 to 68.

[0618] When the integrated image generation device (6400) is implemented as a physical device, the processor (6430) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0619] In addition, in the present invention, when the integrated image generation device (6400) is implemented in the form of an application (program) that runs on an integrated data processing device such as a server, the processor (6430) and memory (6450) included in the integrated image generation device (6400) may be implemented in the form of a virtual machine that implements hardware such as DSPs, microcontrollers, RAM, ROM, HDD, etc. as software (command script).

[0620] Figure 61 is a drawing for explaining a sub-module included in the processor of Figure 60.

[0621] Below, the explanation will be given with reference to Fig. 59.

[0622] Referring to FIG. 61, it can be seen that the processor (6430) includes a first image processing unit (6431), a second image processing unit (6433), a third image processing unit (6435), and an integrated image generation unit (6437). The first image processing unit (6431), the second image processing unit (6433), the third image processing unit (6435), and the integrated image generation unit (6437) illustrated in FIG. 61 are modules that are logically and conceptually separated in order to explain the process performed by the processor (6430) in the process of implementing the method according to the present invention. Therefore, although four sub-modules are illustrated in FIG. 61, the processor (6430) may include fewer than four or more than four sub-modules depending on the embodiment. In addition, the first image processing unit (6431), the second image processing unit (6433), the third image processing unit (6435), and the integrated image generation unit (6437) of FIG. 61 are sub-modules of the processor (6430), and thus, like the processor (6430), they can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0623] When images are collected from the first camera (CA1), the second camera (CA2), the third camera (CA3), and the fourth camera (CA4) of the ship (6001') according to the present invention, the first image processing unit (6431) can extract overlapping feature points between the second images to the fourth images collected from the second camera (CA2) to the fourth camera (CA4).

[0624] The second image processing unit (6433) can generate a target image by matching the second to fourth images based on the feature points extracted by the first image processing unit (6431). The second image processing unit (6433) is equipped with at least one image registration algorithm and sufficient hardware and resources to execute the algorithm in order to match images generated from different cameras. Here, the target image refers to data that processes the appearance information of the first target (ob1) or the second target (ob2) shielded by the chimney (6020) when observed from the wheelhouse (6010) (or, when photographing the chimney (6020) with a camera in the wheelhouse (6010)) into an image, and can be generated by matching the second to fourth images.

[0625] The third image processing unit (6435) can identify an obstacle area in the first image using a preset object recognition algorithm. Here, the obstacle area may be the area occupied by the chimney (6020) in the first image. Depending on the embodiment, the obstacle area may be wider or narrower than a preset value. The obstacle area will be described later with reference to FIGS. 64 and 65.

[0626] The integrated image generation unit (6437) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the target image generated by the second image processing unit (6433). At this time, the first image and the target image can be integrated by the third image processing unit (6435) based on a specific obstacle area.

[0627] Figure 62 is a flowchart showing the process of generating an integrated image by an integrated image generation device, by unit process.

[0628] Hereinafter, based on FIGS. 60 and 61, the process of generating an integrated image by the integrated image generation device (6400) and the sub-modules included in the integrated image generation device (6400) will be described for each unit process.

[0629] The integrated image generation device (6400) can acquire images from each camera and store the data in memory (6450) (S6605). Here, each camera refers to the first camera (CA1) to the fourth camera (CA4) described above.

[0630] The integrated image generation device (6400) can calculate internal parameters and distortion coefficients of each camera (S6610). In addition, the integrated image generation device (6400) can additionally calculate external parameters between cameras, wherein the external parameters may be rotation and translation vectors (S6615).

[0631] The integrated image generation device (6400) can define an obstacle area in the first image collected from the first camera (CA1) (S6620). Typically, the obstacle area is the area occupied by the rotor sail (6100) seen in the bow direction when the image is captured by the first camera (CA1) adjacent to the wheelhouse (6010). The obstacle area in the first image can be classified as a separate layer from the remaining areas of the first image and managed separately.

[0632] The integrated image generation device (6400) can first correct the misalignment of each individual camera (S6625). At this time, the integrated image generation device (6400) can identify and correct the rotation angle of each camera through information linkage with the sensor of the ship (6001') according to the present invention. Subsequently, the integrated image generation device (6400) can secondarily correct the misalignment of each individual camera (S6630). At this time, the integrated image generation device (6400) can correct the roll misalignment based on the sea level, and can correct the pitch misalignment through the visual correlation between the sea level and the singular point of the structure of the ship (6001') according to the present invention.

[0633] The integrated image generation device (6400) can extract common feature points from the second to fourth images captured by the second camera (CA2) to the third camera (CA4) (S6635). After synchronizing the viewpoints of the second to fourth images, the integrated image generation device (6400) can set at least a portion of a continuously observed object as a feature point and extract it. For example, if the second image and the third image include a chimney (6020) in common, the feature points between the two images can be extracted from the chimney (6020) of the ship (6001') according to the present invention, if the second image and the fourth image include a first object (ob1) in common, the feature points between the two images can be extracted from the first object (ob1) of the ship (6001') according to the present invention, and if the third image and the fourth image include a second object (ob2) in common, the feature points between the two images can be extracted from the second object (ob2) of the ship (6001') according to the present invention.

[0634] The integrated image generation device (6400) can align the second to fourth images based on the matching of extracted feature points (S6640). The integrated image generation device (6400) can project the aligned "target image" onto the rear camera view (S6645).

[0635] The integrated image generation device (6400) can generate an integrated image by replacing (S6650) the obstacle area in the first image of the first camera (CA1) with the target image (aligned image) generated in the previous step (S6655). The integrated image generation device (6400) can transparently synthesize only the obstacle area in the integrated image and control the output of the integrated image through electronic devices in the wheelhouse (6010) (S6660).

[0636] Figure 63 is a drawing showing an example of a first image generated by a first camera.

[0637] FIG. 63 illustrates a third image (6710) as an example, which is an image showing the result of looking in the direction of the stern from inside the wheelhouse (6010).

[0638] The third image (6710) is an image assuming one frame that constitutes the first image captured by the first camera installed adjacent to the wheelhouse (6010) or when the crew inside the wheelhouse (6010) observes the stern direction, and as illustrated in FIG. 63, most of the third image (6710) includes image information about the chimney (6020). That is, referring to FIG. 63, it can be seen that the field of view for the blind spot (blind spot) behind the chimney (6020) is hardly secured due to the chimney (6020). The integrated image generation device (6400) can designate the area where the chimney (6020) is located in the first image as an obstacle area, as described in FIG. 61. As described above, the integrated image generation device (6400) stores a unique object recognition algorithm to selectively recognize the chimney (6020) in the first image and designate it as an obstacle area.

[0639] Figure 64 is a drawing showing an example of a second image generated by a second camera.

[0640] More specifically, FIG. 64 is an example of a fourth image (6810) that represents the result of a frame taken in the stern direction by the second camera (CA2) installed on the port side of the bridge wing. Referring to FIG. 60, it can be seen that the fourth image (6810) includes some common features with the third image generated by the third camera (CA3) and the fourth image generated by the fourth camera (CA4). The fourth image (6810) of FIG. 64 includes a first object (ob1) located close to the port side of the ship (6001') according to the present invention.

[0641] Figure 65 is a drawing showing an example of a third image generated by a third camera.

[0642] More specifically, FIG. 65 is an example of a fifth image (6910) that represents a result of a frame taken in the stern direction by a third camera (CA3) installed on the starboard side of the bridge wing. Referring to FIG. 60, it can be seen that the fifth image (6910) includes some common features with the second image generated by the second camera (CA2) and the fourth image generated by the fourth camera (CA4). The fifth image (6910) of FIG. 65 includes a second object (ob2) located close to the starboard side of the ship (6001') according to the present invention.

[0643] Figure 66 is a drawing showing an example of a fourth image generated by a fourth camera.

[0644] More specifically, FIG. 66 is an exemplary sixth image (7010) that represents the result of capturing by the fourth camera (CA4) installed at the very end of the stern or the funnel (6020) of the ship (6001') according to the present invention in one frame. Referring to FIG. 60, it can be seen that the sixth image (7010) includes some common features with the fourth image (6810) of FIG. 64 and the fifth image (6910) of FIG. 65. Since the fourth camera (CA4) is installed at the very end of the stern or the funnel (6020), the common features in the sixth image (7010) with the fourth image (6810) and the fifth image (6910) are mainly located at the sea surface, and according to an embodiment, as illustrated in FIG. 66, the sixth image (7010) may include some of the first object (ob1) located at the sea surface.

[0645] The integrated image generation device (6400) may generate a target image by matching the feature points of the fourth image (6810) and the fifth image (6910) as shown in FIGS. 64 and 65, and may also generate a target image by matching the fourth image (6810) to the sixth image (7010) according to an embodiment.

[0646] Figure 67 is a drawing showing an integrated image according to the present invention as an example.

[0647] In the seventh image (7110) of FIG. 67, it can be seen that not only the first object (ob1) floating on the sea surface behind the chimney (6020) regardless of the height and size of the chimney (6020), but also the second object (ob2) standing on the aft deck of the ship (6001') according to the present invention can be observed. In the seventh image (7110) of FIG. 67, the area occupied by the chimney (6020) can be set as an obstacle area and then made transparent. In the first image, the area where the chimney (6020) is located is classified as an "obstacle area", and then the obstacle area is replaced with an object image by the integrated image generating device (6400), and transparency is applied only to the layer constituting the obstacle area, so that as a result, objects located behind the chimney (6020) can also be displayed in the integrated image. In Fig. 67, a transparent chimney (6020') is indicated with a different reference number from the existing chimney (6020).

[0648] Figure 68 is a flowchart illustrating an example of a method for generating an integrated image for a ship according to the present invention.

[0649] Since the method according to Fig. 68 can be implemented by the integrated image generation device (6400), the processor (6430) and the sub-modules included in the processor (6430) described in Figs. 60 and 61, the method will be described below with reference to Figs. 60 to 67, and any description that overlaps with the content already described will be omitted.

[0650] The integrated image generation device (6400) can collect first images, second images, third images, and fourth images from the first camera (CA1), second camera (CA2), and third camera (CA3) of the ship (S7210).

[0651] The integrated image generation device (6400) can extract overlapping feature points between the second image to the fourth image collected from the second camera (CA2) to the fourth camera (CA4) (S7230).

[0652] The integrated image generation device (6400) can generate a target image by aligning the second to fourth images based on the extracted feature points (S7250).

[0653] The integrated image generation device (6400) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the generated target image (S7270).

[0654] A ship's bridge wings, the protruding sections on either side of the bridge, perform several crucial functions related to ship operation. In particular, they assist navigators and captains in safely navigating the ship in special situations, such as when navigating narrow ports or straits.

[0655] Bridgewings are installed on both sides of a ship's bridge, allowing navigators and captains to easily view the entire hull and its surroundings. This allows them to easily check obstacles on either side of the ship, other vessels, and the distance to the dock. Furthermore, bridgewings provide a view of the hull, both to the left and right, and to the rear, which is not available from the center, allowing for accurate measurement of distances and angles when berthing. Bridgewings allow precise steering in tight quarters, especially during berthing and unberthing, allowing for precise adjustments to the hull's position. Some bridgewings also feature simple steering gear, engine control levers, communication equipment, and emergency alarms, allowing for immediate action when necessary. Furthermore, bridgewings provide a space for crew members to send and confirm visual and audible signals, facilitating close cooperation during berthing operations. They also facilitate direct communication with the port pilot, ensuring a safe berthing according to the pilot's instructions.

[0656] Meanwhile, although the bridge wing is usually located at the highest point on the ship, if the view is obstructed by other installations or other obstacles on the ship, the function of the bridge wing may be significantly reduced. In addition, the presence of the bridge wing may cause wind resistance when the ship moves, which may have a negative effect on the ship's speed and fuel efficiency, which can be pointed out as points that need to be improved.

[0657] Figure 69 is a drawing for explaining a bridge wing installed on a conventional ship.

[0658] FIG. 69 illustrates a perspective view (7110) of a ship, a front view (7130) of a ship, and a side view (7150) of a ship, respectively. The perspective view (7110) of a ship is an image representing the result of observing the ship from an oblique line, and the perspective view (7110) of a ship clearly depicts a ship having a wheelhouse (7010) for controlling the direction and speed of the ship and operating and monitoring various equipment and systems for navigation, a bridge wing consisting of a port (7011) and a starboard (7012) side of the wheelhouse, and a bridge tower (7013) supporting the wheelhouse (7010) and the bridge wing while providing an access route to the wheelhouse (7010) and the bridge wing, which are installed on the hull (7020).

[0659] The front view (7130) and side view (7150) of the vessel also depict the aforementioned wheelhouse (7010), the bridge wing consisting of the port (7011) and starboard (7012) sides of the wheelhouse, the bridge tower (7013) and the hull (7020), respectively.

[0660] As shown in Fig. 69, the bridge tower (7013) on the ship is formed high, which may hinder the maintenance of a high ship speed due to a large resistance to wind, and for the same reason, may have a negative effect on fuel efficiency. Therefore, a new concept ship with improved ship speed and fuel efficiency is needed by lowering the height of the bridge tower (7013) and omitting the port (7011) and starboard (7012) sides of the bridge wing that extend to both sides.

[0661] Figure 70 is a diagram schematically showing the structure of an improved ship that improves the structure of a conventional ship to implement a method according to the present invention.

[0662] The perspective view (7210), the front view (7230) and the side view (7250) of the improved vessel of FIG. 70 illustrate the improved wheelhouse (7010') and hull (7020). Referring to the perspective view (7210), the front view (7230) and the side view (7250) of the vessel, the improved wheelhouse (7010') omits the bridge so that the port (7011) and starboard (7012) sides of the bridge wing of FIG. 69 are no longer visible, and the height of the wheelhouse (7010) is also significantly lowered so that almost all of the bridge tower (7013) except for the wheelhouse (7010) is omitted as in FIG. 69. The improved wheelhouse (7010') still functions as a crew living area.

[0663] The method according to the present invention proposes a method for generating a monitoring image to compensate for the lack of monitoring information on a vessel due to the omission of a bridge wing, and can be implemented by a device that generates an integrated monitoring image by integrating sensing information collected from additionally installed sensors based on an improved wheelhouse (7010'). Hereinafter, the device for generating an integrated monitoring image according to the present invention will be abbreviated as an integrated image generation device.

[0664] Figure 71 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0665] The integrated image generation device (7300) illustrated in Fig. 71 can collect information from sensors installed on a ship, process the collected information, and generate an integrated image containing monitoring information of the ship. The integrated image can be played back on the image playback device of the improved wheelhouse (7010'), and the integrated image generally contains various monitoring information of the ship that cannot be confirmed from the wheelhouse due to various installations and obstacles on the ship in a ship that does not have a bridge wing.

[0666] In addition, the integrated image generation device (7300) according to the present invention generates an integrated image and transmits it to a stationary electronic device or portable electronic device used by a captain or navigator in an improved wheelhouse (7010') so that the integrated image can be played back on the electronic device. The captain or navigator can easily obtain information necessary for navigating the vessel through the integrated image output through the electronic device. The integrated image can help the captain or crew who have viewed the integrated image navigate the vessel, and ultimately, can improve the overall visibility of a vessel with a bridge wing omitted according to the present invention.

[0667] Referring to FIG. 71, it can be seen that the integrated image generation device (7300) includes a communication unit (7310), a processor (7330), and a memory (7350).

[0668] The communication unit (7310) may include one or more components that enable wired / wireless communication with external devices. For example, the communication unit (7310) may include at least one piece of hardware necessary to implement short-range communication, such as Wi-Fi or Bluetooth, in a network provided by a communication network, or to implement various communications, including the Internet, when a LAN cable is connected.

[0669] The memory (7350) is a hardware that stores various data processed within the integrated image generation device (7300), and can store a program for processing and controlling the processor (7330). The memory (7350) may include a random access memory (RAM) such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM, a Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

[0670] The processor (7330) can control the overall operation of the integrated image generation device (7300). For example, the processor (7330) can control the operation of the input unit (not shown), display (not shown), communication unit (7310), memory (7350), etc. included in the integrated image generation device (7300) by executing programs stored in the memory (7350).

[0671] As an example, the processor (7330) may collect first type sensor information of a first type sensor attached to a first area of ​​a ship regardless of the type of ship, collect second type sensor information of a second type sensor attached to a second area of ​​the ship determined based on the type of ship, identify the type of ship based on the collected second type sensor information, and generate an integrated image including monitoring information of the type of ship identified based on the collected first type sensor information and the collected second type sensor information. A specific process of the processor (7330) will be described later with reference to FIGS. 72 to 79.

[0672] When the integrated image generation device (7300) is implemented as a physical device, the processor (7330) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0673] In addition, in the present invention, when the integrated image generation device (7300) is implemented in the form of an application (program) that runs on an integrated data processing device such as a server, the processor (7330) and memory (7350) included in the integrated image generation device (7300) may be implemented in the form of a virtual machine that implements hardware such as DSPs, microcontrollers, RAM, ROM, HDD, etc. as software (command script).

[0674] Figure 72 is a drawing for explaining a sub-module included in the processor of Figure 71.

[0675] Referring to FIG. 72, it can be seen that the processor (7330) includes a first sensor information collection unit (7331), a second sensor information collection unit (7333), a ship type identification unit (7335), and an integrated image generation unit (7337). The first sensor information collection unit (7331), the second sensor information collection unit (7333), the ship type identification unit (7335), and the integrated image generation unit (7337) illustrated in FIG. 72 are modules that are logically and conceptually separated in order to explain the process performed by the processor (7330) in the process of implementing the method according to the present invention. Therefore, although four sub-modules are illustrated in FIG. 72, the processor (7330) may include fewer than four or more than four sub-modules depending on the embodiment.

[0676] In addition, the first sensor information collection unit (7331), the second sensor information collection unit (7333), the ship type identification unit (7335), and the integrated image generation unit (7337) of FIG. 72 are sub-modules of the processor (7330), and thus, like the processor (7330), they can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0677] The first sensor information collection unit (7331) can collect first-type sensor information from a first-type sensor installed in the first area of ​​a vessel, regardless of the vessel type. Here, installation in the first area of ​​a vessel, regardless of the vessel type, means that the first area is a common area or space for all vessels.

[0678] For example, the first area may be either the engine room or the bridge room of the ship. In particular, if the first area is the wheelhouse, it may be an improved wheelhouse (7010') that is omitted from the bridge wing described in FIG. 2. The improved wheelhouse (7010') has a characteristic in that the bridge wing is omitted and the height of the bridge tower (7013) that includes the improved wheelhouse (7010') is lower than the first height, which is the height of the highest installation on the ship, as already described in FIG. 70.

[0679] The first type sensor is a word used to refer to a sensor installed in the first area, and in the present invention, the first type sensor may be one of a camera and a LiDAR (Light Detection And Ranging). The first type sensor information is information that comprehensively refers to the result sensed by the first type sensor, and for example, if the first type sensor is a camera, the first type sensor information may be an image.

[0680] The second sensor information collection unit (7333) can collect second type sensor information of a second type sensor installed in a second area of ​​a ship determined based on the type of ship. Here, the second area is a concept contrasting with the first area described above. If the first area exists regardless of the type of ship, the second area may be the location of an installation or equipment that may or may not necessarily exist depending on the type of ship. For example, a wind-powered ship must be equipped with at least one rotor sail, and a ship to which OCCS is applied must be equipped with an LCO2 tank. The second area may be the rotor sail, the LCO2 tank, or an area adjacent thereto. The second sensor information collection unit (7333) can operate to selectively collect only the second type sensor information by not receiving information of the first type sensor installed in the first area, or by discarding it even if it is received.

[0681] The vessel type identification unit (7335) can identify the vessel type using the collected second type sensor information. The vessel type identification unit (7335) receives sensing values ​​(second type sensor information) from sensors (second type sensors) installed on the vessel, and reads metadata included in the sensing values ​​to identify the type of sensor and its installation location. For example, if the received second type sensor information includes metadata called 'RS1', the second sensor information collection unit (7333) can determine that the vessel type is a wind-powered vessel because the information was received from a sensor installed adjacent to rotor sail No. 1. Since the required monitoring information varies depending on the vessel type, the method according to the present invention first determines the vessel type as described above, and then generates an integrated image including monitoring information matching the vessel type according to the process described below.

[0682] The integrated image generation unit (7337) can generate an integrated image including monitoring information of a type of vessel identified based on the collected first type sensor information and second type sensor information. The integrated image generation unit (7337) can generate an integrated image by analyzing the image included in the first type sensor information collected by the first sensor information collection unit (7331) and the second type sensor information collected by the second sensor information collection unit (7333), recognizing at least one object in the image, and adding visual marks to objects of a preset type among the recognized objects. The visual marks may be bounding boxes that are displayed with different colors depending on the type of object. The integrated image with the added visual marks will be described later with reference to FIG. 77.

[0683] As an optional embodiment, although not illustrated in FIG. 72, the third sensor information collection unit may be capable of moving through user input and collecting third type sensor information of a third type sensor including a camera. The third type sensor information collected by the third sensor information collection unit may be transmitted to a ship type identification unit (7335) and an integrated image generation unit (7337), and may be utilized to identify the type of ship and generate an integrated image. Here, the third type sensor may be a drone capable of flight, and the third type sensor information may be an image taken of the ship from the sky while the drone maintains a flight state. The third type sensor information will be described later in FIG. 78.

[0684] Figure 73 is a flowchart showing the process of generating an integrated image by an integrated image generation device, by unit process.

[0685] Since Fig. 73 can be implemented by the integrated image generation unit (7337) described in Fig. 72, any description overlapping with the description in Fig. 72 will be omitted, and below, with reference to Figs. 71 and 72, the process of the integrated image generation unit (7337) generating an integrated image will be described for each unit process.

[0686] The integrated image generation unit (7337) can acquire images from each camera and store the data in memory (7350) (S7505). Here, each camera refers to a first type sensor and a second type sensor installed on each ship.

[0687] The integrated image generation unit (7337) can calculate internal parameters and distortion coefficients of the first type sensor and the second type sensor (S7510). In addition, the integrated image generation unit (7337) can additionally calculate external parameters between cameras, wherein the external parameters may be rotation and translation vectors (S7515).

[0688] The integrated image generation unit (7337) can first correct the misalignment of each individual camera (S7520). At this time, the integrated image generation unit (7337) can identify and correct the rotation angle of each camera through information linkage with the first type sensor and the second type sensor of the ship. Subsequently, the integrated image generation unit (7337) can secondarily correct the misalignment of each individual camera (S7525). At this time, the integrated image generation unit (7337) can correct the roll misalignment based on the sea level, and can correct the pitch misalignment through the visual correlation between the unique points of the ship's structure and the sea level.

[0689] The integrated image generation unit (7337) can extract common feature points from the information collected from the first type sensor and the second type sensor (S7530). If there are no common feature points among all the information, the integrated image generation unit (7337) can determine the segmented images to be included in the integrated image based on the types and importance of the objects included in the image.

[0690] The integrated image generation unit (7337) can perform image processing to add bounding boxes to each of the divided images by applying an object recognition algorithm after determining the segmented images to be included in the integrated image (S7535), and can finally integrate the determined segmented images into a single integrated image (S7545). The integrated image generation unit (7337) can control the output of the integrated image through electronic devices in the wheelhouse (7010) (S7550).

[0691] Figure 74 is a drawing for explaining the first type sensor and the first area.

[0692] A vessel to which the method according to the present invention is applied includes an improved wheelhouse (7010') with a low height. The front view (7610) of Fig. 74 exemplarily illustrates the improved wheelhouse (7010') and first type sensors (7050) installed on the hull (7020). When the vessel according to the present invention is viewed from the front, a total of four first type sensors are observed.

[0693] The side view (7630) of FIG. 74 also exemplarily shows that the first type sensors are installed in the improved wheelhouse (7010') and the hull (7020). When the ship according to the present invention is viewed from the side, it can be seen that the first type sensors can be installed in the front and rear of the improved wheelhouse (7010'), respectively. According to the front view (7610) of FIG. 74 and the side view (7630) of FIG. 74, the first areas where the first type sensors (7050) are installed are the improved wheelhouse (7010') and the bow of the ship, and the first type sensors (7050) can be installed symmetrically with respect to each other with respect to the ship center line as shown in FIG. 74 in order to obtain precise sensing results.

[0694] Referring to FIG. 74, the total number of first type sensors installed in the first area of ​​the ship according to the present invention is six, but depending on the embodiment, the number of first type sensors may be less than six or more than six.

[0695] Figure 75 is a schematic drawing showing an example of a first type sensor and a second type sensor installed on an OCCS-applied ship.

[0696] The first image (7710) and the second image (7730) of FIG. 75 show a side view and a plan view, respectively, of a vessel to which an OCCS (Onboard Carbon Capture and Storage System) is applied.

[0697] OCCS (Oil Capture and Condensation Systems) for ships is a system that captures carbon dioxide (CO2) directly from ship exhaust gas. OCCS was developed as part of efforts to reduce the environmental impact of the shipping industry, particularly in compliance with international regulations aimed at reducing greenhouse gas emissions from ships. Vessels equipped with OCCS are referred to herein as "OCCS-equipped vessels." OCCS-equipped vessels can significantly reduce carbon emissions by capturing CO2 from exhaust gas and blocking it before it is released into the atmosphere. Furthermore, OCCS-equipped vessels can store the captured CO2 in a liquid form onboard, allowing it to be recycled in industrial processes or appropriately disposed of upon arrival at port. Furthermore, OCCS-equipped vessels can further reduce CO2 emissions compared to non-equipped vessels, making it easier to comply with International Maritime Organization (IMO) environmental regulations. Thus, although OCCS technology is still in its infancy, it could play a key role in helping the shipping industry transition to more environmentally friendly operations.

[0698] In Fig. 75, the wheelhouse (7010a) of the OCCS-applied ship is given a different reference number to distinguish it from the wheelhouses described in Figs. 69 and 70.

[0699] As illustrated in Fig. 75, the OCCS-applied vessel includes a small LCO2 tank (7301) on the stern side and a large LCO2 tank (7304) on the bow side. In particular, the large LCO2 tank (7304) on the bow side is not illustrated in Fig. 75, but refers to a tank that integrates the two small LCO2 tanks on the bow side that were previously separated. When the large LCO2 tank (7304) is placed on the bow side of the OCCS-applied vessel, a blind spot (blind spot) is created due to the size of the large LCO2 tank (7304), and there is a problem in that people or other objects behind the large LCO2 tank (7304) cannot be recognized in the improved wheelhouse (7010').

[0700] As shown in the first image (7710) and the second image (7730) of FIG. 75, the first type sensors may be installed symmetrically in two each at the front and rear of the wheelhouse (7010a) of the OCCS-applied ship, and at least one may be installed on each left and right of the bow of the OCCS-applied ship. In addition, as shown in the first image (7710) and the second image (7730) of FIG. 75, the second type sensors may be installed symmetrically in at least two small LCO2 tanks (7301) on the stern of the OCSS-applied ship, and at least two large LCO2 tanks (7304) on the bow of the OCSS-applied ship. Since both the first type sensors and the second type sensors are installed symmetrically with respect to the bow and stern of the OCCS-applied ship, monitoring information for a specific direction of the OCCS-applied ship may not be omitted. In particular, since the information collected from the first type sensor and the second type sensor respectively includes metadata that can determine whether the area where the sensor is installed is the first area or the second area that exists only in a specific type of ship, the ship type identification unit (7335) can accurately identify the type of ship based on the second type sensor information collected by the second sensor information collection unit (7333).

[0701] Figure 76 is a schematic drawing showing an example of a first type sensor and a second type sensor installed on a container ship.

[0702] The third image (7810) and the fourth image (7830) of FIG. 76 show a side view and a plan view of the container ship, respectively.

[0703] A container ship is a cargo vessel capable of transporting large quantities of containers loaded with goods. Their primary role is to transport these containers safely and quickly to their destination. Container ships can be categorized into cellular container ships, roll-on-ro (Ro-Ro) container ships, and partial container ships.

[0704] Cellular container ships feature fixed cells (a lattice structure) for loading containers, allowing them to be stacked vertically in the ship's holds and decks. Most large container ships typically follow the cellular container ship architecture, which offers the advantage of fast and efficient loading and unloading. Ro-ro container ships are designed to load and unload containers directly onto the ship, similar to vehicles, using a roll-on / roll-off system. Ro-ro container ships allow containers to be loaded and unloaded from trailers or trucks, simplifying dockside operations. Partial container ships are designed to carry not only containers but also general cargo. Partial container ships are multimodal carriers designed to carry containers in some areas and general cargo or bulk cargo in other areas.

[0705] The container ship illustrated in Fig. 76 is a cellular container ship in which the container ship wheelhouse (7010b) is arranged closer to the bow than the stern, the container ship funnel (7020b) is installed closer to the stern than the bow, and the entire area except for the container ship wheelhouse (7010b) and the container ship funnel (7020b) is configured as an area for loading containers. However, the present invention is not limited to a specific type of container ship, and thus, depending on the embodiment, the present invention may also be applied to a ro-ro container ship and a partial container ship. Here, the container ship funnel (7020b) refers to a funnel-shaped structure that safely discharges exhaust gas generated from the engine of the container ship to the outside.

[0706] In Fig. 76, the container ship wheelhouse (7010b) is given a different reference number to distinguish it from the wheelhouses described in Figs. 69 and 70. The container ship of Fig. 76 includes a low-height container ship wheelhouse (7010b) on the bow side and a funnel (7020b) on the stern side.

[0707] As shown in the third image (7810) and the fourth image (7830) of FIG. 76, the first type sensors in the container ship may be installed symmetrically in two each at the front and rear of the wheelhouse (7010b) of the container ship, and one or more may be installed on each of the left and right sides of the bow of the container ship.

[0708] Additionally, as shown in the third image (7810) and the second image (7830) of FIG. 76, at least two type 2 sensors may be symmetrically installed on the container ship funnel (7020b).

[0709] Since both the Type 1 sensor and the Type 2 sensor are installed symmetrically with respect to the forward and stern lines of the container ship, monitoring information regarding a specific direction of the container ship may not be omitted. In particular, since the information collected from the Type 1 sensor and the Type 2 sensor respectively includes metadata that can determine whether the area where the sensor is installed is the First Area or the Second Area that exists only in a specific type of ship, the ship type identification unit (7335) can accurately identify the type of ship based on the Type 2 sensor information collected by the Second Sensor Information Collection Unit (7333). For example, if the information collected by the Second Sensor Information Collection Unit (7333) includes the metadata 'FnC1', the ship type identification unit (7335) can recognize it as 'Type 2 sensor No. 1 installed on the funnel of a container ship' and determine that the ship type is a container ship.

[0710] Figure 77 is a schematic drawing showing an example of a first type sensor and a second type sensor installed on a wind-powered vessel.

[0711] The fifth image (7910) and the sixth image (7930) of FIG. 77 show a side view and a plan view, respectively, of a wind-powered vessel.

[0712] A wind-powered vessel is a vessel equipped with a rotor sail. A rotor sail is a wind-assisted propulsion device installed on a vessel's deck that can generate propulsive force using the wind. A rotor sail is a cylindrical structure that generates additional propulsion, thereby reducing the vessel's fuel consumption and carbon emissions. When a rotor sail, which rotates through an electric motor, encounters the wind blowing around the vessel, a pressure difference is created around the rotor sail, and the Magnus effect resulting from this pressure difference generates forward propulsive force. At least one rotor sail (7100) is installed, and in FIG. 77, four rotor sails (7100) are installed across the bow and stern of the vessel, but the number and arrangement of rotor sails (7100) may vary depending on the embodiment.

[0713] The wind-powered vessel of Fig. 77 may include a wind-powered vessel wheelhouse (7010c), a wind-powered vessel funnel (7020c), and at least one rotor sail (7100). Here, the wind-powered vessel funnel (7020c) refers to a funnel-shaped structure that safely discharges exhaust gas generated from the engine of the wind-powered vessel to the outside. Since the wind-powered vessel is not a type that completely omits the engine, the wind-powered vessel funnel (7020c) is essential to efficiently process exhaust gas generated when the auxiliary engine or auxiliary equipment operates. By appropriately designing the position and height of the wind-powered vessel funnel (7020c) so that the exhaust gas is appropriately discharged through the wind-powered vessel funnel (7020c), the efficiency of the rotor sail (7100) is not reduced, so that the maneuverability of the wind-powered vessel can be improved.

[0714] As illustrated in the fifth image (7910) and the sixth image (7930) of FIG. 77, at least two second type sensors (7070) may be symmetrically installed in an area adjacent to each rotor sail (7100) and on a container ship funnel (7020c) of a wind-powered vessel. Since both the first type sensors (7050) and the second type sensors (7070) are installed symmetrically around the bow and stern of the wind-powered vessel, monitoring information regarding a specific direction of the wind-powered vessel may not be omitted.

[0715] In particular, since the rotor sail (7100) has the characteristic of rotating at high speed to generate the Magnus effect, the second type sensor (7070) is symmetrically installed in an area a predetermined distance away from the rotor sail (7100) so as to obtain information on objects shielded by the size and height of the rotor sail (7100).

[0716] Since the information collected from the first type sensor and the second type sensor each includes metadata that can determine whether the area where the sensor is installed is the first area or the second area that exists only in a specific type of ship, the ship type identification unit (7335) can accurately identify the type of ship based on the second type sensor information collected by the second sensor information collection unit (7333). For example, if the information collected by the second sensor information collection unit (7333) includes metadata called 'RS1-2', the ship type identification unit (7335) can recognize it as 'second type sensor number 2 installed adjacent to rotor sail number 1' and determine that the ship type is a wind-powered ship.

[0717] Figure 78 is a drawing showing an example of an integrated image generated by an integrated image generation device.

[0718] The integrated image generation device (7300) can generate an integrated image by integrating at least two or more segmented images into a single image and outputting the image, as shown in FIG. 78. In particular, the integrated image generation unit (7337) of the integrated image generation device (7300) can generate an integrated image based on the collected first type sensor information and second type sensor information, and can determine and identify the importance and objects of the images included in the integrated image through an AI model.

[0719] The integrated image of Figure 78 includes a first segmented image (8010), a second segmented image (8030), a third segmented image (8050), and a fourth segmented image (8070).

[0720] The first segmented image (8010) is a planar image of the ship's deck taken from above the ship, and can be acquired using a drone, a third type of sensor. The drone is a sensor that can move through user input and includes a camera, allowing it to take pictures of the ship from above, providing 2D / 3D information about various installations and objects on the ship's deck.

[0721] The second segmented image (8030) is an image generated by processing an image collected by a second type sensor (7070) installed in the second area of ​​the ship. The integrated image generation unit (7337) analyzes the image included in the second segmented image (8030) through an object recognition algorithm to recognize objects in the image, and generates a bounding box based on the type or importance of the recognized objects and can apply it to each object. The integrated image generation unit (7337) can track and recognize objects in the image using a Kalman filter-based object recognition algorithm, but depending on the embodiment, it can recognize objects in the image through other types of object recognition algorithms such as YOLO (You Only Look Once), Faster R-CNN (Region-based Convolutional Neural Networks), and SSD (Single Shot Multibox Detector), and distinguish objects through bounding boxes.

[0722] In the second segmented image (8030), the first bounding box (8031) may represent the result of recognizing a person, the second bounding box (8033) may represent the result of recognizing a mobile device including a vehicle, and the third bounding box (8035) may represent the result of recognizing an unidentified mobile object. The first bounding box (8031) to the third bounding box (8035) may be expressed in different colors.

[0723] The third segmented image (8050) is an image that enlarges and displays the object indicated by the third bounding box (8035) in the second segmented image (8030). The user can confirm an unidentified mobile object through the third segmented image (8050) and take appropriate action if it is determined to be dangerous.

[0724] The fourth segmented image (8070) is an image received from the first type sensor (7050) of the ship, and may be an image taken of the bow from the wheelhouse (7010').

[0725] By checking the first segmented image (8010) to the fourth segmented image (8070) included in the integrated image at a glance, the user can quickly obtain necessary information during the operation of the ship. As illustrated in FIG. 74, the device according to the present invention can generate an integrated image including a higher level of monitoring information than that of a conventional ship with a bridge wing, even if the bridge wing is omitted, and provide the generated image to the user. The number of segmented images included in the integrated image may vary depending on the type of ship, the number of second type sensors installed on the ship, the number of objects classified into the third bounding box (8035), etc., and may be determined by the AI ​​model included in the integrated image generation unit (7337).

[0726] Figure 79 is a flowchart illustrating an example of a method according to the present invention.

[0727] Since the method according to Fig. 79 can be implemented by the integrated image generation device (7300) described in Figs. 71 and 72 and the sub-modules included in the integrated image generation device (7300), the method will be described below with reference to Figs. 71 to 78, and any description that overlaps with the content already described will be omitted.

[0728] The integrated image generation device (7300) can collect first type sensor information from first type sensors installed in the first area of ​​the ship (S8110).

[0729] The integrated image generation device (7300) can collect second type sensor information from second type sensors installed in the second area of ​​the ship (S8130).

[0730] The integrated image generation device (7300) can identify the type of ship by analyzing the second type sensor information collected in step S1130 (S8150).

[0731] The integrated image generation device (7300) can generate an integrated image containing monitoring information corresponding to the identified vessel type (S8170).

[0732] Since the user can obtain most of the monitoring information necessary for operating the ship through the integrated image generated in step S8170, according to the present invention, the user can stably operate the ship including a low bridge tower and an omitted bridge wing.

[0733] Figures 80a and 80b are schematic drawings showing an example of a conventional container ship.

[0734] Fig. 80a is an example of the starboard side of a container ship. The container ship of Fig. 80a is considered to be a cellular container ship with its wheelhouse (8010) positioned closer to the bow than the stern, and only containers are loaded. The container ship of Fig. 80a has containers loaded forward of the wheelhouse (8010). As illustrated in Fig. 80a, conventional container ships tend to have their containers loaded at lower heights closer to the bow of the container ship in order to satisfy the visibility rule of container ships, which tends to result in a lower loading capacity.

[0735] Fig. 80b is a drawing showing a container ship in plan view. Referring to Fig. 80b, if containers are loaded high, a blind spot (blind spot) is created at a location beyond the viewing angle α in the wheelhouse (8010), and there is a problem in that other ships or floating objects in the blind spot cannot be discovered or are discovered late.

[0736] Meanwhile, air draft refers to the restriction applied when a ship crosses a bridge. For example, the air draft of a container ship passing through the Panama Canal is approximately 65 m high. In order not to exceed the height of 65 m, the height of containers loaded in the middle and aft areas of the container ship must be loaded at a height 1 to 2 tiers lower than the maximum loadable height.

[0737] FIGS. 81a and 81b are schematic drawings showing the structure of an improved container ship that improves the structure of a conventional container ship in order to implement a method according to the present invention.

[0738] Figure 81a schematically illustrates a structure in which a camera facing the rear of a container ship is installed adjacent to the wheelhouse (8010) to enable the container ship according to the present invention to load more containers than conventional container ships. In Figure 81a, the camera installed in the wheelhouse (8010) may be referred to as a first camera, as described below.

[0739] Figure 81b is a plan view showing the structure of a container ship according to the present invention. In Figure 81b, the container ship according to the present invention includes a first loading area (AR1), a second loading area (AR2), and a third loading area (AR3). The first loading area (AR1), the second loading area (AR2), and the third loading area (AR3) are the same in that they are areas where a plurality of containers are loaded, but their relative positions are different based on the positions of the wheelhouse (8010) and the funnel (8020) of the container ship according to the present invention. The first loading area (AR1) refers to a loading area located at the bow of the container ship according to the present invention. The second loading area (AR2) refers to a loading area located between the wheelhouse (8010) and the funnel (8020) of the container ship according to the present invention. The third loading area (AR3) refers to a loading area located behind the funnel (8020) of the container ship according to the present invention. That is, the first loading area (AR1) is located at the very front of the container ship according to the present invention, the third loading area (AR3) is located at the very rear of the container ship according to the present invention, and the second loading area (AR2) is located between the first loading area (AR1) and the third loading area (AR3).

[0740] Referring to FIGS. 81A and 81B, it can be seen that the container ship according to the present invention has a total of five cameras installed facing rearward. One embodiment of the present invention may be implemented through a first camera (CA1), a second camera (CA2), and a third camera (CA3), and, depending on the embodiment, may be implemented through a first camera (CA1), a second camera (CA2), a third camera (CA3), a fourth camera (CA4), and a fifth camera (CA5).

[0741] According to the present invention, a container ship may have a first camera installed inside a wheelhouse (8010) and at a location adjacent to the wheelhouse (8010), and a second camera (CA2) and a third camera (CA3) may be installed on the port (8011) and starboard (8012) sides of the bridge wing of the container ship, respectively, facing the rear of the container ship according to the present invention. The second camera (CA2) and the third camera (CA3) are installed to supplement the first camera (CA1) of the wheelhouse (8010), whose field of view toward the rear is reduced due to containers additionally loaded in the second loading area (AR2), and the specific operations of the second camera (CA2) and the third camera (CA3) will be described with reference to FIGS. 82 and 83.

[0742] As an example, the fourth camera (CA4) and the fifth camera (CA5) are each installed toward the rear of the container ship according to the present invention, and may be installed on the left and right sides of the funnel (8020) of the container ship according to the present invention, as illustrated in FIG. 81b. Hereinafter, the fourth camera (CA4) and the fifth camera (CA5) are assumed to be installed on the left and right sides of the funnel (8020), respectively. The specific operations of the fourth camera (CA4) and the fifth camera (CA5) will be described with reference to FIGS. 82 and 83.

[0743] The first camera (CA1) to the fifth camera (CA5) illustrated in FIGS. 81a and 81b are considered devices that not only can capture an object to generate an image, but also have a built-in communication module that can transmit the generated image to an external device via wired / wireless communication. In addition, for convenience of explanation, the images generated by the first camera (CA1) to the fifth camera (CA5) will be abbreviated as the first image to the fifth image hereinafter.

[0744] As illustrated in FIG. 81b, a container ship according to the present invention may be a cellular container ship, including at least three cameras, a wheelhouse (8010) located closer to the bow than the stern and including a bridge wing, and a funnel (8020) located closer to the stern than the bow. However, since the method according to the present invention is not limited by the type of container ship, other types of container ships may be used, including at least three cameras, depending on the embodiment.

[0745] The container ship according to the present invention has a differentiated feature in that it has several additional cameras that were not installed in conventional container ships, and further includes a device that creates an integrated image by aligning images taken by several cameras, thereby ensuring improved visibility compared to conventional container ships and enabling the loading of containers.

[0746] The first camera (CA1) can capture a portion of the containers loaded at the rear of the wheelhouse (8010) of the container ship according to the present invention and a portion of the sea surface to generate a first image. The second camera (CA2) and the third camera (CA3) can be installed symmetrically around the ship center line based on the same distance from the first camera (CA1). Referring to FIG. 81b, the distance between the first camera (CA1) and the second camera (CA2) can be the same as the distance between the first camera (CA1) and the third camera (CA3).

[0747] The first camera (CA1) cannot capture objects beyond its field of view, and the area that is blocked by containers stacked high in the second loading area (AR2) or third loading area (AR3) and is not captured by the first camera (CA1) is referred to as the “blind zone of the rear-loaded containers.” Hereinafter, the locations where the second camera (CA2) and the third camera (CA3) are installed in the container ship according to the present invention are referred to as the second position and the third position, respectively.

[0748] The second camera (CA2) can capture a portion of the blind zone of the rear loading container based on the second position of the second camera (CA2) to generate a second image. In addition, the third camera (CA3) can capture another portion of the blind zone of the rear loading container based on the third position of the third camera (CA3) to generate a third image. The device according to the present invention can generate an integrated image through a process of aligning the second image and the third image to generate a target image and integrating the target image into the first image. The integrated image generated as described above can display an object in the blind zone of the rear loading container in the first image, and a specific process of generating the integrated image will be described later with reference to FIG. 84.

[0749] If the object in the blind zone of the rear loading container is far away from the second camera (CA2) and the third camera (CA3) or is at a different angle from the installation direction of the second camera (CA2) and the third camera (CA3), the object in the blind zone of the rear loading container may not be sufficiently identified by the second camera (CA2) and the third camera (CA3). In this case, the container ship according to the present invention can additionally collect the fourth image and the fifth image through the fourth camera (CA4) and the fifth camera (CA5), respectively, and reflect the information included in the fourth image and the fifth image in the process of generating the integrated image.

[0750] The method according to the present invention can generate an integrated image with improved visibility by using images collected by the first camera (CA1), the second camera (CA2), the third camera (CA3), the fourth camera (CA4), and the fifth camera (CA5) described in FIG. 82. Hereinafter, the device for generating the integrated image will be referred to as an integrated image generating device (8300). The integrated image generated by the integrated image generating device (8300) will be described in detail with reference to FIGS. 82 to 85.

[0751] Figure 82 is a block diagram showing an example of an integrated image generation device according to the present invention.

[0752] The integrated image generation device (8300) illustrated in FIG. 82 can collect images from cameras (at least one of the first to fifth cameras) installed on a container ship according to the present invention, and process the collected images to generate an integrated image. The integrated image can be played back on an image playback device in the wheelhouse (8010), and the integrated image includes objects at the rear of the container ship according to the present invention that cannot be confirmed in the wheelhouse (8010) due to containers loaded in the second loading area (AR2) and the third loading area (AR3).

[0753] The integrated image generation device (8300) according to the present invention generates an integrated image and transmits it to an electronic device used by a captain or navigator in the wheelhouse (8010) so that the integrated image can be played back on the electronic device. The captain or navigator can easily obtain information necessary for navigating the ship through the integrated image output through the electronic device. The integrated image can assist the captain or crew who check the integrated image in the wheelhouse (8010) in navigating the ship, and ultimately, can improve the overall visibility of the container ship according to the present invention.

[0754] Referring to FIG. 82, it can be seen that the integrated image generation device (8300) includes a communication unit (8310), a processor (8330), and a memory (8350).

[0755] The communication unit (8310) may include one or more components that enable wired / wireless communication with an external device. For example, the communication unit (8310) may include at least one piece of hardware necessary to implement short-range communication, such as Wi-Fi or Bluetooth, in a network provided by a communication network, or to implement various communications, including the Internet, when a LAN cable is connected. The communication unit (8310) may receive images from the first camera (CA1) to the fifth camera (CA5).

[0756] The memory (8350) is hardware that stores various data processed within the integrated image generation device (8300), and can store a program for processing and controlling the processor (8330). The memory (8350) may include a random access memory (RAM) such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM, a Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

[0757] The processor (8330) can control the overall operation of the integrated image generation device (8300). For example, the processor (8330) can control the operation of the input unit (not shown), display (not shown), communication unit (8310), memory (8350), etc. included in the integrated image generation device (8300) by executing programs stored in the memory (8350).

[0758] As an example, the processor (8330) may collect images from the first camera, the second camera, and the third camera of the container ship, extract overlapping feature points between the second and third images collected from the second and third cameras, match the second and third images based on the extracted feature points to generate a target image, and integrate the first image collected from the first camera and the target image to generate an integrated image. The specific process of the processor (8330) will be described later with reference to FIGS. 83 to 88.

[0759] When the integrated image generation device (8300) is implemented as a physical device, the processor (8330) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0760] In addition, in the present invention, when the integrated image generation device (8300) is implemented in the form of an application (program) that runs on an integrated data processing device such as a server, the processor (8330) and memory (8350) included in the integrated image generation device (8300) may be implemented in the form of a virtual machine that implements hardware such as DSPs, microcontrollers, RAM, ROM, HDD, etc. as software (command script).

[0761] Figure 83 is a drawing for explaining a sub-module included in the processor of Figure 82.

[0762] Below, the explanation will be given with reference to Fig. 82.

[0763] Referring to FIG. 83, it can be seen that the processor (8330) includes a first image processing unit (8331), a second image processing unit (8333), a third image processing unit (8335), and an integrated image generation unit (8337). The first image processing unit (8331), the second image processing unit (8333), the third image processing unit (8335), and the integrated image generation unit (8337) illustrated in FIG. 83 are modules that are logically and conceptually separated in order to explain the process performed by the processor (8330) in the process of implementing the method according to the present invention. Therefore, although four sub-modules are illustrated in FIG. 83, the processor (8330) may include fewer than four or more than four sub-modules depending on the embodiment.

[0764] In addition, the first image processing unit (8331), the second image processing unit (8333), the third image processing unit (8335), and the integrated image generation unit (8337) of FIG. 83 are sub-modules of the processor (8330), and thus, like the processor (8330), they can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0765] When images are collected from the first camera (CA1), the second camera (CA2), the third camera (CA3), the fourth camera (CA4), and the fifth camera (CA5) of the container ship, the first image processing unit (8331) can extract overlapping feature points between the second images to the fifth images collected from the second camera (CA2) to the fifth camera (CA5).

[0766] The second image processing unit (8333) can generate a target image by matching the second image and the third image (or the second to fifth images) based on the feature points extracted by the first image processing unit (8331). The second image processing unit (8333) is equipped with at least one image registration algorithm and sufficient hardware and resources to execute the algorithm in order to match images generated from different cameras. Here, the target image refers to data that processes the appearance information of an object (the first object (ob1) of FIG. 7) that is shielded by a container loaded high in the second loading area (AR2) or the third loading area (AR3) when observing the rear of the container ship according to the present invention through the first camera (CA1) of the wheelhouse (8010) into an image, and can be generated by matching the second and third images, or by matching the second to fifth images.

[0767] The third image processing unit (8335) can identify an obstacle area in the first image using a preset object recognition algorithm. Here, the obstacle area may be an area occupied by containers that were not loaded in a conventional container ship in the first image but were additionally loaded in the second loading area (AR2) or the third loading area (AR3) in the container ship according to the present invention. The obstacle area will be described later in FIG. 86.

[0768] The integrated image generation unit (8337) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the target image generated by the second image processing unit (8333). At this time, the first image and the target image can be integrated by the third image processing unit (8335) based on a specific obstacle area.

[0769] Figure 84 is a flowchart showing the process of generating an integrated image by an integrated image generation device, by unit process.

[0770] Since Fig. 84 can be implemented by the integrated image generation device (8300) described in Fig. 82, any description overlapping with the contents described in Fig. 82 and Fig. 83 will be omitted, and below, with reference to Fig. 82 and Fig. 83, the integrated image generation device (8300) and the process of generating an integrated image by the sub-module included in the integrated image generation device (8300) will be described for each unit process.

[0771] The integrated image generation device (8300) can acquire images from each camera and store the data in memory (8350) (S8505). Here, each camera refers to the first camera (CA1) to the fifth camera (CA5) described above.

[0772] The integrated image generation device (8300) can calculate internal parameters and distortion coefficients of each camera (S8510). In addition, the integrated image generation device (8300) can additionally calculate external parameters between cameras, wherein the external parameters may be rotation and translation vectors (S8515).

[0773] The integrated image generation device (8300) can define an obstacle area in the first image collected from the first camera (CA1) (8S520). Typically, the obstacle area is an area occupied by containers loaded higher than before in the second loading area (AR2) or the third loading area (AR3) when an image is captured by the adjacent first camera (CA1) of the wheelhouse (8010). The obstacle area in the first image can be classified as a separate layer from the remaining areas of the first image and managed separately.

[0774] The integrated image generation device (8300) can first correct the misalignment of each individual camera (S8525). At this time, the integrated image generation device (8300) can identify and correct the rotation angle of each camera through information linkage with the sensor of the container ship according to the present invention. Subsequently, the integrated image generation device (8300) can secondarily correct the misalignment of each individual camera (S8530). At this time, the integrated image generation device (8300) can correct the roll misalignment based on the sea level, and can correct the pitch misalignment through the visual correlation between the sea level and the unique point of the structure of the container ship according to the present invention.

[0775] The integrated image generation device (8300) can extract common feature points from the second to fifth images captured by the second camera (CA2) to the fifth camera (CA5) (S8535). The integrated image generation device (8300) can synchronize the viewpoints of the second to fifth images, and then set at least a portion of an object that is continuously observed as a feature point and extract it. For example, if the second and third images commonly include a buoy or another vessel that is shielded by containers loaded higher than before in the second loading area (AR2) or the third loading area (AR3), the feature points between the two images can be extracted from the buoy or the other vessel. In this process, when the integrated image generation device (8300) operates only for the second loading area (AR2), the feature points can be extracted only from the second and third images captured by the second camera (CA2) and the third camera (CA3).

[0776] The integrated image generation device (8300) can align the second and third images (or, depending on the embodiment, the second to fifth images) based on the matching of extracted feature points (S8540). The integrated image generation device (8300) can project the aligned "target image" onto the rear camera view (S8545).

[0777] The integrated image generation device (8300) can generate an integrated image by replacing the obstacle area in the first image of the first camera (CA1) with the target image (aligned image) generated in the previous step (S8550). The integrated image generation device (8300) can transparently synthesize only the obstacle area in the integrated image (S8555) and control the output of the integrated image through electronic devices in the wheelhouse (8010) (S8560).

[0778] Figure 85 is a drawing showing an example of the rear view of a conventional container ship.

[0779] Conventional container ships do not load containers higher than about 65 m to comply with air draft when navigating across canals or bridges such as the Panama Canal, so it was necessary to secure a rearward view of the container ship. Referring to the rearward view (8610) of a conventional container ship in FIG. 85, when photographing the rear from the wheelhouse of a conventional container ship, it can be seen that the rearward view is sufficiently secured so that a first object (ob1) moving at the rear of the container ship can be observed. Here, the first object (ob1) refers to an object at sea to which the present invention is not applied or other objects excluding a container ship to which the present invention is applied, and although it is depicted as a ship in FIG. 85, depending on the embodiment, it may be any one of a buoy, a sea creature, or other driftwood other than a ship.

[0780] FIG. 86 is a diagram illustrating second and third images generated by the second and third cameras, respectively.

[0781] The second image (8710) of FIG. 86 schematically illustrates one frame of a second image generated by photographing the rear of a container ship according to the present invention from a second position by a second camera (CA2), and referring to FIG. 81b, it can be seen that the second image includes some common features (sea surface at the rear, sky at the rear, containers at the rear, etc.) with the third image generated by the third camera (CA3).

[0782] The third image (8730) of FIG. 86 schematically illustrates one frame of a third image generated by photographing the rear of a container ship according to the present invention from a third position by a third camera (CA3). Referring to FIG. 81b, it can be seen that the third image includes some common features (sea surface at the rear, sky at the rear, containers at the rear, etc.) with the second image generated by the second camera (CA2).

[0783] In FIGS. 85 and 86, only the second and third images generated by the second camera (CA2) and the third camera (CA4) are described, but the fourth camera (CA4) and the fifth camera (CA5) are also similar to the second camera (CA2) and the third camera (CA4) in that they are installed symmetrically with the bow and stern as the center line and include the rear sea surface of the ship, the rear sky, and containers loaded at the rear, and therefore the fourth and fifth images generated by the fourth camera (CA4) and the fifth camera (CA5) can also be applied similarly to the contents described in FIGS. 85 and 86.

[0784] Figures 87a and 88B are diagrams schematically showing the results of comparing a first image generated by a first camera and an integrated image generated in a container ship according to the present invention.

[0785] The first image (8810) of FIG. 87a schematically illustrates one frame of a first image generated by photographing the rear of a container ship according to the present invention at a first position by a first camera (CA1), and referring to FIG. 81b, it can be seen that the first image includes some common features (sea surface at the rear, sky at the rear, containers at the rear, etc.) with the second and third images generated by the second camera (CA2) and the third camera (CA3).

[0786] In particular, the first image (8810) of FIG. 87a, unlike FIG. 85, exemplifies a case where containers are loaded in excess in the loading area at the rear of the wheelhouse. In the first image (8810), the containers are loaded higher than in FIG. 85, and thus the visibility toward the rear is worse compared to FIG. 85. More specifically, in the first image (8810), only information about containers is increased in the image, and information about objects on the sea surface is greatly reduced, so only the blind spot of the rear-loaded containers described in FIG. 81a is widened, and the container ship according to the present invention cannot naturally comply with the aforementioned "air draft". In the first image (8810) of FIG. 87a, the uppermost containers loaded higher than before in the second loading area (AR2) or the third loading area (AR3) can be set as obstacle areas (8200) in the first image, and the obstacle areas (8200) in the first image can be made transparent in the process of generating an integrated image through an image matching process, as described in FIG. 84. The integrated image generating device (8300) stores a unique object recognition algorithm to selectively recognize the positions of additionally loaded containers in the first image and designate them as obstacle areas (8200).

[0787] On the other hand, the integrated video image (8830) of FIG. 87b schematically represents one frame of the integrated video, and the integrated video image (8830) represents an image for the same time point t as the first image (8810). In the integrated video image (8830), all of the uppermost container layers designated as obstacle areas (8200) in the first image (8810) are made transparent and converted into transparent obstacle areas (8200'), so that the first object (ob1) in the blind spot of the rear-loaded container can be observed. That is, since a rear view can be secured even if the "air draft" is not observed, the container ship according to the present invention can load containers higher than existing container ships.

[0788] Figure 88 is a diagram schematically showing the results of comparing the loading capacity of containers of a conventional container ship and a container ship according to the present invention.

[0789] The first comparative image (8910) of FIG. 88 is an image showing the loading capacity of containers of a conventional container ship, and the second comparative image (8930) of FIG. 88 is an image showing the loading capacity of containers of a container ship according to the present invention.

[0790] As illustrated in Figure 88, the container ship according to the present invention can secure a view to the rear while loading an additional number of containers corresponding to the additional loading capacity (8900) compared to a conventional container ship.

[0791] Since the method according to the present invention can be implemented by the integrated image generation device (8300), the processor (8330) and the sub-modules included in the processor (8330) described in FIGS. 82 and 83, the method will be described below with reference to FIGS. 82 to 85, and any description that overlaps with the contents already described will be omitted.

[0792] The integrated image generation device (8300) can collect first, second, and third images from the first camera (CA1), second camera (CA2), and third camera (CA3) of the container ship. At this time, fourth and fifth images from the fourth camera (CA4) and fifth camera (CA5) may be added.

[0793] The integrated image generation device (8300) can extract overlapping feature points between the second and third images collected from the second camera (CA2) and the third camera (CA3). At this time, feature points may also be extracted from the fourth and fifth images.

[0794] The integrated image generation device (8300) can generate a target image by aligning the second and third images based on the extracted feature points. Depending on the embodiment, the fourth and fifth images may also be considered when generating the target image.

[0795] The integrated image generation device (8300) can generate an integrated image by integrating the first image collected from the first camera (CA1) and the generated target image.

[0796] The embodiments of the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. At this time, the medium may include a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a ROM, a RAM, a flash memory, etc.

[0797] Meanwhile, the computer program may be specifically designed and constructed for the present invention, or may be one known and available to those skilled in the computer software field. Examples of computer programs may include not only machine language code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

[0798] The specific implementations described in the present invention are exemplary embodiments and do not limit the scope of the present invention in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components illustrated in the drawings are merely representative of functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, unless specifically mentioned as “essential,” “important,” etc., a component may not be absolutely necessary for the application of the present invention.

[0799] The use of the term "above" and similar referential terms in the specification of the present invention (especially in the claims) may refer to both singular and plural. Furthermore, if a range is described in the present invention, it includes inventions that apply individual values ​​within the range (unless otherwise stated), and is equivalent to describing each individual value constituting the range in the detailed description of the invention. Finally, unless the order of the steps constituting the method according to the present invention is explicitly stated or otherwise stated to the contrary, the steps may be performed in any appropriate order. The present invention is not necessarily limited by the order in which the steps are described. The use of all examples or exemplary terms (e.g., "for example," etc.) in the present invention is merely intended to illustrate the present invention in detail, and the scope of the present invention is not limited by the examples or exemplary terms, unless otherwise defined by the claims. Furthermore, those skilled in the art will appreciate that various modifications, combinations, and variations can be made within the scope of the appended claims or their equivalents, depending on design conditions and factors.

[0800] One embodiment of the present invention can be used in the industry of manufacturing next-generation ships.

Claims

1. A step of collecting images from the first camera, second camera, and third camera of the ship; A step of extracting overlapping feature points between the second and third images collected from the second and third cameras; A step of generating a target image by aligning the second image and the third image based on the extracted feature points; and A method for generating an image of a ship with improved visibility, comprising: a step of generating an integrated image by integrating a first image collected from the first camera and the generated target image.

2. In paragraph 1, The above first camera, A method for generating an image of a vessel with improved visibility, the image being a camera installed adjacent to the wheelhouse of the vessel.

3. In paragraph 1, The second camera and the third camera, A method for generating an image of a vessel with improved visibility, the image being installed forward-facing on the bow of the vessel.

4. In paragraph 1, The second camera and the third camera, A method for generating an image of a ship with improved visibility, the image being installed symmetrically around the ship center line of the ship.

5. In paragraph 1, The steps of collecting the above video are: Collect more images from the fourth and fifth cameras of the above vessel, The step of extracting the above feature points is: Extracting overlapping feature points between the fourth and fifth images collected from the fourth and fifth cameras, The step of generating the above target image is: A method for generating an image of a ship with improved visibility, wherein the second to fifth images are aligned based on the extracted feature points to generate a target image.

6. In paragraph 5, The above fourth camera and the above fifth camera, A method for generating an image of a vessel with improved visibility, the image being installed further rearward than the second camera and the third camera.

7. In paragraph 1, The above vessel, A method for generating an image of a vessel with enhanced visibility, which is a cellular container ship with the wheelhouse positioned closer to the bow than the stern.

8. In paragraph 1, The second camera and the third camera, A method for generating an image of a vessel with improved visibility, the image being installed at a position exceeding a preset first forward viewing angle of the wheelhouse of the vessel.

9. In paragraph 1, The step of generating the above integrated image is: A method for generating an image of a ship with improved visibility, wherein, in the process of generating the above integrated image, at least some of the objects displayed in the first image are processed to be transparent.

10. In paragraph 9, The area that is made transparent in the first image above is A method for generating an image of a vessel with improved visibility, the image including an area exceeding the first forward viewing angle based on the wheelhouse.

11. A computer-readable recording medium storing a program for executing the method according to paragraph 1.

12. Memory in which at least one program is stored; and By executing at least one program, a processor is included that performs an operation, The above processor, Collect images from the ship's first camera, second camera, and third camera, Extracting overlapping feature points between the second and third images collected from the second and third cameras, The second image and the third image are aligned based on the extracted feature points to create a target image, A device for generating an image of a ship with improved visibility by integrating a first image collected from the first camera and the generated target image to generate an integrated image.

Images