Monitoring system and ship

Abstract

To provide a monitoring system and ship capable of improving monitoring accuracy and reducing a monitoring load on a user.SOLUTION: A monitoring system comprises: a distance information calculation section for calculating distance information concerning each one of a plurality of picture elements included in at least one object image between a first image and a second image based on the first image photographed by a first camera included in a stereo camera and the second image photographed by a second camera included in the stereo camera; and a map information generation section for generating three-dimensional map information obtained by mapping a periphery of the stereo camera on a three-dimensional space based on color information related to each one of the picture elements and the distance information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a surveillance system and a ship. [Background technology]

[0002] Conventionally, for example, on ships, a monitoring system that grasps the surrounding situation using sensor fusion technology has been used (see, for example, Patent Document 1). In sensor fusion technology, a plurality of types of sensors are used in combination to detect the surrounding situation. Examples of sensors used in sensor fusion technology include an AIS (Automatic Identification System), radar (for example, millimeter wave radar), and an AI (Artificial Intelligence) camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-170010 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it has been difficult for the sensors used in the sensor fusion technology described above to completely monitor the surrounding situation. For example, when monitoring using AIS, it is impossible to detect obstacles other than ships or ships that do not have AIS. Furthermore, when monitoring using radar, there is a possibility of false detection due to the reflection of radar waves on the sea surface or ghosts.

[0005] Therefore, in systems using sensor fusion technology that combines such sensors, the monitoring accuracy of each individual sensor is insufficient, which can result in insufficient monitoring of the surroundings, forcing users of the monitoring system to actually visually check the surroundings, potentially increasing the burden on the users.

[0006] The present invention has been made in consideration of the above circumstances, and aims to provide a monitoring system and a ship that can improve monitoring accuracy and reduce the monitoring burden on the user. [Means for solving the problem]

[0007] In order to solve the above problem, a surveillance system according to a first aspect of the present invention includes a distance information calculation unit that calculates distance information for each of a plurality of pixels included in a target image, which is at least one of a first image taken by a first camera of a stereo camera and a second image taken by a second camera of the stereo camera, based on the first image and the second image, and a map information generation unit that generates three-dimensional map information that maps the surroundings of the stereo camera onto a three-dimensional space based on the color information and the distance information for each of the plurality of pixels.

[0008] In addition, according to a second aspect of the present invention, the monitoring system of the first aspect further comprises a display unit that displays the three-dimensional map information in a form that can be visually recognized by the user.

[0009] Furthermore, according to a third aspect of the present invention, the surveillance system of the first or second aspect further includes a plurality of the stereo cameras, wherein the distance information calculation unit calculates the distance information based on the first images and the second images captured by the stereo cameras, and the stereo cameras are arranged so that the combined imaging ranges of the stereo cameras cover 360° of the surrounding area.

[0010] Furthermore, aspect 4 of the present invention is a surveillance system according to any one of aspects 1 to 3, further comprising a plurality of the stereo cameras, wherein the distance information calculation unit calculates the distance information based on the plurality of first images and the plurality of second images taken by the plurality of stereo cameras, and the first camera and the second camera of each of the plurality of stereo cameras are arranged in a direction intersecting the direction in which the plurality of stereo cameras are arranged.

[0011] Furthermore, in aspect 5 of the present invention, in the surveillance system of any one of aspects 1 to 4, the map information generation unit acquires shaking information relating to shaking of the stereo camera, and generates the three-dimensional map information with the shaking of the stereo camera corrected based on the shaking information.

[0012] Furthermore, in aspect 6 of the present invention, in the surveillance system of any one of aspects 1 to 5, a detection unit is further provided that detects obstacles by image processing of the target image, and the map information generation unit synthesizes information about the obstacles detected by the detection unit onto the three-dimensional map information.

[0013] Furthermore, according to a seventh aspect of the present invention, the monitoring system of any one of the first to sixth aspects further comprises a calibration unit that calibrates the stereo camera based on the distance to an object whose distance is known.

[0014] Furthermore, aspect 8 of the present invention relates to the surveillance system of aspect 7, wherein the first camera and the second camera are arranged in a direction tilted relative to the horizontal direction, the object is a horizon, and the calibration unit calibrates the stereo camera based on the horizon included in the first image and the horizon included in the second image.

[0015] A ship according to a ninth aspect of the present invention is equipped with the monitoring system according to any one of the first to eighth aspects. [Effects of the Invention]

[0016] According to the above aspects of the present invention, it is possible to provide a monitoring system and a ship that can improve monitoring accuracy and reduce the monitoring burden on the user. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing a system configuration of a monitoring system according to an embodiment of the present invention. [Figure 2] 1 is a perspective view showing a stereo camera according to an embodiment of the present invention; [Figure 3] FIG. 10 is a diagram illustrating an example of a method for arranging a plurality of stereo cameras. [Figure 4] FIG. 10 is a diagram illustrating an example of a method for generating three-dimensional map information. [Figure 5] FIG. 10 is a diagram illustrating an example of auxiliary information. [Figure 6] FIG. 10 is a diagram illustrating an example of a composite display. [Figure 7] 4 is a flowchart illustrating an example of processing performed in the monitoring system according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] A monitoring system according to an embodiment of the present invention will be described in detail below with reference to the drawings.

[0019] <Monitoring system> Fig. 1 is a block diagram showing the system configuration of a monitoring system 1 according to an embodiment of the present invention. As shown in Fig. 1, the monitoring system 1 according to this embodiment includes a sensor unit 10, an interface unit 20, a storage unit 30, and a processing unit 40. The sensor unit 10 according to this embodiment has a plurality of stereo cameras 11. The portions of the monitoring system 1 relating to the interface unit 20, the storage unit 30, and the processing unit 40 may be configured using information devices such as a smartphone, a tablet, a personal computer, or a dedicated device.

[0020] The monitoring system 1 is a system that generates information for monitoring the surroundings of the stereo camera 11 based on images captured by the stereo camera 11. The stereo camera 11 is installed on a ship, for example, and the monitoring system 1 is used to monitor the oceans and rivers through which the ship navigates, the surrounding airspace, etc. However, the use of the monitoring system 1 is not limited to monitoring oceans and rivers and can be changed as appropriate. For example, the stereo camera 11 may be installed on an aircraft, and the monitoring system 1 may be used for air traffic control. The monitoring system 1 may also be used for land-based monitoring and aerial monitoring from land.

[0021] Stereo camera 11 has first camera 111 and second camera 112. Each of first camera 111 and second camera 112 captures an image. Hereinafter, an image captured by first camera 111 may be referred to as a first image, and an image captured by second camera 112 may be referred to as a second image. Stereo camera 11 outputs the captured first image and second image to processing unit 40.

[0022] Each of the first image and the second image includes a plurality of pixels. Each pixel has color information indicating the color of the object reflected in that pixel. Note that the term "color information" in this specification is not particularly limited as long as it is information that changes depending on the color of the object. For example, the color information may be information having all of hue, saturation, and brightness (color information), or may be information having only brightness (grayscale information).

[0023] The first camera 111 and the second camera 112 face in approximately the same direction (see also FIG. 2). More specifically, the optical axis of the first camera 111 and the optical axis of the second camera 112 are approximately parallel. Note that the terms "approximately the same direction" and "approximately parallel" used in this specification also include cases where they can be considered to be "the same direction" and "parallel" respectively if manufacturing errors are removed. Hereinafter, the direction in which the cameras 111 and 112 face may be referred to as the shooting direction.

[0024] First camera 111 and second camera 112 are arranged at an interval in a direction intersecting the shooting direction (intersecting direction) (see also FIG. 2). Furthermore, first camera 111 and second camera 112 are arranged so that the shooting range of first camera 111 and the shooting range of second camera 112 overlap.

[0025] FIG. 2 is a perspective view showing the stereo camera 11 according to this embodiment. As shown in FIG. 2, the stereo camera 11 according to this embodiment has three camera sets P1 to P3. Each of the three camera sets P1 to P3 includes one first camera 111 and one second camera 112. The number of camera sets (sets of the first camera 111 and the second camera 112) included in the stereo camera 11 can be changed as appropriate as long as it is one (one set) or more. Furthermore, each camera set may have three or more cameras.

[0026] The stereo camera 11 according to this embodiment further includes a rectangular parallelepiped housing 113. The housing 113 houses three camera sets P1 to P3. The three camera sets P1 to P3 are exposed to one surface 113a (hereinafter referred to as the imaging surface 113a) of the housing 113. Therefore, the imaging directions of the three camera sets P1 to P3 are the same.

[0027] The imaging surface 113a of the housing 113 has a shape that extends in the gravity direction (vertical direction) Z. The gravity direction (vertical direction) Z refers to a direction parallel to gravity. On the imaging surface 113a, the first camera set P1 is located at both ends in the gravity direction Z. Also, on the imaging surface 113a, the second camera set P2 is located inside the first camera set P1 in the gravity direction Z, and the third camera set P3 is located inside the second camera set P2 in the gravity direction Z.

[0028] In each of the camera sets P1 to P3, the first camera 111 and the second camera 112 are arranged at an interval in the gravity direction Z. Specifically, the first camera 111 is located above the second camera 112. In other words, the shooting direction of each of the cameras 111, 112 is parallel to the horizontal direction.

[0029] Each of the three camera sets P1 to P3 functions as a stereo camera including a first camera 111 and a second camera 112. The three camera sets P1 to P3 may be used differently depending on, for example, the purpose of the stereo camera 11 (e.g., the environment in which the stereo camera 11 is installed, the size of the monitoring range of the monitoring system 1, etc.). For example, the three camera sets P1 to P3 may be used differently depending on the time of day, such as the first camera set P1 for daytime use, the second camera set P2 for evening use, and the third camera set P3 for night use. Alternatively, the three camera sets P1 to P3 may be used differently depending on the size of the monitoring range, such as the first camera set P1 for long-distance use, the second camera set P2 for medium-distance use, and the third camera set P3 for short-distance use.

[0030] FIG. 3 is a diagram showing an example of a method for arranging a plurality of stereo cameras 11. The stereo cameras 11 have different shooting directions. As shown in FIG. 3, in the monitoring system 1 according to this embodiment, the stereo cameras 11 are arranged in a circle when viewed from the gravity direction Z (i.e., in a plan view). As a result, the stereo cameras 11 are arranged so that when the shooting ranges of the stereo cameras 11 are combined, they cover a 360° surrounding area. Note that the range (observation range) covered by the stereo cameras 11 does not have to be 360° and can be changed as appropriate. For example, the observation range may be 90°, 180°, 270°, or another angle.

[0031] Specifically, the multiple stereo cameras 11 are arranged and fixed along the outer circumferential surface of a cylindrical fixed object 114. As a result, the direction in which the multiple stereo cameras 11 are arranged (arrangement direction A) coincides with the circumferential direction C of the fixed object 114. By intersecting the direction in which the cameras 111 and 112 are lined up (gravity direction Z) with the arrangement direction A (circumferential direction C) in this way, it becomes easier to reduce the space required to arrange the multiple stereo cameras 11.

[0032] The type of fixed object 114 is not particularly limited, and may be, for example, a radar mast of a ship, etc. The arrangement of stereo camera 11 can also be changed as appropriate.

[0033] As shown in FIG. 1, a sensor unit 10 according to this embodiment has a plurality of stereo cameras 11, a tilt sensor 12, and a distance sensor 13.

[0034] The tilt sensor 12 acquires the tilt of the stereo camera 11. Specifically, the tilt sensor 12 acquires the tilt of the stereo camera 11 with respect to a horizontal plane. The tilt sensor 12 outputs the acquired tilt to the processing unit 40.

[0035] Note that one tilt sensor 12 may be provided for each of the multiple stereo cameras 11. In this case, the sensor unit 10 may have the same number of tilt sensors 12 as the number of stereo cameras 11 (i.e., multiple tilt sensors 12). Alternatively, one tilt sensor 12 may be provided for the multiple stereo cameras 11. In this case, the sensor unit 10 may have only one tilt sensor 12.

[0036] The distance sensor 13 acquires the distance to a target object present in the vicinity. The distance sensor 13 outputs the acquired distance to the processing unit 40. When the monitoring system 1 is used to monitor an ocean or a river, examples of targets whose distances are acquired by the distance sensor 13 include drifting objects, buoys, ships, etc.

[0037] Examples of distance sensors 13 include AIS, radar, and combinations of these. AIS is a system that transmits and receives information (ship information) related to ships equipped with AIS via radio waves, and exchanges ship information between ships and between ships and land-based facilities. Radar is a device that emits radio waves into the surrounding area and obtains the distance to surrounding objects based on the returned radio waves.

[0038] The interface unit 20 presents information to a user who uses the monitoring system 1 and accepts operations from the user. The interface unit 20 according to this embodiment includes a display unit 21 and an input unit 22.

[0039] The input unit 22 is configured using existing input devices such as a keyboard, a pointing device (mouse, tablet, etc.), a button, a touch panel, etc. The input unit 22 is operated by a user when inputting the user's instructions to the monitoring system 1. The input unit 22 may be configured in any way as long as it is capable of inputting the user's instructions to the monitoring system 1.

[0040] The display unit 21 displays information in a form that can be viewed by the user. That is, the display unit 21 converts information generated by a generating unit 44 (described later, for example, three-dimensional map information M and auxiliary information E1 and E2) into light and displays it. The display unit 21 may be an image display device such as an LCD display or an organic EL (Electro Luminescence) display. The display unit 21 may be configured as a touch panel integrated with the input unit 22.

[0041] The storage unit 30 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 30 stores data used by the processing unit 40. The storage unit 30 stores data required when the processing unit 40 performs processing.

[0042] The processing unit 40 comprehensively controls the operation of the monitoring system 1. The processing unit 40 is configured using a processor such as a CPU (Central Processing Unit) and a memory (main storage device). The processing unit 40 according to this embodiment functions as a distance information calculation unit 41, a calibration unit 42, a detection unit 43, and a generation unit 44 by the processor executing a program.

[0043] The distance information calculation unit 41 calculates distance information for each of a plurality of pixels included in the target image based on the first image captured by the first camera 111 and the second image captured by the second camera 112. Here, the target image is at least one of the first image and the second image. The selection of the target image may be performed by the distance information calculation unit 41, for example. The distance information is information indicating the distance between the target object reflected in the pixel of interest and the stereo camera 11. The distance information may also include orientation information indicating the orientation of the target object as viewed from the stereo camera 11.

[0044] The distance information calculation unit 41 according to this embodiment calculates distance information based on a plurality of first images and a plurality of second images captured by a plurality of stereo cameras 11. More specifically, for each of the plurality of stereo cameras 11, a target image is selected from the first image and the second image. Then, for each of the plurality of stereo cameras 11, distance information for each of the plurality of images included in the selected target image is calculated.

[0045] Specifically, the distance information calculation unit 41 may calculate the distance information of each pixel based on the parallax between the first image and the second image, the focal length of the lenses used in the cameras 111 and 112, the distance between the optical axes of the cameras 111 and 112, etc.

[0046] The calibration unit 42 calibrates the stereo camera 11 based on the distance to an object whose distance is known (hereinafter referred to as a calibration object). For example, if the stereo camera 11 is used for a long period of time, the parameters of the stereo camera 11 (such as the focal length of the lenses used in the cameras 111 and 112 and the distance between the optical axes of the cameras 111 and 112) may gradually change. Therefore, if the changes in the parameters are not taken into consideration, the accuracy of the calculation of distance information by the distance information calculation unit 41 may gradually decrease. To prevent such a decrease in calculation accuracy, the calibration unit 42 calibrates the stereo camera 11, that is, updates the information of the parameters of the stereo camera 11 used by the distance information calculation unit 41.

[0047] For example, the calibration unit 42 can adopt, as the calibration object, a target whose distance has been acquired by the distance sensor 13. Specifically, when a radar or an AIS is used as the distance sensor 13, the calibration unit 42 can adopt, as the calibration object, a target (such as a ship) captured by the radar or the AIS. That is, the calibration unit 42 can calibrate the stereo camera 11 based on the distance to the calibration object acquired by the distance sensor 13 such as a radar or an AIS.

[0048] Furthermore, depending on the positional relationship between the first camera 111 and the second camera 112, the calibration unit 42 can use the horizon as the calibration object. Specifically, if the direction in which the first camera 111 and the second camera 112 are aligned is tilted with respect to the horizontal direction in which the horizon extends, calibration using the horizon as the calibration object is possible. This is because if the alignment direction of the cameras 111 and 112 is tilted with respect to the horizontal direction, the position of the horizon will be shifted between the first image and the second image, making it possible to detect parallax.

[0049] In this embodiment, as described above, the direction in which the cameras 111 and 112 are aligned is the direction of gravity Z, and is inclined with respect to the horizontal direction (specifically, perpendicular to it). This allows for such calibration. In particular, since the direction in which the cameras 111 and 112 are aligned is perpendicular to the horizontal direction, the accuracy of the calibration can be improved compared to when the direction in which the cameras 111 and 112 are aligned forms an acute angle with the horizontal direction. The distance to the horizon can be calculated based on the height at which the stereo camera 11 is installed and the diameter of the Earth.

[0050] The calibration by the calibration unit 42 may be started in response to an instruction from the user (specifically, the user's operation of the input unit 22). Alternatively, the calibration by the calibration unit 42 may be started automatically based on a predetermined execution cycle or the like.

[0051] The detection unit 43 detects obstacles by image processing of the target image. For example, in a monitoring system 1 in which the stereo camera 11 is installed on a ship, an obstacle refers to a target that hinders the navigation of the ship. Examples of targets in this case include drifting objects, buoys, other ships, etc. Specifically, the detection unit 43 may detect obstacles from the target image by image processing using AI.

[0052] The generating unit 44 generates information to be displayed on the display unit 21. The generating unit 44 according to this embodiment has a function as a map information generating unit 441 and a function as an auxiliary display information generating unit 442.

[0053] The map information generating unit 441 generates three-dimensional map information M that maps the surroundings of the stereo camera 11 onto a three-dimensional space based on color information and distance information related to each of a plurality of pixels included in the target image. An example of a method for generating the three-dimensional map information M will be specifically described below with reference to FIG. 4.

[0054] As described above, color information of each pixel is acquired by stereo camera 11, and distance information of each pixel is acquired by distance information calculation unit 41. Map information generation unit 441 generates three-dimensional map information M based on a set of the acquired color information and distance information.

[0055] As shown in FIG. 4, first, the position of the stereo camera 11 is set to a reference point O in three-dimensional space. Then, based on distance information for each pixel included in the target image, a point corresponding to that pixel is plotted at a position that is a distance away from the reference point O. In the illustrated example, a point corresponding to pixel X1 in the target image is plotted at a point that is a distance D1 away from the reference point O in three-dimensional space. Similarly, a point corresponding to pixel X2 in the target image is plotted at a point that is a distance D2 away from the reference point O in three-dimensional space.

[0056] In this case, the color of the point plotted in the three-dimensional space is determined based on the color information of the corresponding pixel. For example, the color of the point corresponding to pixel X1 corresponds to the color information of pixel X1 in the target image. Similarly, the color of the point corresponding to pixel X2 corresponds to the color information of pixel X2 in the target image.

[0057] Note that the color information of a pixel and the color of a point do not necessarily have to match. For example, if the color information of a pixel is information (color information) that includes all of hue, saturation, and brightness, the color of the point plotted in three-dimensional space may be displayed in grayscale. In this case, the map information generating unit 441 may extract brightness from the color information to determine the color of the point displayed in grayscale. In this way, it is sufficient if there is some correspondence between the color information of a pixel and the color of the point.

[0058] By the plotting process described above, the target object Y, the water surface, etc. reflected in the target image are plotted in three-dimensional space based on optical information, that is, the image captured by the camera. In other words, the target object Y, the water surface, etc. are plotted in three-dimensional space as they actually appear to the human eye. By adopting a method using optical information in this way, it is possible to avoid the problems that arise when using only radar or AIS in a surveillance system, and to accurately display surrounding information in three-dimensional space.

[0059] By performing the above-described plotting process on all of the multiple target images corresponding to the multiple stereo cameras 11, three-dimensional map information M including information on the 360° surroundings of the ship is generated, as shown in FIG. 4. Note that boundary line B shown in FIG. 4 indicates the boundary line of the imaging range of the stereo cameras 11. In other words, the area surrounded by two adjacent boundary lines B corresponds to the imaging range of one stereo camera 11. The angle of view (field of view) θ of each stereo camera 11 may be, for example, approximately 15° to 20°. Note that boundary line B may or may not be displayed on the three-dimensional map information M.

[0060] Furthermore, the three-dimensional map information M displayed on the display unit 21 may be configured so that the user can change the viewpoint by operating the input unit 22, for example. In other words, the map information generation unit 441 may output the three-dimensional map information M to the display unit 21 in a form that allows such viewpoint change. For example, in FIG. 4, the viewpoint is set so that the ship on which the stereo camera 11 is installed is viewed from above, but the height of the viewpoint (i.e., the depression angle) and the horizontal position of the viewpoint may be changeable. However, the viewpoint of the three-dimensional map information M may be configured so that it cannot be changed.

[0061] Furthermore, when generating the three-dimensional map information M, the map information generation unit 441 may acquire shaking information related to the shaking of the stereo camera 11 and generate the three-dimensional map information M in a state in which the shaking of the stereo camera 11 is corrected based on the shaking information. That is, the map information generation unit 441 may correct the position of the plot (point) corresponding to each pixel based on the shaking information so that the shaking of the stereo camera 11 is not reflected in the three-dimensional map information M.

[0062] Specifically, the map information generating unit 441 may acquire (calculate) shake information of the stereo camera 11 based on information about the tilt of the stereo camera 11 acquired by the tilt sensor 12. Alternatively, the map information generating unit 441 may detect a reference (shake reference) from the target image by image processing, and acquire (calculate) shake information based on the position of the shake reference. The shake reference may be, for example, the sea surface (water surface). Then, the map information generating unit 441 may correct the shake of the stereo camera 11 based on the shake information acquired (calculated) by these methods. When the sea surface (water surface) is used as the shake reference, the map information generating unit 441 may correct the shake of the stereo camera 11 by regarding the sea surface (water surface) detected from the target image as a horizontal plane.

[0063] Even if the map information generating unit 441 does not have such a correction function, the same effect can be obtained by providing the stereo camera 11 with a mechanism (such as a gimbal) that structurally suppresses shaking of the cameras 111 and 112. However, when such a mechanism is introduced, the structure of the stereo camera 11 may become complex or large. By having the map information generating unit 441 perform shake correction based on data processing as described above, such problems can be avoided.

[0064] The auxiliary display information generation unit 442 generates auxiliary information. The auxiliary information is information that indicates the surroundings of the stereo camera 11, and is information that is different from the three-dimensional map information M. However, the monitoring system 1 does not necessarily have to be equipped with the auxiliary display information generation unit 442.

[0065] FIG. 5 is a diagram showing examples of auxiliary information (auxiliary information E1, E2). Auxiliary information E1 is image information generated based on information acquired by radar. Auxiliary information E1 may include information based on ARPA (Automatic Radar Plotting Aids). Auxiliary information E2 is image information generated based on information acquired by AIS. Auxiliary information E2 may include information based on an electronic chart. However, the content of the auxiliary information can be changed as appropriate.

[0066] The generation unit 44 may output both the three-dimensional map information M generated by the map information generation unit 441 and the auxiliary information E1 and E2 generated by the auxiliary display information generation unit 442 to the display unit 21. For example, the generation unit 44 may be configured so that a state in which the three-dimensional map information M is displayed on the display unit 21, a state in which the auxiliary information E1 is displayed on the display unit 21, and a state in which the auxiliary information E2 is displayed on the display unit 21 can be switched by the user operating the input unit 22, etc. In other words, the generation unit 44 may output the information M, E1, and E2 to the display unit 21 in such a switchable form.

[0067] Furthermore, the map information generation unit 441 may combine information about the obstacle detected by the detection unit 43 onto the generated three-dimensional map information M. The map information generation unit 441 may output the three-dimensional map information M into which the information about the obstacle has been combined in this way to the display unit 21. In other words, the display unit 21 may combine the information about the obstacle detected by the detection unit 43 onto the three-dimensional map information M and display it.

[0068] FIG. 6 is a diagram showing an example of such a composite display. As shown in FIG. 6, in this composite display, icons I based on information about obstacles detected by the detection unit 43 are composited onto three-dimensional map information M. Each icon I indicates an obstacle detected by the detection unit 43. In the example shown in the figure, the icons I are pop-ups that are displayed near the position of the corresponding obstacle on the three-dimensional map information M. The icons I may include an image, a name, or the like indicating the corresponding obstacle. Furthermore, in the example of FIG. 6, in addition to the icons I, targets T based on information acquired by AIS or ARPA are also displayed. Each target T indicates an obstacle captured by AIS or ARPA using a predetermined shape (e.g., a diamond, a triangle, a circle, or the like).

[0069] <Processing performed in the monitoring system> Fig. 7 is a flowchart showing an example of processing performed in the monitoring system 1 according to this embodiment. The processing of the flowchart shown in Fig. 7 is started, for example, when a user instructs the monitoring system 1 to start generating map information. The instruction to start generating map information may be given, for example, by the user operating the input unit 22.

[0070] (Step 101) First, the process of step S101 is performed. In the process of step S101, the stereo camera 11 captures a first image and a second image. The stereo camera 11 outputs the captured first image and second image to the processing unit 40.

[0071] (Step S102) After the process of step S101 is performed, the process of step S102 is performed. In the process of step S102, the distance information calculation unit 41 calculates distance information for each of a plurality of pixels included in the target image, which is at least one of the first image and the second image. Details of the method of calculating the distance information are as described above.

[0072] (Step S103) After the processing of step S102 is performed, the processing of step S103 is performed. In the processing of step S103, map information generation unit 441 generates three-dimensional map information M based on color information and distance information related to each of the multiple pixels. Details of the method of generating three-dimensional map information M are as described above. Map information generation unit 441 outputs the generated three-dimensional map information M to display unit 21.

[0073] (Step S104) After the process of step S103 is performed, the process of step S104 is performed. In the process of step S104, display unit 21 displays three-dimensional map information M output by map information generation unit 441 and presents it to the user. After the process of step S104 is performed, the process of the flowchart ends.

[0074] The above-described processing of steps S101 to S104 may be repeated multiple times at predetermined time intervals. In other words, the information displayed on display unit 21 may be continuously updated in real time by repeatedly capturing images and generating three-dimensional map information M.

[0075] <effect> Next, the operation of the monitoring system 1 configured as above will be described.

[0076] Conventionally, for example, on ships, monitoring systems that grasp the surrounding situation using sensor fusion technology have been used (see, for example, Patent Document 1). In sensor fusion technology, multiple types of sensors are used in combination to detect the surrounding situation. Examples of sensors used in sensor fusion technology include AIS and radar. However, monitoring using AIS and radar does not provide sufficient accuracy, and there are cases where sufficient monitoring accuracy cannot be obtained even when these sensors are combined. As a result, users of the monitoring system have to actually visually check the surrounding situation, which can increase the burden on the users.

[0077] To address this issue, in the monitoring system 1 according to this embodiment, information on surrounding targets and the like is acquired by photographing them with a camera. This allows for accurate acquisition of information on targets and the like that can be at least visually confirmed by humans (i.e., optically confirmed). Therefore, monitoring accuracy can be improved compared to using only radar, which may be subject to error, or AIS, which cannot detect ships not equipped with radar. Furthermore, the acquired target information is provided in a form that is mapped in three-dimensional space. This allows the user to easily recognize targets and the like. Therefore, the monitoring burden on the user can be reduced.

[0078] <Summary> As described above, the monitoring system 1 according to this embodiment includes a distance information calculation unit 41 that calculates distance information for each of a plurality of pixels included in a target image, which is at least one of a first image captured by a first camera 111 included in the stereo camera 11 and a second image captured by a second camera 112 included in the stereo camera 11, based on the first image and the second image, and a map information generation unit 441 that generates three-dimensional map information M that maps the periphery of the stereo camera 11 onto a three-dimensional space, based on the color information and distance information for each of the plurality of pixels.

[0079] This configuration makes it possible to provide a monitoring system 1 that can improve monitoring accuracy and reduce the monitoring burden on the user. The generated three-dimensional map information M may be subjected to analysis by AI or the like. Converting the acquired information into the above-described three-dimensional map information M also has the effect of making such analysis easier to perform.

[0080] Moreover, the monitoring system 1 according to this embodiment further includes a display unit 21 that displays the three-dimensional map information M in a form that can be visually recognized by the user. With this configuration, the user can easily perform monitoring by checking the three-dimensional map information M displayed on the display unit 21.

[0081] Furthermore, the distance information calculation unit 41 calculates distance information based on the plurality of first images and the plurality of second images captured by the plurality of stereo cameras 11. That is, images captured by the plurality of stereo cameras 11 are used to generate the three-dimensional map information M. This allows the monitoring range to be expanded compared to when only images captured by a single stereo camera 11 are used. In particular, when long-distance cameras are used as the stereo cameras 11, the angle of view (field of view) of each stereo camera 11 tends to be small. Even in such a case, using a combination of the plurality of stereo cameras 11 makes it easier to ensure a monitoring range.

[0082] The monitoring system 1 according to this embodiment further includes a plurality of stereo cameras 11, which are arranged so that the combined imaging ranges of the plurality of stereo cameras 11 cover a 360° surrounding area. This configuration allows the user to easily monitor a 360° surrounding area.

[0083] Furthermore, the first camera 111 and the second camera 112 of each of the stereo cameras 11 are aligned in a direction intersecting with the arrangement direction A in which the stereo cameras 11 are arranged. This configuration makes it easier to reduce the space required to arrange the stereo cameras 11.

[0084] Furthermore, the map information generating unit 441 acquires vibration information relating to the vibration of the stereo camera 11, and generates the three-dimensional map information M in a state where the vibration of the stereo camera 11 has been corrected based on the vibration information. According to this configuration, the visibility of the three-dimensional map information M can be improved.

[0085] Moreover, the monitoring system 1 according to this embodiment further includes a detection unit 43 that detects obstacles by image processing of the target image, and the map information generation unit 441 combines information about the obstacles detected by the detection unit 43 onto the three-dimensional map information M. With this configuration, the user can perform monitoring based on both the information from the detection unit 43 and the three-dimensional map information M. This can further improve the monitoring accuracy.

[0086] The monitoring system 1 according to this embodiment further includes a calibration unit 42 that calibrates the stereo camera 11 based on the distance to an object whose distance is known. This configuration can prevent a decrease in the accuracy of distance information calculation by the distance information calculation unit 41.

[0087] Moreover, the monitoring system 1 according to this embodiment further includes a distance sensor 13 that acquires the distance to the target object, and the calibration unit 42 calibrates the stereo camera 11 based on the distance acquired by the distance sensor 13. With this configuration, the stereo camera 11 can be easily calibrated.

[0088] Furthermore, first camera 111 and second camera 112 are aligned in a direction tilted relative to the horizontal direction, the target is the horizon, and calibration unit 42 calibrates stereo camera 11 based on the horizon included in the first image and the horizon included in the second image. With this configuration, calibration of stereo camera 11 can be easily performed without providing a dedicated sensor or the like for calibration.

[0089] The technical scope of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.

[0090] For example, in the above embodiment, pixels included in the target image correspond one-to-one to points in three-dimensional space, but the correspondence between pixels and points is not limited to this. For example, multiple pixels may correspond to one point. In this case, for example, a single piece of color information and a single piece of distance information may be calculated for a predetermined number of pixels by calculating an average of the color information and distance information of each pixel. Then, based on the color information and distance information calculated in this way, a single point corresponding to the predetermined number of pixels may be plotted in three-dimensional space.

[0091] Furthermore, the monitoring system 1 may not include the sensor unit 10 (stereo camera 11, tilt sensor 12, and distance sensor 13). In this case, the monitoring system 1 may include an acquisition unit that acquires information output from the sensor unit 10 and is provided outside the monitoring system 1. Similarly, the monitoring system 1 may not include the interface unit 20 (display unit 21 and input unit 22). In this case, the monitoring system 1 may include a transmission / reception unit that transmits and receives information to and from the interface unit 20 provided outside the monitoring system 1.

[0092] The above-mentioned acquisition unit and transmission / reception unit may be configured by a communication unit that performs data communication with the outside of the monitoring system 1 via a network. The network may be a network using wireless communication or a network using wired communication. The network may be configured using the Internet or a local area network (LAN). The network may also be configured by combining multiple networks.

[0093] In addition, it is possible to replace the components in the above-described embodiments with well-known components as appropriate, and the above-described embodiments and variations may be combined as appropriate, without departing from the spirit of the present invention.

[0094] All or part of the functions of the processing unit 40 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0095] The above program may be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., SSDs: Solid State Drives), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may also be transmitted via telecommunications lines. [Explanation of symbols]

[0096] REFERENCE SIGNS LIST 1... Surveillance system 11... Stereo camera 111... First camera 112... Second camera 12... Tilt sensor 13... Distance sensor 21... Display unit 41... Distance information calculation unit 42... Calibration unit 43... Detection unit 441... Map information generation unit M... Three-dimensional map information

Claims

1. a distance information calculation unit that calculates distance information for each of a plurality of pixels included in a target image, which is at least one of a first image captured by a first camera included in a stereo camera and a second image captured by a second camera included in the stereo camera, based on the first image and the second image; a map information generation unit that generates three-dimensional map information that maps an area around the stereo camera onto a three-dimensional space based on color information related to each of the plurality of pixels and the distance information, Surveillance system.

2. a display unit that displays the three-dimensional map information in a form that can be visually recognized by a user; The monitoring system of claim 1 .

3. Further comprising a plurality of the stereo cameras, the distance information calculation unit calculates the distance information based on the first images and the second images captured by the stereo cameras; The plurality of stereo cameras are arranged so that the imaging ranges of the plurality of stereo cameras are combined to cover 360° of the surroundings. The monitoring system of claim 1 .

4. Further comprising a plurality of the stereo cameras, the distance information calculation unit calculates the distance information based on the first images and the second images captured by the stereo cameras; the first camera and the second camera included in each of the plurality of stereo cameras are aligned in a direction intersecting a direction in which the plurality of stereo cameras are arranged; The monitoring system of claim 1 .

5. the map information generation unit acquires vibration information relating to vibration of the stereo camera, and generates the three-dimensional map information in a state in which the vibration of the stereo camera is corrected based on the vibration information. The monitoring system of claim 1 .

6. a detection unit that detects an obstacle by image processing of the target image, the map information generation unit synthesizes information about the obstacle detected by the detection unit onto the three-dimensional map information. The monitoring system of claim 1 .

7. The system further includes a calibration unit that calibrates the stereo camera based on a distance to an object whose distance is known. The monitoring system of claim 1 .

8. the first camera and the second camera are aligned in a direction tilted with respect to a horizontal direction, the object is a horizon, the calibration unit calibrates the stereo camera based on a horizontal line included in the first image and a horizontal line included in the second image. The monitoring system of claim 7.

9. A vessel equipped with the monitoring system according to any one of claims 1 to 8.

Images