Information processing method, information processing device, and computer program

Abstract

[Problem] To provide an information processing method, an information processing device, and a computer program that enable effective identification of objects on water. [Solution] The information processing method for identifying an object-on-water around a ship comprises: acquiring photographed images obtained by photographing the surroundings of the ship via a plurality of photographing devices of different types installed on the ship; inputting the photographed images acquired from each of the plurality of photographing devices to a trained model that outputs results of identifying the objects on water included in the photographed image when the photographed images are input; acquiring the results of identifying the objects on water output from the trained model; and outputting the results of identifying the objects on water included in each of the plurality of photographing devices.

Description

[Technical Field]

[0001] The present invention relates to an information processing method, an information processing device, and a computer program for identifying an object on water around a ship. [Background technology]

[0002] For navigation, ships need to be able to identify objects on the water, such as other ships, that are present in the vicinity. Conventionally, technologies for identifying objects on the water have been developed. Patent Document 1 discloses a technology that makes it easy to identify objects on the water by displaying an image of the exterior of the ship in association with the results of radar identification. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2020 / 0018848 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technologies for identifying objects on the water are still insufficient compared to visual identification by mariners. For example, radar-based technologies have problems such as the generation of false images or the inability to identify objects on the water. For this reason, there is a demand for technologies that can more effectively identify objects on the water.

[0005] The present invention has been made in consideration of the above circumstances, and its purpose is to provide an information processing method, an information processing device, and a computer program that enable effective identification of water-borne objects. [Means for solving the problem]

[0006] The information processing method of the present invention is characterized in that it acquires photographic images of the surroundings of a ship using a plurality of different types of photographing devices installed on the ship, inputs the photographic images acquired from each of the plurality of photographing devices into a trained model that outputs a discrimination result for water-surface objects included in the photographic images when the photographic images are input, obtains the discrimination result for water-surface objects output by the trained model, and outputs the discrimination result for water-surface objects included in each of the plurality of photographing devices.

[0007] The information processing device of the present invention is characterized by comprising an image acquisition unit that acquires photographed images of the surroundings of a ship taken by a plurality of different types of photographing devices, a discrimination result acquisition unit that inputs photographed images acquired from each of the plurality of photographing devices into a trained model that outputs a discrimination result of an object on the water contained in the photographed image when the photographed image is input, and acquires the discrimination result of an object on the water output by the trained model, and an output unit that outputs the discrimination result corresponding to each of the plurality of photographing devices.

[0008] The computer program of the present invention is characterized in that it causes a computer to execute a process of acquiring photographic images of the surroundings of a ship taken by a plurality of different types of photographic devices, inputting photographic images acquired from each of the plurality of photographic devices into a trained model that outputs a discrimination result for an object on the water contained in the photographic image when the photographic image is input, obtaining the discrimination result for an object on the water output by the trained model, and outputting the discrimination result corresponding to each of the plurality of photographic devices.

[0009] In one aspect of the present invention, images captured by multiple camera devices installed on a ship are input to a trained model, and the trained model outputs a classification result for objects on water. By using the trained model, it is possible to classify objects on water without relying on the visual inspection of crew members. By using multiple camera devices installed at multiple locations on the ship, it is possible to effectively classify objects on water that exist in a wide area around the ship. Furthermore, by using multiple camera devices of different types, it is possible to effectively classify more objects on water than can be classified using a single camera device. [Effects of the Invention]

[0010] The present invention has excellent effects such as effectively identifying water objects present around a ship and facilitating safe navigation of the ship. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram showing a ship. [Figure 2] 1 is a block diagram showing an example of the configuration of an information processing system for identifying water objects around a ship; [Figure 3] FIG. 2 is a block diagram illustrating an example of an internal functional configuration of the information processing device. [Figure 4] This is a conceptual diagram showing the function of a trained model. [Figure 5] 10 is a flowchart illustrating an example of a process for outputting a discrimination result of an object on water. [Figure 6] FIG. 2 is a schematic diagram showing an example of a captured image. [Figure 7] 10A and 10B are schematic diagrams showing an example of output of a photographed image and a discrimination result of an object on water. [Figure 8] 10A and 10B are schematic diagrams showing examples of outputs of a radar image, a photographed image, and a discrimination result of an object on water. [Figure 9] 10A and 10B are schematic diagrams showing examples of a radar image in which a portion is designated and a captured image in which the corresponding portion is highlighted; [Figure 10] 10 is a flowchart illustrating an example of a process for re-learning a trained model. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present invention will now be described in detail with reference to the drawings showing embodiments thereof. FIG. 1 is a schematic diagram showing a ship. The ship 1 is equipped with multiple image capturing devices. In this embodiment, the multiple image capturing devices capture images of the area around the ship 1, and identify water objects contained in the captured images. Water objects are objects that exist on the water around the ship 1, such as other ships or buoys.

[0013] The multiple imaging devices include a forward camera 11, an infrared camera 12, a PTZ camera 13 capable of panning, tilting, and zooming, a starboard camera 14, and a port camera 15. The forward camera 11 uses visible light to image the area in front of the vessel 1. The infrared camera 12 uses infrared light to image the area in front of the vessel 1. Use of the infrared camera 12 makes it easier to image objects that are difficult to image using visible light, such as objects on the water at night. The PTZ camera 13 can image an area in any direction at any magnification. The forward camera 11, infrared camera 12, and PTZ camera 13 are installed at the front of the vessel 1.

[0014] Starboard camera 14 is installed on the starboard side of ship 1, and port camera 15 is installed on the port side of ship 1. Starboard camera 14 uses visible light to photograph the area facing the starboard side of ship 1. Port camera 15 uses visible light to photograph the area facing the port side of ship 1. The relative positions of the areas photographed by forward camera 11, infrared camera 12, starboard camera 14, and port camera 15 with respect to ship 1 are fixed. In this way, the multiple photographing devices include photographing devices installed at multiple positions within ship 1 and include different types of photographing devices.

[0015] By using the front camera 11, starboard camera 14, and port camera 15, it is possible to capture images of multiple areas around the vessel 1, thereby reducing the risk of overlooking the identification of waterborne objects based on the captured images. The angle of view of the front camera 11, starboard camera 14, and port camera 15 is preferably wide. For example, the orientation in a horizontal plane is expressed as an angle with the front of the vessel 1 defined as 0 degrees, and the front camera 11 captures images of areas around the vessel 1 from 300 degrees to 360 degrees and 0 degrees to 60 degrees, the starboard camera 14 captures images of areas from 30 degrees to 150 degrees, and the port camera 15 captures images of areas from 210 degrees to 330 degrees. The wide angle of view of the front camera 11, starboard camera 14, and port camera 15 allows a wide area to be captured with a single camera. This allows for efficient identification of waterborne objects.

[0016] The forward camera 11, the infrared camera 12, the starboard camera 14, or the port camera 15 may be configured to be able to change the position of the area to be photographed. The multiple photographing devices provided on the vessel 1 may include cameras other than the forward camera 11, the infrared camera 12, the PTZ camera 13, the starboard camera 14, and the port camera 15. The multiple photographing devices may not include any of the forward camera 11, the infrared camera 12, the PTZ camera 13, the starboard camera 14, and the port camera 15.

[0017] FIG. 2 is a block diagram showing an example of the configuration of an information processing system 200 for identifying water objects around a vessel 1. The information processing system 200 includes an information processing device 2. The information processing device 2 is a computer. A plurality of image capturing devices, including a forward camera 11, an infrared camera 12, a PTZ camera 13, a starboard camera 14, and a port camera 15, are connected to the information processing device 2. A radar 16 and an AIS (Automatic Identification System) 17 provided on the vessel 1 are also connected to the information processing device 2. The radar 16 scans the area around the vessel 1 with radio waves, identifies water objects based on the reflected radio waves, and generates a radar image including the identification results. The AIS 17 is a device for exchanging vessel information between vessels, and receives vessel information, including information identifying the other vessels, from other vessels via wireless communication.

[0018] Furthermore, the information processing system 200 includes a display device 31 and an operation device 32. The display device 31 displays images. The display device 31 is, for example, a liquid crystal display or an EL display (Electroluminescent Display). The operation device 32 receives input of information by receiving operations from users such as crew members of the vessel 1. The operation device 32 is, for example, a touch panel, a keyboard, or a pointing device. The display device 31 and the operation device 32 may be integrated.

[0019] FIG. 3 is a block diagram showing an example of the internal functional configuration of the information processing device 2. The information processing device 2 executes an information processing method for identifying water objects around the vessel 1. The information processing device 2 includes a calculation unit 21, a memory 22, a drive unit 23, a storage unit 24, and an interface unit 25. The calculation unit 21 is configured using, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a multi-core CPU. A display device 31 and an operation device 32 are connected to the calculation unit 21. The memory 22 stores temporary data generated in conjunction with calculations. The memory 22 is, for example, a RAM (Random Access Memory). The drive unit 23 reads information from a recording medium 20 such as an optical disc or a portable memory. The storage unit 24 is non-volatile, for example, a hard disk or a non-volatile semiconductor memory.

[0020] The calculation unit 21 causes the drive unit 23 to read the computer program 241 recorded on the recording medium 20, and stores the read computer program 241 in the storage unit 24. The calculation unit 21 executes processing required for the information processing device 2 in accordance with the computer program 241. The computer program 241 may be a computer program product. The computer program 241 may be downloaded from outside the information processing device 2. Alternatively, the computer program 241 may be pre-stored in the information processing device 2. In these cases, the information processing device 2 does not need to be equipped with the drive unit 23.

[0021] The interface unit 25 is connected to the front camera 11, the infrared camera 12, the PTZ camera 13, the starboard camera 14, and the port camera 15. The front camera 11, the infrared camera 12, the PTZ camera 13, the starboard camera 14, and the port camera 15 create photographed images of their respective areas around the vessel 1 and input the images to the information processing device 2. The interface unit 25 accepts the input photographed images. By accepting the photographed images with the interface unit 25, the information processing device 2 acquires the photographed images. The front camera 11, the infrared camera 12, the PTZ camera 13, the starboard camera 14, and the port camera 15 continuously create photographed images or periodically repeatedly create photographed images. The information processing device 2 continuously acquires photographed images or periodically repeatedly acquires photographed images.

[0022] Furthermore, the interface unit 25 is connected to the radar 16 and the AIS 17. The radar 16 generates a radar image and inputs it to the information processing device 2. The interface unit 25 accepts the input radar image, and the information processing device 2 acquires the radar image. The radar 16 repeatedly generates radar images, and the information processing device 2 repeatedly acquires the captured images. The AIS 17 receives ship information from other ships, and inputs AIS data based on the received ship information to the information processing device 2. The AIS data includes information indicating the type of the other ship. The interface unit 25 accepts the input AIS data, and the information processing device 2 acquires the AIS data. Each time the AIS 17 receives ship information, the AIS 17 inputs the AIS data to the information processing device 2, and the information processing device 2 acquires the AIS data.

[0023] The information processing device 2 may be configured with multiple computers, data may be stored in a distributed manner across the multiple computers, and processing may be executed in a distributed manner across the multiple computers. The information processing device 2 may also be realized by multiple virtual machines provided within a single computer.

[0024] The information processing device 2 includes a trained model 242 that has been trained. The trained model 242 is realized by the calculation unit 21 executing information processing in accordance with the computer program 241. For example, the storage unit 24 stores data recording parameters of the trained model 242 that has been trained in advance, and the calculation unit 21 uses the parameters to execute information processing in accordance with the computer program 241, thereby realizing the trained model 242. The trained model 242 may be configured by hardware. Alternatively, the trained model 242 may be provided outside the information processing device 2, and the information processing device 2 may execute processing using the external trained model 242.

[0025] FIG. 4 is a conceptual diagram illustrating the function of the trained model 242. Images captured by the forward camera 11, the infrared camera 12, the PTZ camera 13, the starboard camera 14, or the port camera 15 are input to the trained model 242. The trained model 242 has been trained in advance by machine learning so that, when a captured image is input, it outputs a discrimination result that identifies an object on water contained in the captured image. The discrimination result output by the trained model 242 includes the position of the object on water in the captured image, the type of the object on water, and the confidence level of the type of the object on water. For example, the trained model 242 is configured using YOLO (You Only Look Once). The trained model 242 may also be a model using a method other than YOLO, such as R-CNN (Region Based Convolutional Neural Networks) or segmentation.

[0026] The trained model 242 may include a plurality of trained models that correspond one-to-one to the forward camera 11, the infrared camera 12, the PTZ camera 13, the starboard camera 14, and the port camera 15. Each of the plurality of trained models has been individually trained in advance so as to output a discrimination result of an object on the water when an image captured by a corresponding imaging device is input.

[0027] The information processing device 2 performs a process of identifying water objects present around the vessel 1 and outputting the identification results of the water objects. FIG. 5 is a flowchart showing an example of a process of outputting the identification results of the water objects. Hereinafter, step is abbreviated as S. The calculation unit 21 of the information processing device 2 executes the process in accordance with the computer program 241.

[0028] The front camera 11, infrared camera 12, PTZ camera 13, starboard camera 14, and port camera 15 take photographs to create photographed images, and input the photographed images to the information processing device 2. The information processing device 2 acquires the photographed images by receiving the photographed images input from each photographing device through the interface unit 25 (S101). The information processing device 2 may acquire only the photographed images created by some of the photographing devices. The calculation unit 21 stores the photographed images in the memory unit 24.

[0029] Fig. 6 is a schematic diagram showing an example of a captured image. Fig. 6 shows an example of a captured image 41 created by the front camera 11. The captured image 41 includes a part of the hull 51 of the vessel 1. The captured image 41 also includes a plurality of waterborne objects 52. The processing of S101 corresponds to the image acquisition unit.

[0030] The information processing device 2 inputs a captured image to the trained model 242 (S102). In S102, the calculation unit 21 inputs the captured image to the trained model 242 and causes the trained model 242 to execute processing. If the trained model 242 includes multiple trained models that correspond one-to-one to multiple image capture devices, the calculation unit 21 inputs the captured image input from each image capture device to the trained model corresponding to the image capture device. In response to the input of the captured image, the trained model 242 outputs a discrimination result that discriminates an object on water included in the input captured image. The information processing device 2 acquires the discrimination result of the object on water output by the trained model 242 (S103). The calculation unit 21 stores the discrimination result of the object on water in the memory unit 24. The discrimination result includes the position of the object on water in the captured image, the type of the object on water, and the confidence level of the type of the object on water.

[0031] The information processing device 2 then determines whether the certainty factor of the type of water object included in the discrimination result falls within a predetermined range (S104). The predetermined range is a range in which the certainty factor is somewhat low. For example, the predetermined range is a range equal to or less than a predetermined threshold. For example, the second threshold is set to a value smaller than the threshold, and the predetermined range is a range equal to or greater than the second threshold and equal to or less than the threshold. The predetermined range is pre-stored in the storage unit 24 or included in the computer program 241. In S104, the calculation unit 21 compares the certainty factor with the predetermined range. If the discrimination result includes certainty factors for multiple types of water object, the calculation unit 21 determines whether the certainty factor for each water object falls within the predetermined range. If the same water object is included in multiple captured images, the calculation unit 21 may determine whether the maximum certainty factor among the certainty factors obtained based on the multiple captured images falls within the predetermined range, or may determine whether the average value or variance of the certainty factors falls within the predetermined range.

[0032] If the certainty factor is within the predetermined range (S104: YES), the information processing device 2 controls the PTZ camera 13 to take a close-up image of the water object whose type certainty factor is within the predetermined range (S105). In S105, the calculation unit 21 identifies the position of the water object whose type certainty factor is within the predetermined range based on the captured image, and transmits a control signal from the interface unit 25 to the PTZ camera 13 to control the PTZ camera 13 to take a close-up image of the water object whose type certainty factor is within the predetermined range. The PTZ camera 13 adjusts its orientation in accordance with the control signal and takes a close-up image of the water object whose type certainty factor is within the predetermined range.

[0033] The PTZ camera 13 takes photographs to create photographed images, which are then input to the information processing device 2. The information processing device 2 acquires the photographed images by receiving the photographed images input from the PTZ camera 13 at the interface unit 25 (S106). The calculation unit 21 stores the photographed images in the memory unit 24. The photographed images include enlarged versions of water-based objects whose type certainty falls within a predetermined range. The photographed image acquired in S101 corresponds to the first photographed image, and the photographing device that created the first photographed image corresponds to the first camera. The photographed image acquired in S106 corresponds to the second photographed image.

[0034] The information processing device 2 then inputs the captured image acquired in S106 to the trained model 242 (S107). If the trained model 242 includes multiple trained models that correspond one-to-one to multiple image capture devices, the calculation unit 21 inputs the captured image to the trained model for the PTZ camera 13. The trained model 242 outputs a discrimination result for the above-water object included in the input captured image. The information processing device 2 acquires the discrimination result for the above-water object output by the trained model 242 (S108). In S108, the calculation unit 21 stores the discrimination result for the above-water object in the storage unit 24. Because the above-water object is enlarged, the features of the above-water object become more prominent in the captured image. Therefore, a discrimination result with a higher degree of certainty can be obtained based on the enlarged captured image of the above-water object. In S108, a discrimination result that the captured image does not include the above-water object can also be obtained. The processes of S107 to S108 correspond to the discrimination result acquisition unit.

[0035] If the confidence level is not within the predetermined range in S104 (S104: NO), or after S108 is completed, the information processing device 2 outputs the photographed image and the discrimination result of the surface-of-water object (S109). In S109, the calculation unit 21 outputs the photographed image and the discrimination result of the surface-of-water object by displaying an image including the photographed image and the discrimination result of the surface-of-water object on the display device 31. In S109, the calculation unit 21 outputs the multiple photographed images and the discrimination result of the surface-of-water object included in each photographed image. The processing of S109 corresponds to the output unit.

[0036] Fig. 7 is a schematic diagram showing an example of captured images and output of the results of discrimination of objects on the water. Fig. 7 shows an example of an image displayed on the display device 31. The displayed images include an image 41 captured by the front camera 11, an image 42 captured by the infrared camera 12, an image 43 captured by the PTZ camera 13, an image 44 captured by the starboard camera 14, and an image 45 captured by the port camera 15.

[0037] Each captured image is accompanied by a label 46 indicating the type of camera used to create the image. By visually checking the label 46, the user can confirm which camera was used to create each captured image. Each captured image input to the information processing device 2 by each camera is associated with identification information that identifies the camera. The calculation unit 21 generates a label 46 indicating the type of camera used to create the captured image based on the identification information associated with the captured image, and displays an image including the label 46 attached to each captured image on the display device 31.

[0038] As a result of the determination of the water object, a mark indicating the position of the water object in the photographed image and information indicating the type of the water object are output. For example, as shown in FIG. 7, a bounding box 53 surrounding the water object 52 included in the photographed image is displayed as a mark indicating the position of the water object in the photographed image. Markers other than the bounding box 53, such as arrows, may also be output. By visually recognizing the bounding box 53, the user can know that a water object is present around the vessel 1.

[0039] For example, as shown in FIG. 7 , text information 54 indicating the type of water object 52 is displayed as information indicating the type of water object. For example, the type of water object may be determined as either a ship, a navigational aid, a lighthouse, an aquaculture facility, a buoy, a bridge girder, a bridge pier, or other non-watercraft object, or an unknown object. For example, if the certainty factor is equal to or less than a second threshold, the type of water object may be determined as an unknown object. For example, a water object determined to be a ship may be determined as either a merchant ship, a cruise ship, a crane ship, a tugboat, a gravel carrier, a fishing boat, a pleasure boat, or other watercraft, or an unknown watercraft. A navigational aid or a buoy may be further classified. Furthermore, for example, the trained model 242 may be trained to determine the type of water object 52 that is smaller than a predetermined size when the certainty factor is within a predetermined range. By visually checking the text information 54, the user can know the type of water object present around the ship 1. Information indicating the type of water object may be output in addition to the text information 54. For example, the different types of water-borne objects may be represented by changing the color of the bounding box 53 or the text information 54 depending on the type of water-borne object.

[0040] The image 41 taken by the front camera 11 and the image 42 taken by the infrared camera 12 capture almost the same area. However, because different light is used, an object on the water 52 included in one image may not be included in the other image. The image 41 taken by the front camera 11, the image 44 taken by the starboard camera 14, and the image 45 taken by the port camera 15 capture different areas. However, if there is an overlapping area between the areas captured by the two camera devices, the same object on the water 52 may be included in the two images.

[0041] Fig. 7 shows an example in which the same surface object 52, which is a fishing boat, is included in the image 41 captured by the front camera 11 and the image 44 captured by the starboard camera 14. Fig. 7 shows an example in which the same surface object 52, which is a buoy, is included in the image 41 captured by the front camera 11 and the image 45 captured by the port camera 15.

[0042] 7 also includes a photographed image 43 obtained by enlarging and capturing an underwater object 52 included in a photographed image 41 captured by the front camera 11 using the PTZ camera 13. The one underwater object 52 included in the photographed image 41 appears very small, and its type is determined to be unknown, with a certainty factor falling within a predetermined range. The processes of S105 to S108 result in a photographed image 43 obtained by enlarging and capturing the water object using the PTZ camera 13, and a water object discrimination result is obtained. In the photographed image 43 created by the PTZ camera 13, the one underwater object 52 included in the photographed image 41 is enlarged. The type of the water object 52, which was determined to be unknown in the discrimination result based on the photographed image 41, is now determined to be a cruise ship in the discrimination result based on the photographed image 43. When a discrimination result is obtained based on the photographed image 43 captured by the PTZ camera 13 that the type of the water object is an unknown object, text information 54 indicating that the water object is an unknown object is displayed. When the determination result indicates that the photographed image 43 does not include an object on the water, the calculation unit 21 displays information indicating that the photographed image 43 does not include an object on the water on the display device 31. In this case, the calculation unit 21 may display information indicating that the photographed image 43 does not include an object on the water, without displaying the photographed image 43.

[0043] By using multiple camera devices installed at multiple locations within the vessel 1, the camera devices capture different areas, making it possible to capture a wider area than can be captured by a single camera device. This makes it possible to effectively identify water objects that exist in a wide area around the vessel 1. Furthermore, by using multiple camera devices of different types, it is possible to clearly capture water objects that cannot be captured or are captured unclearly by one camera device using another type of camera device. This makes it possible to effectively identify more water objects than can be identified using a single camera device. For example, by using the PTZ camera 13 to zoom in and capture water objects that would be unclear when captured by other camera devices, accurate identification results can be obtained.

[0044] The information processing device 2 then outputs the radar image, the photographed image, and the discrimination result of the surface object (S110). The radar 16 scans the surroundings of the ship 1 with radio waves, discriminates the surface object based on the reflected radio waves, generates a radar image including the discrimination result, and inputs the radar image to the information processing device 2. The information processing device 2 acquires the photographed image by accepting the input radar image at the interface unit 25. The calculation unit 21 stores the acquired radar image in the memory unit 24. In S110, the calculation unit 21 displays an image including the radar image, the photographed image, and the discrimination result of the surface object on the display device 31.

[0045] FIG. 8 is a schematic diagram showing an example of output of a radar image, a photographed image, and a result of discrimination of an object on the water. FIG. 8 shows an example of an image displayed on the display device 31, including a radar image 6, a photographed image, and a result of discrimination of an object on the water. The displayed image includes the radar image 6, a photographed image 41 created by the forward camera 11, a photographed image 44 created by the starboard camera 14, and a photographed image 45 created by the port camera 15. The photographed image 41 is positioned in front of the radar image 6, the photographed image 44 is positioned to the right of the radar image 6, and the photographed image 45 is positioned to the left of the radar image 6. By displaying each photographed image and the radar image 6 in a comparative manner in this way, the correspondence between the contents of the radar image 6 and the photographed image can be easily understood. Other photographed images may also be displayed in comparison with the radar image 6.

[0046] The radar image 6 includes an image 61 of an object on the water that exists around the vessel 1. The radar image 6 also includes a result of discrimination of the object on the water determined by the radar 16. If the result of discrimination of the object on the water determined by the radar 16 matches the result of discrimination of the object on the water output by the trained model 242, both discrimination results are output in the same format. That is, as the discrimination result of the object on the water included in the radar image 6, a mark indicating the position of the object on the water in the radar image 6 and information indicating the type of the object on the water are output. For example, similar to the discrimination result of the object on the water superimposed on the captured image, a bounding box 53 surrounding the image 61 of the object on the water and text information 63 indicating the type of the object on the water in text are displayed. The content of the text information 63 indicating the type of the object on the water is the same as the content of the text information 54 indicating the type of the corresponding object on the water 52. The color of the bounding box 53 or the text information 54 for the image 61 of the surface object may be adjusted to be the same as the color of the bounding box 53 or the text information 54 for the corresponding surface object 52. Note that the result of the surface object discrimination based on the AIS data obtained from the AIS 17 may be used as the result of discrimination of the surface object to be displayed superimposed on the radar image 6.

[0047] By outputting the results of water object discrimination using radar 16 and the results of water object discrimination based on the captured images, the results of water object discrimination obtained by the two different methods are compared. By outputting the results of water object discrimination using radar 16 and the results of water object discrimination based on the captured images in the same format, it is easy to confirm that the results of water object discrimination obtained by the two different methods match. It is also possible to detect errors in the results of water object discrimination. For example, if the radar image 6 contains an image 61 of a water object but the corresponding water object 52 is not included in the captured image, it can be determined that the image 61 of the water object is a false image. In this way, water objects are more reliably identified. In S110, the captured image 42 captured by the infrared camera 12 and the captured image 43 captured by the PTZ camera 13 may also be displayed.

[0048] The information processing device 2 then accepts designation of a portion within the radar image 6 (S111). In S111, the calculation unit 21 accepts the designation by the user operating the operation device 32. For example, the calculation unit 21 displays a designation graphic, such as a cursor or a bounding box, for designating a portion within the radar image 6 on the display device 31, superimposed on the radar image 6. The calculation unit 21 changes the position of the designation graphic in accordance with the operation from the user accepted by the operation device 32, and accepts the designation of a portion within the radar image 6.

[0049] The information processing device 2 then outputs (S112) the results of identifying the water object included in the corresponding portion of the photographed image that corresponds to the specified portion of the radar image 6. In S112, an emphasis figure such as a bounding box for emphasizing the corresponding portion in the photographed image is displayed on the display device 31, superimposed on the photographed image.

[0050] FIG. 9 is a schematic diagram showing an example of a radar image 6 with a designated portion and a captured image with the corresponding portion highlighted. A designated image 64 for designating a portion including an image 61 of an object on the water is displayed overlaid on the radar image 6. In the example shown in FIG. 9, the designated image 64 is a figure that surrounds the designated portion with a double line. Also, captured images 41 and 43 including the corresponding portion are displayed, and an emphasis figure 55 for highlighting the corresponding portion is displayed overlaid on the captured images 41 and 43. In the example shown in FIG. 9, the emphasis figure 55 is a figure that surrounds the corresponding portion with a double line. Text information 54 is displayed as the result of identifying an object on the water 52 included in the corresponding portion.

[0051] A portion of the radar image 6 is output in comparison with a corresponding portion in the captured image, allowing the user to confirm how each portion in the radar image 6 looks visually. For example, the user can confirm in the captured image an object on water 52 corresponding to an image 61 included in the radar image 6. The user can also easily compare the result of the discrimination of an object on water using the radar 16 with the result of the discrimination of an object on water based on the captured image. In S112, a captured image that does not include a portion corresponding to the specified portion in the radar image 6 may also be displayed. After S112 is completed, the information processing device 2 ends the process of outputting the discrimination result of an object on water.

[0052] The process of S109, the process of S110, and the processes of S111 to S112 may be executed in an order different from that shown in the example of Fig. 5, and may be executed in any order according to the operation from the user received by the operation device 32, or may be executed alternately and repeatedly. The execution of any one of the process of S109, the process of S110, and the processes of S111 to S112 may be omitted. The processes of S101 to S109 are executed as needed, and the discrimination results of the surface object are used for the navigation of the vessel 1.

[0053] The information processing device 2 can retrain the trained model 242 using the discrimination results of the above-water objects. FIG. 10 is a flowchart showing an example of the process of retraining the trained model 242. The information processing device 2 generates training data that associates a captured image created by one of the image capture devices with the discrimination results of the above-water objects (S21). In S21, the calculation unit 21 generates training data including a data set that associates a captured image with the discrimination results of the above-water objects based on the captured image created by a different image capture device from the image capture device that captured the captured image. For example, the calculation unit 21 associates the captured image 41 created by the front camera 11 with the discrimination results of the above-water objects based on the captured image 43 created by the PTZ camera 13.

[0054] In S21, the calculation unit 21 generates training data including a data set associating the captured image with the discrimination result of the object on the water using the radar 16. The calculation unit 21 also generates training data including a data set associating the captured image with the discrimination result of the object on the water using the AIS 17. In S21, the calculation unit 21 generates training data including a plurality of data sets associating the captured image with the discrimination result of the object on the water, and stores the training data in the memory unit 24.

[0055] The information processing device 2 then uses the training data to retrain the trained model 242 (S22). In S22, the calculation unit 21 inputs the photographed image included in the training data to the trained model 242 and retrains the trained model 242. The trained model 242 outputs a discrimination result for the water object in response to the input of the photographed image. The calculation unit 21 acquires the discrimination result for the water object output by the trained model 242 and adjusts the calculation parameters of the trained model 242 so as to reduce the error between the discrimination result for the water object associated in the training data with the photographed image input to the trained model 242 and the discrimination result for the water object output by the trained model 242. In other words, the parameters are adjusted so that a discrimination result for the water object that is substantially the same as the discrimination result for the water object associated with the photographed image is output.

[0056] The calculation unit 21 repeats processing using multiple data sets included in the training data to adjust the parameters of the trained model 242, thereby re-learning the trained model 242. The calculation unit 21 stores data recording the final adjusted parameters in the storage unit 24. In this way, the re-learned trained model 242 is generated. After S22 is completed, the information processing device 2 ends the process of re-learning the trained model 242.

[0057] In the processes of S21 to S22, the trained model 242 is retrained using the photographed image and the discrimination result of the water object based on the photographed image taken by a different photographing device as training data, thereby adjusting the trained model 242 so that it can output a more accurate discrimination result. Also, by using the photographed image and the discrimination result of the water object using the radar 16 or the AIS 17 as training data, it is possible to adjust the trained model 242 using a discrimination result different from the discrimination result based on the photographed image. Note that the processes of S21 to S22 may be executed by a device other than the information processing device 2.

[0058] As described above in detail, in this embodiment, images captured by multiple camera devices installed on the ship 1 are input to the trained model 242, and the trained model 242 outputs the results of identifying objects on the water. By using the trained model 242, it is possible to identify objects on the water without relying on the visual inspection of the crew. By using multiple camera devices installed at multiple positions within the ship 1, it is possible to capture images of a wide area, making it possible to effectively identify objects on the water that exist in a wide area around the ship 1. Furthermore, by using multiple camera devices of different types, it is possible to effectively identify many objects on the water. By effectively identifying objects on the water, it is possible to facilitate safe navigation of the ship 1.

[0059] In this embodiment, the radar 16 and the AIS 17 are connected to the information processing device 2, but the radar 16 and the AIS 17 may be independent of the information processing device 2. For example, the radar 16 and the AIS 17 may be connected to the display device 31 without going through the information processing device 2, and the radar image 6 and the result of discrimination of an object on the water using the radar 16 or the AIS 17 may be displayed independently of the processing by the information processing device 2.

[0060] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the claims are also included in the technical scope of the present invention. [Explanation of symbols]

[0061] 1 ship 11. Front camera 12 Infrared camera 13 PTZ cameras 14 Starboard Camera 15 Port side camera 2. Information processing equipment 20 Recording Media 242 trained models 31 Display device 41, 42, 43, 44, 45 Photographed images 52 Water objects 53 Bounding Box 54 Textual Information 6. Radar imagery 61 Image of an object on the water

Claims

1. A plurality of different types of photographing devices installed on a ship acquire photographed images of the surroundings of the ship, Inputting photographed images acquired from each of the plurality of photographing devices into a trained model that outputs a discrimination result of an underwater object included in the photographed image when the photographed image is input, Obtain the discrimination result of the water object output by the trained model, outputting a result of distinguishing an object on the water included in the photographed image captured by each of the plurality of photographing devices; the plurality of photographing devices include a first camera and a PTZ camera capable of panning, tilting, and zooming; The discrimination result of the water object output by the trained model includes a certainty factor of the type of the water object, A first photographed image acquired from the first camera is input into the trained model, and a discrimination result of the water object output by the trained model is obtained; If the certainty factor included in the obtained discrimination result is within a predetermined range, a second photographed image obtained by enlarging and photographing the water object included in the first photographed image is obtained from the PTZ camera; The second captured image is input to the trained model, and a discrimination result of the water object output by the trained model is obtained; Output the obtained discrimination result. An information processing method comprising:

2. The discrimination result includes the type of the water object.

2. The information processing method according to claim 1,

3. the plurality of photographing devices include a forward camera that photographs an area in front of the vessel, an infrared camera that photographs using infrared rays, the PTZ camera, a starboard camera that photographs an area facing the starboard side of the vessel, and a port camera that photographs an area facing the port side of the vessel; The discrimination result is output together with information indicating the type of the imaging device.

3. The information processing method according to claim 1 or 2.

4. The discrimination result is output by outputting the photographed image, a mark indicating the position of the water object contained in the photographed image in the photographed image, and information indicating the type of the water object.

4. The information processing method according to claim 1, wherein:

5. A radar image including the discrimination result of the object on the water using the radar installed on the ship is compared with the discrimination result of the object on the water output by the trained model to which a photographed image of the area included in the range scanned by the radar is input, and the comparison result is output.

5. The information processing method according to claim 1, wherein:

6. Accepting designation of a portion within the radar image; outputting a discrimination result of an object on the water included in a corresponding part in the photographed image corresponding to the part included in the discrimination result of an object on the water output by the trained model; 6. The information processing method according to claim 5,

7. When the result of discrimination of an object on the water using the radar matches the result of discrimination of an object on the water output by the trained model, both discrimination results are output in the same manner.

7. The information processing method according to claim 5 or 6.

8. The trained model is retrained using training data including photographed images and discrimination results of water-borne objects based on photographed images taken by a photographing device of a different type from the photographing device that took the photographed images.

8. The information processing method according to claim 1, wherein:

9. The trained model is retrained using training data including images captured by the imaging device and the results of discrimination of water objects using radar or an AIS (Automatic Identification System) installed on the ship.

9. The information processing method according to claim 1, wherein:

10. an image acquisition unit that acquires images of the surroundings of the ship captured by a plurality of different types of photographing devices; a discrimination result acquisition unit that inputs photographed images acquired from each of the plurality of photographing devices into a trained model that outputs a discrimination result of an object on water included in the photographed image when the photographed image is input, and acquires the discrimination result of the object on water output by the trained model; an output unit that outputs the determination results corresponding to each of the plurality of imaging devices, the plurality of photographing devices include a first camera and a PTZ camera capable of panning, tilting, and zooming; The discrimination result of the water object output by the trained model includes a certainty factor of the type of the water object, The discrimination result acquisition unit inputs the first photographed image acquired from the first camera into the trained model and acquires a discrimination result of the water object output by the trained model; When the certainty factor included in the obtained discrimination result is within a predetermined range, the image acquisition unit acquires a second photographed image from the PTZ camera, the second photographed image being an enlarged image of the water object included in the first photographed image, The discrimination result acquisition unit inputs the second captured image into the trained model and acquires a discrimination result of the water object output by the trained model; The output unit outputs the obtained determination result.

1. An information processing device comprising:

11. A plurality of different types of photographic devices capture images of the surroundings of the ship, Inputting photographed images acquired from each of the plurality of photographing devices into a trained model that outputs a discrimination result of an underwater object included in the photographed image when the photographed image is input, Obtain the discrimination result of the water object output by the trained model, Outputting the discrimination results corresponding to each of the plurality of imaging devices Have the computer execute the process, the plurality of photographing devices include a first camera and a PTZ camera capable of panning, tilting, and zooming; The discrimination result of the water object output by the trained model includes a certainty factor of the type of the water object, Furthermore, the first captured image acquired from the first camera is input into the trained model, and the discrimination result of the water object output by the trained model is obtained; If the certainty factor included in the obtained discrimination result is within a predetermined range, a second photographed image obtained by enlarging and photographing the water object included in the first photographed image is obtained from the PTZ camera; The second captured image is input to the trained model, and a discrimination result of the water object output by the trained model is obtained; Output the obtained discrimination result. A computer program that causes a computer to execute a process.

Images