Program, information processing method, information processing device, and model generation method

Abstract

[Problem] To provide a program and the like capable of suitably estimating wind information. [Solution] This program causes a computer to execute processing for acquiring an image including a water surface, and estimating wind information by inputting the acquired image to a learning model that outputs wind information pertaining to wind when receiving the image. Preferably, the image is obtained by imaging a water surface around a ship and acquired in association with identification information about the ship. When the wind information is estimated, a combination of the estimated wind information and the identification information is stored at any time, and the wind information is output to a user terminal 2 corresponding to the identification information.

Description

Program, information processing method, information processing device and model generation method

[0001] The present invention relates to a program, an information processing method, an information processing device, and a model generation method.

[0002] Wind information (e.g., wind speed class) is important for safe ship operation. For example, Patent Document 1 discloses a ship information providing device that classifies sea conditions (wind speed) during ship operation into multiple sea condition ranks and displays operating information related to ship operation in a distinguishable manner for each sea condition rank.

[0003] Patent No. 7067997

[0004] In one aspect, an object is to provide a program or the like that can suitably estimate wind information.

[0005] In one aspect, the program causes a computer to acquire an image including a water surface and input the acquired image into a learning model that outputs wind information related to wind when the image is input, thereby estimating the wind information.

[0006] This allows wind information to be suitably estimated by using the learning model.

[0007] In one aspect, the image is an image of the water surface surrounding the ship, and the program acquires the image in association with the identification information of the ship, and when the program estimates the wind information, it stores the combination of the estimated wind information and the identification information at any time and outputs the wind information to a user terminal corresponding to the identification information.

[0008] This allows the wind information estimation results to be stored in association with the vessel, and the estimation results to be presented to the user.

[0009] In one aspect, the program outputs a warning to the user terminal when the wind information is equal to or greater than a predetermined threshold.

[0010] This allows the user to be warned in an appropriate manner.

[0011] In one aspect, the wind information includes wind force scale, wind speed, wave height, or wind direction.

[0012] This makes it possible to estimate wind force scale, wind speed, wave height, or wind direction.

[0013] In one aspect, the learning model is generated based on training data generated by assigning the wind information assigned to a video consisting of multiple images captured in chronological order to each of the images that make up the video.

[0014] This allows wind information to be appropriately assigned to images and a learning model to be generated.

[0015] In one aspect, the program acquires wind speed data, wave height data, or wind direction data at the time the image was captured, and estimates the wind information by inputting the image and the wind speed data, wave height data, or wind direction data into the learning model.

[0016] This allows wind information to be more appropriately estimated by adding wind speed data, wave height data, or wind direction data.

[0017] In one aspect, the program acquires multiple images taken at a predetermined time interval, estimates the wind information at multiple points in time by inputting each of the acquired images into the learning model, and determines the wind information to be used as the estimated result based on the wind information at the multiple points in time.

[0018] This makes it possible to reduce erroneous estimation of wind information by synthesizing wind information at multiple points in time.

[0019] In one aspect, the program calculates a statistical value of the wind information based on the wind information at a plurality of points in time, and determines the calculated statistical value as the estimation result.

[0020] This makes it possible to preferably reduce erroneous estimation of wind information by calculating statistical values ​​of wind information.

[0021] In one aspect, the program determines the wind information to be used as the estimated result by inputting the estimated wind information at multiple points in time into a second learning model that outputs wind information to be used as the estimated result when the wind information at multiple points in time is input.

[0022] As a result, by using the second learning model, it is possible to effectively reduce erroneous estimation of wind information.

[0023] In one aspect, the image includes a sky region.

[0024] This allows wind information to be estimated by also referring to cloud movement information, etc.

[0025] In one aspect, the program generates a mask image by masking areas other than the water surface from the image, and estimates the wind information by inputting the mask image into the learning model.

[0026] This makes it possible to improve the accuracy of estimating wind information.

[0027] In one aspect, the program acquires wind speed data, wave height data, or wind direction data at the time the image was captured, and determines whether the estimated wind information is correct or not based on the wind speed data, wave height data, or wind direction data.

[0028] This makes it possible to determine whether the wind information estimation result is correct or not.

[0029] In one aspect, the program receives input of whether the wind information estimation result is correct or incorrect, and updates the learning model based on the image and the correct or incorrect estimation result.

[0030] This allows for improved accuracy in estimating wind information using the learning model.

[0031] In one aspect, the image is an image of the water surface surrounding the ship, and the program acquires ship information about the ship and determines the navigation risk based on the wind information estimated by the learning model and the ship information.

[0032] This allows the navigation status of the ship to be grasped.

[0033] In one aspect, the information processing method involves a computer acquiring an image including a water surface, inputting the acquired image into a learning model that outputs wind information regarding wind when the image is input, and estimating the wind information.

[0034] This allows wind information to be estimated appropriately.

[0035] In one aspect, the information processing device is an information processing device that includes a control unit, and the control unit acquires an image including a water surface and estimates the wind information by inputting the acquired image into a learning model that outputs wind information regarding wind when the image is input.

[0036] This allows wind information to be estimated appropriately.

[0037] In one aspect, the model generation method involves a computer acquiring training data in which wind information relating to wind is associated with an image including a water surface, and generating a learning model that outputs the wind information when the image is input based on the training data.

[0038] This makes it possible to construct a learning model that can appropriately estimate wind information.

[0039] FIG. 1 is an explanatory diagram showing an example of the configuration of a weather estimation system. FIG. 2 is a block diagram showing an example of the configuration of a server. FIG. 3 is an explanatory diagram showing an example of the record layout of a ship DB and an image DB. FIG. 4 is an explanatory diagram showing an overview of a learning model. FIG. 5 is an explanatory diagram related to the annotation processing of wind information. FIG. 6 is an explanatory diagram related to the estimation processing of wind information. FIG. 7 is an explanatory diagram showing an example of the display of a water surface image. FIG. 8 is a flowchart showing the procedure for the generation processing of a learning model. FIG. 9 is a flowchart showing the procedure for the estimation processing of wind information. FIG. 10 is a diagram showing an overview of embodiment 2. FIG. 11 is a flowchart showing the procedure for the estimation processing of wind information according to embodiment 2. FIG. 12 is an explanatory diagram showing an overview of embodiment 3. FIG. 13 is a flowchart showing the procedure for the estimation processing of wind information according to embodiment 3. FIG. 14 is a table showing the correspondence between wind volume classes and wind speeds and wave heights.

[0040] The present invention will be described in detail below with reference to the drawings illustrating embodiments. (Embodiment 1) Fig. 1 is an explanatory diagram showing an example of the configuration of a weather estimation system. In this embodiment, a weather estimation system will be described that uses a learning model 50 (see Fig. 4) constructed by machine learning to estimate wind information (e.g., wind force scale) from an image including the water surface around a ship (hereinafter referred to as a "water surface image"). The weather estimation system includes an information processing device 1, a user terminal 2, and a ship system 3. The information processing device 1 and the ship system 3 are communicatively connected via a network N such as satellite communication or LTE (Long Term Evolution).

[0041] The information processing device 1 is an information processing device capable of various information processing and transmitting and receiving information, such as a server computer or a personal computer. In this embodiment, the information processing device 1 is assumed to be a server computer, and for simplicity, will be referred to as the server 1 below. As will be described later, the server 1 generates a learning model 50 that outputs wind information when a water surface image is input by learning training data in which correct wind information is associated with a water surface image. When the server 1 acquires a water surface image from the ship system 3, it inputs the water surface image into the learning model 50 to estimate wind information.

[0042] The ship system 3 is a system that collects various navigation data including water surface images, and includes a control device 31, a camera 32, and various devices 33, 34, 35, etc. The control device 31 is a device that collects navigation data from the camera 32 and the devices 33, 34, 35, etc. and transmits the navigation data to the server 1 via the network N. The camera 32 is an imaging device installed on the ship, and one or more cameras are installed on the ship. The devices 33, 34, 35 are devices that can acquire navigation data other than water surface images (e.g., ship position information, course, wind speed, wave height, navigation speed, etc.), such as a VDR (Voyage Data Recorder), anemometer, wave height gauge, etc. The server 1 acquires this navigation data from the ship system 3 in association with the ship's identification information and stores it in a database as needed.

[0043] In this embodiment, the camera 32 is described as being installed on a ship, but the camera 32 may be installed on land (on the coast), for example. In other words, the water surface image is not limited to an image captured from a ship, and may be an image of an area including the water surface.

[0044] The user terminal 2 is a terminal device used by a user of the system, such as a personal computer, a smartphone, or a tablet terminal. The user of the system is, for example, a ship manager (management company), but may also be a crew member of the ship. As will be described later, upon receiving a request from the user terminal 2, the server 1 outputs navigation data including the estimated wind information to the user terminal 2 (see FIG. 7 ).

[0045] 2 is a block diagram showing an example configuration of the server 1. The server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary memory unit 14. The control unit 11 has one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), or a graphics processing unit (GPU), and performs various information processing, control processing, and the like by reading and executing programs stored in the auxiliary memory unit 14. The main memory unit 12 is a temporary storage area such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), and temporarily stores data required for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing and transmits and receives information to and from the outside.

[0046] The auxiliary memory unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores programs (program products) and other data necessary for the control unit 11 to execute processing. The auxiliary memory unit 14 also stores a learning model 50, a ship DB 141, and an image DB 142. The learning model 50 is a machine learning model that has learned predetermined training data, and is a model that outputs wind information when a water surface image is input. The ship DB 141 is a database that stores information about each ship from which navigation data is collected. The image DB 142 is a database that stores water surface images acquired from each ship.

[0047] The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may be a multi-computer consisting of multiple computers, or may be a virtual machine virtually constructed by software.

[0048] Furthermore, in this embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit that accepts operation input, a display unit that displays images, etc. Furthermore, the server 1 may be provided with a reading unit that reads a portable storage medium 1a such as a CD (Compact Disk)-ROM or a DVD (Digital Versatile Disc)-ROM, and may read and execute a program from the portable storage medium 1a.

[0049] 3 is an explanatory diagram showing an example of the record layout of the ship DB 141 and the image DB 142. The ship DB 141 includes a ship ID column, a ship name column, a ship data column, and a navigation data column. The ship ID column stores a ship ID for identifying each ship. The ship name column, the ship data column, and the navigation data column each store, in association with the ship ID, the ship name, basic ship data (e.g., ship type, size, weight, etc.), and navigation data (e.g., data acquisition date and time, latitude, longitude, course, wind speed, wind direction, wave height, navigation speed, etc.) at the time of data acquisition.

[0050] The image DB 142 includes a camera ID column, a list of installed vessels, a camera name column, and an image data column. The camera ID column stores a camera ID for identifying each camera 32. The list of installed vessels, the camera name column, and the image data column each store the vessel ID of the vessel on which the camera 32 is installed, the camera name, and water surface image data, in association with the camera ID. The image data column stores the water surface image data and the wind force scale estimated from the water surface image, in association with, for example, the date and time the water surface image was acquired.

[0051] Fig. 4 is an explanatory diagram showing an overview of the learning model 50. Fig. 4 illustrates how wind information (wind force scale) is output from the learning model 50 when a water surface image is input. An overview of this embodiment will be described below.

[0052] The learning model 50 is a machine learning model that has learned predetermined training data and outputs wind information when a water surface image is input. Specifically, the learning model 50 is a convolutional neural network (CNN) constructed by deep learning.

[0053] The learning model 50 may be a neural network other than a CNN. The learning model 50 may also be a machine learning model other than a neural network, such as a decision tree or a support vector machine (SVM).

[0054] The water surface image is an image (still image) captured by the camera 32 installed on the ship as described above, and is an image that includes at least the water surface in its imaging range. Specifically, the water surface image includes not only the water surface but also the sky area in its imaging range. By including the sky area in its imaging range, wind information can be estimated taking into account not only the state of the water surface but also the state of clouds, such as information on cloud movement.

[0055] The learning model 50 may input not only the water surface image but also wind speed data, wave height data, or wind direction data acquired when capturing the water surface image. By adding these data to the input, wind information can be more appropriately estimated.

[0056] In addition, although the water surface image is described as a still image in this embodiment, the water surface image may be a video consisting of multiple still images. In this case, for example, the learning model 50 may be a 3D-CNN or the like, which can use the video as input.

[0057] The server 1 uses training data in which correct wind information is associated with a group of training water surface images to generate a learning model 50. Specifically, as described below, the server 1 annotates a video consisting of multiple water surface images captured in chronological order with correct wind information, and generates the learning model 50 by learning each water surface image that makes up the video.

[0058] In the following description, for the sake of convenience, in order to distinguish between a moving image and each water surface image (still image) that constitutes the moving image, each water surface image that constitutes the moving image will also be referred to as a "frame image."

[0059] Fig. 5 is an explanatory diagram of the annotation process for wind information. Fig. 5 illustrates how wind information is annotated for a video of a predetermined duration (e.g., 10 minutes), and how the wind information is labeled for each frame image constituting the video, thereby learning each frame image.

[0060] The camera 32 captures one frame image at a predetermined time interval (for example, one frame image per minute). The server 1 acquires a video made up of a plurality of frame images captured in this way in chronological order as training images.

[0061] The server 1 extracts a group of frame images spanning a predetermined time period (e.g., 10 minutes) from the video. Because the wind force scale does not change over a short period of time, the server 1 accepts an operation input to collectively assign correct wind information to the extracted group of frame images spanning the predetermined time period. In this way, the server 1 assigns (labels) correct wind information to each frame image that makes up the video.

[0062] The server 1 uses a group of frame images associated with wind information as described above as training data to generate a learning model 50 that outputs wind information when a water surface image (frame image) is input. The server 1 estimates wind information by inputting training water surface images into the learning model 50, compares the estimated wind information with correct wind information, and adjusts parameters such as the weights between neurons so that the two are similar. The server 1 sequentially supplies a group of training water surface images to the learning model 50 to perform learning, and ultimately generates a learning model 50 with optimized parameters.

[0063] The server 1 estimates wind information from water surface images using the learning model 50 generated as described above. Specifically, as described below, the server 1 estimates wind information at each of multiple time points by inputting multiple water surface images (frame images) captured at predetermined time intervals into the learning model 50, and calculates a statistical value (e.g., the most frequent value) of the estimation results at each time point to determine the wind information to be used as the final estimation result.

[0064] Fig. 6 is an explanatory diagram of the wind information estimation process, which illustrates how wind information is estimated from water surface images (frame images) at multiple points in time, and the final wind information estimation result is determined by integrating the estimation results from each point in time.

[0065] The server 1 constantly acquires navigation data (ship's position information, course, wind speed, etc.) including water surface images from the ship system 3. For example, the server 1 may acquire a water surface image (frame image) one by one each time it is captured, or may acquire a predetermined period of water surface images all at once as a video. The server 1 stores navigation data other than water surface images in the ship DB 141 in association with the ship's identification information (ship ID).

[0066] The server 1 estimates wind information at the time each water surface image (frame image) was captured by inputting multiple water surface images captured at a predetermined time interval (e.g., every minute) into the learning model 50. The server 1 then calculates statistical values ​​for the wind information at each time point within a predetermined period of time (e.g., 10 minutes). The statistical value is, for example, the most frequent value, but it may also be an average value, etc. The server 1 determines the calculated statistical value as the estimated result of the wind information at each time point. Because wind information does not change over a short period of time, erroneous estimation can be reduced by performing post-processing to integrate the estimated results of wind information at multiple times.

[0067] The server 1 stores the water surface images acquired from the vessel system 3 and the estimated results of wind information predicted from the water surface images in the image DB 142, in association with the identification information of the vessel and the camera 32. The server 1 performs this processing as needed, and stores the water surface images sent from the vessel system 3 and the estimated results of wind information predicted from the water surface images in the image DB 142 as needed.

[0068] 7 is an explanatory diagram showing an example of a display of a water surface image. In response to access from the user terminal 2, the server 1 outputs the navigation data of the ship stored in each database to the user terminal 2.

[0069] For example, as shown in Fig. 7, the server 1 outputs a video of a water surface image to the user terminal 2. The server 1 displays on the video navigation data, such as wind information corresponding to the currently displayed frame image, and the ship's latitude, longitude, and course, allowing the user to view the navigation data including the wind information.

[0070] In addition, the server 1 may output a warning to the user terminal 2 when the wind information predicted using the learning model 50 exceeds a predetermined threshold. This makes it possible to more appropriately present the navigation status of the ship.

[0071] 8 is a flowchart showing the procedure for generating the learning model 50. The process for generating the learning model 50 by machine learning will be described with reference to FIG. 8. The control unit 11 of the server 1 acquires training water surface images (step S11). Specifically, the control unit 11 acquires a video consisting of a plurality of frame images captured in chronological order.

[0072] The control unit 11 receives an operation input for assigning correct wind information to a video of a predetermined duration (step S12). As a result, the control unit 11 assigns wind information to each frame image constituting the video. The control unit 11 uses the group of frame images with wind information assigned, i.e., the group of water surface images, as training data.

[0073] Based on the training data, the control unit 11 generates a learning model 50 that outputs wind information when a water surface image is input (step S13). For example, the control unit 11 generates a CNN as the learning model 50. The control unit 11 estimates wind information by inputting training water surface images into the learning model 50, and optimizes parameters such as the weights between neurons so that the estimated wind information approximates the correct wind information. In this way, the control unit 11 generates the learning model 50. The control unit 11 ends the series of processes.

[0074] FIG. 9 is a flowchart showing the procedure for estimating wind information. The processing for estimating wind information using the learning model 50 will be described with reference to FIG. 9 . The control unit 11 of the server 1 acquires navigation data including water surface images from the ship system 3 (step S31). For example, the control unit 11 may acquire water surface images (frame images) one by one, or may acquire multiple water surface images for a predetermined period of time all at once. The control unit 11 acquires various navigation data including water surface images (e.g., ship position information, course, wind speed, etc.) in association with identification information (ship ID and camera ID) of the ship and the camera 32. For example, the control unit 11 stores navigation data other than water surface images in the ship DB 141 in association with the ship identification information.

[0075] The control unit 11 estimates wind information related to wind by inputting the acquired water surface images into the learning model 50 (step S32). For example, the wind information is a wind force scale. The control unit 11 estimates wind information at multiple points in time by inputting multiple water surface images (frame images) captured at predetermined time intervals into the learning model 50.

[0076] The control unit 11 calculates a statistical value of the wind information (e.g., a mode value) based on the estimated wind information at the multiple time points (step S33). The control unit 11 determines the calculated statistical value as the estimation result of the wind information at each of the multiple time points.

[0077] The control unit 11 stores the water surface image and the estimated wind information in the image DB 142 in association with the identification information of the ship and the camera 32 (step S34), and ends the series of processes.

[0078] Although the above description has been given assuming that wind force scale is estimated as wind information, this embodiment is not limited to this, and wind speed, wave height, or wind direction may be estimated as wind information. For example, the learning model 50 may estimate (output) wind speed, wave height, or wind direction in addition to wind force scale. This allows the navigation state of the ship to be more appropriately understood.

[0079] As described above, according to the first embodiment, wind information can be suitably estimated.

[0080] (Embodiment 2) In this embodiment, a form will be described in which an area other than the water surface (for example, a sky area) included in a water surface image is masked. Note that the same reference numerals will be used to denote the same contents as in Embodiment 1, and the description thereof will be omitted.

[0081] Fig. 10 is an explanatory diagram showing an overview of embodiment 2. Fig. 10 illustrates how, before inputting a water surface image into the learning model 50, a mask image is generated by masking areas other than the water surface (e.g., sky area) from the water surface image, and the mask image is input into the learning model 50 to estimate wind information (wind force scale).

[0082] In this embodiment, the server 1 does not input the water surface image acquired from the vessel system 3 directly to the learning model 50, but inputs a mask image in which areas other than the water surface are masked from the water surface image to the learning model 50. Specifically, the server 1 recognizes a sky area from the water surface image and generates a mask image in which the sky area is masked. For example, the server 1 may recognize the sky area using rule-based pattern matching, or may recognize the sky area using a machine learning model that identifies image areas on a pixel-by-pixel basis, such as semantic segmentation. The server 1 estimates wind information by inputting the generated mask image to the learning model 50. Masking areas other than the water surface from the water surface image can improve the accuracy of estimating wind information.

[0083] The area to be masked from the water surface image is not limited to the sky area. For example, if a ship's hull is reflected in the image, the ship's hull may be masked. In this way, the server 1 is only required to be able to mask areas other than the water surface, and the area to be masked is not limited to the sky area.

[0084] 11 is a flowchart showing the procedure for estimating wind information according to the second embodiment. After acquiring navigation data including a water surface image (step S31), the server 1 executes the following process. The control unit 11 of the server 1 generates a mask image from the acquired water surface image by masking areas other than the water surface (step S201). Specifically, the control unit 11 recognizes areas such as the sky and the hull from the water surface image, and generates a mask image by masking those areas. The control unit 11 estimates wind information by inputting the generated mask image into the learning model 50 (step S202). The control unit 11 then proceeds to step S33.

[0085] As described above, according to the second embodiment, the accuracy of estimating wind information can be improved.

[0086] Third Embodiment In this embodiment, a description will be given of an embodiment in which the degree of navigation risk is determined by combining wind information estimated by the learning model 50 with ship information.

[0087] 12 is an explanatory diagram showing an outline of the third embodiment. The outline of this embodiment will be described with reference to FIG.

[0088] As explained in the first embodiment, the server 1 estimates wind information by inputting the water surface image into the learning model 50. In the present embodiment, the server 1 further determines the navigation risk level based on the estimated wind information and ship information about the ship from which the water surface image was obtained.

[0089] The vessel information includes, for example, the type, size, weight, etc. In this embodiment, the type of vessel is used as the vessel information. When wind information is estimated, the server 1 acquires the vessel information from the vessel DB 141.

[0090] 12 illustrates the record layout of a table referenced in calculating the navigation risk. The table includes a wind force class column and a navigation risk column. The wind force class column stores wind force classes. The navigation risk column stores navigation risks in association with wind force classes and ship types.

[0091] The server 1 refers to the table and calculates the navigation risk. For example, if the estimated wind force scale is "3" and the type of ship is a "container ship," the navigation risk will be "1." In this way, the server 1 determines the navigation risk based on wind information (wind force scale) and ship information.

[0092] The server 1 may output the calculated navigation risk to the user terminal 2. For example, if the navigation risk is equal to or greater than a predetermined value, the server 1 determines that the risk is high and outputs a warning to the user terminal 2. This allows the user to be appropriately notified of the navigation status of the ship.

[0093] 13 is a flowchart showing the procedure for estimating wind information according to the third embodiment. After calculating the statistical value of the wind information (step S33), the server 1 executes the following process. The control unit 11 of the server 1 acquires ship information about the ship from which the navigation data was acquired from the ship DB 141 (step S301). The ship information includes, for example, the type, size, and weight of the ship.

[0094] The control unit 11 determines the navigation risk level based on the acquired ship information and the wind information estimated by the learning model 50 (the statistical value calculated in step S33) (step S302). Specifically, the control unit 11 determines the navigation risk level by referring to a table that defines the navigation risk level in association with wind information (wind force class) and ship information (ship type). The control unit 11 stores the water surface image, the estimated wind information, and the navigation risk level in the image DB 142 in association with the identification information of the ship and the camera 32 (step S303), and ends the series of processes.

[0095] As described above, according to the third embodiment, the navigational state of the ship (navigational risk) can be grasped.

[0096] (Variant 1) In embodiment 1, we have described a form in which wind information (wind force scale) is estimated from a water surface image using learning model 50, but the accuracy of the estimated wind information may also be determined based on the wind speed, wave height, wind direction, etc. at the time the water surface image was captured.

[0097] That is, when acquiring water surface images from the vessel system 3, the server 1 also acquires wind speed data, wave height data, or wind direction data as navigation data. The server 1 estimates wind information (wind force scale) by inputting the acquired water surface images into the learning model 50. In this modification, the server 1 compares the estimated wind information with the wind speed data, wave height data, or wind direction data acquired at the same time, and determines whether the wind information estimation result is correct.

[0098] 14 is a table showing the correspondence between wind volume classes and wind speeds and wave heights. Generally, a wind force scale (e.g., the Beaufort scale) defines wind speeds or maximum wave heights corresponding to each wind force scale. By determining whether wind information is correct based on wind speed data or wave height data, it is possible to evaluate whether the estimation results are valid.

[0099] When wind speed, wave height, or wind direction is estimated as wind information, it is sufficient to compare it with wind speed data, wave height data, or wind direction data, respectively.

[0100] (Variant 2) In embodiment 1, we have described a form in which wind information is estimated from a water surface image using the learning model 50, but it is also possible to have a navigator or other person determine whether the estimated wind information is correct or not, and update (re-learn) the learning model 50.

[0101] That is, when the server 1 estimates wind information by inputting a water surface image into the learning model 50, it presents (displays) the estimated wind information to the officer or the like. The server 1 then accepts input of whether the wind information estimation result is correct or incorrect. The server 1 performs re-learning based on the water surface image input into the learning model 50 and the correct or incorrect estimation result input by the officer or the like, and updates parameters such as the weights between neurons. This makes it possible to improve the accuracy of wind information estimation.

[0102] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0103] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any combination, regardless of the citation format. Furthermore, the claims use a format in which a claim cites two or more other claims (multiple claim format), but this is not limited to this. A multiple claim citing at least one other multiple claim (multi-multi claim) can also be used.

[0104] The following additional notes are provided regarding the above-described embodiment.

[0105] (Supplementary Note 1) A program causing a computer to execute a process of acquiring an image including a water surface, and estimating wind information by inputting the acquired image into a learning model that outputs wind information when the image is input. (Supplementary Note 2) The program according to Supplementary Note 1, wherein the image is an image of the water surface around a ship, the image is acquired in association with identification information of the ship, and when the wind information is estimated, the program stores a combination of the estimated wind information and the identification information as needed, and outputs the wind information to a user terminal corresponding to the identification information. (Supplementary Note 3) The program according to Supplementary Note 2, wherein when the wind information is equal to or greater than a predetermined threshold, a warning is output to the user terminal. (Supplementary Note 4) The program according to any one of Supplements 1 to 3, wherein the wind information includes a wind scale, wind speed, wave height, or wind direction. (Supplementary Note 5) The program according to any one of Supplements 1 to 4, wherein the learning model is generated based on training data generated by assigning the wind information assigned to a video consisting of a plurality of the images captured in chronological order to each of the images constituting the video. (Supplementary Note 6) The program according to any one of Supplements 1 to 5, which acquires wind speed data, wave height data, or wind direction data at the time the image was captured, and estimates the wind information by inputting the image and the wind speed data, wave height data, or wind direction data into the learning model. (Supplementary Note 7) The program according to any one of Supplements 1 to 6, which acquires a plurality of the images captured at a predetermined time interval, estimates the wind information at a plurality of time points by inputting each of the acquired images into the learning model, and determines wind information to be an estimated result based on the wind information at the plurality of time points. (Supplementary Note 8) The program according to Supplementary Note 7, which calculates statistical values ​​of the wind information based on the wind information at a plurality of time points, and determines the calculated statistical values ​​as an estimated result. (Supplementary Note 9) The program according to Supplementary Note 7, which determines wind information to be an estimated result by inputting the estimated wind information at a plurality of time points into a second learning model that outputs wind information to be an estimated result when the wind information at a plurality of time points is input. (Supplementary Note 10) The program according to any one of Supplements 1 to 9, wherein the image includes a sky region.(Supplementary Note 11) The program according to any one of Supplements 1 to 10, which generates a mask image by masking an area other than the water surface from the image, and estimates the wind information by inputting the mask image into the learning model. (Supplementary Note 12) The program according to any one of Supplements 1 to 11, which acquires wind speed data, wave height data or wind direction data at the time of capturing the image, and determines whether the estimation result of the wind information is correct based on the wind speed data, wave height data or wind direction data. (Supplementary Note 13) The program according to any one of Supplements 1 to 12, which accepts input of whether the estimation result of the wind information is correct or not, and updates the learning model based on the image and the correct or not of the estimation result. (Supplementary Note 14) The program according to any one of Supplements 1 to 13, which is an image of the water surface around a ship, acquires ship information about the ship, and determines a navigation risk based on the wind information estimated by the learning model and the ship information. (Supplementary Note 15) An information processing method in which a computer executes the process of acquiring an image including a water surface, and estimating wind information by inputting the acquired image into a learning model that outputs wind information related to wind when the image is input. (Supplementary Note 16) An information processing device including a control unit, wherein the control unit acquires an image including a water surface, and estimates the wind information by inputting the acquired image into a learning model that outputs wind information related to wind when the image is input. (Supplementary Note 17) A model generation method in which a computer executes the process of acquiring training data in which wind information related to wind is associated with an image including a water surface, and generating a learning model that outputs the wind information when the image is input based on the training data. term

[0106] Not necessarily all objects or advantages may be achieved in accordance with any particular embodiment described herein. Thus, for example, one skilled in the art will appreciate that a particular embodiment may be configured to operate to achieve or optimize one or more advantages as taught herein without necessarily achieving other objects or advantages as taught or suggested herein.

[0107] All processes described herein may be embodied and fully automated by software code modules executed by a computing system including one or more computers or processors. The code modules may be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods may be embodied in dedicated computer hardware.

[0108] Many other variations beyond those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain operations, events, or functions of any of the algorithms described herein may be performed in a different sequence, added, merged, or omitted entirely (e.g., not all described acts or events are necessary to execute an algorithm). Furthermore, in certain embodiments, operations or events may be performed in parallel rather than sequentially, e.g., via multithreading, interrupt processing, or multiple processors or processor cores, or on other parallel architectures. Furthermore, different tasks or processes may be performed by different machines and / or computing systems that may function together.

[0109] The various illustrative logical blocks and modules described in connection with the embodiments disclosed herein may be implemented or executed by a machine such as a processor. The processor may be a microprocessor, but alternatively, the processor may be a controller, microcontroller, or state machine, or a combination thereof. The processor may include electrical circuitry configured to process computer-executable instructions. In another embodiment, the processor includes an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable device that performs logical operations without processing computer-executable instructions. A processor may also be implemented as a combination of computing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Although described herein primarily with reference to digital technology, a processor may also include primarily analog elements. For example, some or all of the signal processing algorithms described herein may be implemented by analog circuitry or mixed analog and digital circuitry. The computing environment can include any type of computer system, including, but not limited to, a microprocessor, mainframe computer, digital signal processor, portable computing device, device controller, or computer system based on a computational engine within an appliance.

[0110] Unless otherwise specified, conditional language such as "can," "could," "would," or "potential" is understood within the context in which it is generally used to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language does not generally imply that features, elements, and / or steps are required in any manner in one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular embodiment.

[0111] Disjunctive language such as "at least one of X, Y, Z," unless specifically stated otherwise, is understood in its general context to indicate that an item, term, etc. can be either X, Y, Z, or any combination thereof (e.g., X, Y, Z). Thus, such disjunctive language does not generally imply that a particular embodiment requires at least one of X, at least one of Y, or at least one of Z, respectively, to be present.

[0112] Any process descriptions, elements, or blocks in the flow diagrams described herein and / or illustrated in the accompanying drawings should be understood as potentially representing modules, segments, or portions of code, comprising one or more executable instructions for implementing a particular logical function or element in the process. Alternative embodiments are included within the scope of the embodiments described herein, in which elements or functions may be performed out of order, substantially simultaneously, or in reverse order from that shown or described, depending on the functionality involved, as will be understood by those skilled in the art.

[0113] Unless otherwise expressly stated, numeral terms such as "one" should generally be construed to include one or more described items. Thus, phrases such as "one device configured to" are intended to include one or more listed devices. Such one or more listed devices may also be collectively configured to perform the recited reference. For example, "a processor configured to perform the following A, B, and C" may include a first processor configured to perform A and a second processor configured to perform B and C. Additionally, even if a specific number of enumerations of the introduced embodiments are explicitly recited, those skilled in the art should construe such enumerations to typically mean at least the recited number (e.g., the mere enumeration of "two enumerations" without other modifiers typically means at least two enumerations, or two or more enumerations).

[0114] In general, it will be appreciated by those skilled in the art that the terms used herein generally intend "non-limiting" terms (e.g., the term "including" should be interpreted as "including but not limited to at least," the term "having" should be interpreted as "having at least," the term "including" should be interpreted as "including, but not limited to," etc.).

[0115] For purposes of description, the term "horizontal" as used herein is defined as a plane parallel to the plane or surface of the floor of the area in which the described system is used or the plane in which the described method is performed, regardless of its orientation. The term "floor" can be interchanged with the terms "ground" or "water surface." The term "vertical / plumb" refers to a direction perpendicular / vertical to a defined horizontal line. Terms such as "upper," "lower," "below," "top," "side," "higher," "lower," "above," "over," "below," etc. are defined relative to the horizontal plane.

[0116] As used herein, the terms "attach," "connect," "mate," and other related terms, unless otherwise noted, should be interpreted to include detachable, movable, fixed, adjustable, and / or removable connections or couplings. Connections / couplings include direct connections and / or connections with intermediate structures between the two components described.

[0117] Unless otherwise expressly stated, as used herein, numbers preceded by terms such as "approximately," "about," and "substantially" are inclusive of the recited number and also refer to an amount close to the recited amount that performs the desired function or achieves the desired result. For example, "approximately," "about," and "substantially" refer to values ​​less than 10% of the recited numerical value, unless otherwise expressly stated. As used herein, features of the disclosed embodiments preceded by terms such as "approximately," "about," and "substantially" refer to features that have some variability that also perform the desired function or achieve the desired result for that feature.

[0118] Many variations and modifications may be made to the above-described embodiments, and these elements should be understood to be among other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and are protected by the following claims.

[0119] DESCRIPTION OF SYMBOLS 1 Server (information processing device) 11 Control unit 12 Main memory unit 13 Communication unit 14 Auxiliary memory unit 50 Learning model 141 Ship DB 142 Image DB 2 User terminal 3 Ship system 31 Control device 32 Camera 33, 34, 35 Device

Claims

1. A program that causes a computer to execute a process of acquiring an image including a water surface, and estimating wind information by inputting the acquired image into a learning model that outputs wind information when the image is input.

2. The program according to claim 1, wherein the image is an image of the water surface surrounding a ship, the image is acquired in association with identification information of the ship, and when the wind information is estimated, a combination of the estimated wind information and the identification information is stored from time to time, and the wind information is output to a user terminal corresponding to the identification information.

3. The program according to claim 2, further comprising: outputting a warning to the user terminal when the wind information is equal to or greater than a predetermined threshold.

4. The program according to claim 1, wherein the wind information includes a wind scale, a wind speed, a wave height, or a wind direction.

5. The program described in claim 1, wherein the learning model is generated based on training data generated by assigning the wind information assigned to a video consisting of a plurality of the images captured in chronological order to each of the images constituting the video.

6. The program according to claim 1, which obtains wind speed data, wave height data or wind direction data at the time the image was captured, and estimates the wind information by inputting the image and the wind speed data, wave height data or wind direction data into the learning model.

7. The program according to claim 1, which acquires a plurality of images taken at a predetermined time interval, estimates the wind information at a plurality of points in time by inputting each of the acquired images into the learning model, and determines the wind information to be used as an estimated result based on the wind information at the plurality of points in time.

8. The program according to claim 7, further comprising: calculating a statistical value of the wind information based on the wind information at a plurality of points in time; and determining the calculated statistical value as the estimation result.

9. The program described in claim 7, which determines the wind information to be used as an estimated result by inputting the estimated wind information at multiple time points into a second learning model that outputs wind information to be used as an estimated result when the wind information at multiple time points is input.

10. The program of claim 1, wherein the image includes a sky region.

11. The program according to claim 1, which generates a mask image by masking areas other than the water surface from the image, and estimates the wind information by inputting the mask image into the learning model.

12. The program according to claim 1, which acquires wind speed data, wave height data or wind direction data at the time the image is captured, and determines whether the estimated wind information is correct or not based on the wind speed data, wave height data or wind direction data.

13. The program according to claim 1, which accepts input of whether the wind information estimation result is correct or not, and updates the learning model based on the image and the correct or not of the estimation result.

14. The program described in claim 1, wherein the image is an image of the water surface surrounding the ship, ship information regarding the ship is acquired, and a navigation risk is determined based on the wind information estimated by the learning model and the ship information.

15. An information processing method in which a computer executes a process of acquiring an image including a water surface, and estimating wind information by inputting the acquired image into a learning model that outputs wind information when the image is input.

16. An information processing device having a control unit, wherein the control unit acquires an image including a water surface, and estimates the wind information by inputting the acquired image into a learning model that outputs wind information when the image is input.

17. A model generation method in which a computer executes the process of acquiring training data in which wind information is associated with an image including a water surface, and generating a learning model that outputs the wind information when the image is input based on the training data.

Images