Remote sensing interpretation method, system and device and storage medium

Abstract

The invention relates to a remote sensing interpretation method, system and device and a storage medium, and the method comprises the steps: loading a sky map image on a base map, loading a to-be-interpreted remote sensing image on a first map layer, aligning the remote sensing image with the sky map image according to geographic coordinates, determining a target interpretation region in the remote sensing image, and carrying out the interpretation of the target interpretation region; generating position information for indicating the target interpretation area; obtaining a to-be-interpreted parameter of the target interpretation area, and sending the position information and the to-be-interpreted parameter to a server; wherein the to-be-interpreted parameters are used for indicating a target scene and a target interpretation model for interpreting the target interpretation area, an interpretation result of scene information the same as that of the target scene is efficiently and quickly interpreted in the server, and then the client displays the interpretation result in a second layer stacked with the base map and the first layer. According to the method, the information to be obtained can be quickly interpreted from the remote sensing image and visually displayed, and the interpretation process is carried out in the server, so that the hardware requirement of the client is reduced, and the applicability is improved.

Description

Technical Field

[0001] The present application relates to the field of remote sensing interpretation technology, and in particular to a remote sensing interpretation method, system, device and storage medium. Background Art

[0002] The development of remote sensing technology began in the early 20th century. In the early days, it was mainly used in the fields of national defense, military, environmental monitoring, etc. In recent years, with the rapid development of remote sensing technology in my country, remote sensing has been widely used in a variety of flexible and complex earth observation tasks, especially the observation ability of some typical surface elements has reached an unprecedented level, such as geological landslides, roads, buildings, vegetation, water bodies, etc., and remote sensing interpretation is a more important part of it. Remote sensing image interpretation refers to the process of identifying and extracting surface features and phenomenon information by observing, analyzing, comparing and integrating remote sensing images. This process can be achieved through manual interpretation or computer-assisted interpretation. Its main goal is to convert remote sensing data into geographic information required for practical applications, such as land cover type, vegetation growth status, urban building distribution, etc.

[0003] According to surveys conducted by relevant departments in natural resources, forestry, environmental protection and other fields, the remote sensing images of surface elements obtained by provincial departments each year have reached hundreds of TB or even PB levels, and are still growing year by year. Due to the large amount of data, the current stand-alone interpretation method is inefficient and cannot quickly and effectively interpret valuable information from remote sensing image data. Summary of the invention

[0004] In order to solve the technical problem of how to quickly and efficiently interpret remote sensing image data, the present application provides a remote sensing interpretation method, system, device and storage medium.

[0005] In a first aspect, the present application provides a remote sensing interpretation method, applied to a client, the method comprising:

[0006] Loading a Tiandi Map image on a base map, and loading a remote sensing image to be interpreted on a first layer, wherein the remote sensing image and the Tiandi Map image are aligned according to geographic coordinates, and the first layer is above the base map;

[0007] Determining a target interpretation area in the remote sensing image, and generating location information indicating the target interpretation area;

[0008] Acquire the parameters to be interpreted of the target interpretation area, and send the location information and the parameters to be interpreted to the server; wherein the parameters to be interpreted are used to indicate the target scene and target interpretation model for interpreting the target interpretation area;

[0009] Obtaining from the server an interpretation result of the target interpretation area based on the location information, the target scene and the target interpretation model, and displaying the interpretation result on a second layer, where the second layer is above the first layer;

[0010] Optionally, load the Tiandi Map image in the base map and load the remote sensing image to be interpreted in the first layer, including:

[0011] Get access to TiandiTu.

[0012] Loading the map service GeoServer in the framework building unit Cesium; wherein the framework building unit Cesium includes at least a base map, a first layer and a second layer;

[0013] Based on the access rights, the Tiandi Map image is loaded in the base map of the framework building unit Cesium, and based on the map service GeoServer, the remote sensing image is loaded in the first layer;

[0014] Optionally, determining a target interpretation area in the remote sensing image and generating location information indicating the target interpretation area includes:

[0015] Acquire a region of interest drawn in the remote sensing image, and use the region of interest as the target interpretation region;

[0016] generating position information indicating the target interpretation area according to the coordinate information of the target interpretation area;

[0017] Optionally, obtaining the to-be-interpreted parameters of the target interpretation area includes:

[0018] Acquire a first indication message indicating the target scene, and acquire a second indication message indicating the target interpretation model;

[0019] Determine the parameter to be interpreted according to the first indication message and the second indication message;

[0020] Optionally, generating position information indicating the target interpretation area according to the coordinate information of the target interpretation area includes:

[0021] Determine a first array according to the region of interest, wherein the first array includes vertex coordinate information of the region of interest;

[0022] Create a Key-Value pair according to the vertex coordinate information;

[0023] The Key-Value pair is stored in a form data object; wherein the Value value is the vertex coordinate information, and the Key value is the identification information;

[0024] The form data object is used as the location information indicating the target interpretation area.

[0025] In a second aspect, the present application provides a remote sensing interpretation method, applied to a server, the method comprising:

[0026] Acquire location information of a target interpretation area and parameters to be interpreted of the target interpretation area from a client; wherein the parameters to be interpreted are used to indicate a target scene and a target interpretation model for interpreting the target interpretation area;

[0027] Interpreting the target interpretation area according to the location information, the target scene and the target interpretation model to obtain an interpretation result;

[0028] The interpretation result is sent to the client, so that the client displays the interpretation result on a second layer; wherein the second layer is above the first layer, the first layer is above the base map, the base map is loaded with a Tiandi map image, the first layer is loaded with a remote sensing image to be interpreted, and the target interpretation area is determined in the remote sensing image;

[0029] Optionally, the parameters to be interpreted include a first indication message indicating the target scene and a second indication message indicating the target interpretation model; interpreting the target interpretation area according to the location information, the target scene and the target interpretation model to obtain an interpretation result includes:

[0030] Determining the target interpretation model according to the first indication message;

[0031] Determining the target scenario according to the second indication message;

[0032] Interpreting a first scene identical to the target scene in the target interpretation area according to the target interpretation model and the position information, and taking all the first scenes as the interpretation results;

[0033] Optionally, the location information includes a form data object, and the form data object stores a Key-Value pair, wherein the Value value is vertex coordinate information, and the Key value is identification information;

[0034] Interpreting a first scene identical to the target scene in the target interpretation area according to the target interpretation model and the position information, and taking all the first scenes as the interpretation results, comprises:

[0035] Acquire the vertex coordinate information according to the identification information;

[0036] According to the target interpretation model and the target scene, the area enclosed by all the vertex coordinate information is interpreted to obtain the target result;

[0037] The Jsonify function is called to convert the target result into JSON data, and the JSON data is used as the interpretation result.

[0038] In a third aspect, the present application provides a remote sensing interpretation system, the system comprising a client and a server;

[0039] The client is used to load a Tiandi Map image on a base map, and load a remote sensing image to be interpreted on a first layer, wherein the remote sensing image is aligned with the Tiandi Map image according to geographic coordinates, and the first layer is above the base map; determine a target interpretation area in the remote sensing image, and generate location information indicating the target interpretation area; obtain parameters to be interpreted of the target interpretation area, and send the location information and the parameters to be interpreted to a server; wherein the parameters to be interpreted are used to indicate a target scene and a target interpretation model for interpreting the target interpretation area; obtain an interpretation result of the target interpretation area based on the location information, the target scene and the target interpretation model from a server, and display the interpretation result on a second layer, wherein the second layer is above the first layer;

[0040] The server is used to obtain the location information of the target interpretation area and the parameters to be interpreted of the target interpretation area from the client; interpret the target interpretation area according to the location information, the target scene and the target interpretation model to obtain an interpretation result; and send the interpretation result to the client so that the client displays the interpretation result on the second layer.

[0041] In a fourth aspect, the present application provides a remote sensing interpretation device, applied to a client, the device comprising:

[0042] A loading module, used for loading a Tiandi Map image on a base map, and loading a remote sensing image to be interpreted on a first layer, wherein the remote sensing image and the Tiandi Map image are aligned according to geographic coordinates, and the first layer is above the base map;

[0043] A generating module, used for determining a target interpretation area in the remote sensing image, and generating position information indicating the target interpretation area;

[0044] A first acquisition module is used to acquire the parameters to be interpreted of the target interpretation area, and send the location information and the parameters to be interpreted to the server; wherein the parameters to be interpreted are used to indicate the target scene and target interpretation model for interpreting the target interpretation area;

[0045] The display module is used to obtain an interpretation result of the target interpretation area based on the position information, the target scene and the target interpretation model from the server, and display the interpretation result on a second layer, where the second layer is above the first layer.

[0046] In a fifth aspect, the present application provides a remote sensing interpretation device, applied to a server, the device comprising:

[0047] A second acquisition module is used to acquire the location information of the target interpretation area and the parameters to be interpreted of the target interpretation area from the client; wherein the parameters to be interpreted are used to indicate the target scene and target interpretation model for interpreting the target interpretation area;

[0048] An interpretation module, used to interpret the target interpretation area according to the location information, the target scene and the target interpretation model to obtain an interpretation result;

[0049] A sending module is used to send the interpretation result to the client so that the client displays the interpretation result on a second layer, wherein the second layer is above the first layer, the first layer is above the base map, the base map is loaded with a Tiandi map image, the first layer is loaded with a remote sensing image to be interpreted, and the target interpretation area is determined in the remote sensing image.

[0050] In a sixth aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0051] Memory, used to store computer programs;

[0052] The processor is used to implement the steps of the remote sensing interpretation method described in any embodiment of the first aspect when executing the program stored in the memory.

[0053] In a seventh aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the remote sensing interpretation method as described in any embodiment of the first aspect are implemented.

[0054] Beneficial effects of this application:

[0055] The method provided by the embodiment of the present application is to load a Tiandi Map image on a base map, load a remote sensing image to be interpreted on a first layer, wherein the remote sensing image and the Tiandi Map image are aligned according to geographic coordinates, and the first layer is above the base map; determine a target interpretation area in the remote sensing image, and generate location information indicating the target interpretation area; obtain parameters to be interpreted of the target interpretation area, and send the location information and the parameters to be interpreted to a server; wherein the parameters to be interpreted are used to indicate a target scene and a target interpretation model for interpreting the target interpretation area; obtain an interpretation result of the target interpretation area based on the location information, the target scene and the target interpretation model from the server, and display the interpretation result on a second layer, and the second layer is above the first layer. In this method, the client loads the Tiandi Map image on the base map, loads the remote sensing image to be interpreted on the first layer, sends the location information of the target interpretation area of ​​the remote sensing image to be interpreted and the parameters to be interpreted to the server, and efficiently and quickly interprets the interpretation result of the same scene information as the target scene in the server. The client then displays the interpretation result on the second layer stacked with the base map and the first layer. The desired information can be quickly interpreted from the remote sensing image and displayed intuitively, and the interpretation process is carried out in the server, which reduces the hardware requirements of the client and improves applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 A system architecture diagram of a remote sensing interpretation method provided in one embodiment of the present application;

[0059] Figure 2 A flowchart of a remote sensing interpretation method applied to a client provided by an embodiment of the present application;

[0060] Figure 3 A flowchart of a remote sensing interpretation method applied to a server provided in one embodiment of the present application;

[0061] Figure 4 A system architecture diagram of a remote sensing interpretation method provided in another embodiment of the present application;

[0062] Figure 5 A schematic diagram of an overall interaction process provided for an embodiment of the present application;

[0063] Figure 6 A schematic diagram of a detection algorithm integration process provided by an embodiment of the present application;

[0064] Figure 7 A schematic diagram of the structure of a remote sensing interpretation device applied to a client provided in one embodiment of the present application;

[0065] Figure 8 A schematic diagram of the structure of a remote sensing interpretation device applied to a server provided in one embodiment of the present application;

[0066] Fig. 9 A schematic diagram of the structure of an electronic device provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0068] The first embodiment of the present application provides a remote sensing interpretation method, which can be applied to Figure 1 The system architecture shown includes at least a client 101 and a server 102, and the client 101 and the server 102 establish a communication connection.

[0069] The method can be applied to the client 101 or server 102 in the system architecture, wherein the client 101 can be a desktop computer, a tablet computer, a notebook, a supercomputer or other devices, or an intelligent terminal, such as a smart phone, etc., and the server 102 can be a local server, a cloud server, or a server cluster.

[0070] The client 101 is used to load a Tiandi Map image on a base map, load a remote sensing image to be interpreted on a first layer, wherein the remote sensing image and the Tiandi Map image are aligned according to geographic coordinates, the first layer is above the base map, determine a target interpretation area in the remote sensing image, generate location information indicating the target interpretation area, obtain parameters to be interpreted of the target interpretation area, send the location information and the parameters to be interpreted to a server, wherein the parameters to be interpreted are used to indicate a target scene and a target interpretation model for interpreting the target interpretation area, obtain an interpretation result of the target interpretation area based on the location information, the target scene and the target interpretation model from the server, and display the interpretation result on a second layer, wherein the second layer is above the first layer.

[0071] The server 102 is used to obtain the location information of the target interpretation area and the parameters to be interpreted of the target interpretation area from the client, interpret the target interpretation area according to the location information, the target scene and the target interpretation model, obtain the interpretation result, and send the interpretation result to the client, so that the client displays the interpretation result on the second layer.

[0072] Next, based on the system architecture, the remote sensing interpretation method is described in detail.

[0073] When applied to the client, a remote sensing interpretation method, such as Figure 2 ,include:

[0074] Step 201 , loading a Tiandi Map image into a base map, and loading a remote sensing image to be interpreted into a first layer, wherein the remote sensing image and the Tiandi Map image are aligned according to geographic coordinates, and the first layer is above the base map.

[0075] Through tools that support image layer stacking display, the Tiandi Map image and the remote sensing image to be interpreted are aligned according to geographic coordinates and displayed in layers. The Tiandi Map image can be data obtained from the National Geographic Information Service Platform, and its resolution is relatively low. The remote sensing image to be interpreted can be satellite data, and its resolution is higher than that of the Tiandi Map image.

[0076] In one embodiment, loading the Tiandi Map image in the base map and loading the remote sensing image to be interpreted in the first layer includes: obtaining access rights to Tiandi Map; loading the map service GeoServer in the framework building unit Cesium; wherein the framework building unit Cesium includes at least a base map, a first layer and a second layer; loading the Tiandi Map image in the base map of the framework building unit Cesium based on the access rights, and loading the remote sensing image in the first layer based on the map service GeoServer.

[0077] User interaction is carried out in the Web, and map service is the basis for building a remote sensing interpretation platform. In this embodiment, Cesium, a JavaScript open source library that supports multiple OGC specifications, is used as a framework construction unit. Relying on Cesium and the map service Geoserver, the GeoServer map service is loaded in Cesium. Cesium supports stacked display of image layers. Only loading the GeoServer map service is not enough to build a complete Cesium map scene. Therefore, Cesium can be used to load the base map service. The Tiandi Map image can use the Web Map Tile Service (WMTS) to provide map services. The Tiandi Map image is loaded as the base map through Cesium, and the vector map service layer is superimposed on it, that is, the remote sensing image to be interpreted is loaded into the first layer, and the final interpretation result is loaded into the second layer.

[0078] Among them, the first layer and the second layer can also include multiple sub-layers, and the contents to be displayed can be displayed in specific layers without limitation. For example, some sub-layers can display main elements or marking information.

[0079] In this embodiment, before calling the Tiandi Map image service, you need to apply for a Key value on the Tiandi Map official website to implement the call. The calling process can be as follows:

[0080] Download the GeoServer jar package or exe installation package from the GeoServer official website, install it, and deploy it to the local server.

[0081] Apply for access rights to Tiandi Map on the National Geographic Information Service Platform. Tiandi Map provides an API for developers to use. After applying, you can obtain the key value to facilitate loading Tiandi Map data on the front end.

[0082] Load the GeoServer map service in Cesium and pass two parameters: the URL address of the map WMS service (i.e. the Tiandi map image address) and the loaded layer name. The layer name format can be "Workspace: Layer 1, Layer 2...".

[0083] Step 202: determine a target interpretation area in the remote sensing image, and generate location information indicating the target interpretation area.

[0084] The target interpretation area can be the area of ​​interest defined by the user on the layer display interface for the remote sensing image, or it can be the latitude and longitude information of several points (at least three points that are not on a straight line) input, and the area enclosed by these points is used as the target interpretation area.

[0085] In one embodiment, a target interpretation area is determined in a remote sensing image, and location information indicating the target interpretation area is generated, including: obtaining an area of ​​interest drawn in the remote sensing image, taking the area of ​​interest as the target interpretation area, and generating location information indicating the target interpretation area based on coordinate information of the target interpretation area.

[0086] In this embodiment, the client (also called the front end) generates location information indicating the target interpretation area according to the coordinate information of the target interpretation area, so that the server (also called the back end) determines the target interpretation area to be interpreted according to the location information.

[0087] In one embodiment, position information indicating the target interpretation area is generated based on the coordinate information of the target interpretation area, including: determining a first array based on the area of ​​interest, the first array including vertex coordinate information of the area of ​​interest; creating a Key-Value pair based on the vertex coordinate information; storing the Key-Value pair in a form data object (FormData object); wherein the Value value is the vertex coordinate information, and the Key value is identification information; and using the form data object as the position information indicating the target interpretation area.

[0088] In this embodiment, the user draws a region of interest (ROI, ROI region or region of interest) that needs to be interpreted, and sets an Arr array (first array) for the drawing shape. For example, if the drawing shape is a rectangle, the coordinate information of the upper left corner and the lower right corner recorded by the rectangular selection box is assigned to the Arr array.

[0089] The Arr array is transferred to the ROIData object, which can store information such as layer ID information, selection box type, and shape style. The ROIData object is finally handed over to the Entity entity for storage. At this time, the stored data has no effect on the interpretation, and only parameter storage is realized. Specifically, the front end creates a FormData object that stores Key-Value key-value pairs, and assigns the geographic coordinates of the upper left corner and lower right corner of the selection box of the area of ​​interest to the Key value respectively. Here, the key can be set to longitude, latitude, longitude1, latitude1, and the layer layerName set when the area of ​​interest is selected.

[0090] After the front-end interface submits the form data object (specifically, the information stored in the FormData object can be Posted to the back-end server via an http request), the longitude and latitude information will be transmitted to the server, and the longitude and latitude information will be passed into the back-end code as parameters. The server calls the information to perform subsequent interpretation of the data stored in the database, and the interpretation speed can generally be completed in about 10 seconds.

[0091] Step 203, obtaining parameters to be interpreted of the target interpretation area, and sending the location information and the parameters to be interpreted to the server, wherein the parameters to be interpreted are used to indicate a target scene and a target interpretation model for interpreting the target interpretation area.

[0092] The client not only needs to send the location information of the target interpretation area to the server, but also needs to send the parameters to be interpreted, such as the target scene that needs to be interpreted and the target interpretation model to be used. The target scene can be a landslide, mudslide, road, building, vegetation, water body and other scenes, without restriction. For example, if the target scene is a road, the server can interpret all the roads in the target interpretation area. If the target scene is a landslide, the server can interpret and mark the area where the geological landslide occurs in the target interpretation area. The target interpretation model can include Mask-RCNN, DeepLabV3+, FCN, U-Net, Center-Mask and other network prediction models, without restriction.

[0093] In one embodiment, obtaining the parameters to be interpreted in the target interpretation area includes: obtaining a first indication message indicating a target scene, and obtaining a second indication message indicating a target interpretation model, and determining the parameters to be interpreted according to the first indication message and the second indication message.

[0094] In this embodiment, the parameter to be interpreted can be determined by the first indication message of the target scene to be interpreted and the second indication message of the target interpretation model to be used, for example, by setting an identification ID for each target scene in advance, and different ID information corresponds to different target scenes, without limitation. Similarly, identification information can also be set for the target interpretation model in advance, and different identification information corresponds to different target interpretation models, and the form of the identification information is not limited.

[0095] Step 204, obtaining from the server an interpretation result of the target interpretation area based on the location information, the target scene and the target interpretation model, and displaying the interpretation result on a second layer, the second layer being above the first layer.

[0096] In this method, the client loads the Tiandi Map image on the base map, loads the remote sensing image to be interpreted on the first layer, sends the location information of the target interpretation area of ​​the remote sensing image to be interpreted and the parameters to be interpreted to the server, and efficiently and quickly interprets the interpretation result of the same scene information as the target scene in the server. The client then displays the interpretation result on the second layer stacked with the base map and the first layer. The desired information can be quickly interpreted from the remote sensing image and displayed intuitively, and the interpretation process is carried out in the server, which reduces the hardware requirements of the client and improves applicability.

[0097] When applied to a server, a remote sensing interpretation method, such as Figure 3 ,include:

[0098] Step 301: Acquire location information of a target interpretation area and parameters to be interpreted of the target interpretation area from a client, wherein the parameters to be interpreted are used to indicate a target scene and a target interpretation model for interpreting the target interpretation area.

[0099] The server obtains the geographic location from the client to determine the target interpretation area, obtains the parameters to be interpreted, and determines the target scene and target interpretation model for interpreting the target interpretation area according to the parameters to be interpreted. The target scene can be a landslide, mud-rock flow, road, building, vegetation, water body and other scenes, without limitation. The target interpretation model can include Mask-RCNN, DeepLabV3+, FCN, U-Net, Center-Mask and other network prediction models, without limitation.

[0100] Step 302: interpret the target interpretation area according to the location information, the target scene and the target interpretation model to obtain an interpretation result.

[0101] The parameters to be interpreted may include a first indication message indicating a target scene and a second indication message indicating a target interpretation model. The target interpretation area is interpreted according to the location information, the target scene and the target interpretation model to obtain an interpretation result, including: determining the target interpretation model according to the first indication message, determining the target scene according to the second indication message, interpreting a first scene identical to the target scene in the target interpretation area according to the target interpretation model and the location information, and taking all the first scenes as the interpretation result.

[0102] In one embodiment, the location information includes a form data object, and the form data object stores a Key-Value pair, wherein the Value value is the vertex coordinate information, and the Key value is the identification information. A first scene identical to the target scene is interpreted in a target interpretation area according to the target interpretation model and the location information, and all first scenes are used as interpretation results, including: obtaining vertex coordinate information according to the identification information, interpreting the area enclosed by all vertex coordinate information according to the target interpretation model and the target scene to obtain a target result, calling a Jsonify function to convert the target result into JSON data, and using the JSON data as the interpretation result.

[0103] Step 303, sending the interpretation result to the client, so that the client displays the interpretation result on the second layer, wherein the second layer is above the first layer, the first layer is above the base map, the base map is loaded with the Tiandi map image, the first layer is loaded with the remote sensing image to be interpreted, and the target interpretation area is determined in the remote sensing image.

[0104] In this method, the server obtains the location information and the parameters to be interpreted of the target interpretation area from the client, and efficiently and quickly interprets the interpretation result of the same scene information as the target scene in the target interpretation area according to the location information and the parameters to be interpreted, and then sends it to the client to display the interpretation result on the second layer stacked with the base map and the first layer. The desired information can be quickly interpreted from the remote sensing image and displayed intuitively, and the interpretation process is carried out in the server, which reduces the hardware requirements of the client and improves applicability.

[0105] In a specific embodiment, the present application provides a Web technology solution for remote sensing interpretation to address the problem of existing conventional platforms (which require the use of relatively complex neural network models) based on C / S architecture, which is used to implement the construction of a complete online remote sensing interpretation platform and provide sufficiently complete interactive components for users to use.

[0106] The scheme is as Figure 4 , mainly including the following modules: map service loading module, ROI region of interest selection module, scene manager module, neural network model selector module, Vue componentization module, Cesium map engine module, WebGIS visualization module, algorithm generation module and performance testing module.

[0107] Among them, Vue componentization and Cesium map engine are the basic modules for interpretation scenes, algorithm generation is also called detection algorithm integration module, WebGIS is an application form based on the combination of Web technology and geographic information system (GIS), WebGIS visualization is also called the interpretation scene visualization module, and the back-end server can be built based on the cloud server ECS.

[0108] Map service loading module: User interaction in the Web, map service is the basis for building a remote sensing interpretation platform. This patent is based on Cesium, a JavaScript open source library that supports multiple OGC specifications. Relying on Cesium and Geoserver, GeoServer map service is loaded in Cesium. Loading only GeoServer map service is not enough to build a complete Cesium map scene. Cesium supports stacked display of image layers. Therefore, Cesium can be used to load basemap services. Tiandi Map uses WMTS to provide map services. Tiandi Map images are loaded through Cesium as basemaps, and vector map service layers are superimposed on them to display the main elements or marking information. Before calling Tiandi Map image services, you need to apply for the Key value on the Tiandi Map official website to implement the call. Combined with GeoServer images, the basic effects of front-end display can be achieved.

[0109] ROI region of interest selection module: Users select the region of interest by themselves and draw a rectangular selection box in the image map of the interpretation system. The rectangular selection box records the complete longitude and latitude information. After the drawing is completed, a confirmation form will pop up to fill in the layer and its name information, which will be transmitted to the server.

[0110] Scene manager module: In the Web scene constructed in this application, the main function of the scene manager module is to select the interpretation scene being used. The scene manager module can display thumbnails of images. Before using the scene, typical landform scenes are usually produced for simple display, such as typical scenes of landslide areas, typical scenes of mudslide areas, etc. Users can click on the thumbnail of the image to switch to the currently displayed image scene. In the three-dimensional earth built by this application based on Cesium, switching between scenes will be performed through flight simulation.

[0111] Neural network model selector module: Each model in the neural network model selector can be clicked, but only one model can be selected to meet the needs of backend calls. After selecting the model, it needs to be associated with the backend service. The backend routing will provide a specific interpretation interface, mainly including commonly used models (trained) for object detection, change detection, and semantic segmentation. The network prediction models covered mainly include Mask-RCNN, DeepLabV3+, FCN, U-Net, Center-Mask, etc.

[0112] Interpretation scene visualization module: system login interface, enter the username and password that matches the database to log in to the system main interface. The main part is a three-dimensional globe built by Cesium, which can realize three-dimensional zooming and basic map operations; the lower left is the model selector module, where you can select the corresponding network model for intelligent interpretation, the upper right is the layer manager module, and the selected area of ​​interest will be saved in this module, the lower right is the scene manager module, select a specific scene to achieve scene switching, and the right is the map-related operation button. After the user selects the area of ​​interest, select the corresponding model in the model selector, click the detection button, and there will be a Notify notification in the lower right, prompting that the detection is in progress. After the detection is completed, it will respond to the front-end prompt that the detection is completed. The main part of the system displays the specific extraction and interpretation effect.

[0113] Algorithm integration module: The purpose of algorithm service integration is to use the Python framework to integrate the convolutional neural network model and the deep learning framework to build the backend service. The backend service is built using Python's lightweight framework Flask. The reason for using Flask to build the backend service framework is that it can directly call the neural network prediction model without the need for DJL (Deep Java Library) for forward processing.

[0114] Performance test module: The performance test indicators can be divided into two main points. The first point is the time required for background service processing, and the second point is the time interval test from the front end sending a request to the front end displaying the interpretation result. Among them, the background service processing includes image preprocessing, network prediction model call, raster to vector, vector to JSON and other processes. The purpose of performance testing is to evaluate the system's capabilities, identify the weaknesses of the online interpretation system of typical surface elements in remote sensing images, detect problems in the system, and detect the robustness and reliability of the system.

[0115] Specifically, the map service loading module part mainly includes the following steps:

[0116] Download the GeoServer jar package or exe installation package from the GeoServer official website, install it, and deploy it to the local server.

[0117] Apply for access rights to Tiandi Map on the National Geographic Information Service Platform. Tiandi Map provides an API for developers to use. After applying, you can obtain the key value to facilitate loading Tiandi Map data on the front end.

[0118] Load the GeoServer map service in Cesium and pass two parameters: the URL address of the map WMS service (that is, the Tiandi Map image address) and the loaded layer name. The format of the layer name is "Workspace: Layer 1, Layer 2..."

[0119] The ROI region of interest module mainly includes the following steps:

[0120] The user draws the area of ​​interest that needs to be interpreted, sets an Arr array for drawing the shape, and assigns the coordinate information of the upper left corner and lower right corner recorded by the rectangular selection box to the Arr array.

[0121] The Arr array is transferred to the ROIData object, which stores information such as layer ID, selection box type, and shape style. Finally, the ROIData object is saved by the Entity entity. The stored data at this time has no effect on the interpretation, and only realizes parameter storage.

[0122] After the form is submitted on the front-end interface, the longitude and latitude information will be transmitted to the server and passed into the back-end code as parameters. The server calls the information to perform subsequent operations on the data stored in the database.

[0123] The neural network model selector module mainly includes the following steps:

[0124] The user selects a model based on the model selector and clicks Detect.

[0125] The front end first uses the If function to determine whether the selected model (checkModel) is the target interpretation model to be used, such as Mask-RCNN. If it is Mask-RCNN, the modelDetect function is called to perform image interpretation.

[0126] The parameter of the modelDetect function is the interpretation interface provided by the backend routing - a URL link. The front end creates a FormData object that stores Key-Value key-value pairs, and assigns the geographic coordinates of the upper left and lower right corners of the interest area selection box to the Key value respectively. Here, the key is set to longitude, latitude, longitude1, latitude1, and the layer layerName set when the interest area is selected.

[0127] The information stored in the FormData object is posted to the backend server via an http request. The backend server will interpret the information, which usually takes about 10 seconds.

[0128] After the interpretation is completed, the data will be returned to the front end. The front end will traverse the files in the data through a for loop, obtain the internal Json data, and perform subsequent operations.

[0129] The interpretation scene visualization module is mainly for overall interface construction, scene construction and layout, such as Figure 5 , including the following steps:

[0130] Step 501, enter the account and password to match with the backend database. Enter the account and password to match with the backend database, if they match, you can log in to the platform.

[0131] Step 502: Provide basic interactive tools and functions. The platform provides a basic functional interface, including basic interactive tools such as pan, zoom, full screen, switching base map, typical scenes, and nostalgia tools.

[0132] Step 503: The scene manager selects a typical scene. The scene manager is selected to switch to a typical scene, such as a landslide scene or a damaged building scene.

[0133] Step 504: Draw the ROI region. Select the ROI region of interest to start drawing a rectangular selection box of interest, and set the layer name and other information.

[0134] Step 505, select a model in the neural network model. Select a typical neural network model in the neural network model selector, select the detection button, and a notification will pop up in the lower right corner, indicating that the detection is in progress.

[0135] Step 506: Display the result to the front end. When the notification of being detected is changed to detection success, the interpreted effect will be directly overlaid on the three-dimensional earth map service, so that the effect can be seen more intuitively.

[0136] The detection algorithm integration module part, such as Figure 6 , mainly includes the following steps:

[0137] Step 601, receiving a Post request sent by the foreground.

[0138] By selecting the area of ​​interest, the user sends the geographic coordinate information and layer name stored in the Http Request header to the background. After receiving the Post request, the background calls the form object of the Request package to receive the data message. The front-end data is transmitted in the form of K, V key-value pairs. The K value parameter is passed in the get method of the form object, and the corresponding V value information is obtained. Since the transmitted data is received in the form of a String field, the data type is determined here, and data conversion operations are performed according to the data type. The following is the relevant code for receiving request information. The model path code is stored in the form of a field in the formData object, so there is no data conversion problem.

[0139] Step 602, receiving parameters to perform data processing.

[0140] Get the data, crop it, and crop and divide the image data. The cropping code has been encapsulated. You only need to call the specific function interface and pass in the corresponding longitude and latitude information to achieve cropping. It should be noted that due to data encapsulation, the conversion of geographic coordinates to image row and column numbers has been realized inside the cropping. If the formal parameters at this time are not geographic coordinates, it may not be possible to achieve effective cropping results.

[0141] Step 603, calling the detection algorithm to perform interpretation and output Geo-Json.

[0142] The algorithm source code has also been encapsulated. When the backend is executed, it only needs to call the buildingdetiection method to interpret the remote sensing image. In the formal parameters, image_path is the remote sensing image to be called, out_path is the output path to be planned, shp_path is the vector file, and ckpt_path is the prediction model path information received from the front end. In the formal parameters of this function, only the path and output path of the remote sensing image to be interpreted need to be considered. For different algorithms, the parameters are also different, so it is more necessary to integrate the algorithm service and unify and standardize different algorithms.

[0143] Step 604, returning the JSON result after vectorizing the raster.

[0144] Call the Jsonify function to convert the result into JSON and return it to the front end. jsonify is an extension package in flask that can convert data into json data. When processing data, Jsonify serializes the data into JSON using the JSON package in the itsdangerous module. Tracing back to the source, we can find that itsdangerous is implemented by an exception handling mechanism. It will try to introduce simplejson. If this module is not installed, it will call the Python native module. The json data returned by jsonify indicates that the Content-Type is application / json, which is more in line with the HTTP protocol. This is also the reason for using jsonify.

[0145] The above steps 601 to 604 are the detection algorithm service integration process. The above integration code will eventually be deployed on the deep learning application server in the form of a "service".

[0146] In this implementation, in order to solve the problem that the existing C / S architecture intelligent interpretation method cannot realize online and real-time interpretation calculation, a remote sensing intelligent interpretation online recognition platform is built based on Web 3.0, a more vivid interpretation scene is constructed, and a three-dimensional earth is built based on the Cesium.js engine. At the same time, the National Geographic Information Service Platform is called, and the Tiandi Map image is used as the base map service. Such an online scene can more intuitively display the interpretation effect. Various scene managers, selectors and daily operation modules are developed in a componentized manner to facilitate users to perform routine interactions, and with the help of convolutional neural network models, online and intelligent real-time interpretation is realized. The desired information can be quickly interpreted from remote sensing images and displayed intuitively, and the interpretation process is carried out on the server, which reduces the hardware requirements of the client and improves applicability.

[0147] Based on the same technical concept, the second embodiment of the present application provides a remote sensing interpretation device, which can be applied to a client or a server.

[0148] When applied to the client, such as Figure 7 , the device comprises:

[0149] The loading module 701 is used to load the Tiandi Map image into the base map, and load the remote sensing image to be interpreted into the first layer, wherein the remote sensing image and the Tiandi Map image are aligned according to geographic coordinates, and the first layer is above the base map;

[0150] A generating module 702 is used to determine a target interpretation area in the remote sensing image and generate position information indicating the target interpretation area;

[0151] The first acquisition module 703 is used to acquire the parameters to be interpreted of the target interpretation area, and send the location information and the parameters to be interpreted to the server; wherein the parameters to be interpreted are used to indicate the target scene and target interpretation model for interpreting the target interpretation area;

[0152] The display module 704 is used to obtain an interpretation result of the target interpretation area based on the position information, the target scene and the target interpretation model from the server, and display the interpretation result on a second layer, where the second layer is above the first layer.

[0153] In the device, a client loads a Tiandi Map image on a base map, loads a remote sensing image to be interpreted on a first layer, sends the location information of a target interpretation area of ​​the remote sensing image to be interpreted and parameters to be interpreted to a server, and efficiently and quickly interprets an interpretation result of scene information identical to the target scene on the server. The client then displays the interpretation result on a second layer stacked with the base map and the first layer, and can quickly interpret the desired information from the remote sensing image and display it intuitively. The interpretation process is performed on the server, which reduces the hardware requirements of the client and improves applicability.

[0154] When applied to a server, such as Figure 8 , the device comprises:

[0155] The second acquisition module 801 is used to acquire the location information of the target interpretation area and the parameters to be interpreted of the target interpretation area from the client; wherein the parameters to be interpreted are used to indicate the target scene and target interpretation model for interpreting the target interpretation area;

[0156] An interpretation module 802 is used to interpret the target interpretation area according to the position information, the target scene and the target interpretation model to obtain an interpretation result;

[0157] The sending module 803 is used to send the interpretation result to the client so that the client displays the interpretation result on a second layer, wherein the second layer is above the first layer, the first layer is above the base map, the base map is loaded with a Tiandi map image, the first layer is loaded with a remote sensing image to be interpreted, and the target interpretation area is determined in the remote sensing image.

[0158] In the device, a server obtains the location information and the parameters to be interpreted of the target interpretation area from the client, and efficiently and quickly interprets the interpretation result of the same scene information as the target scene in the target interpretation area according to the location information and the parameters to be interpreted, and then sends it to the client to display the interpretation result on the second layer stacked with the base map and the first layer. The desired information can be quickly interpreted from the remote sensing image and displayed intuitively, and the interpretation process is carried out in the server, which reduces the hardware requirements of the client and improves applicability.

[0159] like Fig. 9 As shown, the third embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0160] Memory 113, used for storing computer programs;

[0161] In one embodiment, the processor 111 is used to implement the remote sensing interpretation method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113 .

[0162] The memory and processor in the above electronic device communicate through a communication bus and a communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0163] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0164] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0165] A fourth embodiment of the present application provides a computer-readable medium having a non-volatile program code executable by a processor.

[0166] Optionally, in an embodiment of the present application, a computer-readable medium is configured to store program code for a processor to execute the above method.

[0167] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0168] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0169] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present application, or a combination thereof.

[0170] For software implementation, the technology of this article can be implemented by a unit that performs the functions of this article. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0171] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0173] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0176] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0177] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0178] The above embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present application is within the protection scope of the present application.

Claims

1. A remote sensing interpretation method, characterized in that: Applied to a client, the method comprises: Loading a Tiandi Map image on a base map, and loading a remote sensing image to be interpreted on a first layer, wherein the remote sensing image and the Tiandi Map image are aligned according to geographic coordinates, and the first layer is above the base map; Determining a target interpretation area in the remote sensing image, and generating location information indicating the target interpretation area; Acquire the parameters to be interpreted of the target interpretation area, and send the location information and the parameters to be interpreted to the server; wherein the parameters to be interpreted are used to indicate the target scene and target interpretation model for interpreting the target interpretation area; An interpretation result of the target interpretation area based on the location information, the target scene and the target interpretation model is obtained from the server, and the interpretation result is displayed on a second layer, where the second layer is above the first layer.

2. The method according to claim 1, characterized in that Load the Tiandi Map image in the base map and load the remote sensing image to be interpreted in the first layer, including: Get access to TiandiTu. Loading the map service GeoServer in the framework building unit Cesium; wherein the framework building unit Cesium includes at least a base map, a first layer and a second layer; Based on the access permission, the Tiandi Map image is loaded in the base map of the framework building unit Cesium, and based on the map service GeoServer, the remote sensing image is loaded in the first layer.

3. The method according to claim 1, characterized in that Determining a target interpretation area in the remote sensing image and generating location information indicating the target interpretation area includes: Acquire a region of interest drawn in the remote sensing image, and use the region of interest as the target interpretation region; The position information indicating the target interpretation area is generated according to the coordinate information of the target interpretation area.

4. The method according to claim 1, characterized in that: Obtaining the parameters to be interpreted of the target interpretation area includes: Acquire a first indication message indicating the target scene, and acquire a second indication message indicating the target interpretation model; The to-be-interpreted parameter is determined according to the first indication message and the second indication message.

5. The method according to claim 3, characterized in that: Generating location information indicating the target interpretation area according to the coordinate information of the target interpretation area includes: Determine a first array according to the region of interest, wherein the first array includes vertex coordinate information of the region of interest; Create a Key-Value pair according to the vertex coordinate information; The Key-Value pair is stored in a form data object; wherein the Value value is the vertex coordinate information, and the Key value is the identification information; The form data object is used as the location information indicating the target interpretation area.

6. A remote sensing interpretation method, characterized in that: Applied to a server, the method comprises: Acquire location information of a target interpretation area and parameters to be interpreted of the target interpretation area from a client; wherein the parameters to be interpreted are used to indicate a target scene and a target interpretation model for interpreting the target interpretation area; Interpreting the target interpretation area according to the location information, the target scene and the target interpretation model to obtain an interpretation result; The interpretation result is sent to the client so that the client displays the interpretation result in a second layer; wherein the second layer is above the first layer, the first layer is above the base map, the base map is loaded with a Tiandi map image, the first layer is loaded with a remote sensing image to be interpreted, and the target interpretation area is determined in the remote sensing image.

7. The method according to claim 6, characterized in that The parameters to be interpreted include a first indication message indicating the target scene and a second indication message indicating the target interpretation model; The target interpretation area is interpreted according to the location information, the target scene and the target interpretation model to obtain an interpretation result, including: Determining the target interpretation model according to the first indication message; Determining the target scenario according to the second indication message; A first scene identical to the target scene is interpreted in the target interpretation area according to the target interpretation model and the position information, and all the first scenes are used as the interpretation results.

8. The method according to claim 7, characterized in that The location information includes a form data object, and the form data object stores a Key-Value pair, wherein the Value value is vertex coordinate information and the Key value is identification information; Interpreting a first scene identical to the target scene in the target interpretation area according to the target interpretation model and the position information, and taking all the first scenes as the interpretation results, comprises: Acquire the vertex coordinate information according to the identification information; According to the target interpretation model and the target scene, the area enclosed by all the vertex coordinate information is interpreted to obtain the target result; The Jsonify function is called to convert the target result into JSON data, and the JSON data is used as the interpretation result.

9. A remote sensing interpretation system, characterized in that: The system includes a client and a server; The client is used to load a Tiandi Map image on a base map, and load a remote sensing image to be interpreted on a first layer, wherein the remote sensing image is aligned with the Tiandi Map image according to geographic coordinates, and the first layer is above the base map; determine a target interpretation area in the remote sensing image, and generate location information indicating the target interpretation area; obtain parameters to be interpreted of the target interpretation area, and send the location information and the parameters to be interpreted to a server; wherein the parameters to be interpreted are used to indicate a target scene and a target interpretation model for interpreting the target interpretation area; obtain an interpretation result of the target interpretation area based on the location information, the target scene and the target interpretation model from a server, and display the interpretation result on a second layer, wherein the second layer is above the first layer; The server is used to obtain the location information of the target interpretation area and the parameters to be interpreted of the target interpretation area from the client; interpret the target interpretation area according to the location information, the target scene and the target interpretation model to obtain an interpretation result; and send the interpretation result to the client so that the client displays the interpretation result on the second layer.

10. A remote sensing interpretation device, characterized in that: Applied to a client, the device comprises: A loading module, used for loading a Tiandi Map image on a base map, and loading a remote sensing image to be interpreted on a first layer, wherein the remote sensing image and the Tiandi Map image are aligned according to geographic coordinates, and the first layer is above the base map; A generating module, used for determining a target interpretation area in the remote sensing image, and generating position information indicating the target interpretation area; A first acquisition module is used to acquire the parameters to be interpreted of the target interpretation area, and send the location information and the parameters to be interpreted to the server; wherein the parameters to be interpreted are used to indicate the target scene and target interpretation model for interpreting the target interpretation area; The display module is used to obtain an interpretation result of the target interpretation area based on the position information, the target scene and the target interpretation model from the server, and display the interpretation result on a second layer, where the second layer is above the first layer.

11. A remote sensing interpretation device, characterized in that: Applied to a server, the device comprises: A second acquisition module is used to acquire the location information of the target interpretation area and the parameters to be interpreted of the target interpretation area from the client; wherein the parameters to be interpreted are used to indicate the target scene and target interpretation model for interpreting the target interpretation area; An interpretation module, used to interpret the target interpretation area according to the location information, the target scene and the target interpretation model to obtain an interpretation result; A sending module is used to send the interpretation result to the client so that the client displays the interpretation result on a second layer, wherein the second layer is above the first layer, the first layer is above the base map, the base map is loaded with a Tiandi map image, the first layer is loaded with a remote sensing image to be interpreted, and the target interpretation area is determined in the remote sensing image.

12. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images