Three-dimensional POI matching method based on remote sensing and digital twinning and storage medium

Abstract

The invention discloses a three-dimensional POI matching method based on remote sensing and digital twinning and a storage medium, and the method comprises the steps: building a twinning model sample library according to three-dimensional point-of-interest models in a digital twinning scene, and setting labels for the three-dimensional point-of-interest models, the labels including categories, colors and shapes; performing target identification in the remote sensing image to obtain a target object area and a label of a target object; according to the label of the target object, performing retrieval matching in the twin model sample library to obtain a three-dimensional interest point model corresponding to the target object; and placing the three-dimensional interest point model corresponding to the target object in the target object area. According to the invention, automatic matching and placement of the three-dimensional interest point model of the remote sensing image and the digital twinborn scene can be realized.

Description

Technical Field

[0001] The present invention relates to the field of map processing technology, and in particular to a three-dimensional POI matching method and storage medium based on remote sensing and digital twins. Background Art

[0002] POI (Point of Interest) is a common map indication function that explains a certain place on the map, such as Figure 1 As shown, Figure 1 The POI logo of a hospital on a map is shown in 2D text. With the development of digital twins, more realistic 3D models can be placed on the map, such as Figure 2 As shown, you can even interact with the model and enter its internal state to view the real state and real-time data, such as Figure 3 Therefore, converting digital models of buildings, equipment, etc. in the twin world into 3D POI has become a new solution for POI display.

[0003] However, the digital twin scene is large, especially the metallurgical plant digital twin scene, with a large number of equipment and a complex environment. It takes a long time to manually establish the positional relationship between the digital twin model and the remote sensing image. In addition, the digital twin scene is manually modeled, and some models have problems such as inaccurate placement, inconsistent scaling, and human negligence. It is difficult to directly place the 3D model in the digital twin scene into the satellite map as a POI through traditional coordinate conversion. Therefore, an automatic matching method is needed to place 3D POIs. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a three-dimensional POI matching method and storage medium based on remote sensing and digital twins, which can realize automatic matching and placement of three-dimensional point of interest models of remote sensing images and digital twin scenes.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a three-dimensional POI matching method based on remote sensing and digital twins, comprising: According to the three-dimensional interest point models in the digital twin scene, a twin model sample library is established, and labels are set for each three-dimensional interest point model, wherein the labels include category, color and shape; Perform target recognition in remote sensing images to obtain the target area and target label; According to the label of the target object, searching and matching is performed in the twin model sample library to obtain a three-dimensional interest point model corresponding to the target object; The three-dimensional interest point model corresponding to the target object is placed in the target object area.

[0006] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0007] The beneficial effects of the present invention are as follows: through target recognition, the target area (i.e., the location of the point of interest) in the remote sensing image is identified, and the corresponding three-dimensional point of interest model is obtained based on the label matching of the target object. The text label matching method can reduce the amount of calculation and improve the matching efficiency. There is no need to consider the problem of coordinate conversion, and the accuracy requirements for the digital twin scene are relatively low, while the matching accuracy can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A schematic diagram of a 2D POI mark on a map in the prior art; Figure 2 It is a schematic diagram of a 3D model on a map in the prior art; Figure 3 It is an internal schematic diagram of a 3D model on a map in the prior art; Figure 4 It is a flow chart of a three-dimensional POI matching method based on remote sensing and digital twins of the present invention; Figure 5 This is a flow chart of a method according to Embodiment 1 of the present invention; Figure 6 A top view of a three-dimensional point of interest model and a schematic diagram of its naming according to the first embodiment of the present invention; Figure 7 This is a schematic diagram of target recognition results in Embodiment 1 of the present invention; Figure 8 This is a schematic diagram of a newly added three-dimensional point of interest model for the digital twin scene of the first embodiment of the present invention; Fig. 9 It is a schematic diagram of a top view of layers with different transparency according to the first embodiment of the present invention; Fig.10 FIG. 4 is a schematic diagram of image fusion according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0009] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0010] Please refer to Figure 4 , a three-dimensional POI matching method based on remote sensing and digital twins, comprising: According to the three-dimensional interest point models in the digital twin scene, a twin model sample library is established, and labels are set for each three-dimensional interest point model, wherein the labels include category, color and shape; Perform target recognition in remote sensing images to obtain the target area and target label; According to the label of the target object, searching and matching is performed in the twin model sample library to obtain a three-dimensional interest point model corresponding to the target object; The three-dimensional interest point model corresponding to the target object is placed in the target object area.

[0011] From the above description, it can be seen that the beneficial effects of the present invention are: it can realize automatic matching and placement of three-dimensional point of interest models of remote sensing images and digital twin scenes, thereby improving matching efficiency and matching accuracy.

[0012] Furthermore, the step of respectively setting a label for each 3D interest point model includes: The top view of each 3D interest point model is captured respectively, and the top view of each 3D interest point model is named according to a preset naming rule, wherein the preset naming rule includes: naming according to the category, color, shape and unique index of the 3D interest point model.

[0013] From the above description, it can be seen that by setting labels, subsequent matching is facilitated.

[0014] Furthermore, the target recognition in the remote sensing image to obtain the target area and the label of the target includes: The preset target recognition neural network model is used to perform target recognition on the preset area in the remote sensing image to obtain the target area and the category and shape of the target, and the color of the target is determined based on the mode of each pixel value in the target area.

[0015] Furthermore, the target recognition in the remote sensing image includes: Perform target recognition on the entire preset area in the remote sensing image; Dividing the preset area to obtain two or more small areas; Each small area is enlarged in turn, and the target is identified in the enlarged small area.

[0016] From the above description, it can be seen that the target recognition is first performed on the entire preset area, so that targets with a relatively large area can be identified, and then the small area is enlarged, and the target recognition is performed on the high-definition, small-field-of-view remote sensing image, so that targets with a smaller area can be identified.

[0017] Furthermore, searching and matching in the twin model sample library according to the label of the target object to obtain a three-dimensional interest point model corresponding to the target object includes: Retrieve and match the label of the target object with the label of each three-dimensional interest point model in the twin model sample library; If there is only one 3D interest point model matching the target object in the twin model sample library, the one 3D interest point model is used as the 3D interest point model corresponding to the target object; If more than two three-dimensional interest point models are retrieved in the twin model sample library and match the target object, the similarity between the target object area and the two or more three-dimensional interest point models is calculated respectively, and the three-dimensional interest point model corresponding to the maximum similarity is used as the three-dimensional interest point model corresponding to the target object.

[0018] From the above description, it can be seen that on the basis of label text matching, combined with image similarity matching, the matching speed can be greatly improved and the amount of calculation can be reduced. In addition, the text label + image similarity joint matching method does not need to consider the problem of coordinate transformation, which can ensure the matching accuracy.

[0019] Furthermore, it also includes: Calculate the coordinate transformation relationship between the digital twin scene and the remote sensing image; If a new three-dimensional interest point model is added to the digital twin scene, the corresponding position coordinates of the new three-dimensional interest point model in the remote sensing image are calculated based on the position coordinates of the new three-dimensional interest point model in the digital twin scene and the coordinate conversion relationship, and the new three-dimensional interest point model is placed at the corresponding position coordinates in the remote sensing image.

[0020] From the above description, it can be seen that when a new three-dimensional point of interest model is added to the digital twin scene, the remote sensing POI is reversely updated with the digital twin as a reference, which solves the problem of inconsistent update cycles between remote sensing and digital twin.

[0021] Furthermore, before the newly added three-dimensional interest point model is placed at the corresponding position coordinates in the remote sensing image, it further includes: In the digital twin scene, capture the top view of the newly added 3D POI model and overlay it to the corresponding position coordinates in the remote sensing image.

[0022] Furthermore, after intercepting the top view of the newly added three-dimensional point of interest model in the digital twin scene, the method further includes: Different transparency layers are set for the top view of the newly added three-dimensional point of interest model.

[0023] From the above description, it can be seen that it is convenient to choose to view the top view with different transparency to observe the changes of the target area.

[0024] Furthermore, the calculation of the coordinate transformation relationship between the digital twin scene and the remote sensing image includes: Select three targets in the remote sensing image, and obtain three-dimensional interest point models corresponding to the three targets in the digital twin scene; According to the position coordinates of the three targets in the remote sensing image and the position coordinates of their corresponding three-dimensional interest point models in the digital twin scene, the coordinate transformation relationship between the digital twin scene and the remote sensing image is calculated.

[0025] From the above description, it can be seen that the calculation of the coordinate transformation relationship is achieved through radial transformation.

[0026] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0027] Embodiment 1 Please refer to Figure 5-10 , Embodiment 1 of the present invention is: a three-dimensional POI matching method based on remote sensing and digital twins. Figure 5 As shown, the following steps are included: S1: According to the 3D interest point models in the digital twin scene, a twin model sample library is established, and labels are set for each 3D interest point model respectively.

[0028] The labels include categories, colors, and shapes, as well as unique indexes.

[0029] Specifically, obtain the 3D POI model in the digital twin scene and add it to the twin model sample library. At the same time, capture the top view of the 3D POI model and name it with “category-color-shape-unique index”, such as Figure 6 As shown in FIG. 1 , color refers to the color that appears most frequently in the top view of the 3D interest point model, and shape refers to the shape of the 3D interest point model in the top view.

[0030] S2: Perform target recognition in remote sensing images to obtain the target area and target label.

[0031] Specifically, by performing target recognition through a preset target recognition neural network model, the target object area as well as the category and shape of the target object can be output, and then the mode of each pixel value in the target object area, that is, the color with the highest frequency of occurrence, is calculated as the color of the target object.

[0032] In this embodiment, the entire preset area (such as a mining area or a metal smelter) in the remote sensing image can be firstly identified to obtain a target with a relatively large area, and then the preset area can be divided to obtain a plurality of small areas, and each small area can be enlarged in turn to obtain a remote sensing image with a high definition and a small field of view, so as to identify a target with a relatively small area. Figure 7 shown.

[0033] S3: According to the category and shape of the target object, search and match in the twin model sample library to obtain the three-dimensional interest point model corresponding to the target object.

[0034] Specifically, the labels of the target object are matched with the labels of each 3D interest point model in the twin model sample library. If only one 3D interest point model is retrieved to match the target object, that is, a unique result is retrieved, then the 3D interest point model is directly used as the 3D interest point model corresponding to the target object. If more than two 3D interest point models are retrieved to match the target object, that is, multiple results are retrieved, then the target object area is compared with all the retrieval results for similarity, and the 3D interest point model with the greatest similarity is used as the 3D interest point model corresponding to the target object.

[0035] In some optional embodiments, the perceptual hashing algorithm (pHash) is used to calculate the hash values ​​of the top view of the target object area and the 3D interest point model, and then the similarity is calculated by the hash values. In other optional embodiments, the similarity is calculated by the structural similarity index (SSIM).

[0036] If no results are retrieved, it is assumed that the area has changed, the digital twin scene has been updated, and the matching is completed.

[0037] S4: placing the three-dimensional interest point model corresponding to the target object into the target object area.

[0038] The above are the basic operations of remote sensing image matching. However, the update cycle of map remote sensing images is long, and the update speed is obviously behind the digital twin scene (digital twin scene can be updated in real time). In addition, the resolution of remote sensing images is limited. Even if the remote sensing image is enlarged, some smaller objects are difficult to identify through the model. Whether it is a slow update cycle or failure to identify the target object, it is a case of mismatch between remote sensing and digital twins. Therefore, it is necessary to solve the problem of mismatch between the update cycle of digital twin scenes and remote sensing images.

[0039] Through the above steps, the digital twin model that matches the target recognition result has been placed at the corresponding position of the remote sensing image, and a 3D POI has been established. When a 3D point of interest model is added to the subsequent digital twin scene, or a tiny object in the remote sensing image is not recognized (that is, it exists in the digital twin but not in the remote sensing image), perform the following steps: S5: Calculate the coordinate transformation relationship between the digital twin scene and the remote sensing image.

[0040] Specifically, three target objects are selected in the remote sensing image, and these three target objects have corresponding three-dimensional interest point models in the digital twin scene. Then, based on the position coordinates of the three target objects in the remote sensing image and the position coordinates of their corresponding three-dimensional interest point models in the digital twin scene, the coordinate transformation relationship from the digital twin scene to the remote sensing image is calculated, that is, the affine transformation is calculated.

[0041] S6: When a new 3D point of interest model is added to the digital twin scene, the remote sensing image patch is updated.

[0042] Specifically, when a new 3D POI model is added to the digital twin scene, e.g. Figure 8 As shown in the figure, a new factory building is added to the digital twin scene. Then, a top view of the newly added 3D point of interest model is captured in the digital twin scene, and different transparency layers are set for the top view, such as Fig. 9 As shown, you can choose to view the top view with different transparency levels later to observe the changes in the target area.

[0043] Then, according to the position coordinates of the newly added three-dimensional interest point model in the digital twin scene and the coordinate conversion relationship, the corresponding position coordinates of the newly added three-dimensional interest point model in the remote sensing image are calculated.

[0044] Next, the top view of the newly added 3D interest point model is overlaid on the corresponding position coordinates in the remote sensing image to perform image fusion, such as Fig.10 shown.

[0045] Finally, the newly added 3D point of interest model is placed at the corresponding position coordinates in the remote sensing image to form the final 3D POI, completing the entire matching process.

[0046] Furthermore, for targets that are not identified in the remote sensing image (such as tiny targets), they are considered not to exist, and the three-dimensional interest point models corresponding to these targets in the digital twin scene are considered to be newly added. According to step S6, the corresponding three-dimensional interest point models can be placed at the corresponding positions of the remote sensing image.

[0047] This embodiment uses a two-stage retrieval method that combines text label matching of remote sensing image recognition results with image similarity retrieval to perform remote sensing-digital twin model matching, which greatly improves the retrieval speed and reduces the amount of calculation; it fully considers the problem of inconsistent update cycles between remote sensing and digital twins, and reversely updates remote sensing POIs with digital twins as a reference, giving full play to the advantage of the fast update frequency of digital twins.

[0048] Embodiment 2 The present embodiment is a computer-readable storage medium corresponding to the above embodiment, on which a computer program is stored. When the program is executed by the processor, the various steps of a three-dimensional POI matching method based on remote sensing and digital twins in the above embodiment are implemented, and the same technical effect can be achieved, which will not be repeated here.

[0049] In summary, the present invention provides a three-dimensional POI matching method and storage medium based on remote sensing and digital twins, which matches remote sensing-digital twin models through a two-stage retrieval method that combines text label matching of remote sensing image recognition results with image similarity retrieval, greatly improving the retrieval speed and reducing the amount of calculation; the text label + image similarity joint matching method does not need to consider the problem of coordinate conversion, has low requirements for the accuracy of the digital twin scene, and can ensure matching accuracy. When a new three-dimensional point of interest model is added to the digital twin scene, the remote sensing POI is updated in reverse with the digital twin as a reference, which solves the problem of inconsistent update cycles between remote sensing and digital twins, and gives full play to the advantage of the fast update frequency of digital twins.

[0050] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A three-dimensional POI matching method based on remote sensing and digital twins, characterized in that: include: According to the three-dimensional interest point models in the digital twin scene, a twin model sample library is established, and labels are set for each three-dimensional interest point model, wherein the labels include category, color and shape; Perform target recognition in remote sensing images to obtain the target area and target label; According to the label of the target object, searching and matching is performed in the twin model sample library to obtain a three-dimensional interest point model corresponding to the target object; The three-dimensional interest point model corresponding to the target object is placed in the target object area.

2. The three-dimensional POI matching method according to claim 1, characterized in that: The steps of respectively setting a label for each 3D interest point model include: The top view of each 3D interest point model is captured respectively, and the top view of each 3D interest point model is named according to a preset naming rule, wherein the preset naming rule includes: naming according to the category, color, shape and unique index of the 3D interest point model.

3. The three-dimensional POI matching method according to claim 1, characterized in that: The target recognition in the remote sensing image to obtain the target area and the label of the target includes: The preset target recognition neural network model is used to perform target recognition on the preset area in the remote sensing image to obtain the target area and the category and shape of the target, and the color of the target is determined based on the mode of each pixel value in the target area.

4. The three-dimensional POI matching method according to claim 1, characterized in that: The target recognition in the remote sensing image includes: Perform target recognition on the entire preset area in the remote sensing image; Dividing the preset area to obtain two or more small areas; Each small area is enlarged in turn, and the target is identified in the enlarged small area.

5. The three-dimensional POI matching method according to claim 1, characterized in that: The step of searching and matching in the twin model sample library according to the label of the target object to obtain a three-dimensional interest point model corresponding to the target object includes: Retrieving and matching the label of the target object with the label of each three-dimensional interest point model in the twin model sample library; If there is only one 3D interest point model matching the target object in the twin model sample library, the one 3D interest point model is used as the 3D interest point model corresponding to the target object; If more than two three-dimensional interest point models are retrieved in the twin model sample library and match the target object, the similarity between the target object area and the two or more three-dimensional interest point models is calculated respectively, and the three-dimensional interest point model corresponding to the maximum similarity is used as the three-dimensional interest point model corresponding to the target object.

6. The three-dimensional POI matching method according to claim 1, characterized in that: Also includes: Calculate the coordinate transformation relationship between the digital twin scene and the remote sensing image; If a new three-dimensional interest point model is added to the digital twin scene, the corresponding position coordinates of the new three-dimensional interest point model in the remote sensing image are calculated based on the position coordinates of the new three-dimensional interest point model in the digital twin scene and the coordinate conversion relationship, and the new three-dimensional interest point model is placed at the corresponding position coordinates in the remote sensing image.

7. The three-dimensional POI matching method according to claim 6, characterized in that: Before the newly added three-dimensional interest point model is placed at the corresponding position coordinates in the remote sensing image, the method further includes: In the digital twin scene, capture the top view of the newly added 3D POI model and overlay it to the corresponding position coordinates in the remote sensing image.

8. The three-dimensional POI matching method according to claim 7, characterized in that: After intercepting the top view of the newly added three-dimensional point of interest model in the digital twin scene, the method further includes: Different transparency layers are set for the top view of the newly added three-dimensional point of interest model.

9. The three-dimensional POI matching method according to claim 6, characterized in that: The calculation of the coordinate transformation relationship between the digital twin scene and the remote sensing image includes: Select three targets in the remote sensing image, and obtain three-dimensional interest point models corresponding to the three targets in the digital twin scene; According to the position coordinates of the three targets in the remote sensing image and the position coordinates of their corresponding three-dimensional interest point models in the digital twin scene, the coordinate transformation relationship between the digital twin scene and the remote sensing image is calculated.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images