Digital twin image and real image registration method, device and readable storage medium

Abstract

The present invention provides a method, device and readable storage medium for aligning digital twin images with real images. The present invention is based on Hough transform and has the characteristics of strong resistance to illumination and distortion. Since the first reference straight line of the real image and the second reference straight line in the digital twin image are used as references for alignment, and no image transformation, correction and other processing are required, the alignment accuracy is high, so that the generated fused image is clearer, meeting business needs.

Description

Technical Field

[0001] The present invention belongs to the technical field of image registration, and in particular relates to a method, device and readable storage medium for registering a digital twin image with a real image. Background Art

[0002] With the rapid development of new-generation information technology represented by big data, the Internet of Things and artificial intelligence, the digital upgrade of traditional industries is imminent. In the digital transformation process of natural gas operating companies, in order to realize the visualization of LNG stations, virtual digital twin images are generally constructed by three-dimensional modeling. In order to fully display the details of the real scene, it is often necessary to align and fuse the digital twin images with the real images.

[0003] Since the acquisition of remote sensing images of LNG stations is easily affected by natural factors such as weather changes and light intensity, the real images obtained are often blurry. Therefore, it is difficult to find the registration reference features when using the grayscale-based image registration algorithm to match the digital twin image of the LNG station with the real image, resulting in the generated fusion image being often blurry and difficult to meet business needs. The frequency-domain-based image registration algorithm usually uses the properties of image translation, rotation, and scaling in the frequency domain to solve the transformation parameters to achieve accurate matching of the two images. However, the texture detail features of the digital twin image of the station are largely missing. After image transformation and correction, it is easy to introduce noise and errors, which reduces the accuracy of image registration and the generated fusion image is blurry. Summary of the invention

[0004] Based on this, in order to solve the above technical problems, a method, device and readable storage medium for aligning a digital twin image with a real image are provided.

[0005] The technical solution adopted by the present invention is as follows:

[0006] On the one hand, a method for registering a digital twin image with a real image is provided, comprising:

[0007] S101, performing edge detection on a real image to generate a binary edge map of the real image;

[0008] S102, performing Hough transform on edge points in the binary edge map to identify multiple straight lines in the real image;

[0009] S103, detecting a target area in the real image, taking a straight line among the multiple straight lines that coincides with an edge of the detected target area as a first reference straight line, and finding a second reference straight line corresponding to the first reference straight line from a corresponding target area of ​​the digital twin image;

[0010] S104, selecting corner points of the target area on the first reference straight line and the second reference straight line as key feature points respectively;

[0011] S105, scaling the digital twin image so that it has the same proportion as the real image;

[0012] S106: Overlap key feature points on the first reference straight line with key feature points on the second reference straight line, so that the first reference straight line overlaps with the second reference straight line, and the registration is completed.

[0013] On the other hand, an electronic device is provided, comprising a storage module, wherein the storage module comprises instructions loaded and executed by a processor, wherein when the instructions are executed, the processor executes the above-mentioned method for aligning a digital twin image with a real image.

[0014] On the other hand, a computer-readable storage medium is provided, which stores one or more programs, and when the one or more programs are executed by a processor, the above-mentioned method for aligning a digital twin image with a real image is implemented.

[0015] The present invention is based on Hough transform and has the characteristics of strong resistance to illumination and distortion. Since the first reference straight line of the real image and the second reference straight line in the digital twin image are used as references for registration, and no image transformation, correction and other processing are required, the registration accuracy is high, making the generated fused image clearer and meeting business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0017] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0018] like Figure 1 As shown, the embodiment of this specification provides a method for registering a digital twin image with a real image, including:

[0019] S101 . Perform edge detection on a real image to generate a binary edge map of the real image.

[0020] In this embodiment, edge detection is performed on the real image by using the Canny operator. Since the edge features detected by the Canny operator are usually very complete and the edge feature details are clear, it helps to improve the accuracy of the subsequent Hough transform.

[0021] S102, performing Hough transform on edge points in the binary edge map to identify multiple straight lines in the real image.

[0022] S103: Detect a target area in the real image, take a straight line among multiple straight lines that coincides with an edge of the detected target area as a first reference straight line, and find a second reference straight line corresponding to the first reference straight line from a corresponding target area of ​​the digital twin image.

[0023] The target area refers to an area used as a reference. For example, for an LNG terminal image, the unloading position (rectangular area) therein can be used as the target area.

[0024] The detection of the target area is achieved through a trained detection model. Preferably, the detection model adopts the CornerNet model. The CornerNet model detects the target area by detecting a pair of key points (upper left corner point and lower right corner point). Traditional detection models, whether first-order or second-order detectors, are basically based on anchors for classification and regression, but there are two problems with anchors: first, there are a large number of anchors, but in fact only a small part of them have a high overlap with gtboxes, which leads to problems such as imbalanced positive and negative samples and slow network convergence. Secondly, many anchor-related hyperparameters are introduced, such as quantity, size, aspect ratio, etc., and the reasonable choice of hyperparameters will directly affect the performance of the network. The CornerNet model eliminates the anchor mechanism, and at the same time avoids the above-mentioned problems, and has a higher detection efficiency.

[0025] To determine the first reference straight line, the four corner points of the target area can be determined based on the detected target area, and then a straight line passing through two of the corner points among multiple straight lines can be used as the first reference straight line, such as a straight line passing through the upper left corner point and the lower left corner point as the first reference straight line.

[0026] To determine the second reference straight line, we can first determine the edge where the target area overlaps with the first reference straight line based on the two corner points through which the first reference straight line passes, and then find the corresponding edge of the corresponding target area in the digital twin image based on the edge, and finally use the corresponding edge as the second reference straight line.

[0027] Assuming that the first reference straight line passes through the upper left corner point and the lower left corner point, it can be determined that among the four edges of the target area, the left edge coincides with the first reference straight line. In this way, the corresponding left edge can be found from the corresponding target area of ​​the digital twin image.

[0028] For the digital twin image, the coordinates of the four corner points of the target area are known, so it is easy to find the second reference line.

[0029] S104 , selecting corner points of the target area on the first reference straight line and the second reference straight line as key feature points respectively.

[0030] S105. Scale the digital twin image so that it has the same proportion as the real image.

[0031] S106: Overlap key feature points on the first reference straight line with key feature points on the second reference straight line, so that the first reference straight line overlaps with the second reference straight line, and the registration is completed.

[0032] In order to facilitate the coincidence of the first reference straight line and the second reference straight line, in S104, two corner points of the target area are respectively selected on the first reference straight line and the second reference straight line as key feature points.

[0033] Based on the same inventive concept, an embodiment of the present specification also provides an electronic device, including a storage module, the storage module including instructions loaded and executed by a processor, and when the instructions are executed, the processor executes the steps of various exemplary embodiments of the present invention described in the above-mentioned method for aligning a digital twin image with a real image in the present specification.

[0034] The storage module may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) and / or a cache memory unit, and may further include a read-only memory unit (ROM).

[0035] Based on the same inventive concept, an embodiment of this specification also provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by a processor, they implement the steps of various exemplary embodiments of the present invention described in the above-mentioned method for aligning a digital twin image with a real image.

[0036] Computer readable storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination of the above. More specific examples of computer readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0037] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0038] However, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. As long as they are within the spirit of the present invention, any changes or modifications to the above embodiments will fall within the scope of the claims of the present invention.

Claims

1. A method for registering a digital twin image with a real image, characterized in that: include: S101, performing edge detection on a real image to generate a binary edge map of the real image; S102, performing Hough transform on edge points in the binary edge map to identify multiple straight lines in the real image; S103, detecting a target area in the real image, taking a straight line among the plurality of straight lines that coincides with an edge of the detected target area as a first reference straight line, and finding a second reference straight line corresponding to the first reference straight line from a corresponding target area of ​​the digital twin image: Detecting the target area in the real image using a trained detection model; According to the detected target area, four corner points of the target area are determined; Using a straight line passing through two corner points among the plurality of straight lines as a first reference straight line; Determining, according to two corner points passed by the first reference straight line, an edge where the target area overlaps with the first reference straight line; According to the edge, finding a corresponding edge of a corresponding target area in the digital twin image; Using the corresponding edge as a second reference straight line; S104, selecting corner points of the target area on the first reference straight line and the second reference straight line as key feature points respectively; S105, scaling the digital twin image so that it has the same proportion as the real image; S106: Overlap key feature points on the first reference straight line with key feature points on the second reference straight line, so that the first reference straight line overlaps with the second reference straight line, and the registration is completed.

2. According to claim 1, a method for registering a digital twin image with a real image is characterized in that: The S101 performs edge detection on the real image by using a canny operator to generate a binary edge map of the real image.

3. According to claim 1, a method for registering a digital twin image with a real image is characterized in that: The detection model adopts the CornerNet model.

4. According to claim 1, a method for registering a digital twin image with a real image is characterized in that: The S104 further includes: Two corner points of the target area are selected on the first reference straight line and the second reference straight line as key feature points respectively.

5. An electronic device, characterized in that: It includes a storage module, which includes instructions loaded and executed by a processor, and when the instructions are executed, the processor executes a method for aligning a digital twin image with a real image according to any one of claims 1 to 4.

6. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed by the processor, a method for aligning a digital twin image with a real image as described in any one of claims 1 to 4 is implemented.

Images