Drainage pipeline digital twinning system and method

Abstract

The embodiment of the invention provides a drainage pipeline digital twinning system and method. The drainage pipeline digital twinning system comprises a data acquisition module, a data analysis module, a time sequence marking module and a human-computer interaction interface which are connected in sequence, the data acquisition module is used for acquiring fisheye images through the fisheye robot; determining a plane image corresponding to the fisheye image according to the fisheye image; the data analysis module is used for inputting the plane image into the fault detection model and outputting a fault position and a fault type corresponding to the plane image; the fault detection model is obtained by training based on a fault plane image carrying a preset pipeline fault type in advance; the time sequence marking module is used for adding timestamps to the fault types and the fault positions to obtain time sequence fault information; and the man-machine interaction interface is used for dynamically displaying the time sequence fault information through a pre-carried drainage pipe network three-dimensional model. According to the system, the drainage pipeline digital twinning system is established, pipeline defect positions and defect types are automatically detected through the model, and the detection effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of urban public infrastructure monitoring, and in particular to a drainage pipe digital twin system and method. Background Art

[0002] At present, fault detection of drainage pipe networks involves many pipelines and facilities. Both troubleshooting and management require operators to carry professional equipment for field operations, which is very inconvenient. Furthermore, the existing technology mostly uses manual troubleshooting and paper records to monitor and manage drainage pipe networks, but this method is costly and inefficient, and the monitoring effect cannot be effectively demonstrated. Summary of the invention

[0003] The purpose of the present invention is to provide a digital twin system and method for drainage pipes, so as to alleviate the technical problems existing in the prior art that urban drainage pipes have high costs and low efficiency, and the monitoring effects cannot be effectively displayed, thereby improving the monitoring efficiency and displaying the monitoring effects.

[0004] In the first aspect, an embodiment of the present invention provides a drainage pipe digital twin system, comprising: a data acquisition module, a data analysis module, a timing marking module and a human-computer interaction interface connected in sequence; the data acquisition module is used to acquire fisheye images through a fisheye robot; based on the fisheye image, a plane image corresponding to the fisheye image is determined; the data analysis module is used to input the plane image into a preset fault detection model, and output the fault location and fault type corresponding to the plane image; the fault detection model is pre-trained based on a fault plane image carrying a preset pipeline fault type; the timing marking module is used to add timestamps to the fault type and the fault location to obtain timing fault information; the human-computer interaction interface is used to dynamically display the timing fault information through a pre-installed three-dimensional model of the drainage network.

[0005] In a preferred embodiment of the present invention, the step of determining the plane image corresponding to the fisheye image according to the fisheye image comprises: establishing a spatial rectangular coordinate system on the fisheye image with the center point of the fisheye image as the origin; the Z axis of the spatial rectangular coordinate system is parallel to the plane normal of the plane image and passes through the origin, and the distance between the origin and the center of the fisheye image is a preset distance; based on the spatial rectangular coordinate system, calculating the pixel coordinates of the fisheye image corresponding to the pixel coordinates of the plane image; wherein the pixel coordinates of the fisheye image corresponding to the pixel coordinates of the plane image are calculated based on the following formula:

[0006]

[0007] Among them, xp is the X-axis pixel coordinate of the plane image, y p is the Y-axis pixel coordinate of the plane image, x m is the X-axis pixel coordinate of the fisheye image, y m is the Y-axis pixel coordinate of the fisheye image, d is the above-mentioned preset distance, x0 is the X-axis coordinate of the above-mentioned origin, and y0 is the Y-axis coordinate of the above-mentioned origin.

[0008] In a preferred embodiment of the present invention, the step of determining the plane image corresponding to the fisheye image according to the fisheye image comprises: dividing the fisheye image into a plurality of fisheye sub-images based on preset parameters; calculating the first Euclidean distance between the image center and any point in the undistorted image of the plane image and calculating the second Euclidean distance between the image center and any point in the distorted image of the plane image based on a preset image distortion model; wherein the expression of the image distortion model is: Among them, d c is the angle of incident light of the above fisheye image, θ d is the angle of the incident light after passing through the lens of the fisheye robot, λ i is the i-th distortion parameter, is the number of basic distortion parameters, f is the camera focal length of the above fisheye robot, is a high-order distortion parameter, M is the number of high-order distortion parameters, and ϵ represents a regularization term; according to the first Euclidean distance and the second Euclidean distance, coordinate transformation parameters of the undistorted image and the distorted image are determined; based on the coordinate transformation parameters, the coordinates between the distorted image and the undistorted image are transformed to determine a planar image.

[0009] In a preferred embodiment of the present invention, the step of determining the planar image corresponding to the fisheye image according to the fisheye image comprises: step 1, dividing the fisheye image into a plurality of fisheye sub-images based on preset parameters; step 2, dividing the plurality of fisheye sub-images into a first preset distortion degree, a second preset distortion degree, a third preset distortion degree and a fourth preset distortion degree according to corresponding distortion degrees; the difference between the first preset distortion degree and the second preset distortion degree is less than a preset value; the difference between the third preset distortion degree and the fourth preset distortion degree is less than the preset value; step 3, expressing the first preset distortion degree, the second preset distortion degree, the third preset distortion degree and the fourth preset distortion degree as the first preset distortion degree, the second preset distortion degree, the third preset distortion degree and the fourth preset distortion degree respectively. A first eigenvector, a second eigenvector, a third eigenvector and a fourth eigenvector; step 4, based on a preset loss function and the first eigenvector, the second eigenvector, the third eigenvector and the fourth eigenvector, calculate the sum of the loss values ​​of the above multiple fisheye sub-images; step 5, determine whether the sum of the loss values ​​is greater than a preset threshold; step 6, if greater than, use a reverse flow field to de-warp the above fisheye image, estimate the full-scale flow, and obtain a corrected image through a deformation operation of bilinear sampling; step 7, calculate the sum of the loss values ​​of the above corrected image; step 8, repeat the above steps 1 to 7 until the sum of the loss values ​​of the above corrected image is less than the preset threshold; determine a planar image based on the above corrected image.

[0010] In a preferred embodiment of the present invention, the fisheye image is dewarped using an inverse flow field, and the full-scale flow is estimated, and the step of obtaining a corrected image through a deformation operation of bilinear sampling includes: calculating the corrected image based on the following formula: Jc(i,j)=Jd(fx(i,j), fy(i,j)) wherein Jc is the corrected image, Jd is the distorted image, (i, j) is the integer pixel coordinate in the corrected image, and (fx(i,j), fy(i,j)) is the decimal pixel coordinate in the distorted image.

[0011] In a preferred embodiment of the present invention, based on a preset loss function and the first eigenvector, the second eigenvector, the third eigenvector and the fourth eigenvector, the step of calculating the sum of the loss values ​​of the above-mentioned multiple fisheye sub-images includes: setting the above-mentioned first eigenvector and the above-mentioned second eigenvector as positive examples of each other, setting the above-mentioned third eigenvector and the above-mentioned fourth eigenvector as positive examples of each other; setting the above-mentioned first eigenvector and the above-mentioned third eigenvector and the first eigenvector and the above-mentioned fourth eigenvector as negative examples of each other; setting the above-mentioned second eigenvector and the above-mentioned third eigenvector and the second eigenvector and the above-mentioned fourth eigenvector as negative examples of each other; calculating the loss values ​​of the above-mentioned multiple fisheye sub-images by the following formula:

[0012]

[0013]

[0014] Among them, Loss is the above loss value, i∈R 1×D is Er∈R N×D The feature vector of the i-th fisheye sub-image in N i is the number of index images of the positive example, including the image index with a distortion degree less than the preset value with the i-th fisheye sub-image, τ is a preset hyperparameter, Representation and i is the sum of the similarities of the fisheye sub-images of the image index corresponding to the positive example; i, j represents any fisheye sub-image, Representation and The sum of similarities of the fisheye sub-images corresponding to the negative examples; i, j represent any fisheye sub-image; ; in, For the above The sum of the loss values ​​of the above fisheye images.

[0015] In a preferred embodiment of the present invention, the above-mentioned first eigenvector is [0.2, 0.9]; the above-mentioned second eigenvector is [0.3, 0.8]; the above-mentioned third eigenvector is [0.8, 0.3]; the above-mentioned fourth eigenvector is [0.9, 0.2]; and the sum of the loss values ​​of the above-mentioned multiple fisheye sub-images is 3.592.

[0016] In a preferred embodiment of the present invention, the steps of using an inverse flow field to dewarp the above-mentioned fisheye image, estimating the full-scale flow, and obtaining the corrected image through a deformation operation of bilinear sampling include: introducing an upsampling module of a deep learning algorithm; based on the above-mentioned upsampling module, using an inverse flow field to dewarp the above-mentioned fisheye image, estimating the full-scale flow, and obtaining the corrected image through a deformation operation of bilinear sampling.

[0017] In the second aspect, an embodiment of the present invention also provides a drainage pipe digital twin method, which is applied to the above-mentioned drainage pipe digital twin system, and the above-mentioned method includes: obtaining a fisheye image through a fisheye robot; determining a plane image corresponding to the above-mentioned fisheye image based on the above-mentioned fisheye image; inputting the above-mentioned plane image into a preset fault detection model, and outputting the fault location and fault type corresponding to the above-mentioned plane image; the above-mentioned fault detection model is pre-trained based on a fault plane image carrying a preset pipeline fault type; adding a timestamp to the above-mentioned fault type and the above-mentioned fault location to obtain time series fault information; and dynamically displaying the above-mentioned time series fault information through a pre-installed three-dimensional model of the drainage network.

[0018] The embodiments of the present invention have the following beneficial technical effects: The embodiment of the present invention provides a drainage pipeline digital twin system and method, including: a data acquisition module, a data analysis module, a timing marking module and a human-computer interaction interface connected in sequence; the data acquisition module is used to obtain fisheye images through a fisheye robot; according to the fisheye image, the plane image corresponding to the fisheye image is determined; the data analysis module is used to input the plane image into a preset fault detection model, and output the fault position and fault type corresponding to the plane image; the fault detection model is pre-trained based on the fault plane image carrying the preset pipeline fault type; the timing marking module is used to add timestamps to the fault type and the fault position to obtain timing fault information; the human-computer interaction interface is used to dynamically display the timing fault information through a pre-loaded three-dimensional model of the drainage network. The system establishes a drainage pipeline digital twin system, on the one hand, automatically detects the pipeline defect position and defect type through the model, and on the other hand, displays the above defect situation in real time to improve the effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of the structure of a drainage pipe digital twin system provided by an embodiment of the present invention; Figure 2 A schematic diagram of coordinate transformation between a fisheye image and a plane image provided by an embodiment of the present invention; Figure 3 A structural schematic diagram of a drainage pipe digital twin method provided in an embodiment of the present invention.

[0021] Icons: 11-data acquisition module; 12-data analysis module; 13-time sequence marking module; 14-human-computer interaction interface; 21-plane image; 22-fisheye image; 23-incident light. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0023] The existing technology mostly uses manual inspection and paper records to monitor and manage the drainage network, but this method is costly and inefficient, and the monitoring effect cannot be effectively demonstrated.

[0024] Based on this, an embodiment of the present invention provides a drainage pipeline digital twin system and method. The system establishes a drainage pipeline digital twin system. On the one hand, the system automatically detects the location and type of pipeline defects through a model, and on the other hand, it displays the above defects in real time to improve the effect. For ease of understanding, a drainage pipeline digital twin system is first introduced.

[0025] Example 1 In this embodiment, Figure 1 A schematic structural diagram of a drainage pipe digital twin system provided in an embodiment of the present invention.

[0026] Depend on Figure 1 As can be seen, the system includes: a data acquisition module 11, a data analysis module 12, a timing marking module 13 and a human-computer interaction interface 14 connected in sequence; the data acquisition module 11 is used to obtain fisheye images through a fisheye robot; according to the fisheye image, a plane image corresponding to the fisheye image is determined; the data analysis module 12 is used to input the plane image into a preset fault detection model, and output the fault position and fault type corresponding to the plane image; the fault detection model is pre-trained based on a fault plane image carrying a preset pipeline fault type; the timing marking module 13 is used to add timestamps to the fault type and the fault position to obtain timing fault information; the human-computer interaction interface 14 is used to dynamically display the timing fault information through a pre-installed three-dimensional model of the drainage network.

[0027] In this embodiment, the step of determining the plane image corresponding to the fisheye image according to the fisheye image includes: Taking the center point of the fisheye image as the origin, a spatial rectangular coordinate system is established on the fisheye image; the Z axis of the spatial rectangular coordinate system is parallel to the plane normal of the plane image and passes through the origin, and the distance between the origin and the center of the fisheye image is a preset distance; based on the spatial rectangular coordinate system, the pixel coordinates of the fisheye image corresponding to the pixel coordinates of the plane image are calculated; The pixel coordinates of the fisheye image corresponding to the pixel coordinates of the plane image are calculated based on the following formula:

[0028]

[0029] Among them, x p is the X-axis pixel coordinate of the plane image, y p is the Y-axis pixel coordinate of the plane image, x m is the X-axis pixel coordinate of the fisheye image, y m is the Y-axis pixel coordinate of the fisheye image, d is the above-mentioned preset distance, x0 is the X-axis coordinate of the above-mentioned origin, and y0 is the Y-axis coordinate of the above-mentioned origin; The planar image is determined according to the pixel coordinates of the planar image.

[0030] For ease of understanding, Figure 2 A schematic diagram of coordinate transformation between a fisheye image and a plane image provided by an embodiment of the present invention. Figure 2 As can be seen, the plane image 21 is in the xy plane, and the bottom of the fisheye image 22 is located at the x C Y C plane, the incident light 23 passes through the fisheye image 22 and the plane image 21 in sequence.

[0031] Furthermore, the present embodiment also converts the fisheye image into a plane image in a second manner, that is, according to the fisheye image, a step of determining the plane image corresponding to the fisheye image includes: dividing the fisheye image into a plurality of fisheye sub-images based on preset parameters; calculating a first Euclidean distance between an image center and an arbitrary point in an undistorted image of the plane image and calculating a second Euclidean distance between an image center and an arbitrary point in a distorted image of the plane image based on a preset image distortion model; Among them, the expression of the above image distortion model is:

[0032] Among them, d c is the angle of incident light of the above fisheye image, θ d is the angle of the incident light after passing through the lens of the fisheye robot, λ i is the i-th distortion parameter, is the number of basic distortion parameters, f is the camera focal length of the above fisheye robot, is the high-order distortion parameter, M is the number of high-order distortion parameters, represents the regularization term; According to the first Euclidean distance and the second Euclidean distance, coordinate transformation parameters of the undistorted image and the distorted image are determined; based on the coordinate transformation parameters, coordinates between the distorted image and the undistorted image are transformed to determine a planar image.

[0033] Example: Focal length of a fisheye image f =4mm, distortion parameters λ1=0.1, λ2=0.05, λ3=0.01, λ4=0.005. For a point near the center of the image, assume that its coordinates in the undistorted image are (0,0) and its coordinates in the distorted image are (rd, θd).

[0034] First, calculate the coordinates in the undistorted image: Assumption d c = 10 mm, θ d =30 (0.5 radians).

[0035] According to the distortion model, the Euclidean distance between the image center and any point in the undistorted image is calculated, that is, r c , r c ≈4×(1+0.5+0.01+0.005)𝑟 𝑐 ≈4×1.515𝑟 𝑐 ≈6.06mm.

[0036] Then, calculate the coordinates in the distorted image: Use inverse trigonometric calculations to solve for dc, d 𝑐 = d 𝑐 = ≈1.2675 radians≈73.3.

[0037] At this time, the coordinate transformation between the distorted image and the undistorted image can be obtained.

[0038] Furthermore, the present embodiment also converts the fisheye image into a plane image in a third manner, that is, according to the fisheye image, a step of determining the plane image corresponding to the fisheye image includes: step 1, dividing the fisheye image into a plurality of fisheye sub-images based on preset parameters; step 2, dividing the plurality of fisheye sub-images into a first preset distortion degree, a second preset distortion degree, a third preset distortion degree and a fourth preset distortion degree according to corresponding distortion degrees; the difference between the first preset distortion degree and the second preset distortion degree is less than a preset value; the difference between the third preset distortion degree and the fourth preset distortion degree is less than the preset value; step 3, dividing the first preset distortion degree, the second preset distortion degree, the third preset distortion degree and the fourth preset distortion degree The degree of deformation is respectively represented by the first eigenvector, the second eigenvector, the third eigenvector and the fourth eigenvector; step 4, based on the preset loss function and the first eigenvector, the second eigenvector, the third eigenvector and the fourth eigenvector, calculate the sum of the loss values ​​of the above-mentioned multiple fisheye sub-images; step 5, determine whether the sum of the above-mentioned loss values ​​is greater than the preset threshold; step 6, if greater than, use the reverse flow field to de-distort the above-mentioned fisheye image, and estimate the full-scale flow, and obtain the corrected image through the deformation operation of bilinear sampling; step 7, calculate the sum of the loss values ​​of the above-mentioned corrected image; step 8, repeat the above steps 1 to 7 until the sum of the loss values ​​of the above-mentioned corrected image is less than the above-mentioned preset threshold; according to the above-mentioned corrected image, determine the plane image.

[0039] In this embodiment, the steps of using the reverse flow field to de-distort the fisheye image, estimating the full-scale flow, and obtaining the corrected image through the deformation operation of bilinear sampling include: calculating the corrected image based on the following formula: Jc(i,j)=Jd(fx(i,j),fy(i,j)) Wherein, Jc is the above-mentioned corrected image, Jd is the above-mentioned distorted image, (i, j) is the integer pixel coordinate in the corrected image, and (fx(i,j), fy(i,j)) is the decimal pixel coordinate in the above-mentioned distorted image.

[0040] Further, based on a preset loss function and the first eigenvector, the second eigenvector, the third eigenvector and the fourth eigenvector, the step of calculating the sum of the loss values ​​of the multiple fisheye sub-images includes: setting the first eigenvector and the second eigenvector as positive examples of each other, setting the third eigenvector and the fourth eigenvector as positive examples of each other; setting the first eigenvector and the third eigenvector and the first eigenvector and the fourth eigenvector as negative examples of each other; setting the second eigenvector and the third eigenvector and the second eigenvector and the fourth eigenvector as negative examples of each other; The loss values ​​of the above multiple fisheye sub-images are calculated by the following formula:

[0041]

[0042]

[0043] Among them, Loss is the above loss value, i∈R 1×D is Er∈R N×D The feature vector of the i-th fisheye sub-image in N i is the number of index images of the positive example, including the image index with a distortion degree less than the preset value with the i-th fisheye sub-image, τ is a preset hyperparameter, Representation and i is the sum of the similarities of the fisheye sub-images of the image index corresponding to the positive example; i, j represents any fisheye sub-image, Representation and The sum of similarities of the fisheye sub-images corresponding to the negative examples; i, j represent any fisheye sub-image; ; in, For the above The sum of the loss values ​​of the above fisheye images.

[0044] Furthermore, the steps of using an inverse flow field to dewarp the above-mentioned fisheye image, estimating the full-scale flow, and obtaining a corrected image through a deformation operation of bilinear sampling include: introducing an upsampling module of a deep learning algorithm; based on the above-mentioned upsampling module, using an inverse flow field to dewarp the above-mentioned fisheye image, estimating the full-scale flow, and obtaining a corrected image through a deformation operation of bilinear sampling.

[0045] In this embodiment, the first eigenvector is [0.2, 0.9]; the second eigenvector is [0.3, 0.8]; the third eigenvector is [0.8, 0.3]; the fourth eigenvector is [0.9, 0.2]; and the sum of the loss values ​​of the multiple fisheye sub-images is 3.592.

[0046] For ease of understanding, the present application uses the following example for explanation: assuming that a fisheye image is divided into four fisheye sub-images, the feature vectors corresponding to the fisheye sub-images are represented as P1~P4 respectively, these fisheye sub-images come from different areas of the fisheye image, and each fisheye sub-image has different degrees of distortion, and then contrast loss is used to help the model learn to distinguish these fisheye sub-images with different degrees of distortion.

[0047] Furthermore, P1 and P2 have similar distortion levels, P3 and P4 have similar distortion levels, but P1, P2 have different distortion levels from P3, P4.

[0048] Furthermore, for P1, the positive example is P2, and the negative examples are P3 and P4; for P2, the positive example is P1, and the negative examples are P3 and P4; for P3, the positive example is P4, and the negative examples are P1 and P2; for P4, the positive example is P3, and the negative examples are P1 and P2; temperature parameter: τ=0.1; feature representation example: p1=[0.2,0.9], p2=[0.3,0.8], p3=[0.8,0.3], p4=[0.9,0.2]; for the contrast loss of P1, that is, fs1; calculate the similarity between p1 and the positive example p2: exp(p1⋅p2 / τ); calculate the similarity between p1 and all negative examples: exp(p1p3 / τ)+exp(p1p4 / τ); we can get fs1=0.873. Similarly, we can get fs2=0.921, fs3=0.899, fs4=0.899. From this, we can get that the sum of the contrast losses of the four fisheye sub-images is 3.592.

[0049] In actual cases, it is usually necessary to set a threshold for the sum of image losses. Assume that the threshold is 3. If the calculated loss value is less than 3, it means that the loss of the image during the conversion process can be ignored. In this embodiment, the sum of the losses is 3.592, so the image needs to be corrected.

[0050] First, the fisheye image is dewarped using the reverse flow field, given by Er∈R N×D , estimate the full-scale flow fb∈R H×W×2 , with fb = (fx, fy), the rectified image Ic∈R is obtained by the deformation operation of bilinear sampling H×W ×3 , the calculation formula is as follows: Jc(i,j)=Jd(fx(i,j),fy(i,j)) Wherein, Jc is the above-mentioned corrected image, Jd is the above-mentioned distorted image, (i, j) is the integer pixel coordinate in the corrected image, and (fx(i,j), fy(i,j)) is the decimal pixel coordinate in the above-mentioned distorted image.

[0051] A learnable upsampling module is introduced. Er is first reshaped into shape H / P×W / P×D. Then, two convolutional layers produce a deformation flow fm∈R of scale 1 / P H / P × W / P ×2 Next, two more convolutional layers are used to predict a H / P×W / P×(P×P×9) mask and perform softmax on the weights of the 3×3 neighborhood of each pixel of fm. Finally, the obtained H / P×W / P×P×P×2 map is permuted and reshaped into the full-resolution deformation flow fb∈RH×W ×2 . Er∈R N×D : It represents a matrix, where N is the number of N fisheye sub-images into which the original image is segmented, and D is the feature dimension of each fisheye sub-image. It is a three-dimensional tensor representing the full-scale flow field. H and W are the height and width of the image, respectively. This flow field contains the displacement information of each pixel between the original distorted image and the corrected image. It represents an image after geometric correction. H and W are the height and width of the image, respectively. 3 is the number of channels of the image, which is usually used for color images. In a standard RGB color image, each pixel consists of the values ​​of three color channels, which represent R for red, G for green, and B for blue. Er represents the original image representation matrix, which has a shape of N×D, where N is the number of fisheye sub-images into which the image is divided, D is the feature dimension of each fisheye sub-image, and P represents the size of the fisheye sub-image, which is usually the side length of a square image. For example, P×P means that each image is a square with a width of P pixels and a height of P pixels. H / P and W / P represent the number of rows and columns of the fisheye sub-images into which the image is divided in the height and width directions, respectively. D remains unchanged during the reshaping process and still represents the feature dimension of each fisheye sub-image.

[0052] For ease of understanding, this application also lists the following detailed data for illustration: Assume that the original size of a fisheye image is H=600 pixels and W=800 pixels. Segment it into fisheye sub-images and set P=10. Then there will be a total of N(600 / 10)×(800 / 10)=60×80=4800 fisheye sub-images. The feature dimension D of each fisheye sub-image is assumed to be 128.

[0053] First, Er∈R 4800×128 , which represents the feature representation of 4800 fisheye sub-images into which the fisheye image is segmented.

[0054] Then, a series of operations are performed to estimate the full-scale flow fb∈R 600×800×2 This flow field will contain the displacement information of each pixel between the original distorted image and the corrected image.

[0055] Furthermore, for the corrected image Ic∈R 600×800×3, assuming there is a specific integer pixel coordinate (u, v), such as (100, 200). By calculating the formula Jc(i, j) = Jd(fx(i, j), fy(i, j)), the corresponding value can be obtained from the distorted image Jd according to the predicted fractional pixel coordinate (fx(100, 200), fy(100, 200)) to determine the pixel value of the corrected image at that coordinate. Here, fx and fy are determined by the full-scale flow calculated previously.

[0056] When introducing the learnable upsampling module, Er is first reshaped into an image of shape 60×80×128. Then, two convolutional layers are used to generate a deformation flow fm∈R of scale 1 / P. 60×80×2 Two more convolutional layers are then used to predict a 60×80×(10×10×9) mask, performing a softmax on the weights of the 3×3 neighborhood of each pixel in fm.

[0057] Finally, the obtained 60×80×10×10×2 map is permuted and reshaped into the full-resolution deformation flow fb∈R {600×800×2} .

[0058] Furthermore, the sum of the losses of the corrected image is calculated, and the above operation steps are repeated until the sum of the losses of the corrected image is less than a threshold.

[0059] The embodiment of the present invention provides a drainage pipeline digital twin system, including: a data acquisition module, a data analysis module, a timing marking module and a human-computer interaction interface connected in sequence; the data acquisition module is used to obtain fisheye images through a fisheye robot; according to the fisheye image, the plane image corresponding to the fisheye image is determined; the data analysis module is used to input the plane image into a preset fault detection model, and output the fault position and fault type corresponding to the plane image; the fault detection model is pre-trained based on the fault plane image carrying the preset pipeline fault type; the timing marking module is used to add timestamps to the fault type and the fault position to obtain timing fault information; the human-computer interaction interface is used to dynamically display the timing fault information through a pre-loaded three-dimensional model of the drainage network. The system establishes a drainage pipeline digital twin system, on the one hand, automatically detects the pipeline defect position and defect type through the model, and on the other hand, displays the above defect situation in real time to improve the effect.

[0060] Example 2 Based on the above embodiments, Figure 3 A structural schematic diagram of a drainage pipe digital twin method provided in an embodiment of the present invention.

[0061] Among them, the method is applied to the drainage pipe digital twin system in the above embodiment.

[0062] Depend on Figure 3 As can be seen, the method includes: Step S101: obtaining a fisheye image by a fisheye robot; and determining a plane image corresponding to the fisheye image according to the fisheye image.

[0063] Step S102: input the plane image into a preset fault detection model, and output the fault location and fault type corresponding to the plane image; the fault detection model is pre-trained based on a fault plane image carrying a preset pipeline fault type.

[0064] Step S103: adding a timestamp to the fault type and the fault location to obtain timing fault information.

[0065] Step S104: dynamically displaying the above-mentioned time series fault information through the pre-installed three-dimensional model of the drainage pipe network.

[0066] In this embodiment, the step of determining the plane image corresponding to the fisheye image according to the fisheye image includes: establishing a spatial rectangular coordinate system on the fisheye image with the center point of the fisheye image as the origin; the Z axis of the spatial rectangular coordinate system is parallel to the plane normal of the plane image and passes through the origin, and the distance between the origin and the center of the fisheye image is a preset distance; based on the spatial rectangular coordinate system, calculating the pixel coordinates of the fisheye image corresponding to the pixel coordinates of the plane image; wherein the pixel coordinates of the fisheye image corresponding to the pixel coordinates of the plane image are calculated based on the following formula:

[0067]

[0068] Among them, x p is the X-axis pixel coordinate of the plane image, y p is the Y-axis pixel coordinate of the plane image, x m is the X-axis pixel coordinate of the fisheye image, y m is the Y-axis pixel coordinate of the fisheye image, d is the above-mentioned preset distance, x0 is the X-axis coordinate of the above-mentioned origin, and y0 is the Y-axis coordinate of the above-mentioned origin; the plane image is determined according to the pixel coordinates of the above-mentioned plane image.

[0069] Wherein, the step of determining the plane image corresponding to the fisheye image according to the fisheye image comprises: dividing the fisheye image into a plurality of fisheye sub-images based on preset parameters; calculating a first Euclidean distance between an image center and an arbitrary point in an undistorted image of the plane image and calculating a second Euclidean distance between an image center and an arbitrary point in a distorted image of the plane image based on a preset image distortion model; Wherein, the expression of the image distortion model is:

[0070] Among them, d c is the angle of incident light of the fisheye image, θ d is the angle of the incident light after passing through the lens of the fisheye robot, λ i is the i-th distortion parameter, is the number of basic distortion parameters, f is the camera focal length of the fisheye robot, is the high-order distortion parameter, M is the number of high-order distortion parameters, represents a regularization term; determining coordinate transformation parameters of the undistorted image and the distorted image according to the first Euclidean distance and the second Euclidean distance; based on the coordinate transformation parameters, transforming the coordinates between the distorted image and the undistorted image to determine a planar image.

[0071] Further, according to the above-mentioned fisheye image, the step of determining the plane image corresponding to the above-mentioned fisheye image includes: step 1, dividing the fisheye image into a plurality of fisheye sub-images based on preset parameters; step 2, dividing the plurality of fisheye sub-images into a first preset distortion degree, a second preset distortion degree, a third preset distortion degree and a fourth preset distortion degree according to corresponding distortion degrees; the difference between the first preset distortion degree and the second preset distortion degree is less than a preset value; the difference between the third preset distortion degree and the fourth preset distortion degree is less than the preset value; step 3, respectively representing the first preset distortion degree, the second preset distortion degree, the third preset distortion degree and the fourth preset distortion degree as first eigenvectors The method comprises the following steps: step 4, calculating the sum of the loss values ​​of the plurality of fisheye sub-images based on a preset loss function and the first eigenvector, the second eigenvector, the third eigenvector and the fourth eigenvector; step 5, judging whether the sum of the loss values ​​is greater than a preset threshold; step 6, if greater than, anti-warping the fisheye image using a reverse flow field, estimating the full-scale flow, and obtaining a corrected image through a deformation operation of bilinear sampling; step 7, calculating the sum of the loss values ​​of the corrected image; step 8, repeating steps 1 to 7 until the sum of the loss values ​​of the corrected image is less than the preset threshold; and determining a planar image based on the corrected image.

[0072] Furthermore, the steps of using the reverse flow field to de-distort the fisheye image and estimate the full-scale flow, and obtaining the corrected image through the deformation operation of bilinear sampling include: calculating the corrected image based on the following formula: Jc(i,j)=Jd(fx(i,j),fy(i,j)) Wherein, Jc is the corrected image, Jd is the distorted image, (i, j) is the integer pixel coordinate in the corrected image, and (fx(i, j), fy(i, j)) is the decimal pixel coordinate in the distorted image.

[0073] Further, based on a preset loss function and the first eigenvector, the second eigenvector, the third eigenvector and the fourth eigenvector, the step of calculating the sum of the loss values ​​of the plurality of fisheye sub-images comprises: The first eigenvector and the second eigenvector are set as positive examples of each other, and the third eigenvector and the fourth eigenvector are set as positive examples of each other; The first eigenvector and the third eigenvector, and the first eigenvector and the fourth eigenvector are set as negative examples of each other; The second eigenvector and the third eigenvector, as well as the second eigenvector and the fourth eigenvector are set as negative examples of each other; The loss values ​​of the multiple fisheye sub-images are calculated by the following formula:

[0074]

[0075]

[0076] Among them, Loss is the loss value, i∈R 1×D is Er∈R N×D The feature vector of the i-th fisheye sub-image in N i is the number of index images of the positive example, including the image index having a distortion degree less than the preset value with the i-th fisheye sub-image, τ is a preset hyperparameter, Representation and i is the sum of the similarities of the fisheye sub-images of the image index corresponding to the positive example; i, j represents any fisheye sub-image, Representation and The sum of similarities of the fisheye sub-images corresponding to the negative examples; i, j represent any fisheye sub-image; ; Among them, among them, For the above The sum of the loss values ​​of the above fisheye images.

[0077] Among them, the above-mentioned first eigenvector is [0.2, 0.9]; the above-mentioned second eigenvector is [0.3, 0.8]; the above-mentioned third eigenvector is [0.8, 0.3]; the above-mentioned fourth eigenvector is [0.9, 0.2]; and the sum of the loss values ​​of the above-mentioned multiple fisheye sub-images is 3.592.

[0078] Furthermore, the steps of using an inverse flow field to dewarp the above-mentioned fisheye image, estimating the full-scale flow, and obtaining a corrected image through a deformation operation of bilinear sampling include: introducing an upsampling module of a deep learning algorithm; based on the above-mentioned upsampling module, using an inverse flow field to dewarp the above-mentioned fisheye image, estimating the full-scale flow, and obtaining a corrected image through a deformation operation of bilinear sampling.

[0079] The drainage pipe digital twin device provided in the embodiment of the present invention has the same technical features as the drainage pipe digital twin device method provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the above method embodiment, and will not be repeated here.

Claims

1. A drainage pipeline digital twin system, characterized in that: include: A data acquisition module, a data analysis module, a time series marking module and a human-computer interaction interface connected in sequence; The data acquisition module is used to acquire a fisheye image through a fisheye robot; and determine a plane image corresponding to the fisheye image according to the fisheye image; The data analysis module is used to input the plane image into a preset fault detection model, and output the fault location and fault type corresponding to the plane image; The fault detection model is pre-trained based on a fault plane image carrying a preset pipeline fault type; The timing marking module is used to add a timestamp to the fault type and the fault location to obtain timing fault information; The human-computer interaction interface is used to dynamically display the timing fault information through a pre-installed three-dimensional model of the drainage network.

2. The drainage pipeline digital twin system according to claim 1, characterized in that: The step of determining a planar image corresponding to the fisheye image according to the fisheye image comprises: Taking the center point of the fisheye image as the origin, a spatial rectangular coordinate system is established on the fisheye image; the Z axis of the spatial rectangular coordinate system is parallel to the plane normal of the plane image and passes through the origin, and the distance between the origin and the center of the fisheye image is a preset distance; based on the spatial rectangular coordinate system, the pixel coordinates of the fisheye image corresponding to the pixel coordinates of the plane image are calculated; The pixel coordinates of the fisheye image corresponding to the pixel coordinates of the plane image are calculated based on the following formula: Among them, x p is the X-axis pixel coordinate of the plane image, y p is the Y-axis pixel coordinate of the plane image, x m is the X-axis pixel coordinate of the fisheye image, y m is the Y-axis pixel coordinate of the fisheye image, d is the preset distance, x0 is the X-axis coordinate of the origin, and y0 is the Y-axis coordinate of the origin; A planar image is determined according to the pixel coordinates of the planar image.

3. The drainage pipeline digital twin system according to claim 1, characterized in that: The step of determining a planar image corresponding to the fisheye image according to the fisheye image comprises: Dividing the fisheye image into a plurality of fisheye sub-images based on preset parameters; Based on a preset image distortion model, calculating a first Euclidean distance between an image center and an arbitrary point in an undistorted image of the planar image, and calculating a second Euclidean distance between an image center and an arbitrary point in a distorted image of the planar image; Wherein, the expression of the image distortion model is: Among them, d c is the angle of incident light of the fisheye image, θ d is the angle of the incident light after passing through the lens of the fisheye robot, λ i is the i-th distortion parameter, is the number of basic distortion parameters, f is the camera focal length of the fisheye robot, is the high-order distortion parameter, M is the number of high-order distortion parameters, represents the regularization term; determining coordinate transformation parameters of the undistorted image and the distorted image according to the first Euclidean distance and the second Euclidean distance; Based on the coordinate conversion parameters, the coordinates between the distorted image and the undistorted image are converted to determine a planar image.

4. The drainage pipeline digital twin system according to claim 3 is characterized in that: The step of determining a planar image corresponding to the fisheye image according to the fisheye image comprises: Step 1, dividing the fisheye image into a plurality of fisheye sub-images based on preset parameters; Step 2: dividing the multiple fisheye sub-images into a first preset distortion degree, a second preset distortion degree, a third preset distortion degree, and a fourth preset distortion degree according to corresponding distortion degrees; a difference between the first preset distortion degree and the second preset distortion degree is less than a preset value; a difference between the third preset distortion degree and the fourth preset distortion degree is less than the preset value; Step 3, expressing the first preset distortion degree, the second preset distortion degree, the third preset distortion degree, and the fourth preset distortion degree as a first eigenvector, a second eigenvector, a third eigenvector, and a fourth eigenvector, respectively; Step 4, calculating the sum of loss values ​​of the multiple fisheye sub-images based on a preset loss function and the first eigenvector, the second eigenvector, the third eigenvector, and the fourth eigenvector; Step 5, determining whether the sum of the loss values ​​is greater than a preset threshold; Step 6: if it is greater than, the fisheye image is dewarped using an inverse flow field, and the full-scale flow is estimated, and a corrected image is obtained by a deformation operation of bilinear sampling; Step 7, calculating the total loss value of the corrected image; Step 8, repeating step 1 to step 7 until the sum of the loss values ​​of the corrected image is less than the preset threshold; A planar image is determined based on the corrected image.

5. The drainage pipeline digital twin system according to claim 4, characterized in that: The steps of using an inverse flow field to de-distort the fisheye image, estimating the full-scale flow, and obtaining a corrected image through a deformation operation of bilinear sampling include: The corrected image is calculated based on the following formula: Jc(i,j)=Jd(fx(i,j),fy(i,j)) Wherein, Jc is the corrected image, Jd is the distorted image, (i, j) is the integer pixel coordinate in the corrected image, and (fx(i, j), fy(i, j)) is the decimal pixel coordinate in the distorted image.

6. The drainage pipeline digital twin system according to claim 5, characterized in that: The step of calculating the sum of the loss values ​​of the plurality of fisheye sub-images based on a preset loss function and the first eigenvector, the second eigenvector, the third eigenvector, and the fourth eigenvector comprises: The first eigenvector and the second eigenvector are set as positive examples of each other, and the third eigenvector and the fourth eigenvector are set as positive examples of each other; The first eigenvector and the third eigenvector, and the first eigenvector and the fourth eigenvector are set as negative examples of each other; The second eigenvector and the third eigenvector, as well as the second eigenvector and the fourth eigenvector are set as negative examples of each other; The loss values ​​of the multiple fisheye sub-images are calculated by the following formula: Among them, Loss is the loss value, i∈R 1×D is Er∈R N×D The feature vector of the i-th fisheye sub-image in N i is the number of index images of the positive example, including the image index having a distortion degree less than the preset value with the i-th fisheye sub-image, τ is a preset hyperparameter, Representation and i is the sum of the similarities of the fisheye sub-images of the image index corresponding to the positive example; i, j represents any fisheye sub-image, Representation and The sum of similarities of the fisheye sub-images corresponding to the negative examples; i, j represent any fisheye sub-image; ； in, For the The sum of the loss values ​​of the fisheye images.

7. The drainage pipeline digital twin system according to claim 6, characterized in that: The first eigenvector is [0.2, 0.9]; the second eigenvector is [0.3, 0.8]; the third eigenvector is [0.8, 0.3]; the fourth eigenvector is [0.9, 0.2]; The total loss values ​​of the multiple fisheye sub-images is 3.

592.

8. The drainage pipeline digital twin system according to claim 7, characterized in that: The steps of using an inverse flow field to de-distort the fisheye image, estimating the full-scale flow, and obtaining a corrected image through a deformation operation of bilinear sampling include: Introducing the upsampling module of the deep learning algorithm; Based on the upsampling module, the fisheye image is dewarped by using an inverse flow field, and the full-scale flow is estimated, and a corrected image is obtained by a deformation operation of bilinear sampling.

9. A drainage pipeline digital twin method, characterized in that: Applied to the drainage pipe digital twin system according to any one of claims 1 to 8, the method comprising: Acquire a fisheye image by a fisheye robot; and determine a plane image corresponding to the fisheye image according to the fisheye image; Inputting the plane image into a preset fault detection model, and outputting the fault location and fault type corresponding to the plane image; the fault detection model is pre-trained based on a fault plane image carrying a preset pipeline fault type; Adding a timestamp to the fault type and the fault location to obtain timing fault information; The sequential fault information is dynamically displayed through a pre-installed three-dimensional model of the drainage pipe network.

Images