Tunnel disease repair scheme generation method and device

Abstract

The present disclosure relates to a tunnel disease repair scheme generation method and device, and relates to the technical field of repair scheme prediction. The method comprises: performing disease identification on a target tunnel area original image key frame based on a preset depth target detection model to obtain a disease identification result, and determining an actual occurrence position of an original disease according to the disease identification result; determining an original disease image associated with the original disease based on the actual occurrence position, and determining morphological mask information and disease point cloud data of the original disease based on the original disease image; determining three-dimensional disease geometric parameters of the original disease based on the morphological mask information and the disease point cloud data; determining a recommended repair category of the original disease based on the three-dimensional disease geometric parameters and a disease time sequence image sequence at multiple different time points, and generating a tunnel disease repair scheme based on the recommended repair category. The present disclosure improves the accuracy of the obtained repair scheme.

Description

Technical Field

[0001] This disclosure relates to the field of predictive technology for repair schemes, and more specifically, to a method and apparatus for generating tunnel defect repair schemes. Background Technology

[0002] In existing disease repair schemes, the corresponding network models focus on the static characteristics of the disease and do not pay enough attention to the dynamic development and evolution of the disease, resulting in low accuracy of the obtained disease repair schemes.

[0003] It should be noted that the information in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide a method and apparatus for generating tunnel defect repair schemes, thereby overcoming, to some extent, the problem of low accuracy of defect repair schemes caused by limitations and defects in related technologies.

[0005] According to one aspect of this disclosure, a method for generating a tunnel defect repair scheme is provided, comprising: performing defect identification on keyframes of an original image of a target tunnel region based on a preset depth target detection model to obtain defect identification results, and determining the actual location of the original defect based on the defect identification results; determining an original defect image associated with the original defect based on the actual location, and determining morphological mask information and defect point cloud data of the original defect based on the original defect image; determining three-dimensional defect geometric parameters of the original defect based on the morphological mask information and defect point cloud data, and determining a temporal image sequence of the original defect at multiple different times; determining a suggested repair category of the original defect based on the three-dimensional defect geometric parameters and the temporal image sequence of the defect at multiple different times, and generating a tunnel defect repair scheme based on the suggested repair category.

[0006] In an exemplary embodiment of this disclosure, a defect identification result is obtained by performing defect identification on keyframes of an original image of a target tunnel region based on a preset depth target detection model. This includes: acquiring original tunnel image data of the target tunnel region and extracting original image keyframes from the original tunnel image data; the original tunnel image data is obtained by photographing the tunnel surface of the target tunnel region; downsampling the original image keyframes based on a backbone feature extraction network to obtain multiple first local image features of different scales corresponding to the original image keyframes; fusing the multiple first local image features of different scales corresponding to the original image keyframes based on a feature fusion network to obtain first global image features corresponding to the original image keyframes; and performing object detection on the first global image features corresponding to the original image keyframes based on a target object detection network to obtain a defect identification result corresponding to the original image keyframes. The defect identification result includes a first defect detection box and a first defect category for the original defects included in the original image keyframes.

[0007] In an exemplary embodiment of this disclosure, determining the actual location of the original disease based on the disease identification result includes: inputting a first disease detection box in the original image keyframe of the first frame into a preset trajectory prediction model for model initialization processing to obtain a target trajectory prediction model, and predicting the trajectory prediction results of the first disease detection box in other frames of the original image keyframe based on the target trajectory prediction model; performing position matching on the trajectory prediction results of other frames and the first disease detection box of the original disease in other frames of the original image keyframe to obtain a position matching result, and determining the continuous trajectory of the original disease in the target tunnel region based on the position matching result; determining the original two-dimensional spatial coordinates based on the continuous trajectory of the disease, and determining the original acquisition timestamp of the original image keyframe corresponding to the original disease and the original camera pose parameters used to acquire the original image keyframe; and determining the actual location of the original disease in the target tunnel region based on the original two-dimensional spatial coordinates, the original acquisition timestamp, and the original camera pose parameters.

[0008] In an exemplary embodiment of this disclosure, determining the morphological mask information of the original disease based on the original disease image includes: performing original disease identification on the original disease image based on a preset deep target detection model to obtain a second disease detection box; performing area identification on the second disease detection box to select the detection box with the largest area from the second disease detection box as the disease reference image of the original disease; performing semantic segmentation on the disease reference image based on a preset deep semantic segmentation network to obtain a foreground image and a background image of the disease reference image, and determining the morphological mask information based on the foreground image.

[0009] In an exemplary embodiment of this disclosure, determining the point cloud data of the original disease based on the original disease image includes: determining the target camera intrinsic parameters of the target camera used to acquire the original disease image; the target camera intrinsic parameters include multiple parameters among focal length, principal point offset parameter, radial distortion parameter, and tangential distortion parameter; determining the target camera pose parameters of the original disease image based on the target camera intrinsic parameters and the original disease image using a preset multi-view geometric 3D reconstruction model; determining the depth map of the original disease image based on the target camera pose parameters using the preset multi-view geometric 3D reconstruction model, and fusing the depth maps to obtain the point cloud data of the original disease.

[0010] In an exemplary embodiment of this disclosure, determining the three-dimensional geometric parameters of the original disease based on the morphological mask information and the disease point cloud data includes: determining parameter extraction rules required to extract the two-dimensional geometric parameters of the disease from the morphological mask information according to the first disease category of the original disease, and extracting the two-dimensional geometric parameters of the disease based on the feature extraction rules; mapping the disease morphological feature points in the two-dimensional geometric parameters to the disease point cloud data to match the real three-dimensional coordinate points corresponding to the disease morphological feature points from the disease point cloud data; determining feature point pairs composed of a first two-dimensional feature point and a second two-dimensional feature point from the disease morphological feature points, and determining the two-dimensional pixel distance between the feature point pairs; determining the first three-dimensional coordinate point corresponding to the first two-dimensional feature point and the second three-dimensional coordinate point corresponding to the second two-dimensional feature point from the real three-dimensional coordinate points, and determining the three-dimensional pixel distance between the first three-dimensional coordinate point and the second three-dimensional coordinate point; determining the pixel equivalent based on the two-dimensional pixel distance and the three-dimensional pixel distance, and determining the three-dimensional geometric parameters of the original disease based on the pixel equivalent and the two-dimensional geometric parameters of the disease.

[0011] In an exemplary embodiment of this disclosure, the first defect category includes at least one of a lining crack defect category, a lining spalling defect category, and a water leakage defect category; the feature extraction rules include at least one of a first feature extraction rule corresponding to the lining crack defect category, a second feature extraction rule corresponding to the lining spalling defect category, and a third feature extraction rule corresponding to the water leakage defect category; wherein, extracting two-dimensional defect geometric parameters based on the feature extraction rules includes: extracting the main crack orientation of the original defect corresponding to the lining crack defect category from the morphological mask information based on the first feature extraction rule, and based on... The two-dimensional geometric parameters of the original disease are determined based on the direction of the main crack; the two-dimensional geometric parameters are the maximum width pixel value of the crack; and / or, based on the second feature extraction rule, the minimum bounding rectangle of the original disease corresponding to the lining spalling disease category is determined according to the morphological mask information, and the two-dimensional geometric parameters of the original disease are determined based on the minimum bounding rectangle; the two-dimensional geometric parameters are the spalling disease area; and / or, based on the third feature extraction rule, the two-dimensional geometric parameters of the original disease corresponding to the water leakage disease category are determined according to the morphological mask information; the two-dimensional geometric parameters are the water leakage area.

[0012] In one exemplary embodiment of this disclosure, determining the suggested repair category of the original disease based on the three-dimensional disease geometric parameters and a series of disease time-series images at multiple different times includes: determining the disease urgency of the original disease, and determining a multimodal tunnel disease decision model associated with the original disease based on the disease urgency; generating a first disease feature of the original disease in the semantic dimension based on the three-dimensional disease geometric parameters, and generating a second disease feature of the original disease in the image dimension based on a series of disease time-series images at multiple different times; and inputting the first disease feature and the second disease feature into the multimodal tunnel disease decision model to obtain the suggested repair category of the original disease.

[0013] In one exemplary embodiment of this disclosure, generating a tunnel defect repair scheme based on the suggested repair category includes: generating basic defect information associated with the original defect based on the suggested repair category, three-dimensional defect geometric parameters, and multiple temporal image sequences of the defect at different times; generating contextual information associated with the original defect based on preset model prompt information; and inputting the basic defect information and contextual information into a preset content generation model to obtain the tunnel defect repair scheme; wherein, the tunnel defect repair scheme includes multiple elements such as defect identification information, actual location, three-dimensional defect geometric parameters, defect change trend, defect repair decision suggestions, and corresponding image evidence.

[0014] According to one aspect of this disclosure, an apparatus for generating a tunnel defect repair scheme is provided, comprising: an actual occurrence location determination module, configured to perform defect identification on keyframes of an original image of a target tunnel area based on a preset depth target detection model to obtain defect identification results, and determine the actual occurrence location of the original defect based on the defect identification results; a morphological mask information determination module, configured to determine an original defect image associated with the original defect based on the actual occurrence location, and determine morphological mask information and defect point cloud data of the original defect based on the original defect image; a three-dimensional defect geometric parameter determination module, configured to determine the three-dimensional defect geometric parameters of the original defect based on the morphological mask information and defect point cloud data, and determine a time-series image sequence of the original defect at multiple different times; and a defect repair scheme determination module, configured to determine a suggested repair category of the original defect based on the three-dimensional defect geometric parameters and the time-series image sequence of the defect at multiple different times, and generate a tunnel defect repair scheme based on the suggested repair category.

[0015] This disclosure provides a method for generating a tunnel defect repair scheme. Firstly, it identifies defects in keyframes of the original image of the target tunnel area using a preset depth target detection model, obtaining defect identification results and determining the actual location of the original defect based on these results. Then, it determines the original defect image associated with the defect based on the actual location, and determines the morphological mask information and point cloud data of the original defect based on the original defect image. Next, it determines the three-dimensional geometric parameters of the original defect based on the morphological mask information and the point cloud data, and determines a temporal image sequence of the original defect at multiple different times. Finally, it determines a suggested repair method for the original defect based on the three-dimensional geometric parameters and the temporal image sequence of the defect at multiple different times. The system categorizes tunnel defects and generates repair schemes based on suggested repair categories. By considering time-series image sequences of defects at multiple different times during the repair scheme generation process, it addresses the problem of low accuracy in existing technologies due to insufficient attention to the dynamic evolution of defects, thus improving the accuracy of the obtained repair schemes. Furthermore, by integrating the temporal information of defect development into the repair decision-making process, the model can more comprehensively grasp the evolutionary patterns and development trends of defects, solving problems such as incomplete defect detection, unreasonable repair decisions, and low levels of intelligence in existing technologies. This allows for improved efficiency in generating repair schemes while ensuring their accuracy.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0018] Figure 1 The flowchart illustrates a method for generating a tunnel defect repair scheme according to an exemplary embodiment of the present disclosure.

[0019] Figure 2 The diagram schematically illustrates a structural example of a preset depth target detection model according to an exemplary embodiment of the present disclosure.

[0020] Figure 3 The diagram schematically illustrates an example structure of the backbone feature extraction network in a preset deep object detection model according to an example embodiment of the present disclosure.

[0021] Figure 4 The diagram schematically illustrates an example structure of a feature fusion network in a preset deep object detection model according to an exemplary embodiment of the present disclosure.

[0022] Figure 5 The diagram schematically illustrates an example structure of a pre-defined deep semantic segmentation network according to an exemplary embodiment of the present disclosure.

[0023] Figure 6 The diagram schematically illustrates a structural example of a multimodal tunnel defect decision-making model according to an exemplary embodiment of the present disclosure.

[0024] Figure 7 The illustration shows a scenario example of point cloud data of a disease associated with an original disease according to an example embodiment of the present disclosure.

[0025] Figure 8 The diagram schematically illustrates a scenario example of the main fracture trend direction obtained according to an exemplary embodiment of the present disclosure.

[0026] Figure 9 The diagram schematically illustrates a structural block diagram of an apparatus for generating a tunnel defect repair scheme according to an exemplary embodiment of the present disclosure.

[0027] Figure 10 An electronic device for generating a tunnel defect repair scheme is illustrated according to an example embodiment of the present disclosure. Detailed Implementation

[0028] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of these specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0029] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0030] As a critical node in transportation networks, tunnels are subject to frequent defects due to long-term exposure to vehicle vibrations and geological stresses during their service life, severely impacting their service life and operational safety. Therefore, achieving rapid identification, precise location, and scientific repair decisions for tunnel defects is crucial for ensuring the tunnel's performance throughout its entire lifecycle; this issue has also become a widely discussed research topic in the fields of civil engineering and intelligent inspection.

[0031] Existing disease identification and repair decision-making schemes are mainly divided into two categories. The first category relies on manual inspection. However, this method is inefficient and highly subjective, making it prone to missed detections in the dim and complex environment of tunnels. As a result, some diseases are often only discovered when they have developed to a more serious stage, which seriously damages the overall structure of the tunnel and affects its service life and operational safety. Furthermore, manual inspection requires a large investment of manpower and resources, which is costly and not conducive to the long-term sustainable maintenance and management of tunnels.

[0032] Furthermore, with the rapid development of computer technology and artificial intelligence, tunnel defect detection methods based on deep learning have become the mainstream research direction, thus giving rise to a second type of scheme for defect identification and repair decision-making. However, existing deep learning methods often focus on improving detection accuracy, neglecting the overall intelligent development of the system. As a result, the constructed models can usually only handle single types of problems and lack intelligent linkage between various links, leading to low accuracy of the generated repair decision schemes. In addition, existing network models focus more on the static characteristics of defects, but do not pay enough attention to the dynamic development and evolution of defects. This results in some defects that have become stable being over-repaired, while some rapidly expanding minor defects are not given timely attention, causing unnecessary waste of resources and economic losses.

[0033] This exemplary embodiment first provides a method for generating a tunnel defect repair scheme. This method can run on a server, server cluster, or cloud server, etc. Of course, those skilled in the art can also run the method disclosed herein on other platforms as needed, and this exemplary embodiment does not impose any special limitations on this. Specifically, refer to... Figure 1 As shown, the method for generating this tunnel defect repair plan may include the following steps: Step S110. Based on the depth target detection model, perform disease identification on the key frames of the original image of the target tunnel area to obtain the disease identification result, and determine the actual location of the original disease based on the disease identification result; Step S120. Determine the original disease image associated with the original disease based on the actual location of occurrence, and determine the morphological mask information and disease point cloud data of the original disease based on the original disease image; Step S130. Based on the morphological mask information and the disease point cloud data, determine the three-dimensional geometric parameters of the original disease, and determine the disease time-series image sequence of the original disease at multiple different times; Step S140. Based on the three-dimensional defect geometric parameters and the defect time-series image sequence at multiple different times, determine the suggested repair category of the original defect, and generate a tunnel defect repair scheme based on the suggested repair category.

[0034] In the aforementioned method for generating tunnel defect repair schemes, on the one hand, defect identification results are obtained by performing defect identification on keyframes of the original image of the target tunnel area based on a preset depth target detection model, and the actual location of the original defect is determined based on the defect identification results; then, based on the actual location, the original defect image associated with the original defect is determined, and the morphological mask information and defect point cloud data of the original defect are determined based on the original defect image; furthermore, based on the morphological mask information and defect point cloud data, the three-dimensional geometric parameters of the original defect are determined, and the temporal image sequence of the original defect at multiple different times is determined; finally, based on the three-dimensional geometric parameters of the defect and the temporal image sequence of the defect at multiple different times, the suggested repair category of the original defect is determined. The system generates tunnel repair schemes based on suggested repair categories. By considering time-series images of the tunnel defects at multiple different times during the repair scheme generation process, it addresses the problem of low accuracy in existing technologies due to insufficient attention to the dynamic evolution of defects, thus improving the accuracy of the obtained repair schemes. Furthermore, by integrating the temporal information of defect development into the repair decision-making process, the model can more comprehensively grasp the evolutionary patterns and development trends of defects, solving problems such as incomplete defect detection, unreasonable repair decisions, and low levels of intelligence in existing technologies. This allows for improved efficiency in generating repair schemes while ensuring their accuracy.

[0035] The following will provide a detailed explanation and description of the method for generating tunnel defect repair schemes as described in the exemplary embodiments of this disclosure, with reference to the accompanying drawings.

[0036] First, the technical implementation principles of the exemplary embodiments of this disclosure are explained and described. Specifically, the exemplary embodiments of this disclosure integrate deep learning methods with 3D reconstruction methods, employing multiple deep learning models to complete tasks such as locating, identifying, and segmenting defects, and combining image information and 3D point clouds to extract multidimensional geometric parameter information of the defects. Based on this, a temporal multimodal tunnel defect repair decision algorithm is further constructed, integrating the temporal information of defect development into the repair decision-making process. This allows the model to more comprehensively grasp the evolution law and development trend of defects, solving problems such as incomplete defect detection, unreasonable repair decisions, and low intelligence levels in existing technologies. It reduces the consumption of human and material resources in repair projects, significantly improves the service life and operational safety of tunnels, and realizes automated intelligent identification and repair decision-making for tunnel defects.

[0037] Secondly, the preset depth target detection model involved in the exemplary embodiments of this disclosure will be explained and described. Specifically, refer to... Figure 2As shown, the preset deep target detection model may include a backbone feature extraction network 210, a feature fusion network 220, and an object detection network 230. The roles of each model layer in the disease identification process will be detailed later and will not be elaborated further here.

[0038] In one example embodiment, reference is made to... Figure 3 As shown, the backbone feature extraction network described above may include a convolutional stem layer (Stem) 301, an original feature sampling layer (Layer0) 302, and M feature downsampling layers, where M is a positive integer. Specifically, taking M as an example, the backbone feature extraction network may include a first feature downsampling layer (Layer1) 303, a second feature downsampling layer (Layer2) 304, a third feature downsampling layer (Layer3) 305, and a fourth feature downsampling layer (Layer4) 306. The role of each model layer in the defect prediction process will be detailed later and will not be elaborated further here.

[0039] In one example embodiment, reference is made to... Figure 4 As shown, the feature fusion network described above can consist of an original feature fusion network 401 and M sub-feature fusion networks, where M is a positive integer greater than or equal to 3. The value of M described here corresponds one-to-one with the value of M in the feature downsampling layer. In practical applications, the specific value of M can be set according to actual needs, and this example does not impose any special restrictions on it. Specifically, taking a value of 4 for M as an example, the specific feature fusion network can include a first sub-feature fusion network 402, a second sub-feature fusion network 403, a third sub-feature fusion network 403, and a fourth sub-feature fusion network 404. The roles of each model layer in the defect prediction process will be detailed later and will not be elaborated further here.

[0040] The following will explain and describe the preset deep semantic segmentation network involved in the exemplary embodiments of this disclosure. Specifically, refer to... Figure 5 As shown, the preset deep semantic segmentation network may include multiple first convolutional layers 501, multiple first downsampling layers 502, and multiple first upsampling layers 503; wherein, the output of the last convolutional layer in the multiple first convolutional layers is connected to the input of the first first downsampling layer in the multiple first downsampling layers, and the output of the last first downsampling layer in the multiple first downsampling layers is connected to the input of the first first upsampling layer in the right-hand multiple first upsampling layers; and the number of layers in the multiple upsampling layers and the multiple downsampling layers is the same.

[0041] The following will explain and illustrate the multimodal tunnel defect decision-making model involved in the exemplary embodiments of this disclosure. Specifically, refer to... Figure 6 As shown, the multimodal tunnel defect decision-making model may include an image processing channel 601, a semantic processing channel 602, a feature splicing layer 603, a feature aggregation layer 604, and a classification layer 605; the role of each model layer in the process of determining the repair category will be detailed later, and will not be elaborated further here.

[0042] The following will combine Figures 2-6 right Figure 1 The method for generating the tunnel defect repair scheme shown in the illustration will be further explained and illustrated. Specifically: In step S110, based on a preset depth target detection model, the original image keyframes of the target tunnel area are used to identify defects to obtain defect identification results, and the actual location of the original defects is determined based on the defect identification results.

[0043] In the example embodiment of this disclosure, firstly, based on a preset depth target detection model, the original image keyframes of the target tunnel region are used to identify defects to obtain defect identification results. Specifically, this can be achieved as follows: acquiring the original tunnel image data of the target tunnel region and extracting the original image keyframes from the original tunnel image data; the original tunnel image data is obtained by photographing the tunnel surface of the target tunnel region; downsampling the original image keyframes based on a backbone feature extraction network to obtain multiple first local image features of different scales corresponding to the original image keyframes; fusing the multiple first local image features of different scales corresponding to the original image keyframes based on a feature fusion network to obtain first global image features corresponding to the original image keyframes; performing object detection on the first global image features corresponding to the original image keyframes based on a target object detection network to obtain defect identification results corresponding to the original image keyframes; the defect identification results include a first defect detection box and a first defect category included in the original image keyframes.

[0044] In one example embodiment, raw tunnel image data of the target tunnel area can be collected using an intelligent imaging vehicle. This intelligent imaging vehicle may comprise a vehicle power system, an electrical system, an onboard visual-inertial pose acquisition system, and a photography system. In practical engineering applications, the intelligent imaging vehicle should conduct regular tunnel lining inspections according to engineering requirements and perform multiple targeted shots for defects of varying urgency. During the shooting process, the vehicle should rationally plan its travel path based on the effective shooting angle of the photography equipment, completing a comprehensive shot of the current section before moving to the next section to continue collecting data. Simultaneously, the entire shooting process should maintain a uniform and stable speed, avoiding rapid movement. The camera equipment must scan the entire section to ensure that all locations of the tunnel lining are well-captured. Meanwhile, after completing the full-coverage shooting of the entire tunnel lining (i.e., the target tunnel area), the acquired videos can be converted into standardized video frames in chronological order to obtain the original tunnel image data. In addition, the obtained original tunnel image data contains a three-level association index of timestamp-pose-image frame. Based on this, it is convenient to index the corresponding information in subsequent data processing.

[0045] In one example embodiment, the extraction of original image keyframes from the original tunnel image data can be achieved as follows: the original tunnel image data is decoded into an RGB image sequence, and frames are uniformly extracted according to the timestamp (e.g., 1 frame is taken every 5 frames) to obtain the original image keyframes; this avoids redundancy; on this basis, illumination normalization, denoising (bilateral filtering), and geometric correction (distortion based on intrinsic parameters) are performed on each frame, and a standardized frame sequence including multiple original image keyframes is output. The naming rule for the original image keyframes is: Tunnel ID_Mileage_Timestamp.

[0046] In one example embodiment, the keyframes of the original image are downsampled based on a backbone feature extraction network to obtain multiple first local image features of different scales corresponding to the keyframes of the original image. This can be achieved as follows: basic features are extracted from the keyframes of the original image based on convolutional layers to obtain basic feature extraction results, and initial sampling is performed on the basic feature extraction results based on the original feature sampling layer to obtain initial feature sampling results; the initial feature sampling results are downsampled for the first time based on the first feature downsampling layer among the M feature downsampling layers to obtain first sub-local image features; the first sub-local image features are downsampled for the second time based on the second feature downsampling layer among the M feature downsampling layers to obtain second sub-local image features; the generation process of the second sub-local image features is repeated sequentially to obtain third sub-local image features, ..., Mth sub-local image features. Furthermore, taking a value of M of 4 as an example, the specific process for determining the first local image features at different scales is as follows: the first sub-local image features are obtained by downsampling the initial feature sampling results based on the first feature downsampling layer for the first time, and the second sub-local image features are obtained by downsampling the first sub-local image features based on the second feature downsampling layer for the second time; the third sub-local image features are obtained by downsampling the second sub-local image features based on the third feature downsampling layer for the third time, and the fourth sub-local image features are obtained by downsampling the third sub-local image features based on the fourth feature downsampling layer for the fourth time.

[0047] In one example embodiment, the first global image feature corresponding to the original image keyframe is obtained by fusing multiple first local image features of different scales based on a feature fusion network. This can be achieved as follows: the Mth sub-local image feature among the multiple first local image features of different scales corresponding to the original image keyframe is fused using a feature fusion network to obtain the original global feature; the original global feature and the (M-1)th sub-local image feature among the multiple first local image features of different scales are fused using the first sub-feature fusion network among the M sub-feature fusion networks to obtain the first sub-global feature; the first sub-global feature and the (M-2)th sub-local feature among the multiple first local image features of different scales are fused using the second feature fusion network among the M sub-feature fusion networks to obtain the second sub-global feature; the generation process of the second sub-global feature is repeated sequentially to obtain the third sub-global feature, ..., the Mth sub-global feature. Furthermore, taking a value of M of 4 as an example, the specific implementation process of fusing the first local image features at different scales is as follows: the original global features are obtained by fusing the fourth sub-local image features based on the original feature fusion network, and the first sub-global features are obtained by fusing the original global features and the third sub-local image features based on the first feature fusion network; the second sub-global features are obtained by fusing the first sub-global features and the second sub-local features based on the second feature fusion network, and the third sub-global features are obtained by fusing the second sub-global features and the first sub-local features based on the third feature fusion network; the fourth sub-global features are obtained by fusing the third sub-global features and the initial feature sampling results based on the fourth feature fusion network.

[0048] In one example embodiment, the lesion identification result corresponding to the original image keyframe is obtained by performing object detection on the first global image features corresponding to the original image keyframe based on the target object detection network. This can be achieved as follows: the first global image features are convolved based on a convolutional layer to obtain a first convolutional processing result, and the first convolutional processing result is classified based on an object category detection network to obtain a first lesion category corresponding to the original image keyframe; the first convolutional processing result is bounding box detected based on an object region detection network to obtain a first lesion detection box corresponding to the original lesion of the original image keyframe. Specifically, the convolutional layers described here can be multiple convolutional layers, and the specific number of convolutional layers can be determined according to actual needs. This example does not impose any special restrictions on this. At the same time, the number of convolutional layers can correspond one-to-one with the number of layers of global features described above, and the weights can be shared among the first convolutional layers.

[0049] It should be further clarified here that the first category of defects mentioned above may include, but is not limited to, common defect types such as lining cracks, lining spalling, and water leakage, and may also include other defect categories; this example does not impose any special restrictions on this. Furthermore, before image recognition, the original image keyframes can be compressed to a specified size according to network input requirements, and the trained depth target detection model can be used to detect defects in the image. Moreover, the training of the depth target detection model can be achieved using tunnel defect images that combine self-collected data with publicly available datasets.

[0050] Secondly, the actual location of the original disease is determined based on the disease identification results. Specifically, this can be achieved as follows: the first disease detection box in the first frame of the original image keyframe is input into a preset trajectory prediction model for model initialization processing to obtain a target trajectory prediction model. Based on the target trajectory prediction model, the trajectory prediction results of the first disease detection box in other frames of the original image keyframe are predicted. The positions of the other frame trajectory prediction results and the first disease detection box in the other frames of the original image keyframe are matched to obtain position matching results. Based on the position matching results, the continuous trajectory of the original disease in the target tunnel region is determined. Based on the continuous trajectory of the disease, the original two-dimensional spatial coordinates are determined, and the original acquisition timestamp of the original image keyframe corresponding to the original disease and the original camera pose parameters used to acquire the original image keyframe are determined. Based on the original two-dimensional spatial coordinates, the original acquisition timestamp, and the original camera pose parameters, the actual location of the original disease in the target tunnel region is determined. In practical applications, a pre-defined trajectory prediction model is first used to create a trajectory for the first defect detection box in the keyframe of the original image. During this process, as detection progresses, all keyframes of the original image will generate trajectories one by one using a target tracking algorithm, and matching will be performed based on the predictions of existing trajectories. During the matching process, a new trajectory is created when a target box fails to match. Finally, by statistically analyzing the number of trajectories in the tracking results, the location and frequency of various defects within the tunnel during this inspection are recorded. Finally, the average time between the first and last detection of the same original defect in the keyframe of the original image is used as the defect location time, and continuous defect trajectories are determined based on this location time and location. On this basis, the location of the defect is accurately located using the position information and camera pose information provided by the vehicle-mounted visual inertial pose acquisition system on the intelligent shooting vehicle.

[0051] In step S120, an original disease image associated with the original disease is determined based on the actual location of occurrence, and the morphological mask information and disease point cloud data of the original disease are determined based on the original disease image.

[0052] In this exemplary embodiment, firstly, the original disease image associated with the original disease is determined based on the actual location of occurrence. Specifically, this can be achieved as follows: Firstly, based on the disease's location information (i.e., the actual location of occurrence), the intelligent shooting vehicle is controlled to start from the starting position and sequentially perform targeted enhanced shooting of all disease areas to obtain the original disease image. The starting position mentioned here can be the starting point of the target tunnel area or the actual location of the first original disease. Simultaneously, after the intelligent shooting vehicle reaches the location of the disease, it should conduct comprehensive, multi-angle, and detailed shooting of the disease. Furthermore, sufficient lighting must be ensured during the shooting process to guarantee the clarity and reliability of the acquired images. At the same time, the vehicle-mounted visual inertial pose acquisition system is simultaneously activated to record the camera pose information during the shooting process in real time, including the relative position of the camera and the shooting angle.

[0053] Secondly, the morphological mask information of the original disease is determined based on the original disease image. Specifically, this can be achieved as follows: the original disease image is identified using a preset deep target detection model to obtain a second disease detection box; the area of ​​the second disease detection box is identified to select the detection box with the largest area as the disease reference image of the original disease; the disease reference image is semantically segmented using a preset deep semantic segmentation network to obtain the foreground image and background image of the disease reference image, and the morphological mask information is determined based on the foreground image. Specifically, the process for determining the second disease detection frame described here is largely similar to the process for determining the first disease detection frame described above, and will not be elaborated further here. Simultaneously, during the determination of the disease baseline image, the length and width information of all second disease detection frames can be extracted and the corresponding detection frame area can be calculated. Based on this, the second disease detection frame with the largest detection frame area is taken as the disease baseline image of the original disease. Each original disease corresponds to one original disease baseline image, and each original disease has a unique disease identifier. The original disease baseline image can be stored in association with the unique disease identifier.

[0054] In one example embodiment, semantic segmentation of the disease reference image is performed based on a preset deep semantic segmentation network to obtain the foreground and background images of the disease reference image. This can be achieved as follows: The disease reference image is convolved using multiple first convolutional layers to obtain multiple local features at different scales; multiple first downsampling layers are used to downsample the multiple local features at different scales to obtain regional features of multiple local regions; and multiple first upsampling layers are used to perform semantic segmentation based on the regional features of the multiple local regions to obtain the foreground and background images of the disease reference image. It should also be noted that in practical applications, tunnel disease images combining self-collected data and publicly available datasets can be used to train the deep semantic segmentation model. Furthermore, during training, the model only needs to distinguish between foreground and background, without needing to further subdivide disease categories. The trained model can identify the background in the image as black and the foreground of various diseases as white. In the actual process of semantic segmentation, the baseline disease image captured by the camera can be compressed to a specified size, and the trained deep semantic segmentation model can be used to segment and detect the disease in the image to obtain the foreground image and the background image; on this basis, the disease morphology mask information can be obtained based on the pixels corresponding to the foreground image.

[0055] Furthermore, the point cloud data of the original disease is determined based on the original disease image. Specifically, this can be achieved as follows: The intrinsic parameters of the target camera used to acquire the original disease image are determined; the intrinsic parameters include multiple parameters such as focal length, principal point offset parameter, radial distortion parameter, and tangential distortion parameter; the pose parameters of the target camera in the original disease image are determined using a preset multi-view geometric 3D reconstruction model, based on the intrinsic parameters and the original disease image; the depth map of the original disease image is determined using the preset multi-view geometric 3D reconstruction model, based on the pose parameters, and the depth maps are fused to obtain the point cloud data of the original disease.

[0056] In one example embodiment, determining the target camera intrinsic parameters of the target camera used to acquire the original disease images can be achieved as follows: First, using an intelligent shooting vehicle to capture multiple images of a checkerboard calibration board at different poses (e.g., different angles and / or different distances) at the tunnel site or laboratory in the target tunnel area, fix the focal length and aperture; wherein, the checkerboard calibration board images described herein can be 9 The hexagonal dot specification can be used, but other dot specifications are not specifically limited in this example. The side length of the checkerboard grid can be selected from 30 to 50 mm based on the actual field of view of the tunnel, but other specifications are also acceptable; this example does not impose any special restrictions. It should also be noted that during the image capture of the checkerboard calibration board, it is necessary to ensure that the calibration board is clear and complete in all images, covering the center, edges, and four corners of the image. After the images are captured, a Brown-Conrady distortion model can be used to determine the intrinsic parameters of the target camera. The intrinsic parameters of the target camera included in this model may include, but are not limited to, focal length. as well as Principal point offset parameters as well as Radial distortion parameters , as well as Tangential distortion parameters as well as Based on this, the Zhang Zhengyou method is used to iteratively solve these 9 parameters using the checkerboard calibration board image, so that the reprojection error of all corner points is reduced to within 0.1 pixels. After calibration, this set of intrinsic parameters remains unchanged in the subsequent point cloud reconstruction process of the preset multi-view geometric 3D reconstruction model, so as to ensure that the geometric benchmark of 3D reconstruction is unified and there is no scale drift.

[0057] In one example embodiment, the target camera pose parameters of the original disease image are determined based on the target camera intrinsic parameters and the original disease image using a preset multi-view geometric 3D reconstruction model. This can be achieved as follows: First, the input configuration of the preset multi-view geometric 3D reconstruction model is determined. The input configuration may include, but is not limited to, the target camera intrinsic parameters, a multi-view image sequence composed of the original disease images, and the pose corresponding to each original disease image. Second, feature points are extracted from the original disease images based on the multi-view geometric 3D reconstruction model, and the extracted feature points are matched to obtain feature point pairs. It should be noted that during the matching of specific point pairs, the target camera intrinsic parameters need to be locked in a fixed state. Based on this, a sparse model is determined through incremental reconstruction. The obtained sparse model may include the target camera pose parameters and sparse point cloud data. The target camera pose parameters can be used to characterize the camera position and orientation corresponding to each original disease image. The sparse point cloud data can represent multiple 3D points. Of course, the sparse model may also include scene scale information.

[0058] In one example embodiment, a depth map of the original disease image is determined based on the target camera pose parameters using a preset multi-view geometric 3D reconstruction model. The depth maps are then fused to obtain the original disease point cloud data. This can be achieved as follows: First, the depth map of the original disease image is determined based on the target camera pose parameters using a preset multi-view geometric 3D reconstruction model. Each pixel value in the depth map represents the distance from the corresponding spatial point to the camera. Second, the depth maps of all original disease images are fused to generate a dense 3D point cloud containing millions or even tens of millions of points. This dense 3D point cloud can very finely represent the contours of the disease, the extension of cracks, and other details. The obtained disease point cloud data can be referenced... Figure 7 As shown.

[0059] In step S130, the three-dimensional geometric parameters of the original disease are determined based on the morphological mask information and the disease point cloud data, and the disease time-series image sequence of the original disease at multiple different times is determined.

[0060] In this exemplary embodiment, firstly, the three-dimensional geometric parameters of the original disease are determined based on the morphological mask information and the disease point cloud data. Specifically, this can be achieved as follows: Based on the first disease category of the original disease, parameter extraction rules are determined for extracting the two-dimensional geometric parameters from the morphological mask information, and the two-dimensional geometric parameters are extracted based on the feature extraction rules. The morphological feature points of the disease in the two-dimensional geometric parameters are mapped to the disease point cloud data to match the real three-dimensional coordinate points corresponding to the morphological feature points from the disease point cloud data. From the morphological feature points of the disease, a pair of feature points consisting of a first two-dimensional feature point and a second two-dimensional feature point is determined, and the two-dimensional pixel distance between the feature point pairs is determined; a first three-dimensional coordinate point corresponding to the first two-dimensional feature point and a second three-dimensional coordinate point corresponding to the second two-dimensional feature point are determined from the real three-dimensional coordinate points, and the three-dimensional pixel distance between the first three-dimensional coordinate point and the second three-dimensional coordinate point is determined; a pixel equivalent is determined based on the two-dimensional pixel distance and the three-dimensional pixel distance, and the three-dimensional geometric parameters of the original disease are determined based on the pixel equivalent and the two-dimensional disease geometric parameters.

[0061] In one example embodiment, the feature extraction rules described above may include, but are not limited to, a first feature extraction rule corresponding to the lining crack disease category, a second feature extraction rule corresponding to the lining spalling disease category, and a third feature extraction rule corresponding to the water leakage disease category, etc. Based on this, the extraction of two-dimensional disease geometric parameters based on the feature extraction rules can be achieved as follows: Based on the first feature extraction rule, the main crack orientation of the original disease corresponding to the lining crack disease category is extracted from the morphological mask information, and the original disease is determined based on the main crack orientation. The two-dimensional geometric parameters of the disease are: the maximum width pixel value of the crack; and / or, based on the second feature extraction rule, the minimum bounding rectangle of the original disease corresponding to the lining spalling disease category is determined according to the morphological mask information, and the two-dimensional geometric parameters of the original disease are determined based on the minimum bounding rectangle; the two-dimensional geometric parameters of the disease are the spalling disease area; and / or, based on the third feature extraction rule, the two-dimensional geometric parameters of the original disease corresponding to the leakage disease category are determined according to the morphological mask information; the two-dimensional geometric parameters of the disease are the leakage area.

[0062] The following section will further explain and illustrate the specific process for determining the geometric parameters of three-dimensional defects. Specifically, in practical applications, if it is necessary to determine the geometric parameters of three-dimensional defects, it is first necessary to determine the geometric parameters of two-dimensional defects. Based on this, different feature extraction rules can be used to determine the geometric parameters of two-dimensional defects for different defect categories. For example, for the original defect category of lining cracks, a skeleton extraction algorithm can be used to refine the binary image corresponding to the morphological mask information to extract the longest main crack and count its pixel count. Based on this, after determining the direction of the main crack, the normal length at each pixel of the main crack is scanned one by one to determine the maximum width pixel value of the crack. The obtained direction of the main crack can be referenced... Figure 8 As shown; for example, for the lining spalling disease category, the minimum bounding rectangle of the disease area can be constructed to obtain the minimum bounding length and width of the disease area, and the disease area can be determined by counting the number of pixels in the disease area; for another example, for water leakage disease, the number of pixels in the disease area can be counted to determine the water leakage area. Secondly, after obtaining the two-dimensional disease geometric parameters, the corresponding three-dimensional coordinate points in the reconstructed point cloud can be located by multiple disease morphological feature points in the two-dimensional disease geometric parameters, and then the area and length actually represented by the pixel can be calculated by using the spatial distance between the feature points, thus converting the disease geometric information extracted from the binary image into geometric parameters at the real physical scale.

[0063] Secondly, the time-series image sequence of the original disease at multiple different times is determined. Specifically, this can be achieved by repeatedly photographing the same disease multiple times within a certain period, based on the camera pose when the baseline image of the disease was taken, ensuring that the disease is in the same position at the same scale in each image, thereby obtaining the time-series image sequence of the original disease at multiple different times. It should also be noted that the specific acquisition period for the time-series image sequence of the disease at multiple different times can be a fixed period or determined based on the urgency of the original disease. For example, for primary diseases with a severity level of "super urgent," data is collected hourly, typically every 6 hours, with a daily cycle for model decision-making; for primary diseases with a severity level of "very urgent," data is collected daily, typically every 1 to 2 days, with a weekly cycle for model decision-making; for primary diseases with a severity level of "moderately urgent," data is collected weekly, typically weekly, with a monthly cycle for model decision-making; and for primary diseases with a severity level of "not urgent," data is collected monthly, typically monthly, with a six-month to one-year cycle for model decision-making.

[0064] In step S140, the suggested repair category of the original disease is determined based on the three-dimensional disease geometric parameters and the disease time-series image sequence at multiple different times, and a tunnel disease repair scheme is generated based on the suggested repair category.

[0065] In this exemplary embodiment, firstly, the suggested repair category of the original disease is determined based on the three-dimensional disease geometric parameters and multiple time-series image sequences of the disease at different times. Specifically, this can be achieved as follows: determining the urgency of the original disease, and determining a multimodal tunneling disease decision model associated with the original disease based on the urgency; generating a first disease feature of the original disease in the semantic dimension based on the three-dimensional disease geometric parameters, and generating a second disease feature of the original disease in the image dimension based on multiple time-series image sequences of the disease at different times; inputting the first disease feature and the second disease feature into the multimodal tunneling disease decision model to obtain the suggested repair category of the original disease. Specifically, in practical applications, a corresponding multimodal tunneling disease decision model can be matched according to the urgency of different diseases; wherein, the multimodal tunneling disease decision model can be divided into short-term decision model, medium-term decision model, long-term decision model, and ultra-long-term decision model, and the input parameters of different network models are different. In practical applications, short-term decision-making models primarily handle urgent defects, whose short-term development can severely impact tunnel safety and service life. Medium-term decision-making models mainly decide on repair plans for emerging defects, used to monitor their development. Long-term decision-making models primarily handle long-term stable defects with minimal impact on tunnel safety, used to prevent sudden escalation. Ultra-long-term decision-making models primarily handle long-term stable, minor defects without substantial harm. It should also be noted that the urgency of an initial defect can be determined based on previously identified historical defects in the target tunnel area. For example, comparing previously identified historical defects with the currently identified initial defects reveals a new defect, classifying it as super urgent. If not, the specific three-dimensional geometric parameters of the defect are compared; rapid growth indicates super urgent, moderate growth indicates very urgent, slow growth indicates moderately urgent, and no growth indicates no urgency.

[0066] Furthermore, after matching the multimodal tunnel disease decision-making model, the segmented images taken for the same disease within a certain period can be superimposed in the depth direction according to the time sequence, based on the input requirements of the model branch, to obtain the second disease feature. At the same time, the disease semantic information acquired within this period is stacked in the time step dimension to obtain the first disease feature. Furthermore, the first disease feature includes the disease category, the shooting time, and the geometric information corresponding to different disease categories. The disease category is represented by one-hot encoding, the shooting time is represented by relative time, and the geometric information corresponds to different dimensions of the feature vector. Unused vector dimensions are uniformly set to zero. Based on this, the processed data is input into the temporal multimodal tunnel disease decision-making model. The network is divided into two branches: an image processing channel handles temporal image information (i.e., the second disease feature), and a semantic processing channel handles temporal semantic information (i.e., the first disease feature). The semantic processing channel uses a common recurrent neural network for sequence modeling, while the image processing channel can be modeled using a residual network. Based on this, a feature concatenation layer concatenates the image features output from the image processing channel and the semantic features output from the semantic processing channel to obtain concatenated features. Then, a feature aggregation layer performs global feature integration, linear transformation, nonlinear transformation, and higher-order combination and decision mapping on the concatenated features to obtain the feature aggregation result. Finally, a classification layer classifies the feature aggregation result to obtain the suggested repair category for the original disease. It should also be noted that the process of determining the suggested repair category fully considers the disease's development trend over time, the disease category, and its geometric parameters, thus greatly improving the accuracy of the obtained suggested repair categories.

[0067] Secondly, a tunnel defect repair plan is generated based on the suggested repair categories. Specifically, this can be achieved as follows: Based on the suggested repair categories, three-dimensional defect geometric parameters, and multiple temporal image sequences of defects at different times, basic defect information associated with the original defect is generated; based on preset model prompts, contextual information associated with the original defect is generated, and the basic defect information and contextual information are input into a preset content generation model to obtain the tunnel defect repair plan; wherein, the tunnel defect repair plan includes multiple elements such as defect identification information, actual location, three-dimensional defect geometric parameters, defect change trend, defect repair decision suggestions, and corresponding image evidence. The pre-defined content generation model described here may include an embedding mapping layer, an encoding layer, and a hybrid expert model. Based on this, the specific generation process of the tunnel defect repair scheme can be implemented as follows: First, based on the embedding mapping layer, the basic information and context information of the defect are embedded and mapped to obtain the basic features and context features of the defect; second, based on the encoding layer, the basic features and context features of the defect are encoded to obtain the target defect features; finally, based on the hybrid expert model, the target defect features are used to generate content to obtain the tunnel defect repair scheme.

[0068] Thus, the method for generating tunnel defect repair schemes described in the exemplary embodiments of this disclosure has been fully implemented. Based on the foregoing description, the method for generating tunnel defect repair schemes described in the exemplary embodiments of this disclosure has at least the following advantages: On the one hand, through the synergistic fusion of multiple algorithms, the intelligent imaging vehicle is endowed with the ability to automatically perform the entire process of image acquisition, defect localization, 3D reconstruction, defect morphology extraction, and defect repair decision-making, effectively reducing the input of manpower and material resources during the operation process, realizing accurate identification and intelligent repair decision-making for tunnel structural defects, and providing strong technical support for the intelligent development of tunnel structural health monitoring and defect management; on the other hand, the dynamic impact of defect evolution over time is fully considered in the tunnel defect repair decision-making, introducing temporal information into the network model, and integrating defect morphological characteristics on this basis to establish a multi-dimensional repair decision-making mechanism. This method effectively compensates for the decision-making errors caused by insufficient information in traditional methods, avoids the waste of human and material resources, and significantly improves the efficiency of defect detection and the rationality of repair decisions.

[0069] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0070] This disclosure also provides an apparatus for generating a tunnel defect repair scheme. Specifically, refer to... Figure 9As shown, the device for generating tunnel defect repair schemes described herein may include an actual location determination module 910, a morphological mask information determination module 920, a three-dimensional defect geometric parameter determination module 930, and a defect repair scheme determination module 940.

[0071] The actual location determination module 910 can be used to identify defects in keyframes of the original image of the target tunnel area based on a preset depth target detection model, and determine the actual location of the original defect based on the defect identification result. The morphological mask information determination module 920 can be used to determine the original defect image associated with the original defect based on the actual location, and determine the morphological mask information and defect point cloud data of the original defect based on the original defect image. The three-dimensional defect geometric parameter determination module 930 can be used to determine the three-dimensional defect geometric parameters of the original defect based on the morphological mask information and defect point cloud data, and determine the defect time-series image sequence of the original defect at multiple different times. The defect repair scheme determination module 940 can be used to determine the suggested repair category of the original defect based on the three-dimensional defect geometric parameters and the defect time-series image sequence at multiple different times, and generate a tunnel defect repair scheme based on the suggested repair category.

[0072] In an exemplary embodiment of this disclosure, a defect identification result is obtained by performing defect identification on keyframes of an original image of a target tunnel region based on a preset depth target detection model. This includes: acquiring original tunnel image data of the target tunnel region and extracting original image keyframes from the original tunnel image data; the original tunnel image data is obtained by photographing the tunnel surface of the target tunnel region; downsampling the original image keyframes based on a backbone feature extraction network to obtain multiple first local image features of different scales corresponding to the original image keyframes; fusing the multiple first local image features of different scales corresponding to the original image keyframes based on a feature fusion network to obtain first global image features corresponding to the original image keyframes; and performing object detection on the first global image features corresponding to the original image keyframes based on a target object detection network to obtain a defect identification result corresponding to the original image keyframes. The defect identification result includes a first defect detection box and a first defect category for the original defects included in the original image keyframes.

[0073] In an exemplary embodiment of this disclosure, determining the actual location of the original disease based on the disease identification result includes: inputting a first disease detection box in the original image keyframe of the first frame into a preset trajectory prediction model for model initialization processing to obtain a target trajectory prediction model, and predicting the trajectory prediction results of the first disease detection box in other frames of the original image keyframe based on the target trajectory prediction model; performing position matching on the trajectory prediction results of other frames and the first disease detection box of the original disease in other frames of the original image keyframe to obtain a position matching result, and determining the continuous trajectory of the original disease in the target tunnel region based on the position matching result; determining the original two-dimensional spatial coordinates based on the continuous trajectory of the disease, and determining the original acquisition timestamp of the original image keyframe corresponding to the original disease and the original camera pose parameters used to acquire the original image keyframe; and determining the actual location of the original disease in the target tunnel region based on the original two-dimensional spatial coordinates, the original acquisition timestamp, and the original camera pose parameters.

[0074] In an exemplary embodiment of this disclosure, determining the morphological mask information of the original disease based on the original disease image includes: performing original disease identification on the original disease image based on a preset deep target detection model to obtain a second disease detection box and a second disease category; performing area identification on the second disease detection box to select the detection box with the largest area from the second disease detection box as the disease reference image of the original disease; performing semantic segmentation on the disease reference image based on a preset deep semantic segmentation network to obtain a foreground image and a background image of the disease reference image, and determining the morphological mask information based on the foreground image.

[0075] In an exemplary embodiment of this disclosure, determining the point cloud data of the original disease based on the original disease image includes: determining the target camera intrinsic parameters of the target camera used to acquire the original disease image; the target camera intrinsic parameters include multiple parameters among focal length, principal point offset parameter, radial distortion parameter, and tangential distortion parameter; determining the target camera pose parameters of the original disease image based on the target camera intrinsic parameters and the original disease image using a preset multi-view geometric 3D reconstruction model; determining the depth map of the original disease image based on the target camera pose parameters using the preset multi-view geometric 3D reconstruction model, and fusing the depth maps to obtain the point cloud data of the original disease.

[0076] In an exemplary embodiment of this disclosure, determining the three-dimensional geometric parameters of the original disease based on the morphological mask information and the disease point cloud data includes: determining parameter extraction rules required to extract the two-dimensional geometric parameters of the disease from the morphological mask information according to the first disease category of the original disease, and extracting the two-dimensional geometric parameters of the disease based on the feature extraction rules; mapping the disease morphological feature points in the two-dimensional geometric parameters to the disease point cloud data to match the real three-dimensional coordinate points corresponding to the disease morphological feature points from the disease point cloud data; determining feature point pairs composed of a first two-dimensional feature point and a second two-dimensional feature point from the disease morphological feature points, and determining the two-dimensional pixel distance between the feature point pairs; determining the first three-dimensional coordinate point corresponding to the first two-dimensional feature point and the second three-dimensional coordinate point corresponding to the second two-dimensional feature point from the real three-dimensional coordinate points, and determining the three-dimensional pixel distance between the first three-dimensional coordinate point and the second three-dimensional coordinate point; determining the pixel equivalent based on the two-dimensional pixel distance and the three-dimensional pixel distance, and determining the three-dimensional geometric parameters of the original disease based on the pixel equivalent and the two-dimensional geometric parameters of the disease.

[0077] In an exemplary embodiment of this disclosure, the first defect category includes at least one of a lining crack defect category, a lining spalling defect category, and a water leakage defect category; the feature extraction rules include at least one of a first feature extraction rule corresponding to the lining crack defect category, a second feature extraction rule corresponding to the lining spalling defect category, and a third feature extraction rule corresponding to the water leakage defect category; wherein, extracting two-dimensional defect geometric parameters based on the feature extraction rules includes: extracting the main crack orientation of the original defect corresponding to the lining crack defect category from the morphological mask information based on the first feature extraction rule, and based on... The two-dimensional geometric parameters of the original disease are determined based on the direction of the main crack; the two-dimensional geometric parameters are the maximum width pixel value of the crack; and / or, based on the second feature extraction rule, the minimum bounding rectangle of the original disease corresponding to the lining spalling disease category is determined according to the morphological mask information, and the two-dimensional geometric parameters of the original disease are determined based on the minimum bounding rectangle; the two-dimensional geometric parameters are the spalling disease area; and / or, based on the third feature extraction rule, the two-dimensional geometric parameters of the original disease corresponding to the water leakage disease category are determined according to the morphological mask information; the two-dimensional geometric parameters are the water leakage area.

[0078] In one exemplary embodiment of this disclosure, determining the suggested repair category of the original disease based on the three-dimensional disease geometric parameters and a series of disease time-series images at multiple different times includes: determining the disease urgency of the original disease, and determining a multimodal tunnel disease decision model associated with the original disease based on the disease urgency; generating a first disease feature of the original disease in the semantic dimension based on the three-dimensional disease geometric parameters, and generating a second disease feature of the original disease in the image dimension based on a series of disease time-series images at multiple different times; and inputting the first disease feature and the second disease feature into the multimodal tunnel disease decision model to obtain the suggested repair category of the original disease.

[0079] In one exemplary embodiment of this disclosure, generating a tunnel defect repair scheme based on the suggested repair category includes: generating basic defect information associated with the original defect based on the suggested repair category, three-dimensional defect geometric parameters, and multiple temporal image sequences of the defect at different times; generating contextual information associated with the original defect based on preset model prompt information; and inputting the basic defect information and contextual information into a preset content generation model to obtain the tunnel defect repair scheme; wherein, the tunnel defect repair scheme includes multiple elements such as defect identification information, actual location, three-dimensional defect geometric parameters, defect change trend, defect repair decision suggestions, and corresponding image evidence.

[0080] The specific details of each module in the above-mentioned tunnel defect repair scheme generation device have been described in detail in the corresponding tunnel defect repair scheme generation method, so they will not be repeated here.

[0081] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0082] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0083] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0084] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0085] The following reference Figure 10 To describe an electronic device 1000 according to such an embodiment of the present disclosure. Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0086] like Figure 10 As shown, the electronic device 1000 is manifested in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010), and a display unit 1040.

[0087] The storage unit stores program code that can be executed by the processing unit 1010, causing the processing unit 1010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1010 can perform actions such as... Figure 1 Step S110: Based on a preset depth target detection model, perform defect identification on the keyframes of the original image of the target tunnel area to obtain defect identification results, and determine the actual location of the original defect based on the defect identification results; Step S120: Based on the actual location, determine the original defect image associated with the original defect, and determine the morphological mask information and defect point cloud data of the original defect based on the original defect image; Step S130: Based on the morphological mask information and defect point cloud data, determine the three-dimensional defect geometric parameters of the original defect, and determine the defect time-series image sequence of the original defect at multiple different times; Step S140: Based on the three-dimensional defect geometric parameters and the defect time-series image sequence at multiple different times, determine the suggested repair category of the original defect, and generate a tunnel defect repair scheme based on the suggested repair category.

[0088] Storage unit 1020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 10201 and / or cache memory 10202, and may further include read-only memory (ROM) 10203.

[0089] Storage unit 1020 may also include a program / utility 10204 having a set (at least one) program module 10205, such program module 10205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0090] Bus 1030 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0091] Electronic device 1000 can also communicate with one or more external devices 1100 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1060. As shown, network adapter 1060 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0092] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0093] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this disclosure described in the "Exemplary Methods" section above.

[0094] The program product for implementing the above-described method according to embodiments of the present disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0095] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0096] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0097] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0098] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0099] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0100] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not invented by this disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A method for generating a tunnel defect repair scheme, characterized in that, include: Based on the preset depth target detection model, the original image keyframes of the target tunnel area are used to identify defects and obtain defect identification results. The actual location of the original defects is then determined based on the defect identification results. Based on the actual location of occurrence, determine the original disease image associated with the original disease, and based on the original disease image, determine the morphological mask information and disease point cloud data of the original disease. Based on the morphological mask information and the disease point cloud data, the three-dimensional geometric parameters of the original disease are determined, and the time-series image sequence of the original disease at multiple different times is determined. Based on the three-dimensional geometric parameters of the disease and the time-series image sequence of the disease at multiple different times, the suggested repair category of the original disease is determined, and a tunnel disease repair scheme is generated based on the suggested repair category.

2. The method for generating a tunnel defect repair scheme according to claim 1, characterized in that, Based on a pre-defined depth target detection model, the original image keyframes of the target tunnel region are used to identify defects, resulting in defect identification results, including: The original tunnel image data of the target tunnel area is acquired, and the original image keyframes are extracted from the original tunnel image data; the original tunnel image data is obtained by photographing the tunnel embankment surface of the target tunnel area. Based on the backbone feature extraction network, the keyframes of the original image are downsampled to obtain multiple first local image features of different scales corresponding to the keyframes of the original image. Based on the feature fusion network, multiple first local image features of different scales corresponding to the keyframes of the original image are fused to obtain the first global image features corresponding to the keyframes of the original image; Based on the target object detection network, object detection is performed on the first global image features corresponding to the keyframe of the original image to obtain the disease identification result corresponding to the keyframe of the original image; the disease identification result includes the first disease detection box and the first disease category included in the keyframe of the original image.

3. The method for generating a tunnel defect repair scheme according to claim 2, characterized in that, Based on the disease identification results, determine the actual location of the original disease, including: The original disease detection box in the first frame of the original image keyframe is input into the preset trajectory prediction model for model initialization processing to obtain the target trajectory prediction model, and the trajectory prediction results of the first disease detection box in other frames of the original image keyframe are predicted based on the target trajectory prediction model. The position matching results are obtained by performing position matching on the trajectory prediction results of the other frames and the first disease detection box of the original disease in the original image keyframe of the other frames, and the continuous trajectory of the original disease in the target tunnel area is determined based on the position matching results. Based on the continuous trajectory of the disease, the original two-dimensional spatial coordinates are determined, and the original acquisition timestamp of the original image key frame corresponding to the original disease and the original camera pose parameters used to acquire the original image key frame are determined. The actual location of the original defect in the target tunnel area is determined based on the original two-dimensional spatial coordinates, the original acquisition timestamp, and the original camera pose parameters.

4. The method for generating a tunnel defect repair scheme according to claim 1, characterized in that, Determining the morphological mask information of the original disease based on the original disease image includes: The original disease detection box is obtained by performing original disease identification on the original disease image based on the preset deep target detection model. The area of ​​the second disease detection box is identified to select the detection box with the largest area from the second disease detection box as the disease reference image of the original disease; The disease reference image is semantically segmented based on a preset deep semantic segmentation network to obtain the foreground image and background image of the disease reference image, and the morphological mask information is determined based on the foreground image.

5. The method for generating a tunnel defect repair scheme according to claim 1, characterized in that, Based on the original disease image, the point cloud data of the original disease is determined, including: Determine the target camera intrinsic parameters of the target camera used to acquire the original disease images; the target camera intrinsic parameters include multiple parameters such as focal length, principal point offset parameter, radial distortion parameter, and tangential distortion parameter. Using a preset multi-view geometric 3D reconstruction model, the target camera pose parameters of the original lesion image are determined based on the target camera intrinsic parameters and the original lesion image. Using a preset multi-view geometric 3D reconstruction model, the depth map of the original disease image is determined based on the pose parameters of the target camera, and the depth map is fused to obtain the disease point cloud data of the original disease.

6. The method for generating a tunnel defect repair scheme according to claim 1, characterized in that, Based on the morphological mask information and the disease point cloud data, the three-dimensional geometric parameters of the original disease are determined, including: Based on the first disease category of the original disease, determine the parameter extraction rules required to extract the two-dimensional disease geometric parameters from the morphological mask information, and extract the two-dimensional disease geometric parameters based on the feature extraction rules; Map the morphological feature points of the disease in the two-dimensional disease geometric parameters to the disease point cloud data, so as to match the real three-dimensional coordinate points corresponding to the morphological feature points of the disease from the disease point cloud data. From the disease morphology feature points, determine the feature point pairs composed of the first two-dimensional feature point and the second two-dimensional feature point, and determine the two-dimensional pixel distance between the feature point pairs; Determine the first three-dimensional coordinate point corresponding to the first two-dimensional feature point and the second three-dimensional coordinate point corresponding to the second two-dimensional feature point in the real three-dimensional coordinate points, and determine the three-dimensional pixel distance between the first three-dimensional coordinate point and the second three-dimensional coordinate point; The pixel equivalent is determined based on the two-dimensional pixel distance and the three-dimensional pixel distance, and the three-dimensional disease geometric parameters of the original disease are determined based on the pixel equivalent and the two-dimensional disease geometric parameters.

7. The method for generating a tunnel defect repair scheme according to claim 6, characterized in that, The first disease category includes at least one of the following: lining crack disease category, lining spalling disease category, and water leakage disease category; the feature extraction rule includes at least one of the following: a first feature extraction rule corresponding to the lining crack disease category, a second feature extraction rule corresponding to the lining spalling disease category, and a third feature extraction rule corresponding to the water leakage disease category. The extraction of two-dimensional disease geometric parameters based on the feature extraction rules includes: Based on the first feature extraction rule, the main crack orientation of the original disease corresponding to the lining crack disease category is extracted from the morphological mask information, and the two-dimensional disease geometric parameters of the original disease are determined based on the main crack orientation; the two-dimensional disease geometric parameters are the maximum width pixel value of the crack; and / or Based on the second feature extraction rule, the minimum bounding rectangle of the original disease corresponding to the lining spalling disease category is determined according to the morphological mask information, and the two-dimensional disease geometric parameters of the original disease are determined based on the minimum bounding rectangle; the two-dimensional disease geometric parameters are the spalling disease area; and / or Based on the third feature extraction rule, the two-dimensional geometric parameters of the original disease corresponding to the leakage disease category are determined according to the morphological mask information; the two-dimensional geometric parameters of the disease are the leakage area.

8. The method for generating a tunnel defect repair scheme according to claim 1, characterized in that, Based on the three-dimensional geometric parameters of the lesion and the time-series image sequences of the lesion at multiple different times, the suggested repair category of the original lesion is determined, including: Determine the severity of the original disease, and based on the severity of the disease, determine a multimodal tunnel disease decision model associated with the original disease; Based on the three-dimensional disease geometric parameters, the first disease feature of the original disease in the semantic dimension is generated, and based on the disease time-series image sequence at multiple different times, the second disease feature of the original disease in the image dimension is generated. The first and second disease features are input into the multimodal tunnel disease decision model to obtain the suggested repair category for the original disease.

9. The method for generating a tunnel defect repair scheme according to claim 1, characterized in that, Based on the suggested repair categories, a tunnel defect repair plan is generated, including: Based on the suggested repair category, three-dimensional disease geometric parameters, and disease time-series image sequences at multiple different times, generate basic disease information associated with the original disease; Based on preset model prompts, contextual information associated with the original disease is generated, and the basic disease information and contextual information are input into a preset content generation model to obtain a tunnel disease repair plan; wherein, the tunnel disease repair plan includes multiple elements such as disease identification information, actual location, three-dimensional disease geometric parameters, disease change trend, disease repair decision suggestions, and corresponding image evidence.

10. A device for generating a tunnel defect repair scheme, characterized in that, include: The actual location determination module is used to identify defects in the key frames of the original image of the target tunnel area based on a preset depth target detection model, and to determine the actual location of the original defects based on the defect identification results. The morphological mask information determination module is used to determine the original disease image associated with the original disease based on the actual occurrence location, and to determine the morphological mask information and disease point cloud data of the original disease based on the original disease image. The three-dimensional disease geometric parameter determination module is used to determine the three-dimensional disease geometric parameters of the original disease based on the morphological mask information and disease point cloud data, and to determine the disease time-series image sequence of the original disease at multiple different times. The disease repair scheme determination module is used to determine the suggested repair category of the original disease based on the three-dimensional disease geometric parameters and the disease time-series image sequence at multiple different times, and generate a tunnel disease repair scheme based on the suggested repair category.

Images