A method for inspecting tunnels using a mother-and-child robot

Abstract

The present invention discloses a method for inspecting tunnels with a mother-and-child robot, which belongs to the field of tunnel defect detection. The mother-and-child robot includes a mother machine and a child machine. The method includes: firstly, controlling the mother machine to travel in the tunnel according to a preset route; obtaining a global detection data sequence obtained by a rough inspection device carried by the mother machine to detect and collect the tunnel; inputting the global detection data sequence into a rough defect inspection model to obtain a rough inspection result; releasing the child machine to a fine inspection candidate area; obtaining fine inspection data obtained by the fine inspection device carried by the child machine to detect the fine inspection candidate area; finally, inputting the fine inspection data into a pre-trained fine defect inspection model to obtain a fine inspection result. Such a design effectively improves the detection efficiency while ensuring the accuracy of defect detection.

Description

Technical Field

[0001] The present invention relates to the field of tunnel defect detection, and in particular to a method for inspecting tunnels using a mother-and-child robot. Background Art

[0002] Large-diameter and ultra-long-distance water diversion tunnels in large and extra-large hydropower stations and water diversion projects have complex underwater environments such as turbid water, high water pressure, attached siltation, and local accessibility difficulties. The inspection of large-diameter and long water diversion tunnels has the contradiction between heavy inspection tasks and low inspection efficiency, small defect scale and poor inspection accuracy, and large inspection data volume and low defect feature utilization. The traditional underwater inspection theory and methods cannot meet the requirements of tunnel defect detection. Summary of the invention

[0003] An embodiment of the present invention provides a method for inspecting a tunnel using a mother-and-child robot, wherein the mother-and-child robot includes a mother machine and a child machine, and the method includes:

[0004] Controlling the mother machine to move in the tunnel according to a preset route;

[0005] Acquire a global detection data sequence obtained by a rough detection device carried by a mother machine to detect and collect the tunnel, wherein the rough detection device includes at least one of a sonar sensor and a laser scanning device;

[0006] Inputting the global detection data sequence into a defect rough inspection model to obtain a rough inspection result, wherein the rough inspection result is used to indicate a candidate fine inspection area within the detection range of the rough inspection device where a confidence level of a defect is higher than a preset threshold;

[0007] releasing the sub-machine to the candidate area for fine inspection;

[0008] Acquire precision inspection data obtained by a precision inspection device carried by the sub-machine to detect the precision inspection candidate area, wherein the precision inspection device includes at least one of a high-definition image sensor and a laser ruler measuring instrument;

[0009] The fine inspection data is input into a pre-trained defect fine inspection model to obtain a fine inspection result, and the fine inspection result is used to indicate whether there is a defect in the fine inspection candidate area.

[0010] Compared with the prior art, the beneficial effects provided by the present invention include: by setting up a mother-and-child robot, first obtaining a global detection data sequence collected by the mother machine during the tunnel travel process, and using a defect rough inspection model to perform a rough inspection, and after obtaining a fine inspection candidate area with a high possibility of defects, releasing the child machine to the fine inspection candidate area to obtain the fine inspection data collected by the child machine for the fine inspection candidate area, and then using the defect fine inspection model to analyze the fine inspection data to verify whether there are defects in the fine inspection candidate area, which can effectively reduce the occurrence of false detection and missed detection, and effectively improve the detection efficiency while ensuring the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic flow chart of the steps of a method for inspecting a tunnel using a mother-and-child robot according to an embodiment of the present invention;

[0013] Figure 2 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention;

[0014] Figure 3 A schematic flow chart of the steps of a rough inspection process in a method for inspecting tunnels using a mother-and-child robot provided in an embodiment of the present invention;

[0015] Figure 4 A schematic flow chart of the steps for training a defect precision inspection model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0017] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0018] In order to solve the technical problems in the aforementioned background technology, Figure 1A flow chart of a method for inspecting tunnels using a mother-and-child robot provided in an embodiment of the present invention, the method can be used for computer equipment, which can be connected to the mother machine of the mother-and-child robot through an umbilical cord, and the child machine and the mother machine can be connected by a cable. In some embodiments, the mother-and-child robot can adopt a crawling mother machine and a floating child machine. The mother-and-child robot can collect information of different scales and dimensions to achieve more accurate defect detection while ensuring the stability of the operating posture of the underwater robot. Optionally, the child machine can adopt a thruster layout with 4 groups of horizontal arrangements and 2 groups of vertical arrangements, thereby ensuring that the robot has sufficient power and flexible movements in all directions, and can perform behaviors such as floating, diving, suspending, swimming left and right, and returning to the cage without power.

[0019] Refer to the following Figure 1 , and the method of using the mother-and-child robot to inspect tunnels is introduced in detail.

[0020] Step S201: Control the mother machine to move in the tunnel according to a preset route.

[0021] For example, the mother machine can be controlled to move forward at a constant speed in a certain direction of the tunnel, and stop moving after reaching the target position until the preset inspection time is reached, and then the mother machine can be controlled to move forward at a constant speed in the opposite direction, and stop moving after reaching the initial position. In this way, the mother-child robot can inspect the tunnel at a predetermined frequency or time to ensure the safety of the tunnel.

[0022] Step S202: obtaining a global detection data sequence obtained by detecting and collecting tunnels through a rough inspection device carried by the mother machine.

[0023] Optionally, the rough inspection device includes at least one of a sonar sensor and a laser scanning device.

[0024] Specifically, for larger areas to be inspected, the 3D sonar sensor and laser scanning device carried by the mother machine can realize 360-degree rapid scanning inspection of the tunnel, and then the 2D sonar and side scan sonar can be used to conduct supplementary inspections of the bottom and high-incidence areas of defects on both sides, and combine this information to obtain a time-series global inspection data sequence. Among them, the inspection data with a smaller sequence index corresponds to the more forward position of the mother machine during its travel.

[0025] Step S203: input the global detection data sequence into the defect rough inspection model to obtain a rough inspection result.

[0026] Optionally, the rough inspection result is used to indicate a candidate fine inspection area within the inspection range of the rough inspection device where a confidence level of defects detected is higher than a preset threshold.

[0027] It can be understood that the fine inspection candidate area refers to an area that is determined to have a high probability of defects after a rough inspection.

[0028] The process of obtaining the rough inspection result based on the global inspection data sequence by the defect rough inspection model will be described in detail in subsequent embodiments and will not be elaborated here.

[0029] Step S204: releasing the slave to the candidate area for fine inspection.

[0030] Specifically, the rough inspection result may include the absolute position coordinates and relative position coordinates corresponding to the fine inspection candidate area. After the mother machine moves to the absolute position coordinates, the child machine is released, and the working state of the propeller of the child machine is controlled so that the child machine reaches the relative position coordinates, thereby ensuring that the child machine can be accurately released to the fine inspection candidate area.

[0031] Step S205: Obtaining the refined inspection data obtained by the refined inspection device carried by the slave device through detection of the refined inspection candidate area.

[0032] Optionally, the precision inspection device includes at least one of a high-definition image sensor and a laser ruler measuring instrument.

[0033] The precision inspection data may include image data and laser detection data. Optionally, the image data and laser detection data may be fused, for example, by a pre-trained model to obtain corresponding feature data.

[0034] Specifically, for candidate areas for precision inspection with a higher probability of defects, the sub-machine can get close and take a detailed look through high-definition video, laser and other sensors. For example, the sub-machine can extend a manipulator equipped with a high-definition camera and a laser ruler measuring instrument to get close and observe suspected defects at the top and bottom of the tunnel in order to collect more accurate precision inspection data.

[0035] Step S206: input the precision inspection data into the pre-trained defect precision inspection model to obtain precision inspection results.

[0036] The precision inspection result is used to indicate whether there are defects in the precision inspection candidate area.

[0037] Among them, the defect fine inspection model is used to compare the fine inspection data collected in the fine inspection candidate area with various types of defects to determine whether these types of defects exist in the fine inspection candidate area. If the fine inspection candidate area does not have these types of defects or the possibility of these types of defects existing in the fine inspection candidate area is low, it can be determined that there are no defects in the area.

[0038] It should be noted that tunnel defects may include, for example, lining cracks, water leakage, material degradation, lining voids, lining deformation, lining peeling, etc., and each defect may have some common or different characteristics.

[0039] In an embodiment of the present invention, a mother-and-child robot is set up to first obtain a global inspection data sequence collected by the mother machine during the tunnel travel process, and use a defect coarse inspection model to perform a coarse inspection. After obtaining a fine inspection candidate area with a high possibility of defects, the child machine is released to the fine inspection candidate area to obtain fine inspection data collected by the child machine for the fine inspection candidate area, and then use the defect fine inspection model to analyze the fine inspection data to verify whether there are defects in the fine inspection candidate area. This can effectively detect defects in the tunnel while reducing the occurrence of false detections, thereby ensuring the accuracy of defect detection.

[0040] In some embodiments, reference Figure 3 As shown, Figure 1 The step S203 involved in the above “inputting the global detection data sequence into the defect rough inspection model to obtain the rough inspection result”, i.e. the rough inspection process in the method for inspecting tunnels with a mother-and-child robot, may include the following sub-steps, i.e., steps S2031 to S2034:

[0041] Step S2031: perform feature engineering on the global detection data sequence to obtain multiple serialized first feature representations corresponding to the global detection data sequence.

[0042] In some embodiments, the multiple first feature representations in step S2031 include at least two first feature representations.

[0043] The pre-trained rough defect detection model can be used to process the global detection data sequence to obtain the result of defect detection in the tunnel. That is, the computer device can input the serialized global detection data sequence into the rough defect detection model, and the rough defect detection model analyzes and processes the global detection data sequence and outputs a corresponding indication of the area in the tunnel where defects are more likely to exist.

[0044] It should be noted that the defect rough inspection model in the embodiment of the present invention is a model used to detect areas in the tunnel with a higher possibility of defects based on the input serialized global inspection data sequence; in the embodiment of the present invention, the defect rough inspection model can be pre-trained.

[0045] It should be noted that the defect rough inspection model provided in the embodiment of the present invention mainly includes a primary feature engineering unit, a first feature fusion unit and a second feature fusion unit. Steps S2031, S2032 and S2033 in the embodiment of the present invention are used to introduce the processing of the primary feature engineering unit, the first feature fusion unit and the second feature fusion unit, respectively. The processing of the primary feature engineering unit in the defect rough inspection model is first introduced below.

[0046] The primary feature engineering unit in the defect rough inspection model is used to perform feature engineering on each of the global detection data sequences input into the defect rough inspection model, so as to obtain multiple first feature representations of the sequence corresponding to the global detection data sequence. That is, after the sequenced global detection data sequence is input into the defect rough inspection model, the primary feature engineering unit in the defect rough inspection model will perform feature engineering on each data in the global detection data sequence, and obtain multiple first feature representations corresponding to the global detection data sequence, so as to provide the subsequent first feature fusion unit and the second feature fusion unit for further processing according to the multiple first feature representations.

[0047] In a possible implementation, when the primary feature engineering unit is working, it can first use a two-dimensional filter, such as a two-dimensional convolution structure, to perform feature engineering on each global detection data sequence to obtain the first-level feature representations respectively associated with the global detection data sequences; then, for each global detection data, based on the first-level feature representation corresponding to the global detection data and the first-level feature representation corresponding to the global detection data adjacent to the global detection data, determine the dynamic feature representation corresponding to the global detection data; based on the first-level feature representation and the dynamic feature representation corresponding to the global detection data, determine the second-level feature representation corresponding to the global detection data; and further, based on the second-level feature representations respectively associated with the global detection data sequences, determine multiple first feature representations.

[0048] It should be noted that the first-level feature representation corresponding to the global detection data is a multi-channel feature map obtained by performing feature engineering on the global detection data itself using a two-dimensional filter, which carries the original feature information of the global detection data itself. The dynamic feature representation corresponding to the global detection data is determined based on the difference between the first-level feature representation corresponding to the global detection data itself and the first-level feature representation corresponding to the global detection data adjacent to it in order of arrangement. Since the dynamic feature representation is determined based on the difference between the first-level feature representations respectively associated with adjacent global detection data, the dynamic feature representation can reflect the spatial change information of each position in the global detection data to a certain extent, that is, reflect the dynamic information. The second-level feature representation corresponding to the global detection data is a multi-channel feature map obtained by fusing the first-level feature representation corresponding to the global detection data and the dynamic feature representation, which carries both the original feature information of the global detection data itself and the dynamic information of the global detection data.

[0049] In some embodiments, when the primary feature engineering unit is working, it can first use at least one layer of two-dimensional filters to perform feature conversion and channel expansion processing on each of the input serialized global detection data sequences, and obtain multi-channel feature maps (i.e., first-level feature representations) associated with the global detection data sequences. Since the input global detection data is arranged in sequence, the multi-channel feature maps obtained by processing with two-dimensional filters should also be arranged in the corresponding order, that is, multiple first-level feature representations obtained by processing with two-dimensional filters can constitute a first-level feature representation sequence, and the sequence index of each first-level feature representation in the first-level feature representation sequence is the same as the sequence index of the global detection data associated with each first-level feature representation. In this way, by performing feature engineering on each global detection data using a two-dimensional filter, and then performing subsequent processing using the first-level feature representation obtained by the feature engineering, it is possible to avoid the situation where the output numerical range is unstable after making a difference directly on the original global detection data.

[0050] In some embodiments, after using a two-dimensional filter to perform feature engineering to obtain the first-level feature representations associated with the global detection data sequence, the dynamic feature representation can be further supplemented on the basis of the first-level feature representations associated with each piece of global detection data to generate a second-level feature representation carrying the dynamic feature representation. Specifically, for each piece of global detection data, the dynamic feature representation of the global detection data relative to its adjacent global detection data can be determined based on the first-level feature representation corresponding to the global detection data and the first-level feature representation corresponding to the global detection data adjacent to the global detection data, that is, the dynamic feature representation corresponding to the global detection data can be determined. Furthermore, the first-level feature representation and the dynamic feature representation corresponding to the global detection data can be fused to obtain the second-level feature representation corresponding to the global detection data; for example, the feature values ​​at the same position in the first-level feature representation and the dynamic feature representation can be summed to obtain the corresponding second-level feature representation; the second-level feature representation generated in this way can reflect both the information carried by the global detection data itself and the dynamic information of the global detection data relative to its adjacent global detection data.

[0051] As an example, the dynamic feature representation corresponding to the global detection data can also be determined in the following manner: determine the global detection data that is located before the global detection data and adjacent to the global detection data as the associated detection data of the global detection data; then, according to the channel feature filter, use each eigenvalue in the first-level feature representation corresponding to the global detection data and each eigenvalue in the first-level feature representation corresponding to its associated detection data to take the feature deviation to obtain the dynamic feature representation decision coefficient; and then, according to the dynamic feature representation decision coefficient, weight the decision coefficient of each eigenvalue in the first-level feature representation corresponding to the global detection data to obtain the dynamic feature representation corresponding to the global detection data.

[0052] By way of example, for the i-1 (i is an integer greater than 1)th global detection data Di-1, the i-th global detection data Di and the i+1th global detection data Di+1, feature engineering can be first performed on them using a two-dimensional filter to obtain the first-level feature representation Di-1' corresponding to the i-1th global detection data Di-1, the first-level feature representation Di' corresponding to the i-th global detection data Di, and the first-level feature representation Di+1' corresponding to the i+1th global detection data Di+1. For the i-th global detection data Di, the i-1-th global detection data Di-1 can be determined as its associated detection data, and for the i+1-th global detection data Di+1, the i-th global detection data D can be determined as its associated detection data; when further processing the first-level feature representation Di' corresponding to the i-th global detection data Di, a feature filter can be used to take feature deviations from the feature values ​​at the same position in the first-level feature representation Di-1' and the first-level feature representation Di', and the non-maximum suppressor and mapper are used to process the feature deviation results in turn to obtain the dynamic feature representation decision coefficient corresponding to the first-level feature representation Di'; then, the dynamic feature representation decision coefficient can be obtained using the dynamic feature representation decision coefficient. The feature representation decision coefficient is used to weight each eigenvalue in the first-level feature representation Di' with the corresponding decision coefficient to obtain the dynamic feature representation corresponding to the i-th global detection data Di; furthermore, the dynamic feature representation corresponding to the i-th global detection data Di and the first-level feature representation Di' corresponding to the i-th global detection data Di can be taken as the sum of the eigenvalues, that is, the eigenvalues ​​at the same position in the dynamic feature representation and the first-level feature representation Di' are taken as the sum of the eigenvalues ​​to obtain the second-level feature representation Di" corresponding to the global detection data Di. Similarly, for the i+1-th global detection data Di+1, a similar method can be used to obtain its corresponding second-level feature representation Di+1".

[0053] It should be noted that when the above method is used to determine the second-level feature representations associated with each piece of global detection data, for the first piece of global detection data in the global detection data sequence, since there is no adjacent global detection data (that is, there is no global detection data before it), the above method cannot be used to determine the second-level feature representation corresponding to the first piece of global detection data. At this time, the first-level feature representation corresponding to the first piece of global detection data can be directly used as the second-level feature representation; or, the determination of the second-level feature representation corresponding to the first piece of global detection data can be abandoned, even if the number of the second-level feature representations finally generated is one less than the number of global detection data input.

[0054] The above method is used to integrate dynamic feature representation into the feature map corresponding to the global detection data, so that the second-level feature representations generated respectively associated with each global detection data can correspondingly express the information of the global detection data itself and the dynamic information between it and the associated detection data, so that the information carried in the generated second-level feature representation is richer, and defects can be roughly inspected more accurately.

[0055] In a possible implementation, the primary feature engineering unit performs the above processing on the input global detection data sequence, and after obtaining the second-level feature representations respectively associated with the global detection data sequence, a three-dimensional filter, such as a three-dimensional convolution structure, can be further used to perform spatial short-time feature cascade processing on the second-level feature representations respectively associated with the global detection data sequence. That is, for each piece of global detection data, based on the sequence index of the global detection data and the preset sliding search area, the control detection data related to the global detection data can be determined in the global detection data sequence; then, for each piece of global detection data, a three-dimensional filter is used to perform spatial short-time feature cascade according to the second-level feature representation corresponding to the global detection data and the second-level feature representations respectively associated with each control detection data related to the global detection data, to obtain the third-level feature representation corresponding to the global detection data; and then, based on the third-level feature representations respectively associated with the global detection data sequence, multiple first feature representations are determined.

[0056] It should be noted that the third-level feature representation corresponding to the global detection data can be a multi-channel feature map obtained by performing spatial short-time feature cascade processing on the second-level feature representations respectively associated with multiple adjacent global detection data, that is, the features at the same position in multiple adjacent global detection data are fused together, so that the features at each position in the third-level feature representation are integrated with both the feature information of the global detection data itself and the feature information of the global detection data sequence adjacent to it.

[0057] In some embodiments, after obtaining the second-level feature representations respectively associated with each piece of global detection data, a three-dimensional filter can be used to perform spatial short-time feature cascade processing on each second-level feature representation fused with dynamic information. In specific implementation, for each piece of global detection data, the three-dimensional filter can perform spatial short-time feature cascade processing on the second-level feature representation corresponding to the piece of global detection data and the second-level feature representations respectively associated with each control detection data related to the piece of global detection data, to obtain the third-level feature representation corresponding to the piece of global detection data. The so-called spatial short-time feature cascade processing refers to the fusion of the feature cascades at the same position in the second-level feature representation corresponding to the global detection data and the second-level feature representations respectively associated with each control detection data. The control detection data related to the global detection data here are global detection data located within a preset sliding search area based on the sequence index of the global detection data in the global detection data sequence; for example, assuming that the preset sliding search area is a front and back adjacent one, then for the second global detection data in the global detection data sequence, the first global detection data and the third global detection data in the global detection data sequence are the control detection data related to the second global detection data.

[0058] In order to further improve the feature extraction effect, multiple layers of three-dimensional filters can be deployed in the primary feature engineering unit to increase the nonlinear capability and expand the receptive field, so as to further abstract the underlying features and better extract important features in the global detection data.

[0059] In addition, for global detection data with a larger input size, a maximum pooling layer can be further deployed on the basis of the three-dimensional filter to perform non-maximum suppression on the processing results of the three-dimensional filter, thereby compressing the feature representation and reducing redundant information, while reducing the amount of calculation and video memory usage, saving computing resources.

[0060] In this way, the above-mentioned three-dimensional filter is used to perform spatial short-time feature cascade processing on the second-level feature representations respectively associated with each global detection data, so as to obtain the third-level feature representations respectively associated with each global detection data. The obtained third-level feature representation can carry the time and spatial dependency relationship between adjacent global detection data, so that the generated third-level feature representation carries richer information.

[0061] In a possible implementation, the primary feature engineering unit performs the above processing on the input global detection data sequence, and after obtaining the third-level feature representations respectively associated with the global detection data sequence, the recursive network module, such as a recurrent neural network structure, can be further used to perform spatial domain long-term feature cascade processing on the third-level feature representations respectively associated with the global detection data sequence. Specifically, in response to the global detection data being the first global detection data, the corresponding recursive network module is used to perform multiple rounds of feature engineering according to the third-level feature representation corresponding to the global detection data to obtain the first feature representation corresponding to the global detection data; in the process of multiple rounds of feature engineering, spatial domain feature aggregation and temporal domain feature aggregation corresponding to the global detection data will be generated. In response to the global detection data not being the first global detection data, the global detection data located before the global detection data and adjacent to the global detection data is determined as the associated detection data; and the corresponding recursive network module is used to perform multiple rounds of feature engineering according to the third-level feature representation corresponding to the global detection data, and the spatial domain feature aggregation, temporal domain feature aggregation and first feature representation corresponding to the associated detection data to obtain the first feature representation corresponding to the global detection data.

[0062] It should be noted that the first feature representation corresponding to the global detection data is obtained by performing spatial short-time feature cascade processing on the third-level feature representation corresponding to the global detection data and the third-level feature representations respectively associated with each global detection data before the global detection data. It can reflect the static characteristics and dynamic characteristics of the tunnel in the global detection data, as well as the changes in the water flow background in the detection data.

[0063] Step S2032: Perform feature engineering of spatial domain measurement on each of the multiple first feature representations to obtain feature distribution information respectively associated with the multiple first feature representations; perform feature engineering of time domain measurement based on the feature distribution information respectively associated with the multiple first feature representations to obtain first feature distribution information.

[0064] The defect rough inspection model uses the primary feature engineering unit therein to complete the feature engineering of the input global detection data sequence, and after obtaining the corresponding first feature representation sequence (i.e., multiple first feature representations arranged in chronological order), the multiple first feature representations in the first feature representation sequence can be further processed by first spatial domain compression and then temporal domain compression to obtain the first feature distribution information. That is, the first feature fusion unit can first perform feature engineering of spatial domain measurement on each of the multiple first feature representations in the first feature representation sequence to obtain the feature distribution information associated with the multiple first feature representations respectively; then, based on the feature distribution information associated with the multiple first feature representations respectively, feature engineering of temporal measurement is performed to obtain the first feature distribution information.

[0065] It should be noted that the first feature fusion unit is a processing structure for first compressing the first feature representation sequence in the spatial domain and then in the temporal domain. On the basis that the primary feature engineering unit has extracted adjacent spatiotemporal information for each global detection data in the global detection data sequence, the first feature fusion unit uses the processing of first compressing the first feature representation sequence in the spatial domain and then in the temporal domain to more finely characterize the characteristics of the tunnel in the detection.

[0066] In a possible implementation, when the first feature fusion unit is working, it can first use a two-dimensional filter and a downsampler to perform feature engineering of spatial domain measurement on each first feature representation to obtain feature distribution information corresponding to each first feature representation; then, construct a feature vector based on the feature distribution information respectively associated with multiple first feature representations; then, use multiple one-dimensional filters to perform feature engineering of time domain measurement on the feature vector respectively to obtain multiple time-domain reduced dimensionality feature distribution information; the multiple one-dimensional filters here correspond to different sliding search areas respectively; finally, determine the first feature distribution information based on multiple time-domain reduced dimensionality feature distribution information.

[0067] In some embodiments, when the first feature fusion unit is working specifically, for each first feature representation in the first feature representation sequence, the corresponding two-dimensional filter and downsampler can be used to perform deeper feature engineering on the first feature representation in the spatial domain measurement, so as to extract the tunnel wall texture and sonar echo continuity features in the global detection data from the spatial domain measurement. In the spatial domain measurement, the first feature representation is further abstracted using a two-dimensional filter and a downsampler, so that each piece of global detection data can be characterized in a more detailed state, so that the various distinguishing feature information of areas with defects and areas without defects can be better learned, and the missed judgment caused by factors such as the movement of the mother machine and the change of water flow can be reduced. The two-dimensional filter and downsampler can compress the spatial domain information, so as to convert the first feature representation with multiple channels into the corresponding feature distribution information.

[0068] Optionally, after the first feature fusion unit obtains the corresponding feature distribution information by performing feature engineering for each first feature representation, the first feature fusion unit may construct a feature vector using the feature distribution information respectively associated with each first feature representation. Specifically, the first feature fusion unit may use the first N elements of the feature vector to carry the feature distribution information corresponding to the first first feature representation, use the first N elements after the N+1th element of the feature vector to carry the feature distribution information corresponding to the second first feature representation, and so on.

[0069] After the first feature fusion unit constructs the feature vector, it can use multiple one-dimensional filters corresponding to different sliding search areas and multiple one-dimensional filters respectively associated with the maximum pooling layer to perform feature engineering of time domain measurement on the feature vector, thereby obtaining the time domain dimension reduction feature distribution information respectively associated with each one-dimensional filter. In addition to using two-dimensional filters to characterize the features of global detection data in the spatial domain, the first feature fusion unit can further use multiple one-dimensional filters corresponding to different sliding search areas to characterize the tunnel state changes between each piece of data. The use of multi-scale one-dimensional filters can effectively prevent misjudgment caused by jitter between multiple pieces of data.

[0070] As an example, when the multi-scale one-dimensional filter performs feature engineering of time domain measurement on the feature vector, for each one-dimensional filter, according to the sliding search area corresponding to the one-dimensional filter, the feature distribution information of each feature vector can be collected to obtain the candidate feature distribution information set corresponding to the one-dimensional filter; then, each one-dimensional filter is used to perform feature engineering on each candidate feature distribution information set corresponding to the one-dimensional filter, so as to obtain the time domain feature distribution information respectively associated with each candidate feature distribution information set; further, non-maximum suppression is performed on the time domain feature distribution information respectively associated with each candidate feature distribution information set, so as to obtain the time domain dimensionality reduction feature distribution information.

[0071] After using the above-mentioned multi-scale one-dimensional filter to determine the time-domain dimensionality reduction feature distribution information associated with each one-dimensional filter, the time-domain dimensionality reduction feature distribution information associated with multiple one-dimensional filters can be cascaded to obtain cascaded time-domain dimensionality reduction feature distribution information, and then, the first feature distribution information can be determined based on the cascaded time-domain dimensionality reduction feature distribution information.

[0072] In a possible implementation, the first feature fusion unit can determine the first feature distribution information based on the cascaded time domain dimensionality reduction feature distribution information in the following manner: determine the filtering index based on the feature filter and the cascaded time domain dimensionality reduction feature distribution information; use a mapper (such as setting a corresponding fully connected layer) to process the cascaded time domain dimensionality reduction feature distribution information to obtain the control time domain dimensionality reduction feature distribution information; and then, determine the first feature distribution information based on the filtering index, the cascaded time domain dimensionality reduction feature distribution information, and the control time domain dimensionality reduction feature distribution information.

[0073] Since the convolution operation performed by each one-dimensional filter in the above multi-scale one-dimensional filter is performed separately, the cascaded time-domain dimensionality reduction feature distribution information obtained by simply cascading the time-domain dimensionality reduction feature distribution information generated by each one-dimensional filter is difficult to specifically reflect the information that is more valuable for identifying the area with tunnel defects. In order to make the information carried by the cascaded time-domain dimensionality reduction feature distribution information more specifically reflect the more valuable information for the area with tunnel defects, the cascaded time-domain dimensionality reduction feature distribution information is further processed; the cascaded time-domain dimensionality reduction feature distribution information is gated filtered at the feature value level and the global feature fusion is essentially equivalent to further gated filtering and feature conversion of the output results of each convolution kernel in the multi-scale one-dimensional filter; since the sliding search area of ​​each one-dimensional filter is a hyperparameter, the execution of the above processing can further adaptively fuse the multi-scale features and the feature values ​​of each dimension, so that the first feature distribution information finally obtained is more robust.

[0074] It can be understood that in the above embodiment, a feature fusion method of first compressing the spatial domain and then compressing the temporal domain is adopted for the first feature representation sequence. On the basis that the primary feature engineering unit has extracted the first feature representation sequence carrying the spatiotemporal information of adjacent data according to the global detection data sequence, the first feature fusion unit uses a deep two-dimensional filter to further characterize the spatial characteristics in the tunnel, and then uses a one-dimensional filter to perform time domain compression to obtain the post-fusion spatiotemporal feature distribution information of the detection data.

[0075] Step S2033: Perform feature engineering of time domain measurement on the multiple first feature representations to obtain a time domain fusion feature representation; perform feature engineering of spatial domain measurement on the time domain fusion feature representation to obtain second feature distribution information.

[0076] The defect rough inspection model uses the primary feature engineering unit therein to complete the feature engineering of the input global detection data sequence, and after obtaining the corresponding first feature representation sequence (i.e., multiple first feature representations arranged in chronological order), the second feature fusion unit can be further used to perform time domain compression first and then spatial domain compression according to the multiple first feature representations in the first feature representation sequence, thereby obtaining the second feature distribution information. That is, the second feature fusion unit can first perform feature engineering of time domain measurement according to the multiple first feature representations in the first feature representation sequence to obtain the time domain fusion feature representation; then, the time domain fusion feature representation is subjected to feature engineering of spatial domain measurement to obtain the second feature distribution information.

[0077] It should be noted that the second feature fusion unit is a processing structure for compressing the first feature representation sequence in time domain first and then in space domain. On the basis that the primary feature engineering unit has extracted the spatiotemporal information of adjacent data for each global detection data in the global detection data sequence, the second feature fusion unit uses the first time domain compression and then in space domain compression according to the first feature representation sequence to better characterize the information on the change of spatial position in the tunnel over time, and can effectively capture the area where the mutation occurs.

[0078] In a possible implementation, the first feature representation can be obtained by performing multiple rounds of feature engineering on the low-level feature representations respectively associated with the global detection data using a recursive network module. At this time, the second feature fusion unit can perform time domain compression first and then spatial domain compression according to the first feature representation sequence in the following manner: for each piece of global detection data, obtain multiple intermediate-level feature representations corresponding to the global detection data, where the multiple intermediate-level feature representations are generated in the process of performing multiple rounds of feature engineering on the low-level feature representation corresponding to the global detection data, and these multiple intermediate-level feature representations correspond to different feature engineering levels; then, based on the multiple process feature representations and the multiple first feature representations respectively associated with the global detection data sequence, perform feature engineering of time domain measurement to obtain a time domain fused feature representation; and then, use a two-dimensional filter and a downsampler to perform feature engineering of spatial domain measurement on the time domain fused feature representation to obtain second feature distribution information.

[0079] As introduced in the aforementioned step S2031, the primary feature engineering unit in the defect rough inspection model can first use a two-dimensional filter to perform feature engineering on each piece of global detection data to obtain a first-level feature representation corresponding to the global detection data; then, based on the first-level feature representation corresponding to the global detection data and the first-level feature representation corresponding to the global detection data adjacent to the global detection data, the first-level feature representation corresponding to the global detection data is dynamically supplemented to obtain a second-level feature representation corresponding to the global detection data; then, a three-dimensional filter is used to perform spatial short-time feature cascade processing on the second-level feature representations corresponding to multiple global detection data to obtain a third-level feature representation corresponding to the global detection data; and then, a recursive network module is used to perform multiple rounds of feature engineering on the third-level feature representation corresponding to the global detection data to obtain the first feature representation corresponding to the global detection data. Each time the recurrent neural network performs feature engineering, a process feature representation with the same size as the input third-level feature representation is generated.

[0080] In the processing process of the above-mentioned primary feature engineering unit, the third-level feature representation corresponding to the global detection data obtained can be regarded as the low-level feature representation corresponding to the global detection data; in the process of performing multiple rounds of feature engineering on the third-level feature representation corresponding to the global detection data using the recursive network module, the feature graph generated by each feature engineering can be regarded as the process feature representation corresponding to the global detection data, and different process feature representations correspond to different feature engineering levels. For example, the process feature representation obtained by the first feature engineering corresponds to a feature engineering level of 1, the process feature representation obtained by the second feature engineering corresponds to a feature engineering level of 2, and so on.

[0081] In some embodiments, for each first feature representation, the second feature fusion unit can correspondingly obtain each process feature representation generated in the process of generating the first feature representation from the recursive network module in the primary feature engineering unit; then, based on the obtained process feature representations and each first feature representation, feature engineering of time domain measurement, that is, time domain compression processing, is performed to obtain a time domain fused feature representation.

[0082] As an example, the second feature fusion unit can obtain the above-mentioned time domain fusion feature representation in the following manner: for each feature engineering level, determine the process feature representations corresponding to the feature engineering level among the multiple process feature representations respectively associated with the global detection data sequence; for each feature engineering level, perform non-maximum suppression along the time domain metric on the process feature representations corresponding to the feature engineering level to obtain the time domain feature representation corresponding to the feature engineering level; then, cascade the time domain feature representations respectively associated with various feature engineering levels and the first feature representation with the highest sequence index to obtain the cascaded feature representation; and determine the time domain fusion feature representation based on the cascaded feature representation.

[0083] Optionally, the second feature fusion unit can cascade the time domain feature representations respectively associated with each feature engineering level and the first feature representation with the highest sequence index obtained to obtain a cascade feature representation; and determine the time domain fusion feature representation based on the cascade feature representation. Since the above cascade feature representation is a feature representation that carries multi-dimensional information, accordingly, the time domain fusion feature representation determined based on the cascade feature representation can also reflect the multi-dimensional information carried by the cascade feature representation.

[0084] Considering that the cascade feature representation obtained by simple cascading is difficult to highlight which features are more important, and there is a certain amount of information redundancy, in order to make the determined time domain fusion feature representation better reflect the more important information for tunnel defect area detection, the second feature fusion unit can further fuse the information of each channel in the cascade feature representation, and adjust the decision coefficient of each channel based on the overall global information, so as to strengthen the more important information for tunnel defect area detection in the cascade feature representation according to the adjusted decision coefficient, while reducing redundant information.

[0085] Specifically, the second feature fusion unit can use a residual structure to adjust the decision coefficient; that is, the second feature fusion unit can first perform size compression and feature mapping on the cascade feature representation to obtain a control decision coefficient; then, according to the control decision coefficient, the decision coefficient of each eigenvalue in the cascade feature representation is weighted to obtain a control feature representation; then, the sum of the eigenvalues ​​in the control feature representation and the eigenvalues ​​in the cascade feature representation is taken to obtain a time domain fusion feature representation.

[0086] For example, the second feature fusion unit can input the cascade feature representation of size H*W*C (H is the height of the cascade feature representation, W is the width of the cascade feature representation, and C is the number of channels included in the cascade feature representation) into the residual structure, and the residual structure can use the cascade feature representation to perform size compression and feature mapping in sequence to obtain a comparison decision coefficient, which can reflect the importance of the information carried by each channel in the cascade feature representation. Then, the residual structure can multiply the obtained comparison decision coefficient with the eigenvalue carried by the corresponding channel in the cascade feature representation to obtain a comparison feature representation; using the comparison decision coefficient to weight the eigenvalue carried by the corresponding channel in the cascade feature representation, it can effectively strengthen the information in the cascade feature representation that is more valuable for reference to tunnel defect detection, while weakening the redundant information in the cascade feature representation. In order to avoid excessive loss of the original input information, the residual structure can fuse the control feature representation with the original cascade feature representation, that is, the sum of the eigenvalues ​​in the control feature representation and the eigenvalues ​​in the cascade feature representation is taken to obtain the time domain fusion feature representation. The time domain fusion feature representation can more prominently reflect the information that has reference value for tunnel defect detection while reducing redundant information.

[0087] After the second feature fusion unit obtains the time-domain fusion feature representation by the above processing, it can further use several layers to compress the obtained time-domain fusion feature representation in spatial position to obtain the second feature distribution information. For example, by using multiple layers of standard two-dimensional convolution operations and pooling operations, a deeper level of feature extraction is performed on the time-domain fusion feature representation that retains the spatial structure, and the multi-channel time-domain fusion feature representation is further compressed into a high-level semantic vector of a fixed size, and the high-level semantic vector is used to represent the second feature distribution information.

[0088] Among them, the above-mentioned second feature fusion unit adopts a pre-fusion structure of first time domain compression and then spatial domain compression for the first feature representation sequence. On the basis that the primary feature engineering unit has extracted the first feature representation sequence carrying the spatiotemporal information of adjacent data according to the global detection data sequence, the second feature fusion unit uses the time domain compression and fusion processing of the first feature representation sequence to better characterize the change information of the spatial position in the detection data over time, and then uses a deep two-dimensional filter to spatially characterize the overall situation of the static and dynamic features in the tunnel to obtain the feature distribution of the detection data.

[0089] It should be understood that in practical applications, a second feature fusion unit with other structures can also be designed to complete the time domain compression and then spatial domain compression processing of multiple first feature representations. This application does not make any limitation on the specific working principle of the second feature fusion unit.

[0090] It should be noted that the defect rough inspection model can run the first feature fusion unit first and then the second feature fusion unit, or can run the second feature fusion unit first and then the second feature fusion unit, or can run the first feature fusion unit and the second feature fusion unit at the same time. The present application does not impose any limitation on the operating order of the first feature fusion unit and the second feature fusion unit, that is, the present application does not impose any limitation on the execution order of step S2032 and step S2033.

[0091] Step S2034: Determine a rough inspection result based on the first feature distribution information and the second feature distribution information.

[0092] The defect rough inspection model uses the first feature fusion unit and the second feature fusion unit therein to obtain the first feature distribution information and the second feature distribution information respectively, and can further determine the areas in the tunnel that are more likely to have defects based on the first feature distribution information and the second feature distribution information.

[0093] Exemplarily, the defect rough inspection model can concatenate and merge the first feature distribution information and the second feature distribution information to obtain the target feature distribution information; then, using a mapper, the target feature distribution information is subjected to nonlinear processing and dimensional conversion processing; finally, a classifier is used to output a prediction score, which can reflect whether there are defects in each area in the detection data. For example, in response to the prediction score being lower than a preset confidence threshold, it can be determined that the area is more likely to have defects.

[0094] When using the defect rough inspection model to detect candidate inspection areas in the tunnel that are more likely to have defects, the first feature fusion unit in the model is used to first compress the multiple first feature representations in the spatial domain and then in the temporal domain, so as to more carefully describe the characteristics of the tunnel in a single piece of global inspection data and better characterize the static characteristics of each area of ​​the tunnel. In addition, using the second feature fusion unit in the model, multiple first feature representations are first compressed in the temporal domain and then in the spatial domain, which can better describe the state changes of each area in the tunnel. Finally, the corresponding inspection results are determined by combining the above-mentioned first feature distribution information and the second feature distribution information, so as to achieve a comprehensive consideration of the static and dynamic characteristics of the tunnel when using robots for inspection, thereby ensuring the accuracy of the inspection results determined accordingly.

[0095] In some embodiments, the defect precision inspection model provided by the embodiment of the present invention can be obtained by the following method, for example, referring to Figure 4 As shown, Figure 1 The training of the defect precision inspection model involved in step S206 may include the following steps S401 to S404:

[0096] Step S401: obtaining a first example defect data set and a second example defect data set.

[0097] The first example defect data set and the second example defect data set each include multiple example defect data groups, each example defect data group includes at least one fine inspection defect data of a defect type, and the first example defect data set and the second example defect data set correspond to the same defect type.

[0098] For example, the first example defect data set includes example defect data set 1, example defect data set 2, and example defect data set 3, and example defect data set 1 includes at least one fine-detection defect data associated with category 1, example defect data set 2 includes at least one fine-detection defect data associated with category 2, and example defect data set 3 includes at least one fine-detection defect data associated with category 3. For the second example defect data set, the second example defect data set also includes three example defect data sets, and each example defect data set includes at least one fine-detection defect data associated with category 1, category 2, and category 3, respectively.

[0099] The fine inspection defect data included in the example defect data group corresponding to the same defect type in the first example defect data set and the second example defect data set may be the same or different, and there is no limitation here. Also, for any fine inspection defect data example defect data group in the first example defect data set and the second example defect data set, there is no limitation on the number of fine inspection defect data in the example defect data group.

[0100] As an example, for a certain data set, a certain number of defect types can be determined from the data set, and a certain number of data from each defect type in the data set is obtained as fine inspection defect data to construct a first example defect data set. Further, a certain number of data from each defect type in the data set is also obtained as fine inspection defect data to construct a second example defect data set.

[0101] Among them, if after the initial defect precision inspection model is trained with the first example defect data set and the second example defect data set constructed according to the above method, the corresponding training cost parameter still does not reach the update termination threshold, then a new first example defect data set and a second example defect data set are reconstructed according to the above method, and the model parameters of the initial defect precision inspection model are continued to be updated according to the new first example defect data set and the second example defect data set until the final defect precision inspection model is obtained.

[0102] Step S402: input the training fine inspection data and each fine inspection defect data in the first example defect data set into the initial defect fine inspection model to obtain a first detection association index associated with each defect type of the training fine inspection data, where the training fine inspection data is any fine inspection defect data in the second example defect data set.

[0103] In each training process, the input of the initial defect fine inspection model is each fine inspection defect data in the first example defect data set and one fine inspection defect data in the second example defect data set, and the fine inspection defect data in the second example defect data set input into the initial defect fine inspection model is the training fine inspection data in this training process.

[0104] Alternatively, in each training process, when one fine inspection defect data in the second example defect data set is input into the initial defect fine inspection model as training fine inspection data, for the first example defect data set, at least one fine inspection defect data in each example defect data group can be input into the initial defect fine inspection model, that is, at least one fine inspection defect data corresponding to each defect type in the first example defect data set is input into the initial defect fine inspection model.

[0105] Among them, the initial defect fine inspection model includes a feature engineering network. After the training fine inspection data and the fine inspection defect data in the first example defect data set are input into the initial defect fine inspection model, the first data feature representation of the training fine inspection data and the second data feature representation of each fine inspection defect data in the first example defect data set can be determined according to the feature engineering network of the initial defect fine inspection model.

[0106] Specifically, the second data feature representation of each fine-inspection defect data in the first example defect data set can be determined according to the feature engineering network in the initial defect fine-inspection model. For the second data feature representation of the fine-inspection defect data corresponding to each defect type, the defect feature representation corresponding to the defect type can be determined according to each second data feature representation corresponding to the defect type. The defect feature representation corresponding to each defect type is used to characterize the features of the fine-inspection defect data of the defect type.

[0107] Among them, for each defect type, when determining the defect feature representation corresponding to the defect type, the second data feature representations corresponding to the defect type can be averaged, and the final result can be used as the defect feature representation corresponding to the defect type. Alternatively, the second data feature representations corresponding to the defect type can be feature fused according to the feature fusion network to obtain the defect feature representation corresponding to the defect type.

[0108] In some embodiments, after inputting each fine-inspection defect data in a certain example defect data group in the first example defect data set into the initial defect fine-inspection model, the second data feature representation of each fine-inspection defect data can be determined according to the feature engineering network in the initial defect fine-inspection model, such as determining the second data feature representation of three fine-inspection defect data in the example defect data group. Further, the corresponding feature values ​​of each second data feature representation can be averaged to obtain the defect feature representation corresponding to the defect type of the example defect data group.

[0109] Furthermore, after determining the defect feature representation corresponding to each defect type and the first data feature representation of the training fine inspection data, the feature multi-tuple of the training fine inspection data corresponding to each defect type can be determined based on the defect feature representation corresponding to each defect type and the first data feature representation of the training fine inspection data.

[0110] Among them, the training fine inspection data corresponds to a feature multi-group of each defect type, including a focused data feature representation after focusing processing on the first data feature representation according to the first data feature representation and the defect feature representation corresponding to the defect type, and a focused defect feature representation after focusing processing on the defect feature representation corresponding to the defect type.

[0111] That is, the defect feature representation corresponding to each defect type and the first data feature representation of the training fine inspection data can be input into the focusing processing network in the initial defect fine inspection model to perform focusing processing on the first data feature representation and the defect feature representation corresponding to the defect type respectively, so as to obtain the focused data feature representation and the focused defect feature representation after focusing processing.

[0112] For example, if the defect feature representation corresponding to defect type a is pa, and the first data feature representation of the training fine inspection data is b, then the defect feature representation pa and the first data feature representation b corresponding to defect type a are input into the focusing processing network to obtain the focused data feature representation Ba and the focused defect feature representation Pa after focusing processing, and [Pa, Ba] can be determined as the feature multi-tuple corresponding to defect type a.

[0113] Furthermore, based on the feature multiplicity group corresponding to each defect type of the training fine inspection data, the first detection association index associated with the training fine inspection data and each defect type can be determined. That is to say, for any feature multiplicity group, the first detection association index associated with the training fine inspection data and the defect type corresponding to the feature multiplicity group can be determined based on the feature multiplicity group.

[0114] Among them, for the feature multi-group corresponding to each defect type of the training fine inspection data, the similarity between the focused defect feature representation and the focused data feature representation in the feature multi-group can be determined. The higher the similarity between the focused defect feature representation and the focused data feature representation in the feature multi-group, the higher the probability that the training fine inspection data is associated with the defect type corresponding to the feature multi-group. Based on this, the similarity between the focused defect feature representation and the focused data feature representation in each feature multi-group can be determined according to the first detection network in the initial defect fine inspection model, and the first detection association index associated with the training fine inspection data for each defect type can be determined according to the similarity corresponding to each feature multi-group.

[0115] Among them, the measurement method of the similarity between the focused defect feature representation and the focused data feature representation in each feature multi-group can be determined based on cosine similarity, learnable measurement method, Euclidean distance, Manhattan distance or Minkowski distance, etc., and is not limited here.

[0116] In an embodiment of the present invention, the focus processing network in the initial defect inspection model can be a dynamic graph convolution cross-correlation attention module, or it can be other attention networks, which is not limited here.

[0117] In an embodiment of the present invention, for each defect type, when determining the feature multi-tuple of the training fine inspection data corresponding to the defect type, a cross-correlation function between the defect feature representation corresponding to the defect type and the first data feature representation can be determined, and the cross-correlation function can be used for the association relationship between the defect feature representation and the first data feature representation.

[0118] Furthermore, according to the defect feature representation corresponding to the defect type, the first data feature representation of the training fine inspection data and the cross-correlation function, a first focused feature vector corresponding to the defect type and a second focused feature vector corresponding to the first data feature representation are determined.

[0119] For example, the cross-correlation function is used as the dynamic adjacency matrix of graph convolution, and then the defect feature representation and the first data feature representation corresponding to the defect type are processed according to the graph convolution network to obtain the first focused feature vector corresponding to the defect type and the second focused feature vector corresponding to the first data feature representation.

[0120] Further, the focused defect feature representation corresponding to the defect type can be determined based on the defect feature representation corresponding to the defect type and the first focused feature vector. That is, the defect feature representation corresponding to the defect type is focused based on the first focused feature vector of the defect type to obtain the focused defect feature representation after the focused processing. Similarly, the focused data feature representation corresponding to the training fine inspection data is determined based on the first data feature representation and the second focused feature vector. That is, the first data feature representation of the feature multi-tuple of the second focused feature vector is focused to obtain the enhanced focused data feature representation corresponding to the training fine inspection data.

[0121] Among them, for each defect type, when determining the cross-correlation function between the defect feature representation corresponding to the defect type and the first data feature representation, a first cross-correlation sub-function between the defect feature representation of the defect type and the first data feature representation and a second cross-correlation function between the first data feature representation and the defect feature representation corresponding to the defect type can be determined, and then based on the first cross-correlation sub-function, the second cross-correlation sub-function and the unit matrix, the cross-correlation function between the defect feature representation corresponding to the defect type and the first data feature representation can be determined.

[0122] In some embodiments, for a certain defect type, a cross-correlation function between a defect feature representation corresponding to the defect type and a first data feature representation of the training fine inspection data is determined, and used as a dynamic adjacency matrix of a graph convolutional network in a focusing processing network. The defect feature representation corresponding to the defect type and the first data feature representation are further subjected to graph convolution processing, and finally a first focused feature vector corresponding to the defect type and a second focused feature vector corresponding to the first data feature representation are obtained.

[0123] The defect feature representation corresponding to the defect type is multiplied by the first focused feature vector to obtain a first result, and the defect feature representation corresponding to the defect type is added to the first result as an offset to finally obtain a focused defect feature representation corresponding to the defect type.

[0124] Similarly, the first data feature representation and the second focused feature vector are multiplied to obtain the second result, and the first data feature representation is added to the second result as a bias, and finally the focused data feature representation corresponding to the training fine inspection data is obtained. Thus, the above-mentioned focused defect feature representation and focused data feature representation can be determined as the feature multi-tuple corresponding to the defect type of the training fine inspection data.

[0125] By focusing the defect feature representation of each defect type and the first data feature representation, the characteristics of the target area in the precision inspection defect data can be highlighted and the background feature interference can be reduced, so that the final focused defect feature representation and focused data feature representation have better feature characterization capabilities.

[0126] Step S403: Determine the first true correlation index of the training fine inspection data associated with each defect type according to the target defect type corresponding to the training fine inspection data, and determine the training cost parameter according to each first detection correlation index and each first true correlation index.

[0127] Specifically, the target defect type corresponding to the training fine inspection data is the defect type actually corresponding to the training fine inspection data. According to the target defect type corresponding to the training fine inspection data, it can be determined that the first real correlation index of the training fine inspection data associated with the target defect type is 1, and the first real correlation index associated with other defect types is 0.

[0128] Based on this, a first cost parameter can be determined according to the first detection correlation index and each first true correlation index associated with each defect type in the training fine inspection data, and the first cost parameter can be used as the training cost parameter.

[0129] Optionally, a first predicted defect type of the training fine inspection data can be determined based on a first detection correlation index associated with each defect type in the training fine inspection data, and the actual type of the training fine inspection data (i.e., the target defect type) can be determined based on a first real correlation index associated with each defect type in the training fine inspection data. Furthermore, a first cost parameter can be determined based on the first predicted defect type and the actual defect type corresponding to the training fine inspection data, and the first cost parameter can be used as the training cost parameter.

[0130] In the embodiment of the present invention, for each example defect data group in the second example defect data set, the example defect data group also includes augmented defect data corresponding to multiple augmentation methods for each fine-inspection defect data in the example defect data group. That is to say, for one fine-inspection defect data in each example defect data group in the second example defect data set, the example defect data group also includes augmented defect data corresponding to multiple augmentation methods for the fine-inspection defect data, so that in the model training process, the same fine-inspection defect data corresponding to the augmentation method is used as the training fine-inspection data in different training processes, and while increasing the number of training samples, the defect fine-inspection model finally obtained can perform category prediction for the training fine-inspection data of different augmentation methods, which helps to improve the generalization ability of the model.

[0131] Among them, the above-mentioned augmentation method can be determined according to the actual application scenario requirements and is not limited here.

[0132] In this case, the initial defect precision inspection model also includes an augmented prediction network. During each training process, the feature multiplicity group of the training precision inspection data corresponding to the target defect type (actual defect type) can be determined from the feature multiplicity groups of the training precision inspection data corresponding to each defect type and input into the augmented prediction network to obtain the second detection association index of the training precision inspection data corresponding to each augmentation method.

[0133] Furthermore, the second real correlation index associated with each augmentation method can be determined according to the actual augmentation method of the training refined inspection data, and then the second cost parameter can be determined according to each second detection correlation index and each second real correlation index corresponding to the training refined inspection data. And the training cost parameter can be determined according to the first cost parameter and the second cost parameter.

[0134] In some embodiments, each example defect data group in the first example defect data set and the training fine inspection data in the second example defect data set are input into the feature engineering network in the initial defect fine inspection model to obtain the defect feature representation corresponding to each defect type and the first data feature representation corresponding to the training fine inspection data.

[0135] The defect feature representation corresponding to each defect type and the first data feature representation corresponding to the training fine inspection data are input into the focus processing network to obtain a feature multi-group corresponding to each defect type of the training fine inspection data.

[0136] According to the feature multi-tuple corresponding to each defect type of the training fine inspection data, the first detection correlation index of the training fine inspection data associated with each defect type is obtained through the first prediction network, and the first cost parameter L1 is determined according to the first real correlation index and the first detection correlation index of the training fine inspection data associated with each defect type.

[0137] When the target defect type (actual defect type) corresponding to the training fine inspection data is defect type a, the feature multi-tuple is input into the augmented prediction network to obtain the second detection association index of the training fine inspection data corresponding to each augmentation method.

[0138] After determining the second true correlation index of the training refined inspection data corresponding to each augmentation method, the second cost parameter L2 can be determined according to each second true correlation index and each second detection correlation index, and the training cost parameter L can be determined according to the first cost parameter L1 and the second cost parameter.

[0139] In an embodiment of the present invention, the initial defect fine inspection model also includes a second detection network, which is used to determine a third detection association index of the training fine inspection data associated with each defect type based on a feature multi-tuple corresponding to the target defect type of the training fine inspection data.

[0140] That is, in each training process, the feature multi-tuple of the training fine inspection data corresponding to the target defect type (actual defect type) can be input into the second detection network to obtain the third detection association index of the training fine inspection data associated with each defect type.

[0141] Furthermore, a third cost parameter may be determined based on the first true correlation index and the third detection correlation index of the training fine inspection data associated with each defect type, and a training cost parameter may be determined based on the first cost parameter and the third cost parameter.

[0142] In an embodiment of the present invention, the initial defect inspection model may include a first detection network, a second detection network and an augmented prediction network at the same time, and then the first cost parameter, the second cost parameter and the third cost parameter may be determined according to the above implementation method, thereby determining the training cost parameter according to the first cost parameter, the second cost parameter and the third cost parameter.

[0143] In some embodiments, each example defect data group in the first example defect data group and the training fine inspection data in the second example defect data group are input into the feature engineering network in the initial defect fine inspection model to obtain the defect feature representation corresponding to each defect type and the first data feature representation corresponding to the training fine inspection data.

[0144] The defect feature representation corresponding to each defect type and the first data feature representation corresponding to the training fine inspection data are input into the focus processing network to obtain a feature multi-group corresponding to each defect type of the training fine inspection data.

[0145] According to the feature multi-tuple corresponding to each defect type of the training fine inspection data, the first detection correlation index associated with each defect type of the training fine inspection data is obtained through the first detection network, and the first cost parameter L1 is determined according to the first real correlation index and the first detection correlation index associated with each defect type of the training fine inspection data.

[0146] When the target defect type (actual defect type) corresponding to the training fine inspection data is defect type a, the corresponding feature multi-group is input into the augmented prediction network to obtain the second detection correlation index of the training fine inspection data corresponding to each augmentation method. According to each second real correlation index and each second detection correlation index, the second cost parameter L2 is determined, and the feature multi-group is input into the second detection network to obtain the third detection correlation index of the training fine inspection data associated with each pre-type. According to the first real correlation index and the third detection correlation index associated with each defect type of the training fine inspection data, the third cost parameter L3 is determined.

[0147] The training cost parameter L is determined according to the first cost parameter L1, the second cost parameter L2 and the third cost parameter L3.

[0148] Optionally, the training cost parameter can be determined according to the following method: L = DL1 + [(w2 + λ) L2 + b2] + [(w3 + λ) L3 + b3]. Wherein, w2 is the loss weight of L2, and b2 is the loss correction parameter of L2. w3 is the loss weight of L3, and b2 is the loss correction parameter of L3. D is the loss weight of L1, and λ is the weight correction parameter. D and λ can be determined according to actual needs, such as D can be 1 / 2, and λ can be 1.5, which is not limited here.

[0149] The learning weights corresponding to the augmented prediction network can be used to balance the training tasks of the first detection network and the augmented prediction network. The learning weights corresponding to the second detection network can be used to balance the training tasks of the first detection network and the second detection network. In this way, during the model parameter update process, the learning weights can be used to improve the representation ability of the features output by each network in the initial defect precision inspection model, thereby improving the prediction effect of the final defect precision inspection model.

[0150] In the embodiment of the present invention, when determining any cost parameter among the first cost parameter, the second cost parameter and the third cost parameter, it can be determined according to the cross entropy loss function or other loss functions, which is not limited here.

[0151] Step S404: updating the model parameters of the initial defect fine inspection model according to the first example defect data set and the second example defect data set, terminating the parameter update when the training cost parameter reaches the update termination threshold, and determining the model at the time of terminating the parameter update as the defect fine inspection model.

[0152] After determining the training cost parameters corresponding to training the initial defect precision inspection model, the model parameters of the initial defect precision inspection model can be updated according to the training cost parameters and the first example defect data set and the second example defect data set until the parameter update is terminated when the update termination threshold is reached.

[0153] Among them, the above-mentioned update termination threshold can be the training cost size corresponding to when the training cost parameter reaches convergence, or it can be that the difference between two adjacent training cost parameters for a preset number of consecutive times is less than a preset threshold, etc., which can be determined according to the actual application scenario requirements and is not limited here.

[0154] For each training, the parameter update is terminated when the training cost parameter meets the update termination threshold, and the initial defect precision inspection model at the end of the training is determined as the final defect precision inspection model. If the training cost parameter does not meet the update termination threshold during the training of the initial defect precision inspection model, the model parameters of the initial defect precision inspection model can continue to be updated, and the adjusted model is trained again according to the above method and the training cost parameter is determined, until the parameter update is terminated when the training cost parameter reaches the update termination threshold.

[0155] Optionally, after completing the training and obtaining the final model, the third example defect data set can be determined in the same way as the first example defect data set, and the model at the end of the training can be inferred and verified based on the third example defect data set. The model at the end of the training can be optimized based on the verification results to obtain the final defect precision inspection model.

[0156] The defect types corresponding to the third example defect data set and the first example defect data set and / or the second example defect data set may be the same or different, and the number of fine inspection defect data of each defect type is not limited.

[0157] In the embodiment of the present invention, after obtaining the defect fine inspection model, the actually collected fine inspection data can be input into the defect fine inspection model to obtain the fine inspection result, that is, the fine inspection candidate area is associated with multiple types of fourth detection association indicators, and then the type corresponding to the highest fourth detection association indicator is determined as the defect type of the fine inspection candidate area. It is worth noting that if the highest fourth detection association indicator is still lower than the preset confidence threshold, it can be determined that there is no defect in the fine inspection candidate area.

[0158] On the other hand, an embodiment of the present invention further provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned method of inspecting tunnels with a mother-and-child robot. Figure 2 As shown, Figure 2 The block diagram of the computer device 100 provided in the embodiment of the present invention. The computer device 100 includes a memory 111, a processor 112 and a communication unit 113. To achieve data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are directly or indirectly electrically connected to each other. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0159] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in light of the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended intended application.

Claims

1. A method for inspecting tunnels using a mother-and-child robot, characterized in that: The mother-child robot comprises a mother robot and a child robot, and the method comprises: Controlling the mother machine to move in the tunnel according to a preset route; Acquire a global detection data sequence obtained by a rough detection device carried by a mother machine to detect and collect the tunnel, wherein the rough detection device includes at least one of a sonar sensor and a laser scanning device; Inputting the global detection data sequence into a defect rough inspection model to obtain a rough inspection result, wherein the rough inspection result is used to indicate a candidate fine inspection area within the detection range of the rough inspection device where a confidence level of a defect is higher than a preset threshold; releasing the sub-machine to the candidate area for fine inspection; Acquire precision inspection data obtained by a precision inspection device carried by the sub-machine to detect the precision inspection candidate area, wherein the precision inspection device includes at least one of a high-definition image sensor and a laser ruler measuring instrument; Inputting the fine inspection data into a pre-trained defect fine inspection model to obtain a fine inspection result, wherein the fine inspection result is used to indicate whether there is a defect in the fine inspection candidate area; The step of inputting the global detection data sequence into a defect rough inspection model to obtain a rough inspection result includes: Performing feature engineering on the global detection data sequence to obtain a plurality of first feature representations of serialization corresponding to the global detection data sequence; Performing feature engineering of spatial domain measurement on each of the multiple first feature representations to obtain feature distribution information respectively associated with the multiple first feature representations; performing feature engineering of time domain measurement based on the feature distribution information respectively associated with the multiple first feature representations to obtain first feature distribution information; Performing feature engineering of time domain measurement on the multiple first feature representations to obtain a time domain fusion feature representation; performing feature engineering of spatial domain measurement on the time domain fusion feature representation to obtain second feature distribution information; Determine a rough inspection result based on the first feature distribution information and the second feature distribution information; The defect precision inspection model is obtained by the following method: Acquire a first example defect data set and a second example defect data set, wherein the first example defect data set and the second example defect data set include a plurality of example defect data groups, each of the example defect data groups includes at least one fine inspection defect data of a defect type; Inputting training fine inspection data and each fine inspection defect data in the first example defect data set into an initial defect fine inspection model to obtain a first detection association index associated with each of the defect types by the training fine inspection data, wherein the training fine inspection data is any fine inspection defect data in the second example defect data set; Determine, according to the target defect types corresponding to the training fine inspection data, a first real correlation index associated with each defect type of the training fine inspection data, and determine a training cost parameter according to each of the first detection correlation indexes and each of the first real correlation indexes; Update the model parameters of the initial defect precision inspection model according to the first example defect data set and the second example defect data set, terminate the parameter update when the training cost parameter reaches the update termination threshold, and determine the model when the parameter update is terminated as the defect precision inspection model; The first detection correlation index of the training fine inspection data associated with each of the defect types is determined according to the following method: Determine a first data feature representation of the training fine inspection data and a second data feature representation of each fine inspection defect data in the first example defect data set; Determine the defect feature representation corresponding to each defect type according to the second data feature representation of each fine inspection defect data corresponding to each defect type; Determine a feature tuple corresponding to each defect type of the training fine inspection data, wherein the feature tuple corresponding to each defect type of the training fine inspection data includes a focused data feature representation after focusing the first data feature representation according to the first data feature representation and the defect feature representation corresponding to the defect type, and a focused defect feature representation after focusing the defect feature representation corresponding to the defect type; According to the feature multi-tuple group corresponding to each defect type of the training fine inspection data, a first detection correlation index associated with each defect type of the training fine inspection data is determined.

2. The method according to claim 1, characterized in that The performing feature engineering on the global detection data sequence to obtain a plurality of serialized first feature representations corresponding to the global detection data sequence includes: Using a two-dimensional filter, performing feature engineering on each of the global detection data sequences to obtain first-level feature representations respectively associated with the global detection data sequences; For each piece of global detection data, determine the dynamic feature representation corresponding to the global detection data based on the first-level feature representation corresponding to the global detection data and the first-level feature representation corresponding to the global detection data adjacent to the global detection data; determine the second-level feature representation corresponding to the global detection data based on the first-level feature representation and the dynamic feature representation corresponding to the global detection data; The plurality of first feature representations are determined based on the second-level feature representations respectively associated with the global detection data sequences.

3. The method according to claim 2, characterized in that The determining, based on the first-level feature representation corresponding to the global detection data and the first-level feature representation corresponding to the global detection data adjacent to the global detection data, the dynamic feature representation corresponding to the global detection data comprises: Determine global detection data that is located before the global detection data and adjacent to the global detection data as associated detection data of the global detection data; The feature filter uses each feature value in the first-level feature representation corresponding to the global detection data and each feature value in the first-level feature representation corresponding to its associated detection data to obtain a feature deviation to obtain a dynamic feature representation decision coefficient; According to the dynamic feature representation decision coefficient, each feature value in the first-level feature representation corresponding to the global detection data is weighted by the decision coefficient to obtain the dynamic feature representation corresponding to the global detection data.

4. The method according to claim 2, characterized in that: The determining the plurality of first feature representations based on the second-level feature representations respectively associated with the global detection data sequence comprises: For each piece of global detection data, based on the sequence index of the global detection data and a preset sliding search area, determining control detection data related to the global detection data in the global detection data sequence; For each piece of global detection data, using a three-dimensional filter, according to the second-level feature representation corresponding to the global detection data and the second-level feature representations respectively associated with each of the control detection data related to the global detection data, a spatial domain short-time feature cascade is performed to obtain a third-level feature representation corresponding to the global detection data; Determining the plurality of first feature representations based on the third-level feature representations respectively associated with the global detection data sequences; The determining the plurality of first feature representations based on the third-level feature representations respectively associated with the global detection data sequence comprises: In response to the global detection data being the first piece of global detection data, using a corresponding recursive network module, performing multiple rounds of feature engineering according to the third-level feature representation corresponding to the global detection data to obtain a first feature representation corresponding to the global detection data; in the process of the multiple rounds of feature engineering, generating a spatial domain feature aggregation and a temporal domain feature aggregation corresponding to the global detection data; In response to the fact that the global detection data is not the first global detection data, the global detection data that is located before the global detection data and adjacent to the global detection data is determined as the associated detection data; and using the corresponding recursive network module, multiple rounds of feature engineering are performed according to the third-level feature representation corresponding to the global detection data, and the spatial feature aggregation, time domain feature aggregation and first feature representation corresponding to the associated detection data to obtain the first feature representation corresponding to the global detection data.

5. The method according to claim 1, characterized in that The step of performing feature engineering of spatial domain measurement on each of the plurality of first feature representations to obtain feature distribution information respectively associated with the plurality of first feature representations; and performing feature engineering of time domain measurement based on the feature distribution information respectively associated with the plurality of first feature representations to obtain first feature distribution information includes: Using a two-dimensional filter and a downsampler, perform feature engineering of spatial domain measurement on each of the first feature representations to obtain feature distribution information corresponding to each of the first feature representations; constructing a feature vector based on feature distribution information respectively associated with the plurality of first feature representations; Using multiple one-dimensional filters, respectively perform time-domain measurement feature engineering on the feature vector to obtain multiple time-domain dimension reduction feature distribution information; the multiple one-dimensional filters respectively correspond to different sliding search areas; Determine the first feature distribution information based on the multiple time-domain dimensionality reduction feature distribution information; The method of using multiple one-dimensional filters to perform time domain measurement feature engineering on the feature vectors to obtain multiple time domain dimension reduction feature distribution information includes: For each of the one-dimensional filters, according to the sliding search area corresponding to the one-dimensional filter, the feature distribution information of each feature in the feature vector is collected to obtain a candidate feature distribution information set corresponding to the one-dimensional filter; For each of the one-dimensional filters, using the one-dimensional filter, perform feature engineering on each candidate feature distribution information set corresponding to the one-dimensional filter to obtain time domain feature distribution information respectively associated with each candidate feature distribution information set; perform non-maximum suppression on the time domain feature distribution information respectively associated with each candidate feature distribution information set to obtain the time domain dimensionality reduction feature distribution information; The determining the first feature distribution information based on the multiple time-domain dimensionality reduction feature distribution information includes: Cascading the plurality of time-domain dimensionality reduction feature distribution information to obtain cascaded time-domain dimensionality reduction feature distribution information; Determining a filtering index based on the feature filter and the cascaded time-domain dimensionality reduction feature distribution information; Processing the cascaded time-domain dimensionality reduction feature distribution information using a mapper to obtain control time-domain dimensionality reduction feature distribution information; The first feature distribution information is determined based on the filtering index, the cascaded time-domain dimensionality reduction feature distribution information, and the control time-domain dimensionality reduction feature distribution information.

6. The method according to claim 1, characterized in that The first feature representation is obtained by performing multiple rounds of feature engineering on low-level feature representations respectively associated with the global detection data using a recursive network module; performing feature engineering of time domain measurement according to the plurality of first feature representations to obtain a time domain fusion feature representation; Performing feature engineering of spatial domain measurement on the time domain fusion feature representation to obtain second feature distribution information includes: For each piece of global detection data, obtain multiple intermediate-level feature representations corresponding to the global detection data; the multiple intermediate-level feature representations are generated in the process of performing multiple rounds of feature engineering on the low-level feature representations corresponding to the global detection data, and the multiple intermediate-level feature representations respectively correspond to different feature engineering levels; Performing feature engineering of time domain measurement according to a plurality of process feature representations respectively associated with the global detection data sequence and the plurality of first feature representations to obtain the time domain fusion feature representation; Using a two-dimensional filter and a downsampler, perform feature engineering of spatial domain measurement on the time domain fusion feature representation to obtain the second feature distribution information; The step of performing feature engineering of time domain measurement according to the multiple process feature representations respectively associated with the global detection data sequence and the multiple first feature representations to obtain the time domain fusion feature representation includes: For each feature engineering level, determining, from a plurality of process feature representations respectively associated with the global detection data sequence, each process feature representation corresponding to the feature engineering level; For each feature engineering level, performing non-maximum suppression along the time domain metric on each process feature representation corresponding to the feature engineering level to obtain the time domain feature representation corresponding to the feature engineering level; Concatenate the time domain feature representations associated with the various feature engineering levels and the first feature representation with the highest sequence index to obtain a concatenated feature representation; Based on the cascade feature representation, determining the time domain fusion feature representation; The determining the time domain fusion feature representation based on the cascade feature representation includes: Performing size compression and feature mapping on the cascade feature representation to obtain a comparison decision coefficient; According to the comparison decision coefficient, each feature value in the cascade feature representation is weighted by the decision coefficient to obtain a comparison feature representation; The sum of the feature values ​​of each feature value in the control feature representation and each feature value in the cascade feature representation is correspondingly calculated to obtain the time domain fusion feature representation.

7. The method according to claim 1, characterized in that For each defect type, determining a feature multi-tuple of the training fine inspection data corresponding to the defect type includes: Determine a cross-correlation function between a defect feature representation corresponding to the defect type and the first data feature representation; Determine a first focused feature vector and a second focused feature vector according to a defect feature representation corresponding to the defect type, the first data feature representation, and the cross-correlation function; Determining a focused defect feature representation corresponding to the defect type according to the defect feature representation corresponding to the defect type and the first focused feature vector; Determine the focused data feature representation of the training fine inspection data according to the first data feature representation and the second focused feature vector; for each defect type, determine the cross-correlation function of the defect feature representation corresponding to the defect type and the first data feature representation corresponding to the defect type, including: Determine a defect feature representation corresponding to the defect type and compare it with a first cross-correlation subfunction of the first data feature representation; Determining a second cross-correlation subfunction of the first data feature representation versus a defect feature representation of the defect type; A defect feature representation corresponding to the defect type and a cross-correlation function corresponding to the first data feature representation are determined according to the first cross-correlation sub-function, the second cross-correlation sub-function and a unit matrix.

8. The method according to claim 1, characterized in that For each of the example defect data sets in the second example defect data set, the example defect data set also includes augmented defect data corresponding to multiple augmentation modes for each refined defect data in the example defect data set; the method further includes: Determining, according to the feature multi-tuple of the training refined inspection data corresponding to the target defect type, a second detection correlation index of the training refined inspection data corresponding to each of the augmentation modes; The determining of the training cost parameter according to each of the first detection association indicators and each of the first true association indicators includes: Determine a first cost parameter according to each of the first detection association indicators and each of the first true association indicators; Determining, according to the target augmentation methods corresponding to the training refined inspection data, a second real correlation index of the training refined inspection data associated with each of the augmentation methods; determining a second cost parameter according to each of the second detection association indicators and each of the second true association indicators; Determining a training cost parameter according to the first cost parameter and the second cost parameter; The method further comprises: Determining, according to the feature multi-tuple of the training fine inspection data corresponding to the target defect type, a third detection association index of the training fine inspection data associated with each of the defect types; The determining of the training cost parameter according to the first cost parameter and the second cost parameter includes: Determining a third cost parameter according to each of the third detection association indicators and each of the first true association indicators; A training cost parameter is determined according to the first cost parameter, the second cost parameter and the third cost parameter.

Images