A method for training a model for detecting a small pipe of a forepoling, a method for detecting the small pipe of the forepoling, and a device

Abstract

The application provides a kind of advanced support small catheter detection model training method, detection method and device, comprising: using depth camera to shoot advanced small catheter support image as sample and add label, construct training sample set;Training sample set is input initial detection model and is trained, using feature extraction network to extract image feature, using path aggregation network to obtain fusion feature by the bidirectional fusion of top-down and bottom-up to image feature, using multi-task detection head to decode fusion feature, obtain the prediction category, prediction bounding box information and prediction key point coordinates of each sample in advanced small catheter support, construct joint loss optimization initial detection model, obtain advanced small catheter support detection model;The two-dimensional point coordinates predicted by detection model are projected onto the paired depth map, and the three-dimensional coordinates and adjacent spacing of the advanced small catheter support are calculated using the offset optimization method.The application can improve the detection accuracy, reduce the labor intensity and the degree of work danger.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of tunnel engineering, and in particular to a method and device for training and detecting a small pilot pipe detection model. BACKGROUND

[0002] The small pilot pipe support plays an important role in the pre-support of tunnel engineering. It reduces the water pressure inside the tunnel and enhances the stability of the surrounding rock through the ways of guiding and excluding underground water and grouting reinforcement. It effectively improves the construction conditions, improves the construction safety, and prolongs the service life of the support. Therefore, it is of great significance to accurately detect and position the small pilot pipe, and to determine whether the layout and quantity are in compliance.

[0003] The traditional small pilot pipe support detection method mainly includes the following three kinds:

[0004] (1) Manual inspection: manual inspection is the most common inspection method in the current construction scene. It mainly observes whether the layout and spacing of the small pilot pipe support meet the requirements through visual inspection and tape measurement. This method is simple and direct, and does not require complex equipment, but close-range manual measurement is highly dangerous and subjective.

[0005] (2) Laser scanning: a laser scanner is used to scan the inner surface of the tunnel with high precision to generate a three-dimensional model. This method can obtain the support layout and deformation of the overall tunnel face area, and can provide detailed spatial data. However, the equipment is expensive, the technical requirements are high, the data processing is complex, and professional software and technical personnel are required.

[0006] (3) Measuring instruments: such as total station, laser range finder, etc., which can measure the layout of the small pilot pipe at a distance, and the data is reliable and can be quantitatively analyzed. However, the technical level of the operator is high, and the initial equipment investment is large.

[0007] Thanks to the rapid development of deep learning, computer vision based on deep learning has been widely applied in the field of engineering, such as traffic monitoring, face recognition, and unmanned aerial vehicle tracking. However, due to the complex environment of the tunnel and the limitations of the data, this technology is still in its infancy in the detection of internal structures of the tunnel. SUMMARY

[0008] In view of this, the present application provides a small pilot pipe detection model training method, detection method and device to eliminate or improve one or more defects in the prior art.

[0009] On the one hand, the present application provides a small pilot pipe detection model training method based on computer vision technology, which comprises the following steps:

[0010] construct a training sample set, the training sample set comprising a plurality of samples, each sample comprising an advance small pipe support image taken by a depth camera; adding a label to each sample, the label including the category of the corresponding advance small pipe support, bounding box information and key point coordinates;

[0011] obtain an initial detection model, the initial detection model comprising a feature extraction network, a path aggregation network and a multi-task detection head; input the training sample set into the initial detection module, extract a plurality of levels of image features using the feature extraction network, input the plurality of levels of image features into the path aggregation network respectively, perform bidirectional fusion from top to bottom and from bottom to top to obtain a plurality of levels of fusion features, input the fusion features into the multi-task detection head for feature decoding, splice the predicted features output by each detection head to generate class scores and intersection over union scores for each pixel and perform weighted calculation to obtain comprehensive prediction scores, and sort and filter the comprehensive prediction scores to obtain the predicted category, predicted bounding box information and predicted key point coordinates of the advance small pipe support in each sample;

[0012] train the initial detection model using the training sample set, and build a joint loss of a classification task, a bounding box detection task and a key point detection task to optimize the initial detection model until a preset performance requirement is reached to obtain a final advance small pipe support detection model.

[0013] In some embodiments of the present application, before adding the label to each sample, the method further comprises:

[0014] perform dark light enhancement on the advance small pipe support image in each sample using a pre-trained image enhancement model; wherein the original image and the enhanced image gray mapping relationship expression is:

[0015] LE(I(p i );α)=I(p i )+αI(p i )(1-I(p i ));

[0016] wherein LE(I(p class );α) represents the enhancement effect of pixel point p i ; I(p i ) represents the normalized structure of the gray value of pixel point p ProbIoU ; and alpha represents a trainable curve parameter.

[0017] In some embodiments of the present application, the feature extraction network comprises three stages, the first stage comprises a convolution-batch normalization-activation function integrated module and a structure enhancement module, the second stage comprises a convolution-batch normalization-activation function integrated module and a cross-layer feature fusion module, and the third stage comprises a convolution-batch normalization-activation function integrated module, a cross-layer feature fusion module and a fast pyramid pooling module, and the method further comprises:

[0018] The preprocessed training sample set is input into the feature extraction network, high-frequency structure information is extracted by the first stage to obtain shallow features, semantic information is extracted by the second stage to obtain intermediate features, and semantic information is extracted in high dimension by the third stage, and the features of different receptive fields are aggregated by fast pyramid pooling to enhance the feature representation, thereby obtaining deep features.

[0019] In some embodiments of the present application, the structure enhancement module is used to enhance the target features, comprising:

[0020] In the first stage, the features extracted by the convolution-batch normalization-activation function integrated module are evenly divided into first features and second features in the channel dimension;

[0021] The second features are input into the structure enhancement module to construct multi-scale high-frequency residual features;

[0022] The high-frequency residual features and the second features are input into a preset convolution module for local feature enhancement and aggregation to obtain aggregated features;

[0023] The first features and the aggregated features are spliced in the channel dimension and then input into the next stage.

[0024] In some embodiments of the present application, the joint loss of the classification task, the bounding box detection task and the key point detection task is used to optimize the initial detection model, comprising:

[0025] For the classification task, a binary cross-entropy loss is used for training, and the calculation formula is:

[0026]

[0027] Wherein, L class represents the binary cross-entropy loss; N represents the number of samples; y i represents the true label of the i-th sample; p i represents the predicted probability of each sample, and σ(·) represents the Sigmoid function.

[0028] In some embodiments of the present application, the joint loss of the classification task, the bounding box detection task and the key point detection task is used to optimize the initial detection model, comprising:

[0029] For the bounding box detection task, a cross-entropy loss and a distribution focal loss are used for training.

[0030] The calculation formula of the cross-entropy loss is:

[0031]

[0032] Wherein, L ProbIoU represents the cross-entropy loss; B D represents the Bhattacharyya distance of the real bounding box and the predicted bounding box.

[0033] The calculation formula of the distribution focal loss is:

[0034] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(y-y i )log(S i+i ));

[0035] Wherein, y represents a label value, y i and y i+1 are the integer values on the left and right of the label respectively; S i and S i+i are the probabilities corresponding to y i and y i+1 .

[0036] In some embodiments of the present application, the joint loss of the classification task, the bounding box detection task and the key point detection task is used to optimize the initial detection model, including:

[0037] For the key point detection task, a key point loss is used for training, and the calculation formula is:

[0038]

[0039] Wherein, L kpts represents the key point loss; N kpts represents the number of key points; d n represents the Euclidean distance between the nth predicted key point and the real key point; s represents a scale factor; σ(v n >0) represents a key point visibility label.

[0040] In another aspect, the present application also provides a computer vision technology-based advanced support small catheter detection method, which comprises:

[0041] The depth camera is used to shoot the advanced small pipe support to be detected, so as to obtain an image and a depth map thereof;

[0042] The image is input into the advanced small pipe support detection model trained based on the computer vision technology-based advanced support small pipe detection model training method, so as to obtain prediction category, prediction bounding box information and prediction key point coordinates of the advanced small pipe support.

[0043] The bottom points of the advanced small pipe support are projected onto the registered depth map for positioning; each bottom point is sequentially connected into a curve, and a normal line is constructed for each bottom point on the curve; a new point set is obtained by respectively offsetting a preset number of pixels upward and downward along the normal line direction; for each offset bottom point, a corresponding depth value is obtained and sorted, and the minimum value is selected as the final depth value of the bottom point; and three-dimensional coordinates of each bottom point and distances between adjacent bottom points are calculated according to the new point set and the depth values.

[0044] In some embodiments of the present application, the three-dimensional coordinates of each bottom point and the distances between adjacent bottom points are calculated, including:

[0045] After obtaining the three-dimensional coordinates of each bottom point, the bottom points are sorted according to the horizontal coordinate values, and the distances between adjacent bottom points are calculated, and the calculation formula is:

[0046]

[0047] wherein Dis represents the distance between adjacent bottom points; (x i ,y i ,z i ) represents the three-dimensional coordinates of the i-th bottom point.

[0048] On the other hand, the present application also provides a computer vision technology-based advanced support small pipe detection device, which comprises a processor, a memory and computer programs / instructions stored in the memory, wherein the processor is used to execute the computer programs / instructions, and when the computer programs / instructions are executed, the device realizes the steps of the method according to any one of the above embodiments.

[0049] The application provides a pre-support small catheter detection model training method, a detection method and a device. Firstly, the pre-trained image enhancement model is used to perform dark light enhancement on the pre-support small catheter image, so as to improve the image distinguishability and the artificial standard accuracy. Secondly, a rotating target detection network with vector information is constructed, including a feature extraction network, a path aggregation network and a multi-task detection head. The feature extraction network is used to extract image features, the path aggregation network is used to perform bidirectional fusion from top to bottom and from bottom to top on the image features to obtain fused features, the multi-task detection head is used to decode the fused features to obtain the prediction category, the prediction bounding box information and the prediction key point coordinates of the pre-support small catheter in each sample, and a joint loss is constructed to optimize the initial detection model, and finally the pre-support small catheter detection model is trained. Finally, the two-dimensional point coordinates predicted by the detection model are projected onto the paired depth map, and the offset optimization method is used to calculate the number, three-dimensional coordinates and adjacent spacing of the pre-support small catheter. The detection model and the detection method realized by using the detection model can non-contact and intelligently detect the number, spacing and three-dimensional coordinates of the pre-support small catheter in the tunnel engineering, greatly improve the detection precision and detection efficiency, and effectively improve the engineering construction efficiency and reduce the labor intensity and the operation danger degree.

[0050] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following detailed description and drawings in which

[0051] Those skilled in the art will appreciate that the objects and advantages of the application can be accomplished by the structures specifically described in the specification and claims note below, and will be more clearly understood from the following detailed description with appropriate reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description, serve to explain the principles of the application. In the drawings:

[0053] Figure 1 A step schematic diagram of the pre-support small catheter detection model training method based on the computer vision technology in an embodiment of the application.

[0054] Figure 2 A structure schematic diagram of the pre-support small catheter detection model in an embodiment of the application.

[0055] Figure 3 A structure schematic diagram of the structure enhancement module in an embodiment of the application.

[0056] Figure 4 A schematic diagram of the result of the multi-task detection head in an embodiment of the present application.

[0057] Figure 5 A schematic diagram of the steps of the advanced support small catheter detection method based on computer vision technology in an embodiment of the present application.

[0058] Figure 6 A schematic diagram of the point position offset optimization algorithm in an embodiment of the present application. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the present application clearer, further detailed descriptions will be given to the present application in conjunction with the embodiments and the accompanying drawings. Herein, the illustrative embodiments of the present application and the descriptions thereof are used to explain the present application, but are not intended to limit the present application.

[0060] It should be noted that, in order to avoid the present application being obscured by unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the accompanying drawings, and other details not closely related to the present application are omitted.

[0061] It should be emphasized that the term “comprises / comprising” is used herein to indicate the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0062] It should be noted that, unless otherwise specified, the term “connected” used herein can not only mean direct connection, but also indirect connection in the presence of an intermediate.

[0063] In the following, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0064] In order to solve the problems of high labor intensity, high risk of close operation, high cost of detection equipment, high professional requirements, low automation rate, and low intelligent rate in the existing advanced small catheter support detection method, the present application provides a training method for an advanced support small catheter detection model based on computer vision technology, as shown in Figure 1 The method comprises the following steps S101-S103:

[0065] Step S101: Construct a training sample set, which contains multiple samples, each sample containing an advanced small catheter support image taken by a depth camera; add a label to each sample, wherein the label includes the category of the corresponding advanced small catheter support, the boundary box information, and the key point coordinates.

[0066] Step S102: An initial detection model is constructed, which includes a feature extraction network, a path aggregation network and a multi-task detection head. The training sample set is input into the initial detection module, the multi-level image features are extracted using the feature extraction network, the multi-level image features are input into the path aggregation network respectively, bidirectional fusion from top to bottom and from bottom to top is performed, the multi-level fusion features are obtained, the fusion features are input into the multi-task detection head for feature decoding, the prediction features output by each detection head are spliced, the class score and the intersection over union score of each pixel are generated and weighted calculation is performed, the comprehensive prediction score is obtained, and the prediction category, the prediction bounding box information and the prediction key point coordinates of the advanced small pipe support in each sample are obtained by sorting and screening according to the comprehensive prediction score.

[0067] Step S103: The initial detection model is trained using the training sample set, and a joint loss of the classification task, the bounding box detection task and the key point detection task is used to optimize the initial detection model until the preset performance requirement is reached, and a final advanced small pipe support detection model is obtained.

[0068] In step S101, the advanced small pipe support is photographed using a depth camera to obtain an image, the image is used as a sample, and a training sample set is constructed. Real labels are added to each sample for subsequent model training. The depth camera is a device that can capture the distance information between objects in a scene and the camera. It can not only obtain color information of the image, but also generate a depth value corresponding to each pixel, thereby forming a depth map.

[0069] In some embodiments, considering that the light inside the tunnel is dim and the recognition degree of the collected original image data is poor, if the original image is directly manually labeled, a large number of missed labels and mislabeled labels may be caused. Therefore, before adding labels to each sample, an image enhancement model obtained by pre-training, such as ZeroDCE, is used to perform dark light enhancement on the advanced small pipe support image in each sample, to improve the image recognition degree and thus improve the accuracy of manual labeling.

[0070] Specifically, given an image P, each pixel point of the image P is p i , the gray value of each pixel point is I(·), the gray mapping parameter predicted by the model is α, and the gray mapping relationship between the original image and the enhanced image is shown in formula (1):

[0071] LE(I(p i );α)=I(p i )+αI(p i )(1-I(p i )); (1)

[0072] wherein LE(I(p i );α) represents the pixel point pi the enhancement effect of I(p i )∈[0,1] represents the normalized structure of the gray value of the pixel point p i ; alpha ∈ [-1,1] represents a trainable curve parameter, and all operations are pixel-by-pixel in a channel for an image, that is, each pixel of a given input image has a corresponding mapping curve.

[0073] In order to cope with complex practical application scenarios, formula (1) can be used iteratively, as shown in formula (2):

[0074] LE n (x)=LE n-1 (x)+alpha n LE n-1 (x)(1-LE n-1 (x)); (2)

[0075] Wherein, n ∈ {1,2,3,...,8} represents the number of iterations, when n = 1, LE(·) degenerates into I(·); the remaining physical quantities are consistent with formula (1), and thus will not be described again.

[0076] In step S102, in order to solve the problem of pipe target disappearance caused by the accuracy of the depth map in the actual application process, the present application converts the direct detection of the advanced small pipe support into the detection of the bottom position of the support pipe. However, it is difficult to achieve the purpose by directly using the rotating frame and key point detection, the former cannot obtain the pipe direction information, and the latter has poor detection accuracy in the complex environment of the tunnel. Based on the above problems, the present application designs a rotating target detection network PIPENet with vector information. In the training stage, PIPENet is recorded as an initial detection model, and after training is completed, PIPENet is recorded as an advanced small pipe support detection model. As shown in Figure 2 , PIPENet includes a feature extraction network, a path aggregation network and a multi-task detection head. The advanced small pipe support image to be detected is input into PIPENet for detection.

[0077] In the training stage, the training sample set is input into the initial detection model in sequence or in batches, and the detection result is generated by the feature extraction network, the path aggregation network and the multi-task detection head in sequence.

[0078] In some embodiments, as Figure 2As shown in Figure 5, the feature extraction network consists of three stages. The first stage includes a convolution-batch normalization-activation function (CBS) integrated module and a structure enhancement module. The second stage includes a convolution-batch normalization-activation function integrated module and a cross-layer feature fusion module. The third stage includes a convolution-batch normalization-activation function integrated module, a cross-layer feature fusion module and a fast pyramid pooling module.

[0079] The preprocessed training sample set is input into the feature extraction network. For example, given an input image The high-frequency structural information is extracted in the first stage to obtain shallow features The semantic information is extracted in the second stage to obtain the intermediate features The third stage extracts semantic information in high dimensions and uses fast pyramid pooling to aggregate features of different receptive fields, enhance feature representation, and obtain deep features. Among them, C represents the number of channels, W represents the width of the feature map, and H represents the height of the feature map.

[0080] In some embodiments, the complex environment in which the advanced small-duct support is located may cause the supporting pipeline to be confused with the other engineering structures in the background, and the differences in perspective rules and small-duct support layout caused by the shooting angle may cause the supporting pipelines in the same image to have obvious scale and morphological differences. Therefore, in the present invention, a structural enhancement module for small-duct support is designed in the first stage of the feature extraction network to enhance the pipeline target features.

[0081] like Figure 3 Figure 2 shows the structure of the structure enhancement module. The features extracted by the CBS module are divided equally into first and second features along the channel dimension, denoted as feature X and feature Y, respectively. The second feature is input into the structure enhancement module, where multi-scale high-frequency residual features are constructed based on the Laplacian pyramid. The high-frequency residual features and the original second features are then fed into a multi-kernel strip convolution for local feature enhancement. Exemplarily, this component includes four parallel depthwise separable convolution paths, each with a convolution kernel size of {5×1, 1×5; 7×1, 1×7; 11×1, 1×11; 21×1, 1×21}. Strip convolution with large kernels not only improves the feature capture of tubular objects but also captures the adjacent relationships between pipes. For example, pipes are often arranged along beams, resulting in a linear distribution rather than a planar distribution in the image. The features extracted by each convolutional strip are summed and fed into two consecutive convolutional modules. This aggregates local features from different receptive fields, strengthens information interaction between channels, and generates aggregated features. Finally, the first and aggregated features are concatenated along the channel dimension and fed into the next stage. This operation not only strengthens the pipeline structure and aggregates more gradient flow, but also improves model accuracy.

[0082] The overall process can be represented as formulas (3) to (9):

[0083] X, Y = Split(Z); (3)

[0084] H = g i (Y) - Upsample(g i+1 (Y)); (4)

[0085] Y' = Y + H; (5)

[0086] CSP(Y) = Y' + (Conv(Y') + Y'); (6)

[0087] Y' = CSP(CSP(Y)); (7)

[0088]

[0089] T = Enhance(Y') + X; (9)

[0090] In formulas (3) to (9), Split(Z) represents equal division of the feature Z into two parts from the channel dimension; H represents a high-frequency residual feature; g i (·) represents Gaussian blur, i is the level number in the pyramid; Upsample(·) represents the up-sampling operation; Conv(·) represents 2d convolution; DWConv(·) represents depth separable convolution; Scale represents the convolution size; and T represents the aggregated feature.

[0091] The multi-level image features extracted by the feature extraction network are input into the path aggregation network for one-way top-down and bottom-up bidirectional fusion to obtain multi-level fusion features The three levels of features are respectively used for detecting large targets, medium targets and small targets, and finally input into the multi-task detection head for feature decoding to obtain the detection results.

[0092] In some embodiments, the multi-task detection head constructed in the feature decoding stage of the present application adopts a three-branch detection head, which can simultaneously realize target class recognition, boundary box framing and key direction point detection.

[0093] As Figure 4As shown in the figure, each detection head contains three functions, namely category prediction, bounding box prediction, and key point prediction. The features obtained by the path aggregation network are input into the multi-task detection head, and the features decoded and output by each detection head are spliced ​​to generate the class score and intersection over union (IoU) score of each pixel and perform weighted calculation to obtain a comprehensive prediction score. Sorting and screening are performed according to the comprehensive prediction score. For example, the top 10 positive samples with the comprehensive prediction score of each target are taken to obtain the predicted category, predicted bounding box information, and predicted key point coordinates of the advanced small catheter support.

[0094] In step S103, the initial detection model is trained using the training sample set, and the joint loss of the classification task, bounding box detection task and key point detection task is constructed to optimize the initial detection model until the preset performance requirements are met, thereby obtaining the final advanced small catheter support detection model.

[0095] In some embodiments, for classification tasks, binary cross-entropy loss (BCELoss) is used for training, and the calculation formula is shown in formula (10):

[0096]

[0097] The calculation formula of σ is shown in formula (11):

[0098]

[0099] Among them, L class represents the binary cross entropy loss; N represents the number of samples; y i represents the true classification label of the i-th sample; p i represents the predicted probability of each sample, and σ(·) represents the Sigmoid function.

[0100] In some embodiments, for the bounding box detection task, ProbIoULoss and Distribution Focal Loss (DFL) are used for training;

[0101] The calculation formula of the intersection loss is shown in formula (12):

[0102]

[0103] Among them, L ProbIoU represents the intersection loss; B D Represents the Bhattacharyya distance between the true bounding box and the predicted bounding box;

[0104] The calculation formula of the distributed focus loss is shown in formula (13):

[0105] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(y-y i )log(S i+i )); (13)

[0106] wherein y represents a label value, y i and y i+1 are integer values on the left and right of the label respectively; S i and S i+i are the corresponding probabilities of y i and y i+1 .

[0107] In some embodiments, for the key point detection task, the key point loss is used for training, and the calculation formula is shown in formula (14):

[0108]

[0109] wherein L kpts represents the key point loss; N kpts represents the number of key points; d n represents the Euclidean distance between the nth predicted key point and the real key point; s represents a scale factor; and sigma (v n >0) represents a key point visibility label.

[0110] Thus, the binary cross-entropy loss, the intersection-over-union loss, the distribution focal loss and the key point loss are weighted to obtain the total loss of the entire detection network, and the calculation formula is shown in formula (15):

[0111] Loss=L class +L ProbIoU +DFL(S i ,S i+1 )+L kpts ; (15)

[0112] The total loss is used to train and optimize the initial detection module, and finally the advanced small duct support detection model is obtained.

[0113] The application also provides a small duct detection method based on computer vision technology, as shown in FIG. Figure 5 The method comprises the following steps S201-S203:

[0114] Step S201: a depth camera is used to take a picture of the advanced small duct support to be detected, and an image and a depth map thereof are obtained.

[0115] Step S202: input the image into the advanced small pipe support detection model trained based on the computer vision technology-based advanced pipe support small pipe detection model training method described above, to obtain the prediction category, prediction bounding box information and prediction key point coordinates of the advanced small pipe support photographed by the image.

[0116] Step S203: project the bottom points of the identified advanced small pipe support onto the registered depth map for positioning; connect each bottom point in sequence to form a curve, and construct a normal line for each bottom point on the curve; offset a preset number of pixels, such as 5 pixels, upward and downward along the normal line direction respectively to obtain a new point set; for each offset bottom point, obtain its corresponding depth value and sort it, and select the minimum value as the final depth value of the bottom point; calculate the three-dimensional coordinates of each bottom point and the distance between adjacent bottom points according to the new point set and its depth value.

[0117] In step S201, a depth camera is used to photograph the advanced small pipe support to be detected to obtain its image and depth map.

[0118] In step S202, the advanced small pipe support detection model trained by the training method described above is used to detect the category, bounding box, key point direction and other two-dimensional information of the corresponding advanced small pipe support.

[0119] In step S203, the circumscribed rotating box and direction of the advanced small pipe support are detected by image detection, and the bottom points of each target are projected onto the registered depth map for positioning. Since the bottom points recognized by the target detection algorithm based on deep learning are not necessarily accurate, in the present application, a point position offset algorithm is constructed according to artificial priori, which projects the point positions on the same plane to calculate the distance, as shown in FIG. 8, and specifically: Figure 6

[0120] After obtaining the bottom points (Bx i ,By i ) of each advanced small pipe support, connect each bottom point in sequence to form a curve L, construct a normal line based on each point (Bx i ,By i ), offset a preset number of pixels, such as 5 pixels, upward and downward along the normal line direction to obtain a new point set, for each offset bottom point, obtain its corresponding depth value and sort it, and select the minimum value as the final depth value of the bottom point; calculate the three-dimensional coordinates of each bottom point and the distance between adjacent bottom points according to the new point set and its depth value.

[0121] In some embodiments, for the offset bottom points, the obtained new point set is {(x1,y1),(x2,y2),...,(x​n ,y n )}, whose depth value set is {z1,z2,...,z n}, for each bottom point, use formula (16) to calculate its specific coordinates in camera coordinates:

[0122]

[0123] Among them, (x cam ,y cam ,z cam ) represents the coordinates in the camera coordinate system; (c x ,c y ) represents the coordinates of the image center in the pixel coordinate system; (p x ,p y ) represents the coordinate in the pixel coordinate system; f represents the focal length; α and β represent the correction factors in the horizontal and vertical directions, respectively.

[0124] Then sort each bottom point in the camera coordinate system according to the size of the horizontal coordinate value, and calculate the distance between adjacent points. The calculation formula is shown in formula (17):

[0125]

[0126] Where Dis represents the distance between adjacent bottom points; (x i ,y i ,z i ) represents the three-dimensional coordinates of the i-th bottom point, i∈{0,1,2,...,n}, and n is the number of pipeline targets on an image.

[0127] Corresponding to the above method, the present invention also provides an apparatus, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the apparatus implements the steps of the method described above.

[0128] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0129] Those of ordinary skill in the art will appreciate that the various illustrative components, systems and methods described in connection with the embodiments disclosed herein can be implemented as hardware, software, or both. The particular implementation is dependent on the specific application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application. When implemented in hardware, for example, the hardware can comprise an electronic circuit, an Application Specific Integrated Circuit (ASIC), a suitable firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the application are the program or code segments to perform a specific task. The program or code segments can be stored in a machine-readable medium, or transmitted by a carrier wave as data signals over a transmission medium or communication link.

[0130] It is to be understood that the application is not limited to the particular configurations and processes described herein and shown in the drawings. For simplicity, detailed descriptions of known methods and apparatuses are omitted so as not to obscure the disclosure. In the above-described embodiments, several specific steps are described and illustrated as examples. However, the method processes are not limited to the specific steps described and illustrated, and the order of the steps can be changed, or other steps can be added, or replaced, or eliminated, depending on the application.

[0131] In the present application, features described and / or illustrated with respect to one embodiment can be used in the same or a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0132] The above description is only preferred embodiments of the present application, and is not intended to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A method for training a small catheter detection model for advanced support based on computer vision technology, characterized in that: The method comprises the following steps: Constructing a training sample set, wherein the training sample set includes multiple samples, each sample includes an image of an advanced small conduit support taken by a depth camera; adding a label to each sample, wherein the label includes a category of the corresponding advanced small conduit support, bounding box information, and key point coordinates; Obtain an initial detection model, the initial detection model includes a feature extraction network, a path aggregation network and a multi-task detection head, the feature extraction network includes three stages: the first stage includes a convolution-batch normalization-activation function synthesis module and a structure enhancement module, the second stage includes a convolution-batch normalization-activation function synthesis module and a cross-layer feature fusion module, and the third stage includes a convolution-batch normalization-activation function synthesis module, a cross-layer feature fusion module and a fast pyramid pooling module; input the training sample set into the initial detection model, in the first stage of the feature extraction network, use the convolution-batch normalization-activation function synthesis module to extract the features of the training sample set, and divide them into the first feature and the second feature in the channel dimension, input the second feature into the structure enhancement module to construct a high-frequency residual feature, input the high-frequency residual feature and the second feature together into a preset convolution module for local feature enhancement, and aggregate to obtain an aggregated feature; The first feature and the aggregated feature are spliced ​​in the channel dimension to obtain shallow features containing high-frequency structural information and input into the second stage. The second stage extracts semantic information, obtains intermediate features and inputs into the third stage. The third stage extracts semantic information in high dimensions and uses pyramid pooling to aggregate features of different receptive fields to enhance feature representation and obtain deep features. The feature extraction network inputs the extracted shallow features, the intermediate features and the deep features into the path aggregation network respectively, performs top-down and bottom-up bidirectional fusion to obtain multi-level fusion features, inputs the fusion features into the multi-task detection head for feature decoding, splices the predicted features output by each detection head, generates a class score and an intersection score for each pixel and performs weighted calculation to obtain a comprehensive prediction score. The predicted category, predicted bounding box information and predicted key point coordinates of the advanced small catheter support in each sample are obtained by sorting and screening according to the comprehensive prediction score. The initial detection model is trained using the training sample set, and a joint loss of classification task, bounding box detection task and key point detection task is constructed to optimize the initial detection model until the preset performance requirements are met, thereby obtaining the final advanced small catheter support detection model; wherein, the classification task uses binary cross entropy loss, the bounding box detection task uses intersection loss and distribution focus loss, and the key point detection task uses key point loss.

2. The method for training a small catheter detection model for advanced support based on computer vision technology according to claim 1, characterized in that: Before adding labels to each sample, it also includes: The pre-trained image enhancement model is used to perform dark light enhancement on the advanced small catheter support image in each sample. The grayscale mapping relationship between the original image and the enhanced image is expressed as follows: LE(I(p i );α)=I(p i )+αI(p i )(1-I(p i )); Among them, LE(I(p i ); α) represents the pixel point p i The enhancement effect of I(p i ) represents pixel p i The normalized structure of the grayscale value; α represents the trainable curve parameter.

3. The method for training a small catheter detection model for advanced support based on computer vision technology according to claim 1, characterized in that: The binary cross entropy loss is calculated as: Among them, L class represents the binary cross entropy loss; N represents the number of samples; y i represents the true label of the i-th sample; p i represents the predicted probability of each sample, and σ(·) represents the Sigmoid function.

4. The method for training a small catheter detection model for advanced support based on computer vision technology according to claim 1, characterized in that: The calculation formula of the intersection loss is: Among them, L ProbIoU represents the intersection loss; B D Represents the Bhattacharyya distance between the true bounding box and the predicted bounding box; The calculation formula of the distributed focus loss is: DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(y-y i )log(S i+i )); Among them, y represents the label value, y i and y i+1 are the integer values ​​to the left and right of the label respectively; S i and S i+i y i and y i+1 The corresponding probability.

5. The method for training a small catheter detection model for advanced support based on computer vision technology according to claim 1, characterized in that: The calculation formula of the key point loss is: Among them, L kpts represents the key point loss; N kpts Indicates the number of key points; d n represents the Euclidean distance between the nth predicted keypoint and the true keypoint; s represents the scale factor; σ(v n >0) indicates the key point visibility flag.

6. A method for detecting small catheters for advanced support based on computer vision technology, characterized in that: The method comprises: A depth camera is used to shoot the advanced small conduit support to be inspected to obtain an image and a depth map; Inputting the image into an advanced small duct support detection model trained by the advanced small duct support detection model training method based on computer vision technology according to any one of claims 1 to 5 to obtain the predicted category, predicted bounding box information and predicted key point coordinates of the advanced small duct support; The identified bottom point of the advanced small catheter support is projected onto the aligned depth map for positioning; each bottom point is sequentially connected into a curve, and a normal is constructed for each bottom point on the curve; a preset number of pixels are offset upward and downward along the normal direction to obtain a new point set; for each offset bottom point, its corresponding depth value is obtained and sorted, and the minimum value is selected as the final depth value of the bottom point; the three-dimensional coordinates of each bottom point and the distance between adjacent bottom points are calculated based on the new point set and its depth value.

7. The method for detecting small catheters for advanced support based on computer vision technology according to claim 6, characterized in that: Calculate the three-dimensional coordinates of each bottom point and the distance between adjacent bottom points, including: After obtaining the three-dimensional coordinates of each bottom point, sort them according to the size of the horizontal coordinate value of the bottom point, and calculate the distance between adjacent bottom points. The calculation formula is: Where Dis represents the distance between adjacent bottom points; (x i ,y i ,z i ) represents the three-dimensional coordinates of the i-th bottom point.

8. A device for detecting small conduit for advanced support based on computer vision technology, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is configured to execute the computer program / instructions. When the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

Images