Advanced support small conduit detection model training method and device, and advanced support small conduit detection method and device

Abstract

The invention provides a forepoling small conduit detection model training method, detection method and device, and the method comprises the steps: employing a depth camera to shoot a forepoling small conduit support image as a sample, adding a label, and constructing a training sample set; inputting the training sample set into an initial detection model for training, extracting image features by using a feature extraction network, performing top-down and bottom-up bidirectional fusion on the image features by using a path aggregation network to obtain fused features, and decoding the fused features by using a multi-task detection head to obtain a multi-task detection result; a prediction category, prediction bounding box information and prediction key point coordinates of the advanced small duct support in each sample are obtained, a joint loss optimization initial detection model is constructed, and an advanced small duct support detection model is obtained; and projecting the two-dimensional point coordinates predicted by the detection model to the paired depth map, and calculating the three-dimensional coordinates and adjacent spacing of the advanced small duct support by adopting an offset optimization method. The detection precision can be improved, and the labor intensity and the operation danger degree can be reduced.

Description

Technical Field

[0001] The invention relates to the technical field of tunnel engineering, and in particular to a training method, a detection method and a device for an advanced support small conduit detection model. Background Art

[0002] The advance small pipe support plays an important role in pre-support in tunnel engineering. It reduces the water pressure inside the tunnel and enhances the stability of the surrounding rock by guiding the removal of groundwater and grouting reinforcement. It effectively improves the construction conditions, improves the construction safety, and prolongs the support life. Therefore, it is of great significance to accurately detect and locate the advance small pipe and determine whether its layout and quantity are in compliance.

[0003] The traditional methods for detecting advanced small-duct support mainly include the following three methods:

[0004] (1) Manual inspection: Manual inspection is the main inspection method in current construction scenarios. It mainly uses visual inspection and tape measurement to observe whether the layout of small duct supports and the layout spacing meet the regulations. This method is simple and direct and does not require complex equipment, but close-range manual measurement is more dangerous and the inspection is more subjective.

[0005] (2) Laser scanning: Use a laser scanner to scan the inner surface of the tunnel with high precision to generate a three-dimensional model. This method can obtain the support installation and deformation degree of the entire tunnel face area and provide detailed spatial data. However, the equipment is expensive, the technical requirements are high, the data processing is complex, and professional software and technicians are required.

[0006] (3) Measuring instruments: such as total stations and laser rangefinders, which can measure the layout of small pipes from a distance, provide reliable data, and can be used for quantitative analysis. However, they require high technical skills of operators and require a large initial investment in equipment.

[0007] Thanks to the rapid development of deep learning, computer vision based on deep learning has been widely used in the engineering field, such as traffic monitoring, face recognition, drone tracking, etc. However, due to the complex environment of tunnels and data limitations, this technology is still in its infancy in tunnel internal structure detection. Summary of the invention

[0008] In view of this, the embodiments of the present invention provide a method for training an advanced support small catheter detection model, a detection method and a device to eliminate or improve one or more defects existing in the prior art.

[0009] On the one hand, the present invention provides a method for training a small catheter detection model for advanced support based on computer vision technology, the method comprising the following steps:

[0010] Constructing a training sample set, wherein the training sample set includes a plurality of samples, each sample includes an image of an advanced small conduit support taken by a depth camera; adding a label to each sample, wherein the label includes a category of the corresponding advanced small conduit support, bounding box information, and key point coordinates;

[0011] Obtain an initial detection model, which includes a feature extraction network, a path aggregation network, and a multi-task detection head; input the training sample set into the initial detection module, use the feature extraction network to extract multi-level image features, input the multi-level image features into the path aggregation network respectively, perform top-down and bottom-up bidirectional fusion, obtain multi-level fusion features, input the fusion features into the multi-task detection head for feature decoding, splice the predicted features output by each detection head, generate a class score and an intersection score for each pixel, and perform weighted calculation to obtain a comprehensive prediction score, sort and screen according to the comprehensive prediction score to obtain the predicted category, predicted bounding box information, and predicted key point coordinates of the advanced small catheter support in each sample;

[0012] The initial detection model is trained using the training sample set, and a joint loss of classification task, bounding box detection task and key point detection task is constructed to optimize the initial detection model until the preset performance requirements are met, thereby obtaining the final advanced small catheter support detection model.

[0013] In some embodiments of the present invention, before adding a label to each sample, the method further includes:

[0014] The pre-trained image enhancement model is used to perform dark light enhancement on the advanced small catheter support image in each sample; the grayscale mapping relationship between the original image and the enhanced image is expressed as:

[0015] LE(I(p i );α)=I(p i )+αI(p i )(1-I(p i ));

[0016] Among them, LE(I(p i ); α) represents the pixel point p i The enhancement effect of I(p i ) represents pixel p i The normalized structure of the grayscale value; α represents the trainable curve parameter.

[0017] In some embodiments of the present invention, the feature extraction network includes three stages, the first stage includes a convolution-batch normalization-activation function synthesis module and a structure enhancement module, the second stage includes a convolution-batch normalization-activation function synthesis module and a cross-layer feature fusion module, and the third stage includes a convolution-batch normalization-activation function synthesis module, a cross-layer feature fusion module and a fast pyramid pooling module. The method also includes:

[0018] The preprocessed training sample set is input into the feature extraction network, the first stage extracts high-frequency structural information to obtain shallow features, the second stage extracts semantic information to obtain intermediate features, the third stage extracts semantic information in high dimensions, and uses fast pyramid pooling to aggregate features of different receptive fields, enhance feature representation, and obtain deep features.

[0019] In some embodiments of the present invention, the structure enhancement module is used to enhance target features, including:

[0020] In the first stage, the features extracted by the convolution-batch normalization-activation function comprehensive module are equally divided into first features and second features in the channel dimension;

[0021] Inputting the second feature into the structure enhancement module to construct a multi-scale high-frequency residual feature;

[0022] Inputting the high-frequency residual feature and the second feature together into a preset convolution module for local feature enhancement, and performing aggregation to obtain an aggregated feature;

[0023] The first feature and the aggregated feature are concatenated in the channel dimension and input into the next stage.

[0024] In some embodiments of the present invention, constructing a joint loss of a classification task, a bounding box detection task, and a key point detection task to optimize the initial detection model includes:

[0025] For the classification task, binary cross entropy loss is used for training, and the calculation formula is:

[0026]

[0027] Among them, L class represents the binary cross entropy loss; N represents the number of samples; y i represents the true label of the i-th sample; p i represents the predicted probability of each sample, and σ(·) represents the Sigmoid function.

[0028] In some embodiments of the present invention, constructing a joint loss of a classification task, a bounding box detection task, and a key point detection task to optimize the initial detection model includes:

[0029] For the bounding box detection task, the intersection loss and distribution focus loss are used for training;

[0030] The calculation formula of the intersection loss is:

[0031]

[0032] Among them, L ProbIoU represents the crossover loss; B D Represents the Bhattacharyya distance between the true bounding box and the predicted bounding box;

[0033] The calculation formula of the distribution focus loss is:

[0034] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+i ));

[0035] Among them, y represents the label value, y i and i+1 are the integer values ​​to the left and right of the label respectively; S i and S i+i y i and i+1 The corresponding probability.

[0036] In some embodiments of the present invention, constructing a joint loss of a classification task, a bounding box detection task, and a key point detection task to optimize the initial detection model includes:

[0037] For the key point detection task, key point loss is used for training, and the calculation formula is:

[0038]

[0039] Among them, L kpts Represents the key point loss; N kpts Indicates the number of key points; d n represents the Euclidean distance between the nth predicted keypoint and the true keypoint; s represents the scale factor; σ(v n >0) indicates the key point visibility mark.

[0040] On the other hand, the present invention also provides a method for detecting an advanced support small catheter based on computer vision technology, the method comprising:

[0041] A depth camera is used to shoot the advanced small conduit support to be detected to obtain an image and a depth map thereof;

[0042] Input the image into the advanced small duct support detection model trained by the advanced support small duct detection model training method based on computer vision technology described above, to obtain the predicted category, predicted bounding box information and predicted key point coordinates of the advanced small duct support;

[0043] The identified bottom point of the advanced small catheter support is projected onto the aligned depth map for positioning; each bottom point is connected in sequence into a curve, and a normal is constructed for each bottom point on the curve; a preset number of pixels are offset upward and downward in the direction of the normal to obtain a new point set; for each offset bottom point, its corresponding depth value is obtained and sorted, and the minimum value is selected as the final depth value of the bottom point; the three-dimensional coordinates of each bottom point and the distance between adjacent bottom points are calculated based on the new point set and its depth value.

[0044] In some embodiments of the present invention, calculating the three-dimensional coordinates of each bottom point and the distances between adjacent bottom points includes:

[0045] After obtaining the three-dimensional coordinates of each bottom point, the bottom points are sorted according to the size of the horizontal coordinate values, and the distance between adjacent bottom points is calculated. The calculation formula is:

[0046]

[0047] Where Dis represents the distance between adjacent bottom points; (x i ,y i ,z i ) represents the three-dimensional coordinates of the i-th bottom point.

[0048] On the other hand, the present invention also provides an advanced support small catheter detection device based on computer vision technology, including a processor, a memory and a computer program / instructions stored in the memory, the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of any method mentioned above.

[0049] The present invention provides a training method, a detection method and a device for an advanced support small duct detection model. Firstly, a pre-trained image enhancement model is used to perform dark light enhancement on an advanced small duct support image, so as to improve the image recognizability and thus improve the accuracy of the manual standard. Secondly, a rotating target detection network with vector information is constructed, including a feature extraction network, a path aggregation network and a multi-task detection head. The feature extraction network is used to extract image features. The path aggregation network is used to perform top-down and bottom-up bidirectional fusion of image features to obtain fusion features. The multi-task detection head is used to decode the fusion features to obtain the predicted category, predicted bounding box information and predicted key point coordinates of the advanced small duct support in each sample, and a joint loss optimization initial detection model is constructed. Finally, the advanced small duct support detection model is trained. Finally, the two-dimensional point coordinates predicted by the detection model are projected onto a paired depth map, and the offset optimization method is used to calculate the number, three-dimensional coordinates and adjacent spacing of the advanced small duct support. The detection model provided by the present invention and the detection method implemented by using the detection model can non-contact and intelligently detect information such as the number, spacing, and three-dimensional coordinates of advanced small duct supports in tunnel projects, greatly improving the detection accuracy and efficiency, thereby effectively improving the construction efficiency of the project and reducing the intensity of manual labor and the degree of operational danger.

[0050] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0051] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0053] Figure 1 Schematic diagram of the steps of a method for training an advanced support small catheter detection model based on computer vision technology in one embodiment of the present invention.

[0054] Figure 2 It is a structural schematic diagram of an advanced small catheter support detection model in one embodiment of the present invention.

[0055] Figure 3 FIG. 4 is a schematic diagram of the structure of a structural reinforcement module in an embodiment of the present invention.

[0056] Figure 4 Schematic diagram of the results of a multi-task detection head in one embodiment of the present invention.

[0057] Figure 5 The figure is a schematic diagram of the steps of a method for detecting a small catheter for advanced support based on computer vision technology in one embodiment of the present invention.

[0058] Figure 6 Schematic diagram of a point offset optimization algorithm in one embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0060] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0061] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0062] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0063] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0064] In order to solve the problems of high labor intensity, high risk of close-range operation, high cost of detection equipment, high professional requirements, low automation rate and low intelligence rate in the existing advanced small duct support detection methods, the present invention provides an advanced support small duct detection model training method based on computer vision technology, such as Figure 1 As shown, the method includes the following steps S101 to S103:

[0065] Step S101: construct a training sample set, which includes multiple samples, each sample includes an advanced small catheter support image taken by a depth camera; add a label to each sample, wherein the label includes the category of the corresponding advanced small catheter support, bounding box information and key point coordinates.

[0066] Step S102: construct an initial detection model, which includes a feature extraction network, a path aggregation network and a multi-task detection head. Input the training sample set into the initial detection module, use the feature extraction network to extract multi-level image features, input the multi-level image features into the path aggregation network respectively, perform top-down and bottom-up bidirectional fusion, obtain multi-level fusion features, input the fusion features into the multi-task detection head for feature decoding, splice the prediction features output by each detection head, generate the class score and intersection score of each pixel and perform weighted calculation to obtain a comprehensive prediction score, sort and filter according to the comprehensive prediction score to obtain the prediction category, prediction bounding box information and prediction key point coordinates of the advanced small catheter support in each sample.

[0067] Step S103: Use the training sample set to train the initial detection model, and construct a joint loss of the classification task, bounding box detection task and key point detection task to optimize the initial detection model until the preset performance requirements are met, and the final advanced small catheter support detection model is obtained.

[0068] In step S101, a depth camera is used to shoot the advanced small catheter support to obtain an image, and the image is used as a sample to construct a training sample set. A true label is added to each sample for subsequent model training. Among them, the depth camera is a device that can capture the distance information between the object in the scene and the camera. It can not only obtain the color information of the image, but also generate the depth value corresponding to each pixel, thereby forming a depth map.

[0069] In some embodiments, considering the dim light inside the tunnel, the original image data collected has poor recognition. If the original image is directly labeled manually, a large number of missed labels and mislabeling may occur. Therefore, before adding labels to each sample, a pre-trained image enhancement model, such as ZeroDCE, is used to perform dark light enhancement on the advanced small catheter support image in each sample to improve image recognition and thus improve the accuracy of manual labeling.

[0070] Specifically, given an image P, each pixel of which is p i , the gray value of each pixel is I(·), and the gray mapping parameter predicted by the model is α. Then the gray mapping relationship between the original image and the enhanced image is shown in formula (1):

[0071] LE(I(p i );α)=I(p i )+αI(p i )(1-I(p i )); (1)

[0072] Among them, LE(I(p i ); α) represents the pixel point pi The enhancement effect of I(p i )∈[0,1] represents pixel p i The normalized structure of the grayscale value; α∈[-1,1] represents the trainable curve parameters. For images, all operations are channel-by-channel and pixel-by-pixel, that is, each pixel of a given input image has a corresponding mapping curve.

[0073] In order to cope with complex practical application scenarios, formula (1) can be used iteratively, as shown in formula (2):

[0074] LE n (x) = LE n-1 (x)+α n LE n-1 (x)(1-LE n-1 (x)); (2)

[0075] Where n∈{1,2,3,...,8} represents the number of iterations. When n=1, LE(·) degenerates to I(·). The annotations of other physical quantities are consistent with those in formula (1) and will not be repeated here.

[0076] In step S102, in order to solve the problem of disappearance of pipeline targets due to the accuracy of depth maps in actual applications, the present invention converts the direct detection of the advance small duct support into the detection of the bottom position of the support pipeline. However, it is difficult to achieve the purpose by directly using the rotating frame and key point detection. The former cannot obtain the pipeline direction information, and the latter has poor detection accuracy in the complex environment of the tunnel. Based on the above problems, the present invention specifically designs a rotating target detection network PIPENet with vector information. In the training stage, PIPENet is recorded as the initial detection model. After the training is completed, PIPENet is recorded as the advance small duct support detection model. Figure 2 As shown, PIPENet includes a feature extraction network, a path aggregation network and a multi-task detection head. The image of the advanced small catheter support to be detected is input into PIPENet for detection.

[0077] During the training phase, the training sample set is input into the initial detection model sequentially or in batches, and passes through the feature extraction network, path aggregation network and multi-task detection head in sequence to generate detection results.

[0078] In some embodiments, Figure 2As shown in the figure, the feature extraction network includes three stages. The first stage includes a convolution-batch normalization-activation function (CBS) integrated module and a structure enhancement module. The second stage includes a convolution-batch normalization-activation function integrated module and a cross-layer feature fusion module. The third stage includes a convolution-batch normalization-activation function integrated module, a cross-layer feature fusion module and a fast pyramid pooling module.

[0079] The preprocessed training sample set is input into the feature extraction network. For example, given an input image The high-frequency structural information is extracted in the first stage to obtain shallow features The semantic information is extracted in the second stage to obtain the intermediate features The third stage extracts semantic information in high dimensions, and uses fast pyramid pooling to aggregate features of different receptive fields, enhance feature representation, and obtain deep features. Among them, C represents the number of channels, W represents the width of the feature map, and H represents the height of the feature map.

[0080] In some embodiments, the complex environment of the advanced small duct support may cause the supporting pipeline to be confused with the other engineering structures in the background, and the differences in perspective rules and small duct support layout caused by the shooting angle may cause the supporting pipelines on the same image to have obvious scale and morphological differences. Therefore, in the present invention, a structural enhancement module for small duct support is designed in the first stage of the feature extraction network to enhance the pipeline target features.

[0081] like Figure 3 As shown in Figure 1, it is a schematic diagram of the structure enhancement module. The features extracted by the CBS module are evenly divided into the first feature and the second feature in the channel dimension, which are recorded as feature X and feature Y respectively. The second feature is input into the structure enhancement module, and a multi-scale high-frequency residual feature is constructed based on the Laplacian pyramid. The high-frequency residual feature and the original second feature are then input into the multi-core strip convolution for local feature enhancement. Exemplarily, this part includes 4 parallel depth-separable convolution paths, and the convolution kernel size of each path is {5×1, 1×5; 7×1, 1×7; 11×1, 1×11; 21×1, 1×21}. The strip convolution with a large kernel can not only improve the feature capture capability of tubular objects, but also capture the adjacent relationship between pipes. For example, pipes are often laid out based on beams, so they are distributed linearly rather than in a planar manner in the image. The features extracted by each strip convolution are added together and input into two consecutive convolution modules to aggregate local features of different receptive fields, strengthen information interaction between channels, and obtain aggregated features; finally, the first feature and the aggregated feature are spliced ​​in the channel dimension and input into the next stage. This operation not only enhances the pipeline structure features, gathers more gradient flows, but also improves the model accuracy.

[0082] The overall process can be expressed as formula (3) to formula (9):

[0083] X,Y=Split(Z); (3)

[0084] H=g i (Y)-Upsample(g i+1 (Y)); (4)

[0085] Y′=Y+H; (5)

[0086] CSP(Y)=Y′+(Conv(Y′)+Y′); (6)

[0087] Y′=CSP(CSP(Y)); (7)

[0088]

[0089] T=Enhance(Y′)+X; (9)

[0090] In formula (3) to formula (9), Split(Z) means dividing feature Z into two equal parts from the channel dimension; H represents the high-frequency residual feature; g i (·) represents Gaussian blur, i is the level number in the pyramid; Upsample(·) represents upsampling operation; Conv(·) represents 2d convolution; DWConv(·) represents depthwise separable convolution; Scale represents the convolution size; T represents aggregated features.

[0091] The multi-level image features extracted by the feature extraction network are input into the path aggregation network for a top-down and bottom-up bidirectional fusion to obtain a multi-level fusion feature. The features of these three levels are used to detect large targets, medium targets and small targets respectively, and are finally sent to the multi-task detection head for feature decoding to obtain the detection results.

[0092] In some embodiments, the multi-task detection head constructed in the feature decoding stage of the present invention adopts a three-branch detection head, which can simultaneously realize target category recognition, bounding box selection and key direction point detection.

[0093] like Figure 4As shown in the figure, each detection head contains three functions, namely category prediction, bounding box prediction and key point prediction. The features obtained by the path aggregation network are input into the multi-task detection head, and the features decoded and output by each detection head are spliced ​​to generate the class score and intersection over union (IoU) score of each pixel and perform weighted calculation to obtain a comprehensive prediction score. Sorting and screening are performed according to the comprehensive prediction score, such as taking the top 10 positive samples of the comprehensive prediction score of each target to obtain the predicted category, predicted bounding box information and predicted key point coordinates of the advanced small catheter support.

[0094] In step S103, the initial detection model is trained using the training sample set, and the joint loss of the classification task, bounding box detection task and key point detection task is constructed to optimize the initial detection model until the preset performance requirements are met, thereby obtaining the final advanced small catheter support detection model.

[0095] In some embodiments, for classification tasks, binary cross-entropy loss (BCELoss) is used for training, and the calculation formula is shown in formula (10):

[0096]

[0097] The calculation formula of σ is shown in formula (11):

[0098]

[0099] Among them, L class represents the binary cross entropy loss; N represents the number of samples; y i represents the true classification label of the i-th sample; p i represents the predicted probability of each sample, and σ(·) represents the Sigmoid function.

[0100] In some embodiments, for the bounding box detection task, ProbIoULoss and Distribution Focal Loss (DFL) are used for training;

[0101] The calculation formula of the intersection loss is shown in formula (12):

[0102]

[0103] Among them, L ProbIoU represents the intersection loss; B D Represents the Bhattacharyya distance between the true bounding box and the predicted bounding box;

[0104] The calculation formula of the distribution focus loss is shown in formula (13):

[0105] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+i )); (13)

[0106] Among them, y represents the label value, y i and i+1 are the integer values ​​to the left and right of the label respectively; S i and S i+i y i and i+1 The corresponding probability.

[0107] In some embodiments, for the key point detection task, key point loss is used for training, and the calculation formula is shown in formula (14):

[0108]

[0109] Among them, L kpts represents the key point loss; N kpts Indicates the number of key points; d n represents the Euclidean distance between the nth predicted keypoint and the true keypoint; s represents the scale factor; σ(v n >0) indicates the key point visibility mark.

[0110] Therefore, the binary cross entropy loss, intersection loss, distribution focus loss and key point loss are weighted to obtain the total loss of the entire detection network, which is calculated as shown in formula (15):

[0111] Loss = L class +L ProbIoU +DFL(S i ,S i+1 )+L kpts ; (15)

[0112] The total loss is used to train and optimize the initial detection module, and finally the advanced small duct support detection model is obtained.

[0113] The present invention also provides a method for detecting a small catheter for advanced support based on computer vision technology, such as Figure 5 As shown, the method includes the following steps S201 to S203:

[0114] Step S201: Use a depth camera to photograph the advanced small conduit support to be detected to obtain an image and a depth map thereof.

[0115] Step S202: Input the image into the advanced small duct support detection model trained by the advanced small duct support detection model training method based on computer vision technology described above to obtain the predicted category, predicted bounding box information and predicted key point coordinates of the advanced small duct support captured by the image.

[0116] Step S203: Project the identified bottom point of the advanced small catheter support onto the aligned depth map for positioning; connect each bottom point in sequence into a curve, and construct a normal for each bottom point on the curve; offset a preset number of pixels upward and downward in the normal direction to obtain a new point set; for each offset bottom point, obtain its corresponding depth value and sort them, and select the minimum value as the final depth value of the bottom point; calculate the three-dimensional coordinates of each bottom point and the distance between adjacent bottom points based on the new point set and its depth value.

[0117] In step S201, a depth camera is used to photograph the advanced small catheter support to be detected to obtain its image and depth map.

[0118] In step S202, the advanced small duct support detection model trained by the above training method is used to detect and obtain two-dimensional information such as the category, bounding box, key point direction, etc. of the corresponding advanced small duct support.

[0119] In step S203, the external rotation frame and direction of the advanced small catheter support are obtained through image detection, and the bottom point of each target is subsequently projected onto the registered depth map for positioning. Since the bottom point identified by the target detection algorithm based on deep learning is not necessarily accurate, the present invention constructs a point offset algorithm based on artificial priors, and projects the points on an image onto the same plane to calculate the spacing, such as Figure 6 As shown, specifically:

[0120] Obtain the bottom point (Bx i ,By i ), after i = {1,2,3...,n}, connect each bottom point in sequence into a curve L, based on each point (Bx i ,By i ) construct a normal, and shift a preset number of pixels upward and downward along the normal direction, such as 5 pixels, to obtain a new point set. For each offset bottom point, obtain its corresponding depth value and sort them, and select the minimum value as the final depth value of the bottom point; calculate the three-dimensional coordinates of each bottom point and the distance between adjacent bottom points based on the new point set and its depth value.

[0121] In some embodiments, for the bottom point after the shift, the new point set obtained is set to {(x1, y1), (x2, y2), ..., (xn ,y n )}, whose depth value set is {z1,z2,...,z n}, for each bottom point, use formula (16) to calculate its specific coordinates in camera coordinates:

[0122]

[0123] Among them, (x cam ,y cam ,z cam ) represents the coordinates in the camera coordinate system; (c x ,c y ) represents the coordinates of the image center in the pixel coordinate system; (p x ,p y ) represents the coordinate in the pixel coordinate system; f represents the focal length; α and β represent the correction factors in the horizontal and vertical directions, respectively.

[0124] Then, sort each bottom point in the camera coordinate system according to the size of the horizontal coordinate value, and calculate the distance between adjacent points. The calculation formula is shown in formula (17):

[0125]

[0126] Where Dis represents the distance between adjacent bottom points; (x i ,y i ,z i ) represents the 3D coordinates of the i-th bottom point, i∈{0,1,2,...,n}, and n is the number of pipeline targets in an image.

[0127] Corresponding to the above method, the present invention also provides an apparatus, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus implements the steps of the method described above.

[0128] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0129] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0130] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0131] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for training a small catheter detection model for advanced support based on computer vision technology, characterized in that: The method comprises the following steps: Constructing a training sample set, wherein the training sample set includes a plurality of samples, each sample includes an image of an advanced small conduit support taken by a depth camera; adding a label to each sample, wherein the label includes a category of the corresponding advanced small conduit support, bounding box information, and key point coordinates; Obtain an initial detection model, which includes a feature extraction network, a path aggregation network, and a multi-task detection head; input the training sample set into the initial detection module, use the feature extraction network to extract multi-level image features, input the multi-level image features into the path aggregation network respectively, perform top-down and bottom-up bidirectional fusion, obtain multi-level fusion features, input the fusion features into the multi-task detection head for feature decoding, splice the predicted features output by each detection head, generate a class score and an intersection score for each pixel, and perform weighted calculation to obtain a comprehensive prediction score, sort and screen according to the comprehensive prediction score to obtain the predicted category, predicted bounding box information, and predicted key point coordinates of the advanced small catheter support in each sample; The initial detection model is trained using the training sample set, and a joint loss of classification task, bounding box detection task and key point detection task is constructed to optimize the initial detection model until the preset performance requirements are met, thereby obtaining the final advanced small catheter support detection model.

2. The method for training a small catheter detection model for advanced support based on computer vision technology according to claim 1 is characterized in that: Before adding labels to each sample, it also includes: The pre-trained image enhancement model is used to perform dark light enhancement on the advanced small catheter support image in each sample; the grayscale mapping relationship between the original image and the enhanced image is expressed as: LE(I(p i );α)=I(p i )+αI(p i )(1-I(p i )); Among them, LE(I(p i ); α) represents the pixel point p i The enhancement effect of I(p i ) represents pixel p i The normalized structure of the grayscale value; α represents the trainable curve parameter.

3. The method for training a small-duct detection model for advanced support based on computer vision technology according to claim 1, characterized in that: The feature extraction network includes three stages, the first stage includes a convolution-batch normalization-activation function synthesis module and a structure enhancement module, the second stage includes a convolution-batch normalization-activation function synthesis module and a cross-layer feature fusion module, and the third stage includes a convolution-batch normalization-activation function synthesis module, a cross-layer feature fusion module and a fast pyramid pooling module. The method also includes: The preprocessed training sample set is input into the feature extraction network, the first stage extracts high-frequency structural information to obtain shallow features, the second stage extracts semantic information to obtain intermediate features, the third stage extracts semantic information in high dimensions, and uses pyramid pooling to aggregate features of different receptive fields, enhance feature representation, and obtain deep features.

4. The method for training a small catheter detection model for advanced support based on computer vision technology according to claim 3 is characterized in that: The structure enhancement module is used to enhance target features, including: In the first stage, the features extracted by the convolution-batch normalization-activation function comprehensive module are equally divided into first features and second features in the channel dimension; Inputting the second feature into the structure enhancement module to construct a multi-scale high-frequency residual feature; Inputting the high-frequency residual feature and the second feature together into a preset convolution module for local feature enhancement, and performing aggregation to obtain an aggregated feature; The first feature and the aggregated feature are concatenated in the channel dimension and input into the next stage.

5. The method for training a small conduit detection model for advanced support based on computer vision technology according to claim 1, characterized in that: Constructing a joint loss of the classification task, the bounding box detection task, and the key point detection task to optimize the initial detection model includes: For the classification task, binary cross entropy loss is used for training, and the calculation formula is: Among them, L class represents the binary cross entropy loss; N represents the number of samples; y i represents the true label of the i-th sample; p i represents the predicted probability of each sample, and σ(·) represents the Sigmoid function.

6. The method for training a small conduit detection model for advanced support based on computer vision technology according to claim 1, characterized in that: Constructing a joint loss of the classification task, the bounding box detection task, and the key point detection task to optimize the initial detection model includes: For the bounding box detection task, the intersection loss and distribution focus loss are used for training; The calculation formula of the intersection loss is: Among them, L ProbIoU represents the crossover loss; B D Represents the Bhattacharyya distance between the true bounding box and the predicted bounding box; The calculation formula of the distribution focus loss is: DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(y-y i )log(S i+i )); Among them, y represents the label value, y i and i+1 are the integer values ​​to the left and right of the label respectively; S i and S i+i y i and i+1 The corresponding probability.

7. The method for training a small-duct detection model for advanced support based on computer vision technology according to claim 1, characterized in that: Constructing a joint loss of the classification task, the bounding box detection task, and the key point detection task to optimize the initial detection model includes: For the key point detection task, key point loss is used for training, and the calculation formula is: Among them, L kpts Represents the key point loss; N kpts Indicates the number of key points; d n represents the Euclidean distance between the nth predicted keypoint and the true keypoint; s represents the scale factor; σ(v n >0) indicates the key point visibility mark.

8. A method for detecting small conduits for advanced support based on computer vision technology, characterized in that: The method comprises: A depth camera is used to shoot the advanced small conduit support to be detected to obtain an image and a depth map thereof; Input the image into the advanced small duct support detection model trained by the advanced support small duct detection model training method based on computer vision technology described in claims 1 to 7 to obtain the predicted category, predicted bounding box information and predicted key point coordinates of the advanced small duct support; The identified bottom point of the advanced small catheter support is projected onto the aligned depth map for positioning; each bottom point is connected in sequence into a curve, and a normal is constructed for each bottom point on the curve; a preset number of pixels are offset upward and downward in the normal direction to obtain a new point set; for each offset bottom point, its corresponding depth value is obtained and sorted, and the minimum value is selected as the final depth value of the bottom point; the three-dimensional coordinates of each bottom point and the distance between adjacent bottom points are calculated based on the new point set and its depth value.

9. The method for detecting small conduits for advanced support based on computer vision technology according to claim 8 is characterized in that: The three-dimensional coordinates of each bottom point and the distances between adjacent bottom points are calculated, including: After obtaining the three-dimensional coordinates of each bottom point, the bottom points are sorted according to the size of the horizontal coordinate values, and the distance between adjacent bottom points is calculated. The calculation formula is: Where Dis represents the distance between adjacent bottom points; (x i ,y i ,z i ) represents the three-dimensional coordinates of the i-th bottom point.

10. An advance support small conduit detection device based on computer vision technology, comprising a processor, a memory and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 9.

Images