A cable intelligent traction method

Data is collected through cameras and sensors, optical flow algorithms and neural networks are used to judge foreign object interference, correct acceleration, and draw accurate traction trajectory, which solves the problem of damage during cable traction and improves the accuracy and safety of the traction process.

CN118712958BActive Publication Date: 2024-11-15武汉华源电力设计院有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411194073.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-15
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

During the traction process, cables are susceptible to accidental tension, which leads to over-traction and cannot monitor the surrounding environment in real time, resulting in cable damage, affecting electrical insulation performance, and even causing power system failure.

Method used

The camera is used to collect images, combine acceleration sensors and mechanical sensors, and use optical flow algorithms to obtain the motion trajectory of pixel points. The neural network judges foreign object interference, corrects acceleration through filtering, draws accurate traction trajectory, and calculates the tension threshold to generate an alarm signal.

Benefits of technology

It effectively avoids cable jitter interference, improves the accuracy of trajectory acquisition, ensures the reliability of tension judgment, reduces false alarm signal triggering, and improves the safety and efficiency of the cable traction process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118712958B_ABST
    Figure CN118712958B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of data processing, and in particular to a method for intelligent cable traction, comprising: training a neural network for acquiring foreign objects according to the motion trajectory of each pixel point in an image, and the neural network outputting the interference trajectory when the pixel point is interfered by the foreign object; filtering and correcting the acceleration at each moment according to the motion trajectory of all pixel points outside the interference trajectory to obtain the corrected acceleration at each moment, and drawing the traction trajectory of the cable according to the corrected acceleration; marking the position of the foreign object on the traction trajectory, and calculating the tension threshold according to the traction trajectory, generating an alarm signal when the tension is greater than the tension threshold and there is a foreign object, and transmitting the traction trajectory marked with the foreign object and the alarm signal to a monitoring device. The present invention further improves the accuracy of obtaining the cable traction trajectory.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of data processing, and in particular to an intelligent cable traction method. Background Art

[0002] The cable may be subjected to unexpected tension during the traction process, resulting in over-traction or even causing the traction system to lose control of the cable. In addition, during the general traction process, it is impossible to monitor the surrounding environment in real time and make appropriate adjustments, and it is impossible to take timely response measures according to the actual situation, which will eventually cause the cable to be stretched, twisted or squeezed, thereby reducing the performance of the cable, affecting the electrical insulation performance, and even causing power system failures, resulting in significant economic losses. Therefore, the usual practice is to install a camera on the traction equipment to capture whether there are foreign objects during the traction process. At the same time, sensors are installed on the traction equipment. For example, an acceleration sensor can be used to obtain the trajectory during the traction process, and a mechanical sensor can determine whether the tension is overloaded by using a threshold judgment method. The above approach can deal with the problem of cable damage caused by traction in complex environments.

[0003] However, when the cable is pulled in the pipeline, the cable jitter interference is serious, and the trajectory acquisition during the pulling process is inaccurate, resulting in the inability to mark the location of foreign objects on the trajectory. The inaccurate trajectory will also lead to an incorrect estimation of the contact length between the cable and the pipeline, resulting in errors in the judgment of whether the tension is overloaded. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a cable intelligent traction method.

[0005] A cable intelligent traction method of the present invention adopts the following technical solution:

[0006] An embodiment of the present invention provides a cable intelligent pulling method, the method comprising the following steps:

[0007] When the cable in the pipeline is being pulled, a camera is used to collect images at each moment, an acceleration sensor is used to collect acceleration at each moment, and a mechanical sensor is used to obtain the tension of the cable during the pulling process;

[0008] The motion trajectory of each pixel in the image is obtained by using an optical flow algorithm; a neural network is used to obtain whether there is a foreign object in the image, and the neural network includes the motion trajectory of the pixel when it is interfered by the foreign object when it is inferred, which is recorded as the interference trajectory; the neural network inputs the image and the motion trajectory of the pixel when it is inferred and trained, and the loss function used in training is obtained from the motion trajectory of the pixel;

[0009] According to the motion trajectories of all pixel points outside the interference trajectory, the acceleration at each moment is filtered and corrected to obtain the corrected acceleration at each moment, and the trajectory of the cable during the traction process is drawn according to the corrected acceleration, which is recorded as the traction trajectory;

[0010] The location where the foreign object appears is marked on the traction trajectory, and a tension threshold is calculated based on the traction trajectory. When the tension is greater than the tension threshold and a foreign object is present, an alarm signal is generated, and the traction trajectory marked with the foreign object and the alarm signal are transmitted to a monitoring device.

[0011] Preferably, the loss function used in the training is obtained from the motion trajectory of the pixel points, and the specific steps included are as follows:

[0012] The loss function is equal to the cross entropy loss function plus the trajectory attention loss function, the trajectory attention loss function includes the trajectory attention of each pixel point, the trajectory attention is updated along with the neuron parameters in the neural network when being trained, and the trajectory attention loss function also includes the trajectory difference of each pixel point, the trajectory difference of each pixel point is equal to the average value of the difference between the motion trajectory of each pixel point and the motion trajectory of other pixels.

[0013] Preferably, the specific steps of obtaining the trajectory attention loss function are as follows:

[0014] During the neural network training process, when there is no foreign matter in the input image, the trajectory attention loss function is the square of the mean of the difference between the trajectory difference of all pixels and the trajectory attention; when there is a foreign matter in the input image, the pixels whose trajectory difference is greater than the trajectory attention are obtained and recorded as the first category of pixels, and the pixels whose trajectory difference is less than or equal to the trajectory attention are obtained and recorded as the second category of pixels. The difference between the average trajectory attention of the second category of pixels and the average trajectory attention of the first category of pixels is calculated, and the trajectory attention loss function is negatively correlated with the difference in the average trajectory attention.

[0015] Preferably, the motion trajectory of the pixel point when it is disturbed by foreign matter includes the following specific steps:

[0016] Anomaly detection is performed on the trajectory attention of all pixels to obtain abnormal pixels. The trajectory of the pixel with abnormal trajectory attention is recorded as the motion trajectory when interfered by foreign objects, and is recorded as the interference trajectory. The algorithm used for the anomaly detection is the LOF algorithm.

[0017] Preferably, the filtering and correction of the acceleration at each moment according to the motion trajectories of all pixel points outside the interference trajectory to obtain the corrected acceleration at each moment includes the following specific steps:

[0018] The motion trajectories of all pixel points outside the interference trajectory are recorded as target motion trajectories. Each target motion trajectory contains a displacement length and a displacement direction to construct a two-dimensional vector, which is recorded as a reference motion trajectory. The reference motion trajectory constructed by all target motion trajectories is averaged to obtain the average trajectory at each moment.

[0019] A filter kernel of a preset length is created, filter kernel parameters are obtained according to the difference between the average trajectory and acceleration at all moments in the filter kernel, and the acceleration at each moment is filtered according to the filter kernel and the filter kernel parameters to obtain the corrected acceleration at each moment.

[0020] Preferably, the tension threshold is positively correlated with the length of the traction track.

[0021] Preferably, the trajectory difference of each pixel point is equal to the average value of the difference between the motion trajectory of each pixel point and the motion trajectory of other pixel points, and the specific steps included are as follows:

[0022] The average value of the Euclidean distance between the reference motion trajectory of each pixel and the reference motion trajectory of other pixels is taken as the trajectory difference of each pixel;

[0023] The reference motion trajectory of each pixel point is a two-dimensional vector constructed by the displacement length and the displacement direction contained in the motion trajectory of each pixel point.

[0024] Preferably, the specific steps of obtaining the reference motion trajectory are as follows:

[0025] For the motion trajectory of any pixel point, the motion trajectory includes the displacement lengths and displacement directions at several adjacent moments. All displacement lengths and displacement directions of each pixel point are averaged to obtain the average displacement length and average displacement direction, and the two-dimensional vector formed by the average displacement length and the average displacement direction is used as the reference motion trajectory.

[0026] Preferably, the step of obtaining the filter kernel parameters according to the difference between the average trajectory and acceleration at all times in the filter kernel comprises the following specific steps:

[0027] The average trajectory and acceleration at all moments in the filter kernel are linearly normalized respectively, and the absolute value of the difference between the normalized average trajectory and acceleration at each moment in the filter kernel is calculated, the filter kernel parameter is negatively correlated with the absolute value of the difference, and the sum of the filter kernel parameters is equal to 1;

[0028] The absolute value of the difference between the average trajectory and the acceleration includes the absolute value of the difference between the displacement length and the acceleration magnitude in the average trajectory, and also includes the absolute value of the difference between the displacement direction and the acceleration direction in the average trajectory.

[0029] Preferably, the trajectory attention loss function is negatively correlated with the difference in the average trajectory attention, and the specific steps included are as follows:

[0030] The square of the difference between the average trajectory attention of the second type of pixels and the average trajectory attention of the first type of pixels is recorded as s, and exp(-s) is used as the trajectory attention loss function, where exp() represents an exponential function with a natural constant as the base.

[0031] The beneficial effects of the technical solution of the present invention are:

[0032] The present invention filters and corrects the acceleration at each moment according to the motion trajectory of the pixel point to obtain the corrected acceleration at each moment, and draws the trajectory of the cable during the traction process according to the corrected acceleration, which is recorded as the traction trajectory. This effectively avoids the problem of serious cable jitter interference and inaccurate trajectory acquisition during the traction process when the cable is pulled in the pipeline.

[0033] The present invention uses the motion trajectory of all pixel points except the interference trajectory when obtaining the corrected acceleration at each moment, wherein the interference trajectory is obtained by a neural network for obtaining whether there is a foreign object, and the loss function used in the neural network training is obtained by the motion trajectory of the pixel points; on the one hand, the interference of the motion trajectory of the foreign object pixel points is avoided, and the accuracy and reliability of the corrected acceleration are further ensured; on the other hand, the interference trajectory is obtained by the neural network for obtaining whether there is a foreign object, while ensuring the accuracy of the foreign object judgment, combining the features between the foreign object and the motion trajectory extracted by the neural network, and integrating the acquisition of the foreign object and the interference of the foreign object on the motion trajectory into one neural network task, thereby improving the reliability of the interference trajectory.

[0034] A tension threshold is calculated based on the traction trajectory, and an alarm signal is generated when the tension is greater than the tension threshold and when foreign matter is present. The tension threshold is adaptively calculated based on the acquired accurate and reliable traction trajectory to avoid the problem of false triggering of the alarm signal, such as avoiding the situation where the friction between the cable and the pipeline is too large, resulting in excessive tension and causing a false alarm. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flowchart of a method for intelligent cable pulling provided by one embodiment of the present invention;

[0037] Figure 2Schematic diagram of the traction device. DETAILED DESCRIPTION

[0038] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation, structure, features and effects of a cable intelligent traction method proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0039] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0040] The specific scheme of the intelligent cable traction method provided by the present invention is described in detail below with reference to the accompanying drawings.

[0041] See also Figure 1 , which shows a flowchart of a cable intelligent pulling method provided by an embodiment of the present invention, the method comprising the following steps:

[0042] Step S001: When the cable in the pipeline is being pulled, a camera is used to collect images at each moment, an acceleration sensor is used to collect acceleration at each moment, and a mechanical sensor is used to obtain the tension of the cable during the pulling process.

[0043] In this embodiment, when pulling the cable, a pulling device is used. The cable to be pulled is fixed at one end of the pulling device, and a steel wire rope for pulling is fixed at the other end. The steel wire rope pulls the pulling device to move in the pipeline under the traction of the motor outside the pipeline, thereby achieving the pulling of the cable. Figure 2 As shown, the device has a power supply module, a tension measurement module, an inertial navigation unit, a processor, a camera, and data cables connecting the above modules.

[0044] The power supply module includes a group of batteries (such as lithium-ion batteries) to power other modules. The tension measurement module includes a stress sensor for measuring the stress of the traction device. The stress sensor uses a patch strain gauge to measure the mechanical load. The change in resistance on the strain gauge is converted into a voltage signal through a bridge circuit to obtain the stress magnitude (stress sensor is a well-known technology, and its specific principle is not repeated in this embodiment). In this embodiment, the traction device generates stress during the traction process, so this embodiment records the stress output by the stress sensor as the tension of the cable during the traction process.

[0045] The inertial navigation unit includes an acceleration sensor, which is used to measure the acceleration of the traction device at each moment during the traction process, and is used to describe the position information and motion state data of the device. The camera is placed at the head end of the traction device to collect images in the pipeline. The images collected in this embodiment are grayscale images. In other embodiments, when the images collected are color images, they need to be grayscaled to grayscale images. The processor is used to read and process the data output by the camera, stress sensor, and acceleration sensor.

[0046] In this embodiment, the data (including images, tension, and acceleration) output by the camera, the stress sensor, and the acceleration sensor are collected once every second.

[0047] Step S002: Use the optical flow algorithm to obtain the motion trajectory of each pixel in the image.

[0048] The purpose of this embodiment is to obtain the traction trajectory of the cable during the traction process, wherein the motion trajectory of the pixel points in the image captured by the camera can describe the traction trajectory.

[0049] The specific method is: obtain multiple images at each moment and before that moment (for example, 10 images, other embodiments may use other numbers of images, if there are not 10 images before, no subsequent processing and calculation are performed), and use the optical flow algorithm to process these images to obtain several motion trajectories, each of which contains the displacement length and displacement direction of the pixel point at two adjacent moments. In this embodiment, the motion trajectory of each pixel point in the image collected at each moment is recorded as the motion trajectory of each pixel point, which is used to describe the motion state of each pixel point in the image collected at each moment within a period of time (for example, within the first 10 moments of each moment or on the first 10 images). The optical flow algorithm is a well-known technology and will not be described in detail in this embodiment.

[0050] The cable in the pipeline moves along the pipeline under the traction of the traction device, so the motion trajectory of all pixel points also includes the traction trajectory information of the cable during the traction process.

[0051] Step S003: training a neural network for obtaining whether there is a foreign object in the image, wherein the loss function used in the training is obtained from the motion trajectory of the pixel points.

[0052] In order to detect the movement of the cable in the pipeline during the traction process and avoid damage to the cable, this embodiment needs to detect whether there are foreign objects during the traction process and mark them on the traction track for visual display. Therefore, before obtaining the traction track, this embodiment first detects and determines whether there are foreign objects in the pipeline based on the image, such as debris, plastic fragments, metal fragments, or metal mesh and wire.

[0053] This embodiment uses a neural network to detect foreign objects in an image, such as a ResNet50 network, an SSD network, a DeepLabV3 network, etc.

[0054] Conventionally, the input of the neural network is an image. When the neural network adopts the ResNet50 network structure, the output of the neural network is 0 or 1; 0 represents that there is no foreign body in the image, and 1 represents that there is a foreign body in the image; at this time, the task of the neural network is a binary classification task, the data set for training the network is easy to label, and the network structure is simple, but the specific position or area of ​​the foreign body in the image cannot be located; when the neural network adopts the SSD network structure, the output of the neural network is a bounding box, which is a rectangular box including the foreign body area (the bounding box is not output when there is no foreign body). At this time, the task of the neural network is a positioning task, which can roughly locate the area of ​​the foreign body in the image; however, the data set for training the network is not easy to label, the network structure is relatively complex, the amount of calculation is large, and the specific contour area of ​​the foreign body cannot be described; when the neural network adopts the DeepLabV3 network structure, the output of the neural network is the semantic area of ​​the foreign body (the semantic area is not output when there is no foreign body). At this time, the task of the neural network is a semantic segmentation task, which can segment the specific area of ​​the foreign body in the image; however, the workload of labeling the data set for training the network is large, the network structure has many parameters, and the amount of calculation is large.

[0055] The above network structures can all be used to determine whether there are foreign objects. However, this embodiment takes into account that when obtaining the traction trajectory of the cable during the traction process, the acceleration at each moment and the motion trajectory of each pixel point contain information about the traction trajectory. Therefore, the traction trajectory can be constructed using the acceleration at all moments, and the traction trajectory can be constructed using the motion trajectory of the pixel points in the image collected at all moments. However, due to the serious jitter interference of the traction device during the migration process (especially when encountering foreign objects), the acceleration at each moment contains a lot of noise information. At the same time, when there are foreign objects in the image, the appearance of the foreign objects causes the motion trajectory of each pixel point to not accurately represent the movement of each pixel point along the pipeline, which makes it impossible to directly obtain the traction trajectory based on the motion trajectory of the pixel point or the acceleration at each moment.

[0056] It is also taken into consideration that the above-mentioned neural network can extract the features of the foreign matter in the image when determining whether there is a foreign matter. Therefore, this embodiment further obtains the motion trajectory of the pixel points under the interference of the foreign matter based on the features of the foreign matter extracted by the neural network, thereby avoiding the interference of these foreign matter trajectories on the acquisition of the traction trajectory.

[0057] The specific steps are:

[0058] (1) First, the input of the neural network includes not only the image but also the motion trajectory of all pixels in the image.

[0059] It should be noted that in the image collected at each moment, the motion trajectory of each pixel represents: the displacement length and displacement direction of the pixel at two adjacent moments in a period of time (for example, within the first 10 moments of each moment); therefore, the motion trajectory of each pixel can be expressed as multiple (for example, 9) displacement lengths and displacement directions, and these displacement lengths and displacement directions of each pixel constitute a multi-dimensional vector (for example, a 2×9=18-dimensional vector), then all vectors of all pixels are arranged in sequence according to the pixel position to form a multi-channel trajectory image (for example, an 18-channel trajectory image, that is, each pixel on the trajectory image is an 18-dimensional vector mentioned above). The image collected by the camera at each moment is a single-channel image (that is, a grayscale image), and the single-channel image collected by the camera at each moment and the multi-channel trajectory image are stacked together to form a spliced ​​image (that is, an 18+1=19-channel image), and the spliced ​​image is used as the input of the neural network.

[0060] In other embodiments, in order to further reduce the amount of input data of the neural network, the input of the neural network can be implemented by the following method: only the displacement length and displacement direction of each pixel at each moment compared with the previous moment are obtained to form a two-dimensional displacement vector for each pixel, and the two-dimensional displacement vectors of all pixels constitute a two-channel trajectory image, which is stacked with the single-channel image captured by the camera at each moment into a three-channel stitched image and used as the input of the neural network.

[0061] In some other embodiments, the grayscale value of each pixel in the single-channel image captured by the camera at each moment mentioned above can be multiplied by the displacement length and displacement direction in the two-dimensional displacement vector of each pixel to obtain the superimposed two-dimensional displacement vector of each pixel, and the two-channel trajectory image composed of the superimposed two-dimensional displacement vectors of all pixels can be used as the input of the neural network.

[0062] (2) Construct a data set for training the neural network. The specific method is to collect a large number of images from the pipeline, take each image and the motion trajectory of all pixels in the image as samples, and manually label the samples. All samples and their labels constitute a data set. There are several methods for labeling samples: when the neural network uses the ResNet50 network structure, if there is a foreign object in the sample, it is labeled as 1, and if there is no foreign object, it is labeled as 0; when the SSD network structure is used, if there is a foreign object in the sample, the minimum circumscribed rectangular frame of the foreign object is drawn, and if there is no foreign object, no labeling is performed; when the DeepLabV3 network structure is used, if there is a foreign object in the sample, the contour area of ​​the foreign object is drawn, and if there is no foreign object, no labeling is performed.

[0063] The method of constructing or acquiring a data set is a commonly known method for training a neural network, and the specific process of acquiring the data set will not be described in detail in this embodiment.

[0064] The neural network of this embodiment uses the ResNet50 network structure. Since the amount of calculation is relatively small, the power supply module and the processor in the traction device can be made smaller, so that the entire traction device can be miniaturized, which is more suitable for the traction of cables with relatively small diameters. In other embodiments, the DeepLabV3 network structure can be used. At this time, due to the relatively large amount of calculation, there are requirements for the capacity of the power supply and the data processing capability of the processor (for example, a large volume and large capacity battery pack, or a CPU or storage device with stronger data processing capability, or even a heat dissipation device or heat dissipation component), resulting in a larger volume of the traction device. At this time, it is suitable for the traction of cables with larger diameters.

[0065] (3) The neural network is trained using the data set. During the training process, the neural network's neuron parameters are updated using the stochastic gradient descent algorithm. However, in order for the neural network to obtain the motion trajectory of the pixel points under the interference of foreign objects, the loss function of the neural network during training must also be related to the motion trajectory of the pixel points. That is, it is necessary to construct a loss function for training the neural network based on the motion trajectory of the pixel points, so as to guide the neural network to further learn the features between the foreign objects and the motion trajectory of the pixel points, so that the neural network can obtain the motion trajectory of the pixel points under the interference of foreign objects.

[0066] As an optional example, the loss function is obtained as follows:

[0067] Considering that the cross entropy loss function is a commonly used loss function in neural network training, it can effectively guide the neural network to learn and extract foreign body features. Therefore, this embodiment uses the cross entropy loss function as a part of the loss function for training the neural network.

[0068] The similarity of the motion trajectory between any two pixels is obtained (the similarity used in this embodiment is the cosine similarity between the 18-dimensional vectors corresponding to the motion trajectory obtained above), and the average similarity of the motion trajectory between all pixels is recorded as S1. When there is a foreign object in the image input to the neural network, the loss function is equal to the cross entropy loss function plus S1. At this time, the neural network will minimize the loss function by updating the neuron parameters (S1 is also minimized). At this time, the feature learned by the neural network is: the similarity of the trajectories between different pixels is small when there is a foreign object. When there is no foreign object in the image input to the neural network, the loss function is equal to the cross entropy loss function minus S1. At this time, the neural network will minimize the loss function by updating the neuron parameters (S1 is maximized). At this time, the feature learned by the neural network is: the similarity of the trajectories between different pixels is large when there is no foreign object, and they can all describe the movement of the traction device in the pipeline.

[0069] The loss function obtained through the motion trajectory of the pixel points can ensure that the neural network can learn the features between the motion trajectory of the foreign object and the pixel points, and the features between the motion trajectory of the foreign object and the pixel points include the motion trajectory of the pixel points under the interference of the foreign object. However, the features between the motion trajectory of the foreign object and the pixel points obtained above only include the motion trajectory of the pixel points under the interference of the foreign object, and cannot further specifically describe which pixel points are the motion trajectories of the pixel points under the interference of the foreign object and which are not the motion trajectories of the pixel points under the interference of the foreign object.

[0070] Based on this, as a preferred example, the method for obtaining the loss function is as follows:

[0071] In this embodiment, the cross entropy loss function is used as a part of the loss function for training the neural network, and the other part of the loss function is the trajectory attention loss function. The cross entropy loss function and the trajectory attention loss function are used as the loss function for training the neural network.

[0072] The trajectory attention loss function includes the trajectory attention of each pixel. The trajectory attention can be updated together with the neuron parameters in the neural network when being trained. In this embodiment, a neuron parameter is artificially specified in the neural network multiplied by the gray value of each pixel as the trajectory attention of each pixel. In other embodiments, a neuron parameter multiplied by the mean of all displacement lengths in the motion trajectory of the pixel can be specified as the trajectory attention of each pixel, or a neuron parameter multiplied by the modulus of the corresponding 18-dimensional vector of each pixel can be specified as the trajectory attention of each pixel. In this implementation, the first neuron parameter of the penultimate feature layer of the ResNet50 network structure is used as a specified neuron parameter. In other embodiments, other neuron parameters can be specified, and other methods can be used to calculate the trajectory attention of each pixel. It is only necessary that each pixel corresponds to a trajectory attention (that is, the trajectory attention is associated with the pixel one-to-one), and the trajectory attention can be updated together with the neuron parameters in the neural network when being trained.

[0073] In addition, the trajectory attention loss function also includes the trajectory difference of each pixel, and the trajectory difference of each pixel represents the average value of the difference between the motion trajectory of each pixel and the motion trajectory of other pixels.

[0074] Preferably, the method for obtaining the difference between the motion trajectory of each pixel and the motion trajectory of other pixels includes:

[0075] Among them, the motion trajectory of each pixel point includes the displacement lengths and displacement directions of several adjacent moments. In this embodiment, all the displacement lengths and displacement directions of each pixel point are averaged to obtain the average displacement length and the average displacement direction, and the two-dimensional vector composed of the average displacement length and the average displacement direction is used as the reference motion trajectory of each pixel point. In other embodiments, the two-dimensional vector composed of the displacement lengths and displacement directions of the two most recent adjacent moments in the motion trajectory of each pixel point can be used as the reference motion trajectory of each pixel point. In this embodiment, the Euclidean distance between the reference motion trajectory of each pixel point and the reference motion trajectory of each other pixel point is used as the difference between the motion trajectory of each pixel point and the motion trajectory of each other pixel point.

[0076] The loss function in the above process includes the difference in the trajectory of each pixel, so that the loss function can learn the characteristics between the foreign object and the motion trajectory of each specific pixel through the difference in the trajectory between each pixel and other pixels. In addition, the trajectory attention of each pixel describes whether the impact of the foreign object on the pixel when the foreign object appears in the image needs to be paid attention to or the degree of attention. It is a parameter that can be updated, so as to guide the neural network to further learn the situation where the motion estimation of each pixel is interfered by foreign objects on the basis of learning the characteristics between the foreign object and the motion trajectory of the pixel.

[0077] As a preferred example, the specific method for obtaining the trajectory attention loss function is:

[0078] During the neural network training process, when there is no foreign matter in the input image, the difference between the trajectory difference of each pixel and the trajectory attention of each pixel is taken as the difference of each pixel, and the trajectory attention loss function is the square of the mean of the differences of all pixels; at this time, the neural network minimizes the loss function by updating the neuron parameters, so that the trajectory difference of each pixel approaches the trajectory attention of each pixel, and then when there is no foreign matter in the image, the trajectory attention of the pixel depends on the size of the trajectory difference of the pixel, thereby ensuring that the neural network can extract features from the movement between pixels when there is no foreign matter.

[0079] When there are foreign objects in the input image, the pixels whose trajectory difference is greater than the trajectory attention are obtained and recorded as the first type of pixels. The pixels whose trajectory difference is less than or equal to the trajectory attention are obtained and recorded as the second type of pixels. The difference between the average trajectory attention of the second type of pixels and the average trajectory attention of the first type of pixels is calculated. At this time, the trajectory attention loss function is negatively correlated with the square of the difference. For example, when the square of the difference is recorded as s, the trajectory attention loss function is exp(-s), and exp() represents an exponential function with a natural constant as the base. In other embodiments, as the trajectory attention loss function.

[0080] At this time, the neural network minimizes the loss function by updating the neuron parameters, so that the average trajectory attention of the second type of pixels is significantly different from the average trajectory attention of the first type of pixels, so that the neural network has the ability to clearly distinguish which pixels are disturbed by foreign objects and which are not. In other words, it has the ability to clearly distinguish which pixels are disturbed by foreign objects or not.

[0081] As an optional example, the specific method of obtaining the trajectory attention loss function is:

[0082] Regardless of whether there are foreign objects in the image, the trajectory attention loss function is negatively correlated with the square of the above difference s.

[0083] This method can also enable the neural network to distinguish between the motion trajectory of pixels that are disturbed by foreign objects or the motion trajectory that is not disturbed by foreign objects. However, since the features between foreign objects and the motion trajectory of pixels cannot be extracted in images without foreign objects, the neural network's ability to distinguish in this method is relatively unclear.

[0084] So far, a neural network for training the loss function is obtained. The neural network is trained using the loss function and the data set. Since the training method of the neural network is well known, the specific training process will not be described in detail in this embodiment.

[0085] Step S004, using a neural network to determine whether there is a foreign object in the image; when the neural network is inferred, the movement trajectory of the pixel when it is disturbed by the foreign object is included, which is recorded as the interference trajectory.

[0086] The trained neural network is obtained above, and the images collected by the camera on the traction device at each moment and the motion trajectories of all pixels in the image are input into the trained neural network to obtain whether there are foreign objects in the image.

[0087] It should be noted that the neural network used in this embodiment is a ResNet50 network structure. When the output of the neural network is greater than 0.5, there is a foreign object in the image, otherwise there is no foreign object. In other embodiments, when the SSD network structure or the DeepLabV3 network structure is used, when the output of the neural network has a bounding box or a semantic area, there is a foreign object in the image, and when there is no bounding box or a semantic area, there is no foreign object.

[0088] The image collected at each moment and the motion trajectory of all pixels in the image are input into the trained neural network to obtain whether there are foreign objects in the image. This process is the reasoning process of the neural network, that is, the process of obtaining the output result from the input of the neural network.

[0089] Furthermore, since the loss function is obtained based on the motion trajectory of the pixel points during the training process, the neural network can learn and extract the features between the foreign matter and the motion trajectory of the pixel points, including the motion trajectory of the pixel points under the interference of the foreign matter. Therefore, when the trained neural network is inferring, it includes the motion trajectory of the pixel points under the interference of the foreign matter in the image collected at each moment, that is, the interference trajectory of the pixel points in the image collected at each moment when they are interfered by the foreign matter. This interference trajectory will affect the acquisition of the cable pulling trajectory.

[0090] As an optional example, the method for obtaining the interference trajectory is:

[0091] When using the DeepLabV3 network structure, since the neural network input includes the motion trajectory of the pixels, the semantic area of ​​the foreign object output by the neural network is more accurate than when there is no motion trajectory of the pixel input. Therefore, the pixels in the semantic area can be recorded as pixels with foreign object interference, and the trajectories of these pixels can be recorded as interference trajectories when the pixels are interfered by foreign objects.

[0092] However, this method has some defects: during the cable pulling movement, some pixels are not in the semantic area, but they were in the semantic area at the last moment or several moments before that. In other words, some pixels are not foreign objects, but they were foreign objects at the last moment or several moments before that. These pixels should also be regarded as pixels with foreign object interference, but the above results cannot regard them as pixels with foreign object interference. For another example, some pixels are in the semantic area, but they were outside the semantic area for most of the previous moments. At this time, the movement trajectory of these pixels is still valuable, and it is still important to obtain the cable pulling trajectory. At this time, these pixels should not be regarded as pixels with foreign object interference, but the above results will regard them as pixels with foreign object interference.

[0093] Therefore, the above method is not very accurate.

[0094] As another optional example, the method for obtaining the interference trajectory is:

[0095] Whether using the DeepLabV3 network structure, the ResNet50 network structure, or the SSD network structure, the neural network must contain multiple feature layers. For example, in the ResNet50 network structure, the convolution operation, normalization operation, and activation function are used as a feature layer, and each feature layer outputs a feature map. Each pixel point in the feature map output by each feature layer is a multidimensional vector, which contains the features learned or extracted in the neural network. The feature layer is a well-known technology in neural networks and will not be described in detail in this embodiment.

[0096] In this embodiment, all feature maps of the same size as the input image are obtained, and anomaly detection is performed on the multidimensional vectors of all pixels on these feature maps to obtain abnormal pixels. These abnormal pixels are recorded as pixels interfered by foreign objects, and the trajectories of these pixels are recorded as interference trajectories when the pixels are interfered by foreign objects.

[0097] It should be noted that neural network structures generally have feature maps with the same size as the input image. If the neural network structure used does not have a feature map with the same size as the input image, it is necessary to add such features or feature maps to the neural network structure (for example, directly use the first feature layer of the ResNet50 network structure, or add a feature layer with a convolution kernel size of 1×1, a step size of 1, and no pooling operation before the first feature layer).

[0098] Considering that the cable moves along the pipeline as a whole during the migration process, the interference trajectory of the pixel when it is disturbed by foreign objects is obviously different from the movement trajectory of most other pixels. Therefore, the abnormal pixel is recorded as the pixel with foreign object interference, and the trajectory of the pixel with foreign object interference is recorded as the interference trajectory when the pixel is disturbed by foreign objects. The above anomaly detection algorithm is the LOF (local outlier factor) algorithm.

[0099] However, in the above method, the multidimensional vector of each pixel in the feature map output by each feature layer contains features other than the features when the pixel is interfered by foreign objects, such as foreign object features or pipeline features, etc. Therefore, the interference trajectory obtained above is not very accurate.

[0100] Based on this, as a preferred example, the method for obtaining the interference trajectory is:

[0101] The image and the motion trajectory of all pixels are input into the trained neural network to obtain the trajectory attention of each pixel, perform anomaly detection on the trajectory attention of all pixels, and obtain pixels with abnormal trajectory attention. The trajectory of the pixel with abnormal trajectory attention is recorded as the interference trajectory when interfered by foreign objects.

[0102] In this process, the trajectory attention of each pixel describes whether the impact of foreign objects on the pixel needs to be paid attention to or the degree of attention when foreign objects appear in the image. It can specifically characterize the situation where the motion estimation of each pixel is interfered by foreign objects. Therefore, the interference trajectory obtained above is more reliable. The above anomaly detection algorithm is a LOF (local outlier factor) algorithm.

[0103] Step S005: filtering and correcting the acceleration at each moment according to the motion trajectories of all pixel points outside the interference trajectory to obtain the corrected acceleration at each moment, and drawing the trajectory of the cable during the traction process according to the corrected acceleration, which is recorded as the traction trajectory.

[0104] Considering that the acceleration at each moment can more directly describe the instantaneous motion state of the cable when it is pulled, the noise generated by the jitter or vibration of the cable accumulates over a long period of time, which leads to a certain interference in the motion trajectory reflected by the acceleration over a long period of time. Although the motion trajectory of the pixel points obtained above (especially the motion trajectory of all pixel points outside the interference trajectory) can also describe the motion state of the cable when it is pulled, the motion trajectory is not sensitive to the cable motion state in a short time or short distance, but can describe the approximate motion direction or motion trend of the cable over a long period of time. Therefore, this embodiment filters and corrects the acceleration at each moment according to the motion trajectory of all pixel points outside the interference trajectory to obtain the corrected acceleration at each moment, and then fuses the motion trajectory of the pixel points and the acceleration data to obtain the accurate corrected acceleration at each moment.

[0105] Finally, the attitude solution algorithm is used to obtain the traction trajectory based on the accurate corrected acceleration at each moment. The traction trajectory represents the position of the traction device at each moment (it can also describe the position of the cable at each moment).

[0106] The attitude solution algorithm can solve the attitude information of the carrier (ie, the traction device) in the navigation coordinate system, and solve the position information through integral calculation, and then draw a trajectory. The attitude solution algorithm is a well-known technology and its specific principle will not be described in detail in this embodiment.

[0107] As a preferred example, the method for obtaining the corrected acceleration at each moment includes:

[0108] At each moment, the motion trajectories of all pixels outside the interference trajectory are averaged to obtain the average trajectory at each moment, and a filter kernel with a length of 13 is created (other filter kernel lengths may be set in other embodiments). The difference between the average trajectory at all moments in the filter kernel and the acceleration is used as the filter kernel parameter. The acceleration at each moment is filtered according to the filter kernel and the filter kernel parameters to obtain the corrected acceleration at each moment.

[0109] The average trajectory of all pixels outside the interference trajectory is calculated to obtain the average trajectory at each moment, including:

[0110] The motion trajectory of each pixel point includes the displacement lengths and displacement directions of several adjacent moments. In this embodiment, all displacement lengths and displacement directions of each pixel point are averaged to obtain the average displacement length and the average displacement direction, and the two-dimensional vector formed by the average displacement length and the average displacement direction is used as the reference motion trajectory of each pixel point. In other embodiments, the two-dimensional vector formed by the displacement lengths and displacement directions of the two most adjacent moments in the motion trajectory of each pixel point can be used as the reference motion trajectory of each pixel point.

[0111] The average of the motion trajectories of all pixels outside the interference trajectory is equal to the average of the reference motion trajectories of all pixels outside the interference trajectory, and the average trajectory at each moment is obtained (the average trajectory is also a two-dimensional vector, including two dimensions of displacement length and displacement direction).

[0112] The difference between the average trajectory and acceleration at all times in the filter kernel is used as the filter kernel parameter, including the following steps:

[0113] Considering that acceleration also contains two dimensions, namely, acceleration magnitude and acceleration direction. The displacement length and acceleration magnitude in the average trajectory of all moments in the filter kernel are linearly normalized respectively, in order to remove the dimension, and the absolute value of the difference between the normalized displacement length and acceleration magnitude at each moment is calculated, recorded as y, and exp(-y) is recorded as the movement amplitude trend at each moment (in other embodiments, The motion amplitude trend at each moment is recorded as the motion amplitude trend at each moment), the motion amplitude trend at all moments in the filter kernel is normalized using the softmax normalization method (probability normalization method can be used in other embodiments), and the normalized result at each moment is recorded as the filter kernel parameter. The purpose of this normalization is to make the sum of the filter kernel parameters equal to 1. The acceleration magnitude at all moments is filtered using the filter kernel parameters (that is, the filter kernel parameters in the filter kernel are multiplied by the acceleration magnitude respectively and then summed to obtain the filtering result. The method of filtering according to the filter kernel parameters is well known and will not be described in detail in this embodiment).

[0114] Similarly, the displacement direction and acceleration direction in the average trajectory of all moments in the filter kernel are linearly normalized respectively, in order to remove the dimension, and the absolute value of the difference between the normalized displacement direction and the acceleration direction at each moment is calculated and recorded as x, and exp(-x) is used as the movement direction trend at each moment (in other embodiments, The motion direction trend at each moment is recorded as the motion direction trend at each moment), and the motion direction trend at all moments in the filter kernel is normalized using the softmax normalization method (in other embodiments, the probability normalization method can be used), and the normalized result at each moment is recorded as the filter kernel parameter. The purpose of this normalization is to make the sum of the filter kernel parameters equal to 1. The filter kernel parameters are used to filter the acceleration direction at all moments.

[0115] After filtering the acceleration magnitude and acceleration direction, the corrected acceleration (also including the magnitude and direction dimensions) at each moment is obtained.

[0116] The above filtering is performed based on the difference between the motion trajectory and the acceleration, that is, the above-mentioned x and y. When the difference between the motion trajectory and the acceleration is large (that is, when x and y are small), the filtering process pays more attention to the magnitude and direction of the acceleration at this time. When the difference between the motion trajectory and the acceleration is small (that is, when x and y are large), the filtering process pays less attention to the magnitude and direction of the acceleration at this time, thereby ensuring the accuracy of the filtering.

[0117] As an optional example, the method for obtaining the corrected acceleration at each moment includes:

[0118] Gaussian filtering is performed on the acceleration using Gaussian filter kernels with lengths of 3, 4, 7, 9, 11, 13, and 17 respectively (the acceleration magnitude and acceleration direction are filtered separately). The Pearson correlation coefficient is calculated between the Gaussian filter at each length and the reference motion trajectory of the pixel point (the acceleration magnitude and acceleration direction are calculated with the length and direction of the reference motion trajectory respectively). The filter kernel length with the largest Pearson correlation coefficient is taken as the target length. The acceleration magnitude and acceleration direction are Gaussian filtered using the target length to obtain the corrected acceleration at each moment.

[0119] It should be noted that the displacement direction of the pixel point motion trajectory is a direction angle in the two-dimensional space of the image, and the acceleration direction has three direction angles in the three-dimensional space. In this embodiment, the absolute value of the difference between the displacement direction of the motion trajectory and the normalized result of each direction of the acceleration is calculated respectively, that is, the above-mentioned x. In other embodiments, there is a component acceleration at each direction angle of the acceleration direction. In other embodiments, only the direction angle with the largest component acceleration in the filter kernel can be considered, and the other direction angles do not participate in the above calculation.

[0120] Step S006, mark the location where the foreign object appears on the traction trajectory, and calculate the tension threshold based on the traction trajectory, generate an alarm signal when the tension is greater than the tension advance and there is a foreign object, and transmit the traction trajectory marked with the foreign object and the alarm signal to the monitoring device.

[0121] The moment when the foreign matter appears in the image is obtained, and the position of the traction device at that moment is marked on the traction track.

[0122] At the same time, in order to avoid the cable being damaged by excessive tension during cable pulling, a tension threshold needs to be set. When the tension is greater than the tension threshold, an alarm is triggered. However, considering that the greater the contact length between the cable and the pipeline, the greater the friction between the cable and the pipeline, the tension threshold cannot be set too small to pull the cable. Otherwise, the friction will cause a false alarm when the tension is greater than the tension threshold. At this time, the ratio of the length of the pulling track to the total length of the cable is obtained, recorded as w, and the tension threshold is set to w×th1, where th1 is 0.8 times the maximum tension that the cable can withstand. In other embodiments, th1 can be set to other values, which are not specifically limited in this embodiment.

[0123] When the pulling force is greater than the pulling force advance and there is a foreign object, an alarm signal is generated. The traction track marked with the foreign object and the alarm signal are transmitted to the monitoring device through the data cable and presented to the staff. The staff performs cable traction operations according to the monitoring device. For example, when an alarm signal is received, a smaller pulling force is applied to pull slowly.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cable intelligent pulling method, It is characterized in that The method comprises the following steps: When the cable in the pipeline is being pulled, a camera is used to collect images at each moment, an acceleration sensor is used to collect acceleration at each moment, and a mechanical sensor is used to obtain the tension of the cable during the pulling process; The motion trajectory of each pixel in the image is obtained by using an optical flow algorithm; a neural network is used to determine whether there is a foreign object in the image. When the neural network is inferred, the motion trajectory of the pixel is included when the pixel is disturbed by the foreign object, which is recorded as the interference trajectory; when the neural network is inferred and trained, the image and the motion trajectory of all pixels are input, and the loss function used in training is obtained from the motion trajectory of all pixels; According to the motion trajectories of all pixel points outside the interference trajectory, the acceleration at each moment is filtered and corrected to obtain the corrected acceleration at each moment, and the trajectory of the cable during the traction process is drawn according to the corrected acceleration, which is recorded as the traction trajectory; Marking the location of the foreign object on the traction track, calculating the tension threshold according to the traction track, generating an alarm signal when the tension is greater than the tension threshold and the foreign object is present, and transmitting the traction track marked with the foreign object and the alarm signal to a monitoring device; The motion trajectory of each pixel represents: the displacement length and displacement direction of the pixel at two adjacent moments in a period of time; the motion trajectory is used to describe the motion state of each pixel in the image collected at each moment in a period of time; The tension threshold is positively correlated with the length of the traction track.

2. According to claim 1, a cable intelligent pulling method, It is characterized in that The loss function used in the training is obtained from the motion trajectory of the pixel points, and the specific steps included are as follows: The loss function is equal to the cross entropy loss function plus the trajectory attention loss function, the trajectory attention loss function includes the trajectory attention of each pixel point, the trajectory attention is updated along with the neuron parameters in the neural network when being trained, and the trajectory attention loss function also includes the trajectory difference of each pixel point, the trajectory difference of each pixel point is equal to the average value of the difference between the motion trajectory of each pixel point and the motion trajectory of other pixels.

3. According to claim 2, a cable intelligent pulling method, It is characterized in that The specific steps for obtaining the trajectory attention loss function are as follows: During the neural network training process, when there is no foreign matter in the input image, the trajectory attention loss function is the square of the mean of the difference between the trajectory difference of all pixels and the trajectory attention; when there is a foreign matter in the input image, the pixels whose trajectory difference is greater than the trajectory attention are obtained and recorded as the first category of pixels, and the pixels whose trajectory difference is less than or equal to the trajectory attention are obtained and recorded as the second category of pixels. The difference between the average trajectory attention of the second category of pixels and the average trajectory attention of the first category of pixels is calculated, and the trajectory attention loss function is negatively correlated with the difference in the average trajectory attention.

4. According to claim 2, a cable intelligent pulling method, It is characterized in that The motion trajectory of the pixel when it is disturbed by foreign matter includes the following specific steps: Anomaly detection is performed on the trajectory attention of all pixels to obtain abnormal pixels. The trajectory of the pixel with abnormal trajectory attention is recorded as the motion trajectory when interfered by foreign objects, and is recorded as the interference trajectory. The algorithm used for the anomaly detection is the LOF algorithm.

5. According to claim 1, a cable intelligent pulling method, It is characterized in that The filtering and correction of the acceleration at each moment according to the motion trajectories of all pixel points outside the interference trajectory to obtain the corrected acceleration at each moment includes the following specific steps: The motion trajectories of all pixels outside the interference trajectory are recorded as target motion trajectories. Each target motion trajectory contains a displacement length and a displacement direction to form a two-dimensional vector, which is recorded as a reference motion trajectory. The reference motion trajectory constructed by all target motion trajectories is averaged to obtain the average trajectory at each moment; A filter kernel of a preset length is created, filter kernel parameters are obtained according to the difference between the average trajectory and acceleration at all moments in the filter kernel, and the acceleration at each moment is filtered according to the filter kernel and the filter kernel parameters to obtain the corrected acceleration at each moment.

6. According to claim 2, a cable intelligent pulling method, It is characterized in that The trajectory difference of each pixel point is equal to the average value of the difference between the motion trajectory of each pixel point and the motion trajectory of other pixel points, and the specific steps included are as follows: The average value of the Euclidean distance between the reference motion trajectory of each pixel and the reference motion trajectory of other pixels is taken as the trajectory difference of each pixel; The reference motion trajectory of each pixel point is a two-dimensional vector constructed by the displacement length and the displacement direction contained in the motion trajectory of each pixel point.

7. A cable intelligent pulling method according to claim 5 or 6, It is characterized in that The specific steps of obtaining the reference motion trajectory are as follows: For the motion trajectory of any pixel point, the motion trajectory includes the displacement lengths and displacement directions at several adjacent moments. All displacement lengths and displacement directions of each pixel point are averaged to obtain the average displacement length and average displacement direction, and the two-dimensional vector formed by the average displacement length and the average displacement direction is used as the reference motion trajectory.

8. According to claim 5, a cable intelligent pulling method, It is characterized in that The specific steps of obtaining the filter kernel parameters according to the difference between the average trajectory and acceleration at all times in the filter kernel are as follows: The average trajectory and acceleration at all moments in the filter kernel are linearly normalized respectively, and the absolute value of the difference between the normalized average trajectory and acceleration at each moment in the filter kernel is calculated, the filter kernel parameter is negatively correlated with the absolute value of the difference, and the sum of the filter kernel parameters is equal to 1; The absolute value of the difference between the average trajectory and the acceleration includes the absolute value of the difference between the displacement length and the acceleration magnitude in the average trajectory, and also includes the absolute value of the difference between the displacement direction and the acceleration direction in the average trajectory.

9. According to claim 3, a cable intelligent pulling method, It is characterized in that The trajectory attention loss function is negatively correlated with the difference in the average trajectory attention, and includes the following specific steps: The square of the difference between the average trajectory attention of the second type of pixels and the average trajectory attention of the first type of pixels is recorded as s, and exp(-s) is used as the trajectory attention loss function, where exp() represents an exponential function with a natural constant as the base.