Real-time visual damage detection of synthetic sling

Abstract

A method, apparatus and system for real-time visual damage detection of a synthetic sling, comprising: rolling in or out (701) a synthetic sling by a crane; obtaining (702) a stream of photographic images of the rope as it is rolled in or out by the crane; and detecting (703) a damage in the rope using a convolutional neural network (CNN). Training of the CNN includes: rolling in or out (704) the cord under a tensile load; obtaining (705) a stream of photographic images of the rope when the rope is rolled in or out under a tensile load; obtaining (706) two classification groups formed using the images, the two classification groups comprising a first group of images classified as good and a second group of images classified as bad; preprocessing the two groups of images (707); and feeding (708) the pre-processed image to the CNN.

Description

Technical Field

[0001] The present disclosure generally relates to real-time visual damage detection of synthetic lifting ropes. Background Art

[0002] This section presents useful background information without admitting that any of the techniques described herein represent the state of the art.

[0003] Cranes traditionally use steel wire as hoisting ropes. Synthetic hoisting ropes are an alternative that is particularly suitable for certain applications. Since the hoisting rope forms a single point of failure, it is desirable to monitor the rope for damage.

[0004] There are many different damage detection methods for steel ropes, but fewer methods for synthetic ropes. For example, electromagnetic testing, radiation testing, and eddy current inspection are the main non-destructive testing methods for steel ropes. However, these methods may not be suitable for synthetic ropes. Some other methods, such as ultrasonic guided wave evaluation and optical inspection, may also be useful for synthetic ropes. However, despite research, there is still a need for more reliable and alternative methods to detect damage in synthetic ropes. Summary of the invention

[0005] The attached claims define the scope of protection. Any examples and technical descriptions of devices, products and / or methods in the specification and / or drawings that are not covered by the claims are not presented as embodiments of the present invention, but are presented as background technology or examples for understanding the present invention.

[0006] According to a first example aspect, there is provided a method for real-time visual damage detection of a synthetic lifting rope, comprising: Synthetic slings are reeled in or out by cranes; obtaining a stream of photographic images of the synthetic sling as it is reeled in or out by the crane; and Detecting damage in synthetic lifting ropes using convolutional neural networks.

[0007] According to a second example aspect, there is provided a method for training a convolutional neural network for real-time visual damage detection of synthetic suspension ropes, the method comprising: Synthetic ropes that are reeled in or out of a crane under tensile load; obtaining a stream of photographic images of the synthetic sling as it is reeled in or out under a tensile load; obtaining two classification groups formed using images, the two classification groups comprising: a first group of images classified as good, wherein the synthetic rope is classified as good; and a second group of images classified as bad, wherein the synthetic rope is classified as damaged; Preprocessing the two sets of images; and Feed the preprocessed image to the convolutional neural network.

[0008] The acquisition of the image stream may include taking photos from two or more sides around the synthetic sling. The photos taken from two or more sides around the synthetic sling may be aligned in the longitudinal direction of the synthetic sling.

[0009] The acquisition of the image stream may include preprocessing the photo to reduce the computational complexity in the convolutional neural network. The preprocessing may include histogram equalization. The preprocessing may include converting the photo to a grayscale image, for example, with an 8-bit depth. The preprocessing may include reducing the resolution of the photo to, for example, 32×32 pixels. The reduction in resolution may include downsampling the photo.

[0010] A convolutional neural network may include two consecutive convolutional layers. A convolutional neural network may include two consecutive convolutional layers before any pooling layer such as a MaxPool layer.

[0011] The convolutional neural network can be trained using the Adam optimizer. The Adam optimizer can be as disclosed in DPKingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014, https: / / doi.org / 10.48550 / arXiv.1412.6980 Focus to learn more.

[0012] The convolutional neural network can be trained for at least 100 epochs. The convolutional neural network can be trained for at least 200 epochs. The convolutional neural network can be trained for at least 300 epochs.

[0013] Can be set to attenuate at least 10 -3 A convolutional neural network can be trained with a learning rate that is set to decay dynamically at two or more rates over a corresponding range of epochs. The learning rate can be set to decay as: 10 for epochs 1 to 120 -3 ; 10 for rounds 121 to 150 -4 ; 10 for rounds 151 to 180 -5 , and 10 for rounds 181 to 200 -6 .

[0014] You can train a convolutional neural network with a batch size of 16. You can train a convolutional neural network with a batch size of 64. You can train a convolutional neural network with a batch size of 128.

[0015] The convolutional neural network can be trained using categorical cross-entropy as a loss function. The validation accuracy can be monitored during the training of the convolutional neural network.

[0016] The visual damage check may be performed automatically, for example periodically or repeatedly. The visual damage check may be performed automatically after every N days before starting the first lifting task, N being one or more. The automatically performed visual damage check may include lifting the load to a given top position and lowering the load to the ground or floor to obtain photographs between these two positions. The top position may be at the maximum lifting position of the crane. The bottom position may be up to 2 meters above the ground or floor height.

[0017] According to a third example aspect, there is provided a device comprising at least one processor configured such that the method of the first or second example aspect is performed.

[0018] According to a fourth example aspect, there is provided an apparatus comprising means for performing the method of the first or second example aspect.

[0019] According to a fifth example aspect, a system is provided, the system comprising a device configured to perform the third or fourth example aspect of the method of the first example aspect and a device configured to perform the third or fourth example aspect of the method of the second example aspect.

[0020] According to a sixth example aspect, there is provided a computer program comprising computer executable program code, which when executed by at least one processor causes an apparatus to perform at least the method of the first or second example aspect.

[0021] According to a seventh example aspect, there is provided a computer program product comprising a non-transitory computer readable medium having stored thereon the computer program of the sixth example aspect.

[0022] According to an eighth example aspect, there is provided an apparatus comprising means for performing the method of any preceding aspect.

[0023] According to a ninth example aspect, there is provided a system comprising: Cranes, including synthetic rope hoisting elements; a camera system for taking pictures from at least two different sides of the synthetic rope as it is reeled in or out by the sling element; and The apparatus of claim 14, for real-time visual detection of damage in a synthetic hoist, or for training a convolutional neural network for real-time visual damage detection of synthetic hoist ropes.

[0024] The system may also include a synthetic rope. The synthetic rope may include polyethylene. At least 95 weight % of the synthetic rope may be polyethylene.

[0025] The system may include an online communication interface for sending the visual damage detection information to a remote location. The system may also include a crane control element configured to, if the visual damage detection indicates damage in the synthetic rope, block further use of the crane at least for new lifting tasks. Blocked use may be reset and the crane control element allows new tasks in response to receiving remote or local permission to proceed after verifying the synthetic rope. The verification may be based on a photograph taken by the camera system.

[0026] According to a tenth example aspect, there is provided a method, apparatus, computer program, computer program product or system according to any preceding example aspect, adapted to operate with a steel wire rope instead of a synthetic rope or alternately with a synthetic rope. The same apparatus may be provided with two or more different convolutional neural networks, which are optimized for respective two or more different types of ropes. The model to be used may be selected automatically or manually. The automatic selection may be based at least in part on detection of one or more electrical or magnetic properties of the rope, and / or on images taken of the rope, for example the color, texture and / or gloss of the rope material.

[0027] Providing for detecting damage to steel hoist ropes in nuclear power plants may be particularly advantageous, particularly in tall cranes such as gantry cranes, foundry cranes and high mountain or ground lifts, typically such applications where human access to the crane may be difficult. In nuclear and foundry applications, heat and / or radiation may prevent or make it difficult to provide human access to the crane for rope damage detection.

[0028] Any of the foregoing storage media may include digital data storage, such as a data disk or floppy disk; optical storage; magnetic storage; holographic storage; magneto-optical storage; phase change storage; resistive random access memory; magnetic random access memory; solid electrolyte storage; ferroelectric random access memory; organic storage; or polymer storage. The storage medium may be formed as a device having no other substantial function besides storing memory, or it may be formed as part of a device having other functions, including but not limited to the memory of a computer; a chipset; and a subassembly of an electronic device.

[0029] Different non-binding example aspects and embodiments have been described above. The foregoing embodiments are only used to explain selected aspects or steps that can be utilized in different implementations. Some embodiments may be presented with reference only to certain example aspects. It should be understood that the corresponding embodiments may also be applied to other example aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Some example embodiments will be described with reference to the accompanying drawings, in which:

[0031] Figure 1 schematically illustrates a system according to an example embodiment;

[0032] Figure 2a to Figure 2d shows sample images (good) through different stages of pre-processing according to an example embodiment;

[0033] Figure 3a to Figure 3d shows sample images (good) through different stages of pre-processing according to an example embodiment;

[0034] Figures 4a to 4d shows sample images (bad) through different stages of pre-processing according to an example embodiment;

[0035] Figure 5a to Figure 5d shows sample images (bad) through different stages of pre-processing according to an example embodiment;

[0036] Figure 6 A block diagram showing an apparatus according to an example embodiment; and

[0037] Figure 7a and Figure 7b A flow chart according to an example embodiment is shown. DETAILED DESCRIPTION

[0038] In the following description, the same reference numerals denote the same elements or steps.

[0039] Figure 1 A system according to an example embodiment is schematically illustrated.

[0040] The system 100 includes a camera system 110 including three cameras configured to image the synthetic rope from three different sides at uniform intervals of 120 degrees. The captured images are stored in an image memory 120. A preprocessing circuit 130 enhances the captured images and reduces their resolution and color space to better suit a deep learning model 150, here a convolutional neural network CNN. The preprocessed images are fed to a data separator 140, which separates the preprocessed images into three baskets with representative selections of the preprocessed images. These baskets are test images used by a training element 160 for: testing the deep learning model; training images; and validation images for jointly training the deep learning model or CNN 150. The training element 160 also generates a training model to which the test images are fed. The resulting classification of the test images is fed by the trained model to a performance evaluation and analysis circuit.

[0041] The test images are classified by a trusted classifier (such as a reliably trained deep learning model or one or more human experts) into good (OK) images and bad (not ok, NOK) images representing good and bad parts in the synthetic rope.

[0042] In an example embodiment, pre-processing includes histogram equalization followed by a reduction in color depth, for example, from three 14-bit color channels to one 8-bit grayscale, and a reduction in resolution, for example, by downsampling to 32×32 pixels.

[0043] Figure 2a to Figure 2d Sample images are shown passing through different stages of pre-processing according to an example embodiment. Figure 2a is a grayscale version of the original image of the nearly new synthetic rope. Figure 2b Showing it in 8-bit color depth, Figure 2c This is shown in the form of histogram equalization, Figure 2d It is shown at a resolution reduced to 32x32 pixels.

[0044] Figure 3a to Figure 3d A similar series of more worn synthetic ropes is shown, which have some signs of wear but are still classified as good. Figure 2d and Figure 3d There are obvious differences between them.

[0045] Figures 4a to 4d and Figure 5a to Figure 5d Two sample images (Not OK) through different stages of pre-processing according to an example embodiment are shown. These images clearly visualize, somewhat surprisingly, that 32x32 pixel resolution is sufficient to show damage in a synthetic rope.

[0046] Next, an example of a CNN proposal CNN-p is described. The convolutional neural network includes the following layers in this order: first convolutional layer, second convolutional layer, first maximum pooling layer, first dropout layer, third convolutional layer, second maximum pooling layer, second dropout layer, flatten layer, third dropout layer, first dense layer (fully connected layer), fourth dropout layer, and second dense layer.

[0047] The first convolutional layer is configured with L2 kernel adjustment 0.0005, L2 bias adjustment 0.0005, kernel size 3×3, and activation by rectified linear unit ReLU. The output shape of the first convolutional layer is 30×30×64. The first convolutional layer has 640 parameters.

[0048] The second convolutional layer is configured with L2 kernel adjustment 0.0005, L2 bias adjustment 0.0005, kernel size 3×3, and activation by rectified linear unit ReLU. The output shape of the second convolutional layer is 28×28×64. The second convolutional layer has 36928 parameters.

[0049] The first max pooling layer has pooling of 2 × 2. The first max pooling layer has an output shape of 14 × 14 × 64. The first max pooling layer has 0 parameters.

[0050] The first dropout layer has a rate of 0.4. The first dropout layer has an output shape of 14×14×64. The first dropout layer has 0 parameters.

[0051] The third convolutional layer is configured with L2 kernel adjustment 0.0005, L2 bias adjustment 0.0005, kernel size 3×3, and activation by rectified linear unit ReLU. The output shape of the third convolutional layer is 12×12×64. The third convolutional layer has 36928 parameters.

[0052] The second max pooling layer has pooling of 2 × 2. The second max pooling layer has an output shape of 6 × 6 × 64. The second max pooling layer has 0 parameters.

[0053] The second dropout layer has a rate of 0.4. The second dropout layer has an output shape of 6 × 6 × 64. The second dropout layer has 0 parameters.

[0054] The Flatten layer has an output shape of 2304. The Flatten layer has no parameters.

[0055] The third dropout layer has a rate of 0.4. The third dropout layer has an output shape of 2304. The third dropout layer has 0 parameters.

[0056] The first dense layer is configured with activations through Rectified Linear Units (ReLU). The output shape of the first dense layer is 20. The first dense layer has 46100 parameters.

[0057] The fourth dropout layer has a rate of 0.2. The fourth dropout layer has an output shape of 20. The fourth dropout layer has 0 parameters.

[0058] The second dense layer is configured with activation through Softmax. The output shape of the second dense layer is 2. The second dense layer has 42 parameters.

[0059] In total, the CNN-p layer has 120683 parameters.

[0060] In an example embodiment, the deep learning model is designed to aim for real-time execution and light computational cost. To this end, the layers have two consecutive convolutional layers before the first max pooling layer.

[0061] The CNN-p model is evaluated against two reference models. The two reference models are referred to here as Model A and Model B. Model A is based on the neural network model disclosed in "Surface defect detection for wireropes based on deep convolutional neural network" by Z. Ping, Z. Gongbo, L. Yingming, and H. Zhenzhi in the 14th IEEE International Conference on Electronic Measurement Instruments (ICEMI), 2019, pages 855-860, in particular, the model WRIPDCNN1 is used. Model B is based on the neural network model disclosed in "Automatic Detection of Industrial Wire Rope Surface Damage Using Deep Learning-Based Visual Perception Technology" by P. Zhou, G. Zhou, H. Wang, D. Wang, and Z. He in IEEE Transactions on Instrumentation and Measurement, volume 70, pages 1-11, 2021. As described below, models A and B are used to compare with CNN-p. Model A is modified by changing the output shape to 2 to match the current problem definition, good / bad. In addition, six dropout layers with a rate of 0.5 are added to alleviate the overfitting problem of the model. Model B is modified by increasing the network's original two dropout rates to 0.6 and adding three layers to alleviate overfitting.

[0062] Model A is a convolutional neural network that includes the following layers in this order: first convolutional layer, first max pooling layer, second convolutional layer, second max pooling layer, first dropout layer, third convolutional layer, third max pooling layer, second dropout layer, fourth convolutional layer, fourth max pooling layer, third dropout layer, flatten layer, fourth dropout layer, first dense layer, fifth dropout layer, second dense layer, sixth dropout layer, and third dense layer. The first convolutional layer has a 5×5 kernel and ReLU activation, a 64×64×32 output shape, and 832 parameters. The first max pooling layer has a 2×2 pool, a 32×32×32 output shape, and 0 parameters. The second convolutional layer has a 3×3 kernel and ReLU activation, a 32×32×64 output shape, and 18496 parameters. The second max pooling layer has a 2×2 pool, a 16×16×64 output shape, and 0 parameters. The first dropout layer has a rate of 0.5, a 16×16×64 output shape, and 0 parameters. The third convolutional layer has a 3×3 kernel and ReLU activation, a 16×16×128 output shape, and 73856 parameters. The third max pooling layer has a 2×2 pool, 8×8×128, and 0 parameters. The second dropout layer has a rate of 0.5, an output shape of 8×8×128, and 0 parameters. The fourth convolutional layer has a 3×3 kernel, ReLU activation, an output shape of 8×8×256, and 295168 parameters. The fourth max pooling layer has a 2×2 pool, an output shape of 4×4×256, and 0 parameters. The third dropout layer has a rate of 0.5, an output shape of 4×4×256, and 0 parameters. The flatten layer has an output shape of 4096 and 0 parameters. The fourth dropout layer has a rate of 0.5, an output shape of 4096, and 0 parameters. The first dense layer has a ReLU activation, an output shape of 2560, and 10488320 parameters. The fifth dropout layer has a rate of 0.5, an output shape of 2560, and 0 parameters. The second dense layer has a ReLU activation, an output shape of 768, and 1966848 parameters. The sixth dropout layer has a rate of 0.5, an output shape of 768, and 0 parameters. The third dense layer has a Softmax activation, an output shape of 2, and 1538 parameters. The total number of parameters is 12845058.

[0063] Model B is a convolutional neural network that includes the following layers in this order: first convolutional layer, first max pooling layer, second convolutional layer, second max pooling layer, third convolutional layer, third max pooling layer, first dropout layer, fourth convolutional layer, fourth max pooling layer, second dropout layer, flatten layer, third dropout layer, first dense layer, fourth dropout layer, second dense layer, fifth dropout layer, and third dense layer. The first convolutional layer has a 5×5 kernel and ReLU activation, a 96×96×16 output shape, and 416 parameters. The first max pooling layer has a 2×2 pool, a 48×48×16 output shape, and 0 parameters. The second convolutional layer has a 3×3 kernel and ReLU activation, a 48×48×32 output shape, and 4640 parameters. The second max pooling layer has a 2×2 pool, a 24×24×32 output shape, and 0 parameters. The third convolutional layer has a 3×3 kernel and ReLU activation, a 24×12×64 output shape, and 18496 parameters. The third max pooling layer has a 2×2 pool, 12×12×64, and 0 parameters. The first dropout layer has a rate of 0.6, an output shape of 12×12×64, and 0 parameters. The fourth convolutional layer has a 3×3 kernel, ReLU activation, an output shape of 12×12×96, and 55392 parameters. The fourth max pooling layer has a 2×2 pool, an output shape of 6×6×96, and 0 parameters. The second dropout layer has a rate of 0.6, an output shape of 6×6×96, and 0 parameters. The flatten layer has an output shape of 3456 and 0 parameters. The third dropout layer has a rate of 0.6, an output shape of 3456, and 0 parameters. The first dense layer has a ReLU activation, an output shape of 120, and 414840 parameters. The fourth dropout layer has a rate of 0.6, an output shape of 120, and 0 parameters. The second dense layer has ReLU activation, 32 output shape, and 3872 parameters. The fifth dropout layer has a rate of 0.6, 32 output shape, and 0 parameters. The third dense layer has a Softmax activation, 2 output shape, and 66 parameters. The total number of parameters is 497722.

[0064] The model was trained for 200 epochs using the Adam optimizer. The learning rate was decayed as follows: 10 for epochs 1 to 120 -3 ; 10 for rounds 121 to 150 -4 ; 10 for rounds 151 to 180 -5 , and 10 for rounds 181 to 200 -6 The batch size is 128. The loss function is defined by categorical cross entropy.

[0065] In the performance analysis, model A correctly classified 1735 good images and 1938 bad images, while incorrectly classified 267 good images and 64 bad images. Model B correctly classified 1845 good images and 1936 bad images, while incorrectly classified 138 good images and 66 bad images. The currently developed CNN model, CNN proposal or CNN-p correctly classified 1864 good images and 1931 bad images, while incorrectly classified 138 good images and 71 bad images. For the intended use, avoiding errors that classify images as good may be more important than avoiding errors that classify images as bad. In terms of accuracy, CNN-p also slightly beats model B by 0.948 (relative to 0.944) and beats model A by 0.917. In other words, it is shown that when running in real time at a rate compressed to 40 frames per second using a commercially available Apple MacBook Pro with an M1 Pro chip, a 10-core CPU with an integrated 16-core GPU, a 16-core neural engine, and 16GB of RAM, CNN-p works well even if the experimental code is written in Python 3.9.7 instead of some faster language (such as C++). The model size of CNN-p is only 730kB, compared to the 51.6MB and 2.25MB obtained with models A and B, respectively. CNN-p has 120,683 parameters, while models A and B have 12.8 million and 4.98 million parameters, respectively. Preprocessing also produces a smaller image of 1.05kB, compared to 3.59 and 7.42 for models A and B, respectively. The processing rate is 39.5 frames per second, while models A and B are 32.5 and 33.3, respectively.

[0066] In the receiver operating characteristic curve (ROC), the area under the curve (AUC) of CNN-p was 0.986, the area under the curve (AUC) of model A was 0.976, and the area under the curve (AUC) of model B was 0.986.

[0067] Heatmaps of the proposed CNN Grad-CAM are obtained for two correctly detected damaged ropes. These heatmaps demonstrate the adequacy of the model by focusing on pixels associated with visual detection of damage in the synthetic ropes.

[0068] In an example embodiment, the training dataset is augmented from the training dataset for CNN-p (20008) with different rope sizes and types to improve the generalization ability of the model.

[0069] In an example embodiment, rope position information is integrated for damage location. In an example embodiment, the rope position is determined using the crane's electric drive. In an example embodiment, the rope position is determined using optical analysis based on the output of the camera system 110. In an example embodiment, the rope position determination compensates for rope stretch. For example, the system 100 can estimate the stretch of the rope under the influence of the current load and / or the number of lifting operations and / or the combination of load time and number of lifting operations.

[0070] In an example embodiment, the model is extended to classify a wider range than just good and bad. For example, the model can be extended to classify into three categories, such as good / medium / bad. In this case, one or more intermediate categories can be used for different actions. In an example embodiment, the classification result or the classification result that is worse than good is presented with an indication of the time, the camera in question, and / or the rope position (such as the length position). For example, a rope damage estimated as damaged can be manually verified at the next scheduled maintenance, while a more severe category can be used to trigger a more urgent verification and / or stop the operation of the crane until the rope has been verified and / or replaced with a new rope.

[0071] In an example embodiment, the classification is then verified for at least some of the images. In an example embodiment, the verification of the classification is used to further train the deep learning model to improve its accuracy.

[0072] In an example embodiment, the classification results or better-than-better classification results are shown for a given portion of the rope, such as the portion currently at a given position, for example, at a distance of 20 cm to 1 m from the camera system 110 .

[0073] In an example embodiment, the system 100 enables local connection to the classified images of the rope to display the findings to the service personnel in the field. In an example embodiment, the classification results are displayed as a graph for each camera of the camera system 110 as a function of the rope position when the rope is reeled in or out. In particular, by connecting locally to the system, maintenance personnel can conveniently identify suspected damage locations and manually verify the condition of the rope at the corresponding location. In an example embodiment, the system enables winding of the suspected part at a specific inspection location, stopping winding, and / or displaying the image of the rope that resulted in the classification as bad. In an example embodiment, a deep learning model is used to form a heat map of the most important parts of the image for the classification results, and the heat map is displayed to the maintenance personnel to verify the classification more quickly.

[0074] In an example embodiment, the system 100 is configured to form a series of images of the same rope surface section that has been classified as bad. In an example embodiment, the series of images is displayed as a transition image or video that indicates how the condition of the rope changes over time. The system may also be configured to estimate how the rope will change in the future and / or when the rope condition will reach a predetermined critical condition that satisfies a predetermined condition, such as a minimum rope thickness, a maximum number of broken strands, and / or a minimum contrast between the estimated nominal surface of the rope and the adjacent area around the rope.

[0075] In an example embodiment, images of the rope as captured by each camera from a different side of the rope are presented in parallel for presenting a 360 degree view of the rope.

[0076] In an example embodiment, the same taxonomy is used in both training and production use of the deep learning model.

[0077] In an example embodiment, the method further comprises automatically issuing a signal in response to detecting that the synthetic rope should be replaced. In an example embodiment, the signal is used to stop the operation of the crane for a new lifting task until the synthetic rope has been manually verified.

[0078] In an example embodiment, one device implements any one or more of the image memory 120, the pre-processing circuit 130, the data separator 140, the deep learning model, the training model, and / or the performance evaluation and analysis.

[0079] Figure 6 A block diagram of an apparatus 600 and a mobile device 650 for implementing any one or more of the above elements is shown according to an example embodiment. The apparatus 600 includes a communication interface 610; a processor 620; a user interface 630; and a memory 640.

[0080] In an embodiment, the communication interface 610 includes wired and / or wireless communication circuitry, such as Ethernet; wireless LAN; Bluetooth; GSM; CDMA; WCDMA; LTE; and / or 5G circuitry. The communication interface may be integrated into the device 600, or provided as part of an adapter, card, etc. that may be attached to the device 600. The communication interface 610 may support one or more different communication technologies. The device 600 may also or alternatively include more than one communication interface 610.

[0081] In this document, a processor may refer to a central processing unit (CPU); a microprocessor; a digital signal processor (DSP); a graphics processing unit; an application specific integrated circuit (ASIC); a field programmable gate array; a microcontroller; or a combination of these elements.

[0082] The user interface may include circuitry for receiving input from a user of device 600, for example, via a keyboard; a graphical user interface shown on a display of device 600; speech recognition circuitry; or an accessory device, such as a headset, and for providing output to the user via, for example, a graphical user interface or a speaker.

[0083] The memory 640 includes a working memory 642 and a persistent memory 644 configured to store computer program code 646 and data 648. The memory 640 may include any one or more of the following: read-only memory (ROM); programmable read-only memory (PROM); erasable programmable read-only memory (EPROM); random access memory (RAM); flash memory; data disk; optical memory; magnetic memory; smart card; solid-state drive (SSD); etc. The device 600 may include multiple memories 640. The memory 640 may be configured as part of the device 600 or as an accessory that is inserted into a slot, port, etc. of the device 600 by a user or by another person or by a robot. The memory 640 may be used for the sole purpose of storing data or may be configured as part of the device 600 for other purposes, such as processing data.

[0084] In an example embodiment, the processor 620, memory 640 and computer program code are collectively configured to implement a web server for allowing on-site and / or remote access to the system 100. In an example embodiment, the system is configured to enable the use of a mobile device 650, such as a tablet computer or mobile phone, to display information through the system and / or receive user input, such as through a wireless or wired connection. In an example embodiment, the mobile device can be used to verify or adjust the position and / or orientation of the camera of the camera system 110.

[0085] Technical personnel understand that in addition to Figure 6 In addition to the elements shown in , device 600 may include other elements, such as a microphone; a display; and additional circuits, such as input / output (I / O) circuits; memory chips; application-specific integrated circuits (ASICs); processing circuits for specific purposes, such as source encoding / decoding circuits; channel encoding / decoding circuits; encryption / decryption circuits; etc. In addition, device 600 may include a disposable or rechargeable battery (not shown) for powering device 600 when an external power source is not available.

[0086] In an example embodiment, only a single neural network is used for visual damage detection using only image data. This avoids the complexity that would arise if, for example, multiple separate neural networks were used (optionally additionally incorporating further information, such as some electromagnetic measurements).

[0087] In an example embodiment, a neural network is trained from scratch rather than transferring learning to one or more neural networks.

[0088] In an example embodiment, images are classified without prior other classification methods. In an example embodiment, images are classified into two categories, good or bad.

[0089] In an example embodiment, each image as a whole is classified.

[0090] In an example embodiment, the images are downsized so that processing is significantly accelerated.

[0091] In example embodiments, images are captured at a frame rate of at least 35 frames per second or at least 40 frames per second.

[0092] In example embodiments, real-time visual damage detection may be performed on less powerful computing devices and with less memory and power consumption than known prior art solutions. Furthermore, training may be faster when new rope types are introduced.

[0093] In an exemplary embodiment, the hoisting rope has a circular cross-section. For example, a circular cross-section can help reduce problems caused by the fact that the rope may twist about its longitudinal direction when it is wound onto the rope drum. Round hoisting ropes also differ from belts and chains in that the orientation of any given patch on the rope side changes when in use. Therefore, the location of a particular damage on the rope can change position laterally or be divided into two or more images in a new way. This may further explain why relatively low image resolution seems to work well in real-time visual damage detection. It is worth noting that various damages manifest very differently in different types of ropes and other hoisting media. Even the materials used in the roller elements can have an impact on this. In addition, steel ropes have completely different manifestations of damage. A simple cut in a steel wire is completely different from a fiber-level damage that occurs in a synthetic rope, not to mention other possible types of damage. Therefore, any algorithm developed for steel ropes is obviously far from being able to be properly used as an indicator of the condition or remaining strength of synthetic ropes.

[0094] Next the performance of various damage types is compared between synthetic and wire ropes. Comparison of damage types between synthetic ropes and steel wires: *Both have diameter variations, but they occur for various reasons and can appear very different.

[0095] Figure 7a and Figure 7bA flowchart according to an example embodiment is shown, which shows various possible steps including some optional steps, while other steps may also be included and / or some steps may be performed more than once or in a different order, including: 701. Synthetic slings being reeled in or out by crane. 702. Obtain a stream of photographic images of a composite sling as it is reeled in or out by a crane. 703. Detecting Damage in Synthetic Lifting Ropes Using Convolutional Neural Networks. 704. Synthetic hoisting ropes reeled in or out of cranes under tensile load. 705. Obtain a stream of photographic images of the synthetic rope as it is reeled in or out under a tensile load. 706. Obtain two classification groups formed using the images, the two classification groups comprising: a first group of images classified as good, wherein the synthetic rope is classified as good; and a second group of images classified as bad, wherein the synthetic rope is classified as damaged. 707. Preprocess the two sets of images. 708. Feed the preprocessed image to the convolutional neural network. 709.Using two consecutive convolutional layers in convolutional neural networks. 710. Use two consecutive convolutional layers before any pooling layer such as max pooling layer. 711. When obtaining the image stream, take photos from two or more sides around the synthetic sling, and the photos are aligned in the longitudinal direction of the synthetic sling. 712. When obtaining the image stream, the photos are preprocessed to reduce the computational complexity in the convolutional neural network. 713. In preprocessing, histogram equalization is performed. 714. Preprocessing involves converting photos into grayscale images. 715. In preprocessing, the resolution of the photos was reduced to 32×32 pixels. 716.Use Adam optimizer to train convolutional neural networks. 717. In the training of a convolutional neural network, a learning rate is used that is set to dynamically decay at two or more rates over a corresponding range of rounds. 718. In the training of convolutional neural networks, batch sizes of 16 to 128 are used. 719. In the training of convolutional neural networks, categorical cross entropy is used as the loss function.

[0096] A technical effect of one or more example embodiments is that visual damage detection can be automatically arranged in real time with modest model memory and processing requirements. Another technical effect of one or more example embodiments is that visual damage detection can be automatically performed with good predictive capabilities, in particular with a reduced likelihood of misclassifying damaged rope segments as good. Yet another technical effect of one or more example embodiments is that visual damage detection can be automatically performed without hampering normal use of the crane in which the synthetic rope is installed.

[0097] Any of the foregoing methods, method steps, or combinations thereof may be controlled or performed using hardware; software; firmware; or any combination thereof. The software and / or hardware may be local; distributed; centralized; virtualized; or any combination thereof. Furthermore, any form of computing (including computational intelligence) may be used to control or perform any of the foregoing methods, method steps, or combinations thereof. For example, computational intelligence may refer to any artificial intelligence; neural networks; fuzzy logic; machine learning; genetic algorithms; evolutionary computing; or any combination thereof.

[0098] Various embodiments have been presented.It should be understood that in this document, the words include; comprise; and contain are used as open-ended expressions with no exclusivity intended.

[0099] The foregoing description has provided, by way of non-limiting examples of specific implementations and embodiments, a complete and informative description of the best mode presently contemplated by the inventors for carrying out the invention. However, it will be apparent to those skilled in the art that the invention is not limited to the details of the foregoing embodiments, but may be implemented using equivalent means or in different combinations of embodiments without departing from the features of the invention.

[0100] In addition, some features of the aforementioned disclosed example embodiments may be advantageously used without correspondingly using other features. Therefore, the foregoing description should be considered as merely an illustration of the principles of the present invention, rather than a limitation thereof. Therefore, the scope of the present invention is limited only by the attached patent claims.

Claims

1. A method for real-time visual damage detection of synthetic suspension ropes, comprising: Synthetic slings are reeled in or out by cranes; obtaining a stream of photographic images of the synthetic sling as the synthetic sling is reeled in or out by the crane; as well as Damage in the synthetic lifting rope is detected using a convolutional neural network configured to classify the obtained image as good or bad.

2. A method for training a convolutional neural network for real-time visual damage detection of synthetic suspension ropes, the method comprising: Synthetic ropes that are reeled in or out of a crane under tensile load; obtaining a stream of photographic images of the synthetic sling as the synthetic sling is reeled in or out under the tensile load; obtaining two classification groups formed using the images, the two classification groups comprising: a first group of images classified as good, wherein the synthetic rope is classified as good; and a second group of images classified as bad, wherein the synthetic rope is classified as damaged; Preprocessing the two sets of images; and Feed the preprocessed image to the convolutional neural network.

3. The method according to claim 1 or 2, wherein: The convolutional neural network includes two consecutive convolutional layers.

4. The method according to claim 3, wherein: The consecutive two convolutional layers are before any pooling layer such as a max pooling layer.

5. A method according to any one of the preceding claims, wherein: Acquiring the image stream includes taking photos from two or more sides around the synthetic sling, the photos being aligned in the longitudinal direction of the synthetic sling.

6. A method according to any one of the preceding claims, wherein: Obtaining the image stream includes preprocessing the photo to reduce computational complexity in the convolutional neural network.

7. The method according to claim 6, wherein: The pre-processing includes histogram equalization.

8. The method according to claim 6 or 7, wherein: The pre-processing includes converting the photograph into a grayscale image.

9. The method according to any one of claims 6 to 8, wherein: The pre-processing includes reducing the resolution of the photos to 32×32 pixels.

10. The method according to any one of claims 2 to 9, wherein: The convolutional neural network is trained using the Adam optimizer.

11. The method according to any one of claims 2 to 10, wherein: The convolutional neural network is trained using a learning rate that is set to dynamically decay at two or more rates over a corresponding range of epochs.

12. The method according to any one of claims 2 to 11, wherein: The convolutional neural network was trained with a batch size of 16 to 128.

13. The method according to any one of claims 2 to 12, wherein: The convolutional neural network is trained using categorical cross entropy as the loss function.

14. A device comprising: at least one memory including computer executable program code; as well as At least one processor is configured to, when executing the program code, at least cause the device to perform the method according to any one of the preceding claims.

15. A system comprising: Cranes, including synthetic rope hoisting elements; a camera system for taking pictures from at least two different sides of the synthetic rope as the synthetic rope is reeled in or out by the sling element; as well as The apparatus of claim 14, for real-time visual detection of damage in said synthetic rope, or for training a convolutional neural network for real-time visual damage detection of said synthetic lifting rope.

Images