Defect detection method and defect detection system for engineering equipment

Abstract

This invention provides a defect detection method for engineering equipment, comprising: receiving multimodal data of a component to be inspected from the engineering equipment, the multimodal data including at least one of images, audio, and video collected from the component; performing frame-by-frame processing on the video to separate the images and audio; acquiring the network status with a server; when the network status is connected, uploading the multimodal data to the server; performing collaborative analysis and processing of the multimodal data using a multimodal large model configured on the server combined with a dedicated detection model; and outputting a detection report, wherein the detection report includes: the defect type of the component to be inspected and handling suggestions; when the network status is interrupted, performing offline inference on the multimodal data using a lightweight model to output the defect type of the component to be inspected. This method is optimized for defects in mining equipment, achieving high diagnostic accuracy; and the integration of a professional knowledge base embedded in the large model makes the output results more authoritative and standardized.

Description

Technical Field

[0001] This invention relates to the field of engineering safety technology, and in particular to a defect detection method and system for engineering equipment. Background Technology

[0002] Mine hoists, as core engineering equipment in various mining production systems, are primarily responsible for transporting mined mineral resources to the surface. The operational status of mine hoists directly impacts the production efficiency and safety of the mine. While regular maintenance is the usual method for monitoring the operational status of mine hoists, this approach cannot promptly detect potential safety hazards. With advancements in audio and video processing technology, a solution for real-time monitoring of mine hoist operational status is needed. Summary of the Invention

[0003] This invention provides a method for detecting defects in engineering equipment to solve the technical problem of not being able to detect safety hazards in mine hoists in a timely manner.

[0004] A first aspect of the present invention provides a defect detection method for engineering equipment, comprising: receiving multimodal data of a component to be inspected from the engineering equipment, the multimodal data including at least one of images, audio, and video collected from the component to be inspected; performing frame-by-frame processing on the video to separate the images and audio; acquiring the network status with a server; when the network status is network connected, uploading the multimodal data to the server; performing collaborative analysis and processing on the multimodal data using a multimodal large model configured on the server combined with a dedicated detection model; and outputting a detection report, wherein the detection report includes: the defect type of the component to be inspected and handling suggestions; when the network status is network interrupted, performing offline inference on the multimodal data using a lightweight model to output the defect type of the component to be inspected.

[0005] In one embodiment of the present invention, the dedicated detection model includes an image detection model and an audio analysis model. The image detection model is structurally optimized for the working environment of the engineering equipment and the defect area of ​​the component to be detected, and the audio analysis model is adaptively trained for the background noise characteristics and the sound defect characteristics of the component to be detected.

[0006] In one embodiment of the present invention, the image detection model is structurally optimized, including: adding an attention mechanism to the backbone network of the image detection model, the attention mechanism including a feature attention branch corresponding to the part to be detected, wherein the feature attention branch uses a strip convolutional kernel to enhance the linear texture of the part to be detected in the horizontal / vertical direction in the image, highlighting the structure of the part to be detected, and uses surface correction and a circular convolutional kernel to enhance the texture continuity of the part to be detected; configuring anchor boxes of various sizes adapted to various defects of the part to be detected; and using a dynamic focusing loss function for model training, the dynamic focusing loss function dynamically adjusts the weights according to the size of the anchor boxes, assigning high weights to small-sized anchor boxes.

[0007] In one embodiment of the present invention, the audio analysis model undergoes adaptive training, comprising: configuring a defect feature extraction layer in the audio analysis model, the defect feature extraction layer including a low-frequency enhancement filter for filtering low-frequency noise, an envelope demodulator for extracting low-frequency information from high-frequency resonances included in the audio, and an impulse pulse detector for identifying the audio semantics of the component to be detected; and using adversarial examples to perform noise adversarial training on the audio analysis model, wherein the adversarial examples are generated by mixing recorded background noise with the audio of the real component to be detected, the background noise including at least one of fan noise, water flow noise, and mechanical vibration noise.

[0008] In one embodiment of the present invention, multimodal data is uploaded to a server, and the multimodal data is collaboratively analyzed and processed by a multimodal large model configured on the server in conjunction with a dedicated detection model to output a detection report. This includes: calling the dedicated detection model to obtain preliminary identification results, which include the defect type, location, and confidence level output by the image detection model and / or audio analysis model after processing the image and / or audio; based on the defect type, location, and confidence level, calling the multimodal large model to retrieve the configured safety standard database, and generating a detection report according to the mandatory standard clauses, industry regulations, and fault handling knowledge configured in the safety standard database.

[0009] In one embodiment of the present invention, the component to be tested includes at least one of a steel wire rope and a drive shaft configured in the engineering equipment.

[0010] In one embodiment of the present invention, the defect types corresponding to the drive shaft include at least one of the following: outer ring failure, inner ring failure, rolling element failure, and cage failure, and the defect types corresponding to the wire rope include broken wires.

[0011] In one embodiment of the present invention, the lightweight model is obtained by performing structured pruning and quantization compression on a multimodal large model and a dedicated detection model.

[0012] A second aspect of the present invention provides a defect detection system for engineering equipment, characterized in that it includes: a data acquisition and preprocessing module, a network judgment module, a cloud-based intelligent processing module, and a local offline operation module, wherein the data acquisition and preprocessing module and the network judgment module are configured on a terminal device, and the cloud-based intelligent processing module is configured on a server; the data acquisition and preprocessing module is used to receive multimodal data of the component to be inspected in the engineering equipment, the multimodal data including at least one of images, audio, and video acquired for the component to be inspected; the network judgment module is used to obtain the network status between the terminal device and the server; the intelligent processing module is used to receive the multimodal data from the terminal device when the network status is network connected, and to perform collaborative analysis and processing through a multimodal large model combined with a dedicated detection model, and output a detection report, wherein the detection report includes: the defect type of the component to be inspected and handling suggestions; the local offline operation module is used to perform offline inference on the multimodal data through a lightweight model when the network status is network interrupted, and output the defect type of the component to be inspected.

[0013] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the defect detection method for the engineering device described in the first aspect.

[0014] The beneficial effects of this invention are as follows: By integrating text, image, and audio information through a multimodal large model, it provides more comprehensive analysis. The dual-mode operation mechanism, combining online (cloud-based) and offline (local) operation, effectively addresses network instability issues in mines, ensuring the continuity of operation and maintenance support functions. The multimodal large model is optimized for mine equipment defects, resulting in high diagnostic accuracy; combined with the professional knowledge base embedded in the large model, the output results are more authoritative and standardized. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0016] In the attached diagram:

[0017] Figure 1 This is an application scenario diagram illustrating the detection of the operating status of a mine hoist, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a defect detection system for the operation status of a mine hoist provided in one embodiment of the present invention; Figure 3 This is a flowchart illustrating a defect detection method for the operating status of a mine hoist, provided in one embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of an electronic device for a defect detection system provided in one embodiment of the present invention. Detailed Implementation

[0018] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0019] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0020] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0021] Please see Figure 1 , Figure 1 This is an application scenario diagram illustrating the detection of the operating status of a mine hoist, as provided in an embodiment of the present invention. Figure 1 As shown, XX Mining's XX Mine has developed a maintenance plan for its mine hoists, covering all mine hoists (engineering equipment) at the XX Mine, scheduled for maintenance in June and December each year. The entire maintenance process for a single mine hoist includes: shutdown preparation and equipment isolation, disassembly of core components, testing and diagnosis, and reassembly and commissioning. Although the XX Mine strictly adheres to the mine hoist maintenance plan, hoist malfunctions can occur during maintenance intervals. If a malfunction causes a safety accident, it will result in significant losses for the XX Mine. Modifying the mine hoist maintenance plan, for example, scheduling maintenance in January, April, August, and December each year to increase the frequency of maintenance, may not guarantee against malfunctions and would also significantly increase the XX Mine's operating costs.

[0022] To address the aforementioned problems, one embodiment of the present invention provides a defect detection method for engineering equipment, comprising: receiving multimodal data of a component to be inspected from a mine hoist. The multimodal data includes at least one of the following: images acquired of the wire rope and / or drive shaft of the mine hoist, and audio and video acquired of the drive shaft. The network status between the electronic device acquiring the images, audio, or video and a server is obtained. When the network is connected, the multimodal data is uploaded to the cloud, where a cloud-based multimodal large-scale model, combined with a dedicated detection model, performs collaborative analysis and processing, outputting a detection report. The detection report includes the defect type of the component to be inspected and suggested remediation measures. When the network is interrupted, a local lightweight model is triggered to perform offline inference on the multimodal data, outputting the defect type of the component to be inspected. In some embodiments, the dedicated detection model includes an image detection model and an audio analysis model. The image detection model is structurally optimized for working environments including low light and / or dust, and for the morphological characteristics of the wire rope and drive shaft. The audio analysis model is adaptively trained for background noise characteristics and low-frequency defect characteristics of the drive shaft. (Continue to refer to...) Figure 1 A camera and microphone 11 are configured for the mine hoist 10 to collect images, audio, and video of the wire rope and / or drive shaft 12. The images, audio, and video are then combined with a multimodal large model, a dedicated detection model, or a local lightweight model to generate defect types.

[0023] As can be seen, the multimodal large model integrates text, image, and audio information to provide more comprehensive analysis. The dual-mode operation mechanism, combining online cloud access with local offline operation, effectively addresses network instability issues in mines, ensuring the continuity of operation and maintenance support functions. The multimodal large model is optimized for mine equipment defects, resulting in high diagnostic accuracy; combined with the professional knowledge base embedded in the large model, the output results are more authoritative and standardized.

[0024] Please see Figure 2 , Figure 2 This is a schematic diagram of a defect detection system for the operation of a mine hoist, provided in an embodiment of the present invention. The defect detection system 20 may include a data acquisition and preprocessing module 201, a network judgment module 202, a cloud-based intelligent processing module 203, and a local offline operation module 204. The cloud-based intelligent processing module 203 can be deployed on a cloud server, while the data acquisition and preprocessing module 201, the network judgment module 202, and the local offline operation module 204 can be deployed in the control system configured on the mine hoist. The data acquisition and preprocessing module 201 and the cloud-based intelligent processing module 203 can be connected via network communication.

[0025] The data acquisition and preprocessing module 201 can communicate with cameras and microphones deployed on the mine hoist to achieve image, audio, and video acquisition. The module can also receive text input from maintenance personnel or staff. For example, it can receive descriptions of equipment malfunctions, operating parameters, etc., input by maintenance personnel via a microphone or the input box on the display screen. Image acquisition involves the module capturing images of key components of the mine hoist, such as the wire rope and drive shaft. Audio acquisition involves capturing the operating sound of the drive shaft. Video acquisition involves continuously recording the operating status of the wire rope and drive shaft. The module can also perform preprocessing on the images, audio, and video, such as low-light enhancement, dust filtering, ROI cropping, and audio noise reduction, to more clearly describe the operating status of the mine hoist. The video can be processed in frames per second (n frames / second) to separate the image and audio.

[0026] The network judgment module 202 can detect the network connection between the data acquisition and preprocessing module 201 and the cloud intelligent processing module 203 by sending network heartbeat packets. For example, the network judgment module 202 sends a network heartbeat packet to the cloud intelligent processing module 203 once every 5 seconds. If it does not receive a response from the cloud intelligent processing module 203 after more than 10 times, it determines that the network is interrupted. The data acquisition and preprocessing module 201 can send the acquired data (images, sound and video) to the local offline running module 204 for processing. Otherwise, it sends it to the cloud intelligent processing module 203 for processing.

[0027] The cloud-based intelligent processing module 203 may include a Coze processing unit, a multimodal large model, and various specialized detection models. The Coze processing unit is used for intent recognition, routing and distribution, and result integration. For example, it determines the input modality (text / image / audio / video), directs data to the corresponding model, and generates a comprehensive result based on the outputs of multiple models. The multimodal large model (e.g., DeepSeek-VL) is used to generate detection results / maintenance suggestions and format these results / suggestions. The input to the multimodal large model can be the outputs of various specialized detection models and the search results from the industry knowledge base. The output can be a formatted detection report (e.g., JSON / makedown with natural language). Specialized models may include a YOLOv8 image detection model for identifying defects (broken wires / cracks, etc.) in wire ropes and / or drive shafts. The input to a specialized model can be a pre-processed ROI image, and the output can be the detection results of the wire rope or drive shaft, including defect type, confidence level, condition coordinates, etc. The specialized model can also include the Qwen2Audio audio analysis model, used for defect identification (cracks, etc.) of the drive shaft. The input can be a denoised audio clip (a separate audio clip or audio separated from a video), and the output can be the defect type, and may also include: confidence level, condition coordinates, etc. The cloud-based intelligent processing module 203 can be deployed on large servers such as data centers / cloud servers.

[0028] The local offline execution module 204 can be developed using the Qt framework. It utilizes engines such as ONNX Runtime to load pruned and quantized YOLOv8 and Qwen2Audio models (local lightweight models) and package them into a standalone application. For example, the original model (YOLOv8n / Qwen2Audio-B) undergoes structured pruning (removing 30% of redundant channels) and INT8 quantization (reducing model size by 70%), followed by ONNX format conversion. After processing images, audio, and video by the local offline execution module 204, the generated detection results can describe only the presence of defects in the wire rope and / or drive shaft, such as broken wires in the wire rope or cracks in the drive shaft. The local offline execution module 204 can be deployed in the control system of a mine hoist, such as the industrial control computer of the mine hoist.

[0029] The aforementioned electronic devices such as cameras and microphones can also be configured on mobile terminals. In this case, the aforementioned data acquisition and preprocessing module 201, network judgment module 202, and local offline operation module 204 can be deployed on the mobile terminal. If the mobile terminal is connected to the server that deploys the cloud intelligent processing module 203, the preprocessed images, sounds, videos, or text can be sent to the server.

[0030] In some embodiments, Figure 2 The application scenarios of the detection system 20 shown can include the following two scenarios: Scenario 1: [Maintenance personnel / staff] capture images of the wire rope using a camera and upload them to the data acquisition and preprocessing module 201. The data acquisition and preprocessing module 201 performs preprocessing on the wire rope image, such as low-light enhancement, dust filtering, and ROI cropping. The network judgment module 202 detects that the network is normal and sends the wire rope image to the cloud intelligent processing module 203 on the server. The cloud intelligent processing module 203 receives the wire rope image, calls a dedicated detection model (e.g., YOLOv8), combines it with a multimodal large model (e.g., DeepSeek-VL) and a knowledge base (e.g., the GB20181-2006 standard library), and generates detection results (e.g., a detection report: "A broken wire was detected at point X, located at XX. According to Article XX of GB20181-2006, replacement is recommended, and speed limit operation is required during this period"). The detection results can be displayed on a client or sent to the maintenance personnel's email address.

[0031] Scenario 2: [Maintenance personnel / staff] Take images of the steel wire rope through a camera and upload them to the data acquisition and preprocessing module 201. The network judgment module 202 detects a network anomaly, and the local offline running module 204 loads the quantized YOLOv8 and Qwen2Audio models to process the steel wire rope images and generate detection results (e.g., detection report: "Broken wire detected at X, it is recommended to connect to the network for verification").

[0032] Please see Figure 3 , Figure 3 This is a flowchart illustrating a defect detection method for the operation status of a mine hoist, according to an embodiment of the present invention. The defect detection method includes the following steps.

[0033] Step S301: Receive multimodal data of the components to be tested from the mine hoist.

[0034] In some embodiments, the multimodal data herein includes at least one of the following: images acquired of the wire rope and / or drive shaft of a mine hoist, and audio and video acquired of the drive shaft.

[0035] Step S302: Obtain network status.

[0036] In some embodiments, the network status here can be the network connection status between the electronic device that acquires images, audio, or video and the server.

[0037] Step S303: When the network is connected, the multimodal data is uploaded to the cloud. The cloud-based multimodal large model and the dedicated detection model perform collaborative analysis and processing to output a detection report. The detection report includes the defect type of the component to be detected and the handling suggestions.

[0038] Step S304: When the network is interrupted, the local lightweight model is triggered to perform offline inference on the multimodal data and output the defect type of the component to be detected.

[0039] In some embodiments, the dedicated detection model includes an image detection model and an audio analysis model. The image detection model is structurally optimized for working environments including low light and / or dust, as well as the morphological characteristics of the wire rope and drive shaft. The audio analysis model is adaptively trained for background noise characteristics and low-frequency defect characteristics of the drive shaft.

[0040] The structural optimization of the image detection model here includes: adding an attention mechanism to the backbone network of the image detection model, which includes a wire rope feature attention branch and a drive shaft feature attention branch. The wire rope feature attention branch uses strip-type convolutional kernels to enhance the linear texture in the horizontal / vertical directions and highlight the strand structure of the wire rope. The drive shaft feature attention branch uses surface correction and circular convolutional kernels to enhance texture continuity. Anchor frames of various sizes are configured to adapt to various defects of the wire rope and / or drive shaft. The model is trained using a dynamic focusing loss function, which dynamically adjusts the weights according to the size of the anchor frames, assigning higher weights to smaller anchor frames.

[0041] The attention mechanism suppresses dust through global illumination distribution, eliminates water mist through outlier detection and median filtering, and recovers low light through multi-scale feature fusion.

[0042] The dimensions of the anchor frames here are determined by the aspect ratio, taking into account the different defects of the drive shaft and wire rope.

[0043] The adaptive training of the audio analysis model includes: configuring a driveshaft defect feature extraction layer, which includes a low-frequency enhancement filter to filter low-frequency noise, an envelope demodulator to extract low-frequency information from the high-frequency resonances in the audio, and an impulse pulse detector to identify the semantics of the driveshaft audio; and employing noise adversarial training, which mixes recorded background noise with real driveshaft audio to generate adversarial examples, including fan noise, water flow noise, mechanical vibration noise, etc.

[0044] The output of the audio analysis model includes the corresponding defect types of the drive shaft, including at least one of the following: outer ring failure, inner ring failure, rolling element failure, and cage failure.

[0045] The multimodal large model here is combined with a dedicated detection model for collaborative analysis and processing, including: The system calls upon a dedicated detection model to obtain preliminary defect identification results. These results include the defect type, location, and confidence level of the wire rope and / or rotating shaft, as output by the image detection model and / or audio analysis model. The system then retrieves the built-in safety standard database, which contains mandatory standard clauses, industry regulations, and fault handling knowledge. Based on the matching mandatory standard clauses, industry regulations, and fault handling knowledge, the system generates detection results.

[0046] The local lightweight model here is obtained by performing structured pruning and quantization compression on the multimodal large model and the dedicated detection model. The output of the local lightweight model includes the defect type.

[0047] In some embodiments, an industrial-grade acoustic signature encoder can be configured for the audio analysis model in the professional model (e.g., Qwen2Audio), including a driveshaft defect feature extraction layer, adapting to low-frequency vibration and abnormal noise, and training the audio analysis model against underground noise. This includes embedding adversarial samples of mine background noise (fans, water flow, mechanical vibration) to improve anti-interference capabilities. Finally, the output of the audio analysis model can refine the defect classification of the driveshaft, for example, subdividing it into outer race faults, inner race faults, rolling element faults, cage faults, etc.

[0048] Specifically, in a general downhole scenario, the frequency range of the sound is 85Hz-8kHz, the signal type can be harmonics, the energy distribution can be concentrated in the mid-to-high frequency range, and the time domain characteristics can be continuous and stable. In contrast, in a drive shaft defect scenario, the frequency range of the sound is 10Hz-500Hz (low-frequency vibration), the signal type can be impact pulses, the energy distribution can be weak in the low-frequency range, and the time domain characteristics can be periodic impacts.

[0049] In some embodiments, the encoder of a typical audio analysis model can be: input: the sound waveform corresponding to the audio (16kHz sampling), multiple Conv1D layers (e.g., kernel=n, stride=m) for extracting frame-level features (100Hz frame rate, 640 dimensions); a general audio semantic representation is obtained through a Transformer encoder. For driveshaft defects, the improved encoder can add a driveshaft-specific branch, a driveshaft defect feature extraction layer, which may include: a low-frequency enhancement filter, an envelope demodulator, an impulse detector, etc.; the driveshaft audio semantic representation is fused through a Transformer encoder.

[0050] As can be seen, the improved encoder can acquire the impact pulses generated by drive shaft defects (cracks, etc.) and extract the low-frequency modulation information from the high-frequency resonance through envelope demodulation. The improved encoder outputs a general audio semantic representation and a drive shaft audio semantic representation (including impact pulse characteristics).

[0051] The aforementioned underground noise countermeasures training process may include: mine background noise characteristic analysis and countermeasures training procedures. The mine background noise characteristic analysis includes noise characteristic analysis, for example: noise sources may include: fans (95-105dB, spectral characteristics 50Hz, time domain characteristics: continuous and stable), water flow / drainage (75-85dB, spectral characteristics broadband white noise, time domain characteristics: random fluctuations), mechanical vibration (80-95dB, spectral characteristics broadband white noise, time domain characteristics: random fluctuations), electromagnetic interference (60-70dB, spectral characteristics 50Hz, time domain characteristics: continuous), and personnel operations (75-85dB, spectral characteristics intermediate frequency, time domain characteristics: intermittent). The countermeasures sample generation strategy involves deploying recording equipment, such as: microphone arrays (covering different locations), continuous recording for N hours, and synchronously recording the operating status (speed, load) of the mine hoist in the XX mine. Noise can be categorized based on its source into stationary noise (fans, water flow), periodic noise (mechanical vibration), and impulse noise (personnel operation, electromagnetic interference), etc. Training data preparation: Obtain a driveshaft dataset. This can be a dataset collected in a laboratory with different speeds, loads, and defect types (real driveshafts) mixed with the aforementioned noise sources. Alternatively, training data can be obtained by randomly selecting noise sources corresponding to different noise scenarios and mixing them with driveshaft data. For example, for fan interference scenarios (noise characteristics: 95-105dB, 50Hz, easily masking driveshaft sound), a 50Hz filter can be added to the filter bank; for the mixed noise scenarios of water flow and machinery at the bottom of a mine (noise characteristics: broadband, multiple harmonics, non-stationary), time-frequency masking (spectral gating) can be used as data augmentation.

[0052] For wind turbine interference scenarios, the input is an audio signal from a drive shaft containing wind turbine noise. The improved encoder can separate the 50Hz wind turbine interference signal, extract impact features, and achieve feature representation of most interference. Adversarial training ensures the audio analysis model can identify wind turbine noise. Output: the correct defect type (probability).

[0053] Defect types are defined, including: outer ring failure, which is caused by the rolling element impacting the outer ring when it passes through a defect, improper installation, overload, or poor lubrication; inner ring failure, which is caused by the rolling element impacting the inner ring when it passes through a defect, insufficient shaft fit interference, or thermal deformation; rolling element failure, which is caused by surface defects of the ball / roller rolling element colliding with the inner and outer rings, fatigue spalling, or material defects; and cage failure, which is caused by the rolling element colliding with the cage or the cage breaking, lubrication failure, excessive speed, or wear.

[0054] For example, the frequency characteristics of each defect type are as follows: outer ring faults have a sound frequency of 89.3 Hz, inner ring faults 100.7 Hz, rolling element faults 46.8 Hz, and cage faults 4.7 Hz. This can also be combined with the time-domain waveform characteristics of each defect type: outer ring fault waveform: equally spaced sharp pulses; inner ring fault waveform: pulse amplitude fluctuates periodically; rolling element fault waveform: alternating double pulse sequences; cage fault waveform: low-frequency random impacts + continuous friction noise.

[0055] To ensure that the audio analysis model can output defect types, the output layer of the audio analysis model needs to be improved. After the improved encoder separates the interference signal, the output layer needs to be configured with a classifier, especially the specific classification corresponding to the defect type. For example, the classifier can include Linear(m, n) + LayerNorm + GELU->Linear(n, 4) to represent [outer ring, inner ring, rolling element, cage].

[0056] In some embodiments, for the image detection model in the professional model, a feature attention module for wire rope and drive shaft can be configured to suppress interference from downhole dust, water mist, and low light; optimize the defect anchor frame, reconfigure the anchor frame to accurately match the defect areas of wire rope (broken wire) and drive shaft (crack, wear); and improve the loss function: use the WIoU loss function (dynamic focusing loss function) to improve the detection accuracy of small target defects.

[0057] Among them, visual inspection of mine hoists faces unique environmental interferences. Dust forms a hazy fog, which reduces image contrast, blurs defect edges, and confuses the background with the target. Water mist forms local fogged areas, and light scattering produces false textures, interfering with feature extraction. Insufficient low light causes the loss of details in dark areas and small target defects to be submerged in noise.

[0058] The wire rope and drive shaft feature attention module is embedded in the YOLOv8 backbone network, forming a parallel processing architecture with two branches (wire rope branch and drive shaft branch). The YOLOv8 backbone network uses large-kernel convolution to capture the global illumination distribution in the image, estimate the scene illumination map, suppress dust, and combine outlier detection with median filtering to eliminate water mist. Multi-scale feature fusion restores low-light conditions. For the wire rope branch, strip-type convolution kernels are used to specifically enhance the horizontal / vertical linear textures in the image, highlighting the strand structure features of the wire rope. For the drive shaft branch, geometric prior modeling is performed. The drive shaft is cylindrical, appearing as a rectangle or trapezoid in the image, with a machined surface texture. Defect types include axial cracks and circumferential wear. The processing flow may include: estimating the cylindrical surface parameters of the drive shaft, unfolding the surface region into an approximate plane (surface correction); using a circular convolution kernel to enhance texture continuity and highlight the fracture characteristics of axial cracks, focusing on high-probability areas of interest based on the fact that cracks and other defects often appear in stress concentration areas (shoulder, keyway). Specifically, the input features (including dust, water mist, and low-light interference) are processed through a wire rope and drive shaft feature attention module to obtain enhanced features, which are further matched to defect regions using anchor frames.

[0059] Optimize the defect anchor frames by categorizing all anchor frames according to their aspect ratio, targeting different defects in drive shafts and wire ropes. For small targets such as broken wires and micro-cracks, specialized small target anchors can be added. The anchor frame size can be reduced to cover areas from 4×4 to 16×16 pixels. The optimized anchor frame parameters are as follows: Small targets: Anchor frame 1: 4 pixels wide × 4 pixels high, suitable for: fine broken wires, pitting; Anchor frame 2: 6 pixels wide × 6 pixels high, suitable for: single broken wires, micro-cracks; Anchor frame 3: 8 pixels wide × 16 pixels high, suitable for: short cracks. Medium targets: Anchor frame 4: 12 pixels wide × 36 pixels high, suitable for: local sections of wire rope; Anchor frame 5: 16 pixels wide × 16 pixels high, suitable for: drive shaft end face defects; Anchor frame 6: 20 pixels wide × 60 pixels high (aspect ratio 3:1), suitable for: multiple defect areas in the entire rope; Large target layers: Anchor frames 7-9: covering entire ropes, entire shafts, and multiple ropes side-by-side scenarios.

[0060] The WIoU loss function has been improved. YOLOv8 uses CIoU (CompleteIoU) loss by default, which has the following problems: scale sensitivity prioritizes optimization of large bounding boxes, while training of small bounding boxes is insufficient. The WIoU loss function can focus on small targets, enhance the weight of small targets, and calculate the weight based on the target area. The smaller the area, the higher the loss weight. For example: w = (target area / image area). For wire break detection, when a small bounding box is detected, an additional 20% loss weight is added. It also handles dense defects: when there are multiple defects in the same area, it retains multiple predictions of overlapping areas instead of selecting the best prediction.

[0061] As can be seen, the input includes images of the wire rope and drive shaft (dust + water mist + low light). Stage 1: Attention processing enhances strip textures, suppresses dust, corrects textures, and highlights cracks. Output: Interference-suppressed, target-clear feature map. Stage 2: Anchor frame matching configures various anchor frames, such as long anchor frames to cover the rope body and small anchor frames to capture single-wire breaks. Output: Anchor frames with high recall. Stage 3: WIoU fine-tuning and localization, gradient enhancement for small targets, and accurate identification of the broken wire location. Final output: Precise defect location. For example, in a scene with a single broken wire on the surface of the wire rope (2 pixels wide), without the proposed solution, the broken wire is submerged in the dust texture, and the minimum anchor frame is 8×8, making matching impossible. However, with the proposed solution, a 4×4 anchor frame matches the broken wire, increasing the WIoU weight for small targets and accurately identifying the defect.

[0062] Through improvements to the image detection model, the image detection model (YOLOv8) has been transformed from a general-purpose object detector into a specialized industrial inspection tool capable of accurately locating defects in wire ropes and drive shafts at a reduced cost.

[0063] In some embodiments, for multimodal large models (such as DeepSeek-VL), the multimodal large model can have a built-in safety standard database for mine hoists, realizing a relational database corresponding to a series of steps from defects to standards, contingency plans, and maintenance. After a defect is diagnosed, it can automatically match the corresponding standard clauses and output the test results (such as compliance judgment plus handling requirements).

[0064] Specifically, the dedicated safety standard database for mine hoists can include a national mandatory standards layer, an industry regulation layer, and a fault handling knowledge base. The national mandatory standards layer (legal basis) includes GB 20181-2006 "Safety Requirements for Mine Hoists and Mine Hoisting Windlasses" and AQ1035-2007 "Safety Inspection Specifications for Single-Rope Winding Mine Hoists for Coal Mines". The industry regulation layer (operation guidelines) includes: daily maintenance standards (daily / weekly / monthly / annual inspection item list), maintenance technical standards (major / medium / minor repair content, acceptance standards), and scrapping judgment standards (mandatory scrapping clauses and recommended scrapping clauses). The third layer, the fault handling knowledge base (decision support), includes: a fault mode library (defect types, characteristics, causes), a standard mapping library (fault → corresponding standard clauses), and a handling plan library (standard → mandatory handling requirements and maintenance steps library (plan → detailed operation instructions)).

[0065] For example, for images including broken wires in steel wire ropes, the detection results (detection report) given by the multimodal large model can be "broken wire ratio ≥5%, mandatory replacement is required". At the same time, it can also include disposal suggestions such as "immediately shut down the mine hoist, complete the replacement within 24 hours, acceptance standard: tension difference <5%, trial run for 2 hours", etc.

[0066] The mine hoist's dedicated safety standard database can be integrated with the large model in a vectorized way. For example, "If the proportion of broken wires in the wire rope is ≥5%, it must be replaced", the vector is (wire rope, broken wire, replacement, 5%).

[0067] like Figure 4 As described above, the present invention provides a method for operation Figure 3 The electronic device 40 of the detection system shown may include a memory 401, a processor 402 and a bus, and may also include computer programs stored in the memory 401 and executable on the processor 402, such as the various functional modules of the detection system.

[0068] The memory 401 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 401 can be an internal storage unit of the electronic device 40, such as the portable hard drive of the electronic device 40. In other embodiments, the memory 401 can be an external storage device of the electronic device 40, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 400. Furthermore, the memory 401 can include both internal and external storage units of the electronic device 40. The memory 401 can be used not only to store application software and various types of data installed on the electronic device 40, such as code for data management methods, but also to temporarily store data that has been output or will be output.

[0069] In some embodiments, the processor 402 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 402 is the control unit of the electronic device 40, connecting various components of the electronic device 40 through various interfaces and lines. It executes programs or modules (such as data management modeling programs) stored in the memory 401, and calls data stored in the memory 401 to perform various functions of the electronic device 40 and process data.

[0070] The processor 402 executes the operating system and various installed applications of the electronic device 40. The processor 402 executes the applications to implement the steps in the data management method described above.

[0071] For example, the computer program may be divided into one or more modules, which are stored in the memory 401 and executed by the processor 402 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device 40.

[0072] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The software functional module stored in the storage medium includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute some functions of the data management methods of the various embodiments of this application.

[0073] In one embodiment, a storage medium is provided on which a computer program is stored, and when the computer program is executed by a processor, it can also perform the steps described above when the processor executes the computer program.

[0074] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for defect detection in engineering equipment, characterized in that, include: The system receives multimodal data of the component to be tested from the engineering equipment. The multimodal data includes at least one of images, audio, and video collected from the component to be tested. The video is then processed by frame segmentation to separate the images and audio. Get the network status with the server; When the network status is network connected, the multimodal data is uploaded to the server. The multimodal data is then collaboratively analyzed and processed using a multimodal large model configured on the server combined with a dedicated detection model, and a detection report is output. The detection report includes: the defect type of the component to be detected and handling suggestions. When the network status is network interruption, the multimodal data is subjected to offline inference using a lightweight model to output the defect type of the component to be detected.

2. The method according to claim 1, characterized in that, The dedicated detection model includes an image detection model and an audio analysis model. The image detection model is structurally optimized for the working environment of the engineering equipment and the defect area of ​​the component to be detected. The audio analysis model is adaptively trained for background noise features and sound defect features of the component to be detected.

3. The method according to claim 2, characterized in that, The image detection model undergoes structural optimization, including: An attention mechanism is added to the backbone network of the image detection model. The attention mechanism includes a feature attention branch corresponding to the part to be detected. The feature attention branch uses a strip convolution kernel to enhance the linear texture of the part to be detected in the horizontal / vertical direction, highlighting the structure of the part to be detected. It also uses surface correction and a ring convolution kernel to enhance the texture continuity of the part to be detected. Configure anchor frames of various sizes to adapt to various defects of the component to be inspected; The model is trained using a dynamic focusing loss function, which dynamically adjusts the weights according to the size of the anchor frame, assigning higher weights to the smaller anchor frames.

4. The method according to claim 2, characterized in that, The audio analysis model undergoes adaptive training, including: The audio analysis model is configured with a defect feature extraction layer, which includes a low-frequency enhancement filter for filtering low-frequency noise, an envelope demodulator for extracting low-frequency information from high-frequency resonances in the audio, and an impulse pulse detector for identifying the audio semantics of the component to be detected. The audio analysis model is trained using adversarial examples, wherein the adversarial examples are generated by mixing recorded background noise with real audio of the component to be detected, and the background noise includes at least one of fan noise, water flow noise, and mechanical vibration noise.

5. The method according to claim 2, characterized in that, The process involves uploading the multimodal data to the server, and then using a multimodal large model configured on the server combined with a dedicated detection model to perform collaborative analysis and processing of the multimodal data, outputting a detection report, including: The dedicated detection model is invoked to obtain preliminary identification results, which include the defect type, location, and confidence level output by the image detection model and / or the audio analysis model after processing the image and / or the audio. Based on the defect type, the location, and the confidence level, the multimodal large model is invoked to retrieve the configured safety standard database, and the detection report is generated according to the mandatory standard clauses, industry procedures, and fault handling knowledge configured in the safety standard database.

6. The method according to claim 1, characterized in that, The component to be tested includes at least one of the steel wire rope and the drive shaft configured in the engineering equipment.

7. The method according to claim 6, characterized in that, The defect types corresponding to the drive shaft include at least one of the following: outer ring failure, inner ring failure, rolling element failure, and cage failure; the defect types corresponding to the wire rope include broken wires.

8. The method according to claim 1, characterized in that, The lightweight model is obtained by performing structured pruning and quantization compression on the multimodal large model and the dedicated detection model.

9. A defect detection system for engineering equipment, characterized in that, include: The system includes a data acquisition and preprocessing module, a network judgment module, a cloud-based intelligent processing module, and a local offline operation module. The data acquisition and preprocessing module and the network judgment module are configured on the terminal device, and the cloud-based intelligent processing module is configured on the server. The data acquisition and preprocessing module is used to receive multimodal data of the component to be tested from the engineering equipment. The multimodal data includes at least one of images, audio, and video acquired from the component to be tested. The network determination module is used to obtain the network status between the terminal device and the server; The intelligent processing module is used to receive the multimodal data from the terminal device when the network status is network connected, and to perform collaborative analysis and processing using a multimodal large model combined with a dedicated detection model to output a detection report. The detection report includes: the defect type of the component to be detected and suggested remedial measures. The local offline operation module is used to perform offline inference on the multimodal data through a lightweight model when the network status is network interruption, and output the defect type of the component to be detected.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the defect detection method for engineering equipment as described in any one of claims 1 to 8.

Images