A limited space quality detection method based on multi-modal edge cloud transmission

Abstract

The application discloses a kind of limited space quality detection methods based on multi-modal edge cloud transmission, belong to industrial manufacturing and quality detection technical field, innovatively fusion head-mounted camera, flexible hole detection equipment, step gap detection equipment, build millimeter three-dimensional space perception network, cover personnel accessible and inaccessible area, solve the problem of limited vision of traditional manual detection.Detection equipment as data transfer station, real-time processing multi-modal data (video stream, measurement data, voice), realize high-resolution video local decoding and wrist terminal real-time display by FFmpeg technology.CBAM attention mechanism is introduced in the neck part of YOLOv8, and the scratch feature representation is enhanced by channel and spatial attention weighting;SIoU loss function is used instead of traditional IoU, considering the overlapping area, center distance and aspect ratio of the predicted box and the real box, which significantly improves the detection rate of micro-defects.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of industrial manufacturing and quality detection, and particularly relates to a quality detection method for a restricted space based on multi-modal edge cloud transmission. BACKGROUND

[0002] In the modern high-end manufacturing field, there are a large number of narrow spaces with limited physical accessibility in the high-density device cluster scene, and the in-situ quality detection of the internal precision components faces technical bottlenecks. The traditional detection mode relies on manual visual inspection and basic measurement tools, which not only is limited by the ergonomics constraints to result in low detection efficiency, but also has a very high misjudgment rate in 0.1mm-level micro-defect identification. Human vision and judgment ability is limited, especially in long-time and high-intensity working conditions, which is more likely to cause fatigue and misjudgment.

[0003] Real-time information interaction in the restricted space detection scene also has significant technical bottlenecks. Due to the physical constraints of the space environment, the two-way communication between the detection personnel and the external control center faces signal attenuation, multipath effect and other interferences, resulting in transmission delay of key detection data, and the real-time return demand of high-resolution detection pictures is particularly prominent. Under the existing technical conditions, it is difficult to meet the dynamic image transmission requirement of 30FPS, resulting in the phenomenon of video stream freezing and mosaic, which seriously affects the remote decision-making efficiency. Therefore, constructing a low-delay and high-reliability communication and transmission system has become a core technical requirement to ensure the safety of the restricted space operation.

[0004] With the progress of science and technology and the development of industrial automation, finding a more efficient, accurate and safe detection method has become a problem to be solved in modern industrial production. This not only requires that the new detection method can adapt to the characteristics of the restricted space, but also needs to have the ability of real-time information transmission and remote collaboration to ensure the smooth progress of the detection work and the accuracy of the detection results. Therefore, the present application proposes an intelligent detection system based on multi-modal edge cloud transmission suitable for restricted space. SUMMARY

[0005] The purpose of the present application is to solve the problems of the prior art and provide a quality detection method for a restricted space based on multi-modal edge cloud transmission, which can solve the problems of low quality detection efficiency, poor information transmission, and inability to communicate and collaborate with external personnel in the restricted space.

[0006] The present application is implemented by the following technical solutions:

[0007] A quality detection method for a restricted space based on multi-modal edge cloud transmission, comprising the following steps:

[0008] Step S1: An internal operator carries an edge detection device into a workpiece to be detected, and starts each data acquisition device in the restricted space.

[0009] Step S2: The wrist display terminal worn by the internal operator displays the video pictures of the head-mounted camera and the hole probe device in real time;

[0010] Step S3: The video data of the head-mounted camera and the hole probe device inside the restricted space, the step difference gap detection data, and the real-time voice interaction data are transmitted to the external data processing center through TCP / IP;

[0011] Step S4: The external data processing center performs scratch detection on the received video data, and selectively enters the scratch detection key frame data, video stream data, and step difference gap detection data into the database.

[0012] Preferably, in the step S1, the workpiece to be detected is determined, and the internal operator carries each data acquisition device and edge detection device into the narrow space of the workpiece to be detected; each data acquisition device includes a head-mounted camera, a hole probe device, a step difference gap detection device, and a head-mounted earphone; the edge detection device is connected to each data acquisition device as a data transfer station, and the data acquisition devices are used to scan the inside of the workpiece and transmit to the edge detection device; each set of portable device is equipped with a lighting device, and the operator turns on or off the lighting device according to the need to ensure the usability of the collected data.

[0013] Preferably, in the step S2, the following steps are included:

[0014] Step S21: The edge detection device is equipped with a wrist-type small liquid crystal display screen, and the internal operator views the video pictures of the head-mounted camera and the hole probe device in real time;

[0015] Step S22: The video data stream of the head-mounted high-resolution camera is obtained by using FFmpeg technology;

[0016] Step S23: The input video stream is opened by avformat_open_input, and the video stream parameters contained in the container format are parsed;

[0017] Step S24: The stream metadata is obtained by calling avformat_find_stream_info, and the encoding context of each stream is determined;

[0018] Step S25: The original data packet is extracted from the input video stream by cyclically calling av_read_frame, and is separated and stored to the corresponding queue according to the stream type;

[0019] Step S26: The separated audio Packet sequence and video Packet sequence carry timestamp and stream index information, create a codec context for each stream, and initialize the encoding parameters by avcodec_parameters_to_context.

[0020] Step S27: Call avcodec_find_decoder to obtain H.264 encoder, and open the decoder through avcodec_open2;

[0021] Step S28: Loop to send Packet into the decoder: input compressed data through avcodec_send_packet, and call avcodec_receive_frame to output raw data frame;

[0022] Step S29: Push the data frame decoded by FFmpeg to the wrist display terminal, and the internal operator views the real-time video picture of the head-mounted high-resolution camera;

[0023] Step S210: Push the data frame collected by the hole exploration device to the wrist display terminal, and the internal operator can view the real-time video picture inside the restricted space;

[0024] Step S211: The measurement result of the step difference gap detection device is temporarily stored in the edge detection device memory in the form of a string.

[0025] Preferably, the hole exploration device is used to view the physically inaccessible area in the restricted space, which includes a flexible probe that can be extended into the restricted space and an image acquisition device integrated at the front end of the probe; the outer diameter of the probe of the hole exploration device is not greater than 10 mm, and a flexible guide pipe and an illumination assembly are integrated inside; the guide pipe can realize positive and negative 90° bending turning to adapt to complex space structures; the illumination assembly adopts a high-brightness LED light source, and a diffusion lens is configured to realize uniform illumination; the image acquisition device integrated at the front end of the probe carries a miniature camera with a resolution of 1920x1080, and supports 30 frames per second dynamic video acquisition; the step difference gap detection device is a handheld high-precision non-contact optical measurement component, which is suitable for high-precision detection of step difference and gap of various size range gaps, and the measurement error is less than or equal to ±25μm, and the measurement range is 0.1mm-25mm; the headset is used for voice interaction between the internal operator and the external data center, and the input and output of audio are realized through Windows Multimedia API.

[0026] Preferably, in step S3, the following steps are included:

[0027] Step S31: The edge detection device and the external data processing center are both equipped with Ethernet gigabit network cards, support TCP / IP protocol stack, and realize network communication function through C++;

[0028] Step S32: The external data processing center integrates WinSock network programming interface, including user interface module, data processing module and network communication module;

[0029] Step S33: Edge detection device and external data processing center through the gigabit transmission rate network line data interaction, including data receiving and sending;

[0030] Step S34: According to the different types of internal data acquisition device data collection, determine the type of data receiving and sending, including the transmission of headset video stream, hole detection device video stream transmission, differential gap detection device string transmission and headset audio transmission;

[0031] Step S35: external data processing center based on WinSock create TCP socket: socket(), bind to the specified IP address and port: bind(), listen to the connection request: listen();

[0032] Step S36: external data processing center through accept() receive edge detection device connection, establish Socket handle mapping table, support multiple edge device concurrent communication;

[0033] Step S37: external data processing center cycle call recv() read buffer data, according to the protocol analysis data packet, check through the extraction of the encoded data;

[0034] Step S38: external data processing center will be sent string according to the protocol encapsulation, including adding the starting symbol, length, check bit, end symbol, through send() to the target edge device;

[0035] Step S39: edge detection device create TCP socket, through connect() connection external data processing center IP address and port;

[0036] Step S310: edge detection device according to the protocol encapsulation of the data to be sent, through send() upload to the external data processing center; when receiving external data processing center instruction, analysis data packet and execute corresponding operation;

[0037] Step S311: set the heartbeat packet mechanism, regularly send PING string, detect network connection state, abnormal automatic reconnection;

[0038] Step S312: internal and external network intercommunication makes the external data processing center real-time receiving edge detection device sent by the headset camera video stream, hole detection device video stream, differential gap detection device string and headset audio.

[0039] Preferably, the step S4, comprising the following steps:

[0040] Step S41: The external data processing center makes different operations on the received headset video stream, hole detection device video stream, step difference gap detection device string and headset audio classification;

[0041] Step S42: The headset video stream and the hole detection device video stream are subjected to scratch detection and identification based on the YOLOv8 network, the CBAM attention mechanism is introduced in the Neck part of YOLOv8, and the SIoU loss function is used for parameter updating;

[0042] Step S43: The image to be detected is scaled to 640x640 and sent to the trained neural network model for detection;

[0043] Step S44: The image first comes to the Backbone part in the model, passes through the P1 layer of Backbone, a standard 3x3 Conv module, and the convolution kernel size is 32, so as to obtain a feature map with a channel number of 32;

[0044] Step S45: The feature map passes through the standard 3x3 Conv module of P2 layer and the 2nd layer C2f module of the network, the channel number of the feature map is 64, and C2f adopts more jump connections and feature splicing to make the features more rich;

[0045] Step S46: The feature map passes through the standard 3x3 Conv module of P3 layer and the 4th layer C2f module of the network, the channel number is 128, and C2f is used for further fusion of the feature map;

[0046] Step S47: The feature map passes through the standard 3x3 Conv module of P4 layer and the 6th layer C2f module of the network, the channel number is 256, and C2f is used for further fusion of the feature map;

[0047] Step S48: The feature map passes through the standard 3x3 Conv module of P5 layer and the 8th layer C2f module of the network, the channel number is 512, and C2f is used for further fusion of the feature map;

[0048] Step S49: The feature map passes through the SPPF module, the channel number is 512, which is used for spatial pyramid pooling to realize feature extraction of different scales and capture detailed information of the target at different scales;

[0049] Step S410: The feature map comes to the Neck module in the model, first passes through the upsampling operation to expand the feature map by one time, is spliced with the feature map passing through the 6th layer, and then the spliced feature passes through the 12th layer C2f module of the network, and then passes through the CBAM attention mechanism module to improve the feature representation ability and strengthen the spatial information, and outputs a feature map with a dimension size of 256;

[0050] Step S411: The feature map of the previous layer is expanded by one time through upsampling, spliced with the feature map passing through the 4th layer, and then the spliced feature passes through the 16th layer C2f module of the network, followed by the CBAM attention mechanism module, and the output dimension size is 128;

[0051] Step S412: The feature map of the previous layer passes through a standard 3x3 Conv module, is spliced with the feature map passing through the 13th layer, and then the spliced feature passes through the 20th layer C2f module of the network, followed by the CBAM attention mechanism module, and the output dimension size is 256;

[0052] Step S413: The feature map of the previous layer passes through a standard 3x3 Conv module, is spliced with the feature map passing through the 9th layer, and then the spliced feature passes through the 24th layer C2f module of the network, followed by the CBAM attention mechanism module, and the output dimension size is 512;

[0053] Step S414: The feature map passing through the 17th layer is input into the Head module of the network for scratch target recognition and position positioning;

[0054] Step S415: The feature map passing through the 21st layer is input into the Head module of the network for scratch target recognition and position positioning;

[0055] Step S416: The feature map passing through the 25th layer is input into the Head module of the network for scratch target recognition and position positioning;

[0056] Step S417: The scratch key frame detected by the head-mounted camera video stream and the hole exploration device video stream is automatically input into the external data processing center database, and the historical data can be viewed and traced in real time in the visualization interface;

[0057] Step S418: For the step difference gap detection device string data, it is segmented into detection categories and true values by the external data processing center, and the segmented data is automatically input into the external data processing center database according to the detection time;

[0058] Step S419: The external data processing center is equipped with a professional database visualization interactive interface, and the operation personnel can perform data adding, deleting, modifying and querying operations on the database based on the interface;

[0059] Step S420: For the historical data in the database, representative data is automatically generated to generate a personalized report, and the report function module has a perfect comment embedding mechanism;

[0060] Step S421: The external data processing center and the internal edge device are both equipped with a headset device, thereby building an internal and external real-time voice communication link to ensure that instant and efficient voice information interaction can be achieved between the internal and external.

[0061] Preferably, in the step S42, the calculation formula of the SIoU loss function is as follows:

[0062] ;

[0063] wherein, represents the intersection over union between the predicted box and the real box, Δ represents the angle loss, and Ω represents the distance loss and the aspect ratio loss;

[0064] The calculation formula of the angle loss Δ is as follows:

[0065] ;

[0066] wherein, represents the Euclidean distance between the centers of the predicted box and the real box, represents the diagonal length of the minimum circumscribed rectangle of the predicted box and the real box;

[0067] The calculation formula of the distance loss and the aspect ratio loss Ω is as follows:

[0068] ;

[0069] wherein, and respectively represent the distance of the predicted box and the real box in the width and height directions, and respectively represent the width and height of the real box;

[0070] In the training process, the parameters of the model are updated by minimizing the SIoU loss function.

[0071] Preferably, in the step S42, the overall architecture of YOLOv8 includes three main parts: Backbone, Neck and Head; Backbone is responsible for extracting features from the input image, Neck fuses and enhances the features, and Head is used to predict the category and position of the target.

[0072] Preferably, the CBAM attention mechanism module includes a channel attention module and a spatial attention module.

[0073] Preferably, the channel attention module assigns different weights to each channel by operating on the channel dimension of the feature map, so as to highlight important channel features; first, the input feature map respectively, to obtain two global feature vectors and The calculation formula is as follows:

[0074] ;

[0075] ;

[0076] wherein, represents the value of all channels of the feature map F in the i-th row and the j-th column; is the height of the feature map in the spatial dimension; is the width of the feature map in the spatial dimension;

[0077] The two global feature vectors are respectively input into a shared multi-layer perception, and after ReLU activation function and Sigmoid activation function, a channel attention map M c ∈R C :

[0078] ;

[0079] wherein, σ represents a Sigmoid activation function; v avg is a global feature vector obtained by performing global average pooling on the input feature map F; v max is a global feature vector obtained by performing global maximum pooling (GMP) on the input feature map F;

[0080] Finally, the channel attention map M c is multiplied with the input feature map F channel by channel to obtain a feature map enhanced by channel attention :

[0081] ;

[0082] wherein, M c is multiplied with F element by element.

[0083] Preferably, the spatial attention module then operates on the spatial dimension of the feature map, and assigns different weights to each spatial position to highlight important spatial regions; first, the feature map enhanced by channel attention is respectively subjected to average pooling and maximum pooling in the channel dimension to obtain two spatial feature maps and The calculation formula is as follows:

[0084] ;

[0085] ;

[0086] The two spatial feature maps are spliced in the channel dimension to obtain a new feature map , and a 7*7 convolution layer and a Sigmoid activation function are used to obtain a spatial attention map :

[0087] ;

[0088] Wherein, represents a splicing operation in the channel dimension;

[0089] Finally, the spatial attention map is multiplied element by element with the feature map enhanced by the channel attention to obtain the final feature map enhanced by the CBAM attention mechanism :

[0090] ;

[0091] Wherein, and are multiplied element by element.

[0092] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0093] 1. The limited space quality detection method based on multi-modal edge cloud transmission provided by the present application innovatively integrates a head-mounted camera, a flexible hole detection device and a step gap detection device to construct a millimeter-level three-dimensional space perception network, covers reachable and unreachable areas of personnel, and solves the problem of limited field of view in traditional manual detection. The edge detection device acts as a data transfer station, and real-time processes multi-modal data (video stream, measurement data, voice), realizes high-resolution video local decoding and real-time display on a wrist terminal through FFmpeg technology.

[0094] 2. The limited space quality detection method based on multi-modal edge cloud transmission provided by the present application adopts a 10-gigabit Ethernet TCP / IP protocol stack, supports real-time transmission of high-resolution 30FPS video stream, automatically detects network abnormalities and reconnects through a heartbeat packet mechanism, solves the problem of transmission delay caused by signal attenuation in a limited space, and guarantees that the video stream is not stuck. Differentiated transmission protocols are designed for video streams (head-mounted cameras / hole detection devices), measurement strings (step gap data) and voice data to realize efficient data packaging and analysis, support concurrent communication of multiple edge detection devices (Socket handle mapping table).

[0095] Thirdly, the application provides a limited space quality detection method based on multi-modal edge cloud transmission, a CBAM attention mechanism is introduced in the Neck part of YOLOv8, and the scratch feature representation is enhanced through channel and spatial attention weighting; a SIoU loss function is used to replace the traditional IoU, and the overlapping area, center point distance and width-height ratio of the predicted frame and the real frame are comprehensively considered, so that the micro defect detection rate is significantly improved. The detection data (key frame, measurement value) are automatically entered into the database according to time and category, visual traceability, multi-database switching and personalized report generation (including comment embedding mechanism) are supported, and the traceability and decision support of the detection data are realized. BRIEF DESCRIPTION OF DRAWINGS

[0096] Figure 1 It is a flowchart of the application;

[0097] Figure 2 It is a schematic diagram of the communication between the internal edge detection device and each data acquisition device in the application;

[0098] Figure 3 It is a principle diagram for realizing high-resolution image encoding and decoding based on Ffmpeg in the application;

[0099] Figure 4 It is a schematic diagram of differentiating transmission of various data based on TCP / IP protocol in the application;

[0100] Figure 5 It is an improved YOLOv8 network model diagram in the application;

[0101] Figure 6 It is a principle diagram of CBAM attention mechanism in the application;

[0102] Figure 7 It is a principle diagram of SIoU loss function in the application;

[0103] Figure 8 It is a database framework diagram in the application. DETAILED DESCRIPTION

[0104] The application will be further described in detail below in combination with embodiments, but the embodiments of the application are not limited thereto.

[0105] Embodiment 1

[0106] The embodiment provides a limited space quality detection method based on multi-modal edge cloud transmission, comprising the following steps:

[0107] Step S1: the internal operator carries the edge detection device into the workpiece to be measured, and starts each data acquisition device in the limited space;

[0108] Step S2: The wrist display terminal worn by the internal operator displays the video pictures of the head-mounted camera and the hole probe device in real time.

[0109] Step S3: The video data of the head-mounted camera and the hole probe device, the step difference gap detection data, and the real-time voice interaction data inside the restricted space are transmitted to the external data processing center through TCP / IP.

[0110] Step S4: The external data processing center performs scratch detection on the received video data and selectively enters the scratch detection key frame data, video stream data, and step difference gap detection data into the database.

[0111] Embodiment 2

[0112] The embodiment provides a restricted space quality detection method based on multi-modal edge cloud transmission, comprising the following steps:

[0113] Step S1: The internal operator carries the edge detection device into the workpiece to be detected and starts the data collection devices in the restricted space.

[0114] Step S2: The wrist display terminal worn by the internal operator displays the video pictures of the head-mounted camera and the hole probe device in real time.

[0115] Step S3: The video data of the head-mounted camera and the hole probe device, the step difference gap detection data, and the real-time voice interaction data inside the restricted space are transmitted to the external data processing center through TCP / IP.

[0116] Step S4: The external data processing center performs scratch detection on the received video data and selectively enters the scratch detection key frame data, video stream data, and step difference gap detection data into the database.

[0117] In step S1, the workpiece to be detected is determined, and the internal operator carries the data collection devices and the edge detection device into the narrow space of the workpiece to be detected. The data collection devices include a head-mounted camera, a hole probe device, a step difference gap detection device, and a head-mounted earphone. The edge detection device is connected to the data collection devices as a data transfer station. The data collection devices are used to scan the inside of the workpiece and transmit the data to the edge detection device. Each set of portable devices is equipped with a lighting device. The operator turns on or off the lighting device as needed to ensure the availability of the collected data.

[0118] In step S2, the following steps are included:

[0119] Step S21: The edge detection device is equipped with a wrist-mounted small liquid crystal display screen. The internal operator can view the video pictures of the head-mounted camera and the hole probe device in real time.

[0120] Step S22: Use FFmpeg technology to take the video data stream of the head-mounted high-resolution camera;

[0121] Step S23: Open the input video stream by avformat_open_input, and parse the video stream parameters contained in the container format;

[0122] Step S24: Call avformat_find_stream_info to obtain stream metadata, and determine the encoding context of each stream;

[0123] Step S25: Loop call av_read_frame to extract raw data packets from the input video stream, separate and store them to the corresponding queue according to the stream type;

[0124] Step S26: Separate the audio Packet sequence and the video Packet sequence, carry the timestamp and stream index information, create a codec context for each stream, and initialize the encoding parameters through avcodec_parameters_to_context;

[0125] Step S27: Call avcodec_find_decoder to obtain the H.264 encoder, and open the decoder through avcodec_open2;

[0126] Step S28: Loop to send Packet to the decoder: input compressed data through avcodec_send_packet, and call avcodec_receive_frame to output raw data frames;

[0127] Step S29: Push the data frames decoded by FFmpeg to the wrist display terminal, and the internal operator can view the real-time video picture of the head-mounted high-resolution camera;

[0128] Step S210: Push the data frames collected by the hole exploration device to the wrist display terminal, and the internal operator can view the real-time video picture inside the restricted space;

[0129] Step S211: The measurement results of the step difference gap detection device are temporarily stored in the edge detection device memory in the form of a string.

[0130] The hole exploration device is used for viewing the physically unreachable area of the personnel in the restricted space, and includes a flexible probe capable of extending into the restricted space and an image acquisition device integrated at the front end of the probe; the outer diameter of the probe of the hole exploration device is not greater than 10 mm, and a flexible guide pipe and an illumination assembly are integrated inside the probe; the guide pipe can realize positive and negative 90° bending turning and adapt to complex space structures; the illumination assembly adopts a high-brightness LED light source and is configured with a diffusion lens to realize uniform illumination; the image acquisition device integrated at the front end of the probe is loaded with a miniature camera with a resolution of 1920*1080 and supports 30 frames per second dynamic video acquisition; the step difference gap detection device is a handheld high-precision non-contact optical measurement component, which is suitable for high-precision detection of step differences and gaps of various size range gaps, the measurement error is less than or equal to ± 25 μm, and the measurement range is 0.1 mm-25 mm; the headset is used for voice interaction between the internal operating personnel and the external data center, and the input and output of audio are realized through Windows Multimedia API.

[0131] In the step S3, the following steps are included.

[0132] In step S31, the edge detection device and the external data processing center are both provided with an Ethernet gigabit network card, support TCP / IP protocol stack, and realize network communication function through C++.

[0133] In step S32, the external data processing center integrates WinSock network programming interface, includes user interface module, data processing module and network communication module.

[0134] In step S33, the edge detection device and the external data processing center interact data through gigabit transmission rate network cable, including data receiving and sending.

[0135] In step S34, according to different types of data collected by the internal data acquisition device, the type of data receiving and sending is determined, including transmission of headset video stream, transmission of hole exploration device video stream, transmission of step difference gap detection device string and transmission of headset audio.

[0136] In step S35, the external data processing center creates a TCP socket based on WinSock: socket(), binds a specified IP address and port: bind(), and listens to connection requests: listen().

[0137] In step S36, the external data processing center receives the edge detection device connection through accept(), establishes a socket handle mapping table, and supports multiple edge devices concurrent communication.

[0138] Step S37: The external data processing center reads the buffer data in a loop by calling recv(), parses the data packet according to the protocol, and extracts the encoded data after verification;

[0139] Step S38: The external data processing center encapsulates the to-be-sent string according to the protocol, including adding a start symbol, length, check bit, and end symbol, and sends it to the target edge device through send();

[0140] Step S39: The edge detection device creates a TCP socket and connects the external data processing center IP address and port through connect();

[0141] Step S310: The edge detection device encapsulates the to-be-sent data according to the protocol and uploads it to the external data processing center through send(); when receiving instructions from the external data processing center, it parses the data packet and performs corresponding operations;

[0142] Step S311: Set up a heartbeat packet mechanism to periodically send PING strings to detect network connection status and automatically reconnect in case of exceptions;

[0143] Step S312: The internal and external network intercommunication enables the external data processing center to receive the head-mounted camera video stream, hole exploration device video stream, step difference gap detection device string, and head-mounted earphone audio sent by the edge detection device in real time.

[0144] Among them, the step S4 includes the following steps:

[0145] Step S41: The external data processing center classifies the received head-mounted earphone video stream, hole exploration device video stream, step difference gap detection device string, and head-mounted earphone audio and performs different operations;

[0146] Step S42: For the head-mounted earphone video stream and hole exploration device video stream, perform scratch detection and recognition based on the YOLOv8 network, introduce a CBAM attention mechanism in the Neck part of YOLOv8, and use a SIoU loss function for parameter update;

[0147] Step S43: Scale the image size to be detected to 640x640 and send it to the trained neural network model for detection;

[0148] Step S44: The image first comes to the Backbone part in the model, passes through the P1 layer of Backbone, a standard 3x3 Conv module, and a convolution kernel size of 32, to obtain a feature map with a channel number of 32;

[0149] Step S45: The feature map passes through the standard 3x3 Conv module of the P2 layer and the second layer C2f module of the network, the number of feature map channels is 64, and the C2f adopts more jump connections and feature splicing to make the features more rich;

[0150] Step S46: The feature map passes through the standard 3x3 Conv module of the P3 layer and the fourth layer C2f module of the network, the number of channels is 128, and the C2f is used for further fusion of the feature map;

[0151] Step S47: The feature map passes through the standard 3x3 Conv module of the P4 layer and the sixth layer C2f module of the network, the number of channels is 256, and the C2f is used for further fusion of the feature map;

[0152] Step S48: The feature map passes through the standard 3x3 Conv module of the P5 layer and the eighth layer C2f module of the network, the number of channels is 512, and the C2f is used for further fusion of the feature map;

[0153] Step S49: The feature map passes through the SPPF module, the number of channels is 512, which is used for spatial pyramid pooling to realize feature extraction of different scales and capture detailed information of the target at different scales;

[0154] Step S410: The feature map comes to the Neck module in the model, first passes through the up-sampling operation to expand the feature map by one time, is spliced with the feature map passing through the sixth layer, and then the spliced feature passes through the twelfth layer C2f module of the network, and then passes through the CBAM attention mechanism module, which is used for improving the feature representation ability and strengthening the spatial information, and outputs the feature map with a dimension size of 256;

[0155] Step S411: The feature map of the previous layer is expanded by one time through up-sampling, is spliced with the feature map passing through the fourth layer, and then the spliced feature passes through the sixteenth layer C2f module of the network, and then passes through the CBAM attention mechanism module, and outputs the feature map with a dimension size of 128;

[0156] Step S412: The feature map of the previous layer passes through the standard 3x3 Conv module, is spliced with the feature map passing through the thirteenth layer, and then the spliced feature passes through the twentieth layer C2f module of the network, and then passes through the CBAM attention mechanism module, and outputs the feature map with a dimension size of 256;

[0157] Step S413: The feature map of the previous layer passes through the standard 3x3 Conv module, is spliced with the feature map passing through the ninth layer, and then the spliced feature passes through the twenty-fourth layer C2f module of the network, and then passes through the CBAM attention mechanism module, and outputs the feature map with a dimension size of 512;

[0158] Step S414: input the feature map passing through the 17th layer to the Head module of the network to perform scratch target recognition and position positioning;

[0159] Step S415: input the feature map passing through the 21st layer to the Head module of the network to perform scratch target recognition and position positioning;

[0160] Step S416: input the feature map passing through the 25th layer to the Head module of the network to perform scratch target recognition and position positioning;

[0161] Step S417: the scratch key frame detected by the head-mounted camera video stream and the hole probe device video stream is automatically entered into the external data processing center database, and the historical data can be viewed and traced in real time in the visualization interface;

[0162] Step S418: for the step difference gap detection device string data, it is segmented into detection categories and true values by the external data processing center, and the segmented data is automatically entered into the external data processing center database according to the detection time;

[0163] Step S419: the external data processing center is equipped with a professional database visualization interactive interface, and the operation personnel can perform data adding, deleting, modifying and querying operations on the database based on the interface;

[0164] Step S420: for the historical data in the database, representative data is automatically generated to generate a personalized report, and the report function module has a perfect comment embedding mechanism;

[0165] Step S421: the external data processing center and the internal edge device are both equipped with a headset device, so as to build an internal and external real-time voice communication link to ensure that the internal and external can realize instant and efficient voice information interaction.

[0166] In the step S42, the calculation formula of the SIoU loss function is as follows:

[0167] ;

[0168] Wherein, represents the intersection over union between the predicted box and the real box, Δ represents the angle loss, and Ω represents the distance loss and the aspect ratio loss;

[0169] The calculation formula of the angle loss Δ is as follows:

[0170] ;

[0171] Wherein, represents the Euclidean distance between the center points of the predicted box and the real box, Diagonal length of the minimum bounding rectangle of the prediction box and the real box;

[0172] The calculation formula of the distance loss and the aspect ratio loss Ω is as follows:

[0173] ;

[0174] wherein, and respectively represent the distance of the prediction box and the real box in the width and height directions, and respectively represent the width and height of the real box;

[0175] In the training process, the parameters of the model are updated by minimizing the SIoU loss function.

[0176] Wherein, in the step S42, the overall architecture of YOLOv8 includes three main parts of Backbone, Neck and Head; Backbone is responsible for extracting features from input images, Neck fuses and enhances features, and Head is used to predict the category and position of the target.

[0177] Wherein, the CBAM attention mechanism module includes a channel attention module and a spatial attention module.

[0178] Wherein, the channel attention module assigns different weights to each channel by operating on the channel dimension of the feature map, in order to highlight important channel features; first, the input feature map is respectively globally averaged and globally maximized to obtain two global feature vectors and , whose calculation formula is as follows:

[0179] ;

[0180] ;

[0181] wherein, represents the value of all channels of the feature map F in the i-th row and j-th column; is the height of the feature map in the spatial dimension; is the width of the feature map in the spatial dimension; used to describe the spatial size of the input feature map , wherein C is the number of channels.

[0182] The two global feature vectors are respectively input into a shared multi-layer perceptron, and after ReLU activation function and Sigmoid activation function, the channel attention map M c ∈R C :

[0183] ;

[0184] wherein, σ represents a Sigmoid activation function; v avg is a global feature vector obtained by performing global average pooling on the input feature map F; v max is a global feature vector obtained by performing global maximum pooling (GMP) on the input feature map F;

[0185] Finally, the channel attention map M c is multiplied with the input feature map F F channel by channel to obtain a feature map enhanced by channel attention :

[0186] ;

[0187] wherein, M c is multiplied with F F element by element.

[0188] wherein, the spatial attention module then operates on the spatial dimension of the feature map, and assigns different weights to each spatial position to highlight important spatial regions; first, the feature map enhanced by channel attention F is respectively subjected to average pooling and maximum pooling in the channel dimension to obtain two spatial feature maps and , and the calculation formulae are as follows:

[0189] ;

[0190] ;

[0191] The two spatial feature maps are spliced in the channel dimension to obtain a new feature map , and a spatial attention map M is obtained through a 7x7 convolution layer and a Sigmoid activation function:

[0192] ;

[0193] wherein, represents a splicing operation in the channel dimension;

[0194] Finally, the spatial attention map M is multiplied with the feature map enhanced by channel attention F element by element to obtain a final feature map enhanced by CBAM attention mechanism F :

[0195] ;

[0196] wherein, and element-wise multiplication.

[0197] Embodiment 3

[0198] As Figure 1 shown, the embodiment of the present application provides a limited space quality detection method based on multi-modal edge cloud transmission, comprising:

[0199] S1: The internal operator carries the edge detection equipment into the workpiece to be detected, and starts each data acquisition equipment in the limited space.

[0200] Wherein, step S1 comprises:

[0201] S11: Determine the workpiece to be detected, and carry each data acquisition equipment and edge equipment into the narrow space of the workpiece to be detected by the internal operator.

[0202] S12: The edge equipment is connected with each data acquisition equipment as a data transfer station, and the data acquisition equipment is used to scan the inside of the workpiece and transmit to the edge equipment. (As Figure 2 shown, the internal edge equipment and each data acquisition equipment communication schematic diagram)

[0203] S13: The internal data acquisition equipment includes a head-mounted camera, a hole detection device, a step gap detection device and a head-mounted earphone.

[0204] S14: In order to avoid the problem that the inside of the workpiece is difficult to work due to insufficient light, each set of portable equipment is equipped with a lighting device, and the operator can turn on or off the lighting device according to the need to ensure the availability of the collected data.

[0205] S2: The wrist display terminal worn by the internal operator displays the video pictures of the head-mounted camera and the hole detection device in real time.

[0206] Wherein, step S2 comprises:

[0207] S21: The edge equipment is equipped with a wrist small liquid crystal display screen, and the internal operator can view the video pictures of the head-mounted camera and the hole detection device in real time.

[0208] S22: Use FFmpeg technology to obtain the video data stream of the head-mounted high-resolution camera. (As Figure 3 shown, the Ffmpeg realizes the high-resolution image coding principle diagram)

[0209] S23: Open the input video stream through avformat_open_input, and analyze the video stream parameters contained in the container format.

[0210] S24: Call avformat_find_stream_info to obtain stream metadata, and determine the encoding context of each stream.

[0211] S25: Loop call av_read_frame to extract raw data packets from the input video stream, separate and store them to the corresponding queue according to the stream type.

[0212] S26: Separate the audio Packet sequence and the video Packet sequence, carry the timestamp and stream index information, create a codec context for each stream, and initialize the encoding parameters through avcodec_parameters_to_context.

[0213] S27: Call avcodec_find_decoder to obtain the H.264 encoder, and open the decoder through avcodec_open2.

[0214] S28: Loop to send Packet to the decoder: input compressed data through avcodec_send_packet, and call avcodec_receive_frame to output raw data frames.

[0215] S29: Push the data frames decoded by FFmpeg to the wrist display terminal, and the internal workers can view the real-time video pictures of the head-mounted high-resolution camera.

[0216] S210: The hole exploration device is used to view the physically inaccessible area of the personnel in the restricted space, which includes a flexible probe that can extend into the restricted space and an image acquisition device integrated at the front end of the probe.

[0217] S211: The outer diameter of the probe is not greater than 10mm, and a flexible catheter and a lighting assembly are integrated inside. The catheter can realize positive and negative 90° bending turning to adapt to complex space structures. The lighting assembly uses high-brightness LED light source and is configured with a diffusion lens to realize uniform illumination and avoid strong light reflection.

[0218] S212: The image acquisition device integrated at the front end of the probe is equipped with a miniature camera with a resolution of 1920x1080, supporting 30 frames per second dynamic video acquisition.

[0219] S213: Push the data frames collected by the hole exploration device to the wrist display terminal, and the internal workers can view the real-time video pictures inside the restricted space.

[0220] S214: The step difference gap detection device is a handheld high-precision non-contact optical measurement component, which is suitable for high-precision detection of step differences and gaps of various size range gaps, and the measurement error is less than or equal to ± 25 μm, and the measurement range is 0.1 mm-25 mm.

[0221] S215: The measurement result of the step difference gap detection device is temporarily stored in the edge device memory in the form of a string.

[0222] S216: The headset is used for voice interaction between internal workers and external data center, and the input and output of audio are realized through Windows Multimedia API.

[0223] Step S3: The video data of the head-mounted camera and the hole exploration device, the step difference gap detection data, and the real-time voice interaction data in the restricted space are transmitted to the external data processing center through TCP / IP.

[0224] Wherein, step S3 comprises:

[0225] S31: The internal edge device and the external data processing center are both equipped with Ethernet gigabit network cards, support TCP / IP protocol stack, and realize network communication function through C++.

[0226] S32: The external data processing center integrates WinSock network programming interface, including user interface module, data processing module and network communication module.

[0227] S33: The internal edge device and the external data processing center interact data through gigabit transmission rate network cable, including data receiving and sending.

[0228] S34: According to the different types of data collected by the internal data acquisition device, the type of data receiving and sending is determined, including the transmission of headset video stream, the transmission of hole exploration device video stream, the transmission of step difference gap detection device string and the transmission of headset audio. (As shown in the TCP / IP protocol differentiation transmission of various data schematic diagram) Figure 4

[0229] S35: The external data processing center creates TCP socket based on WinSock: socket(), binds the specified IP address and port: bind(), and listens to the connection request: listen().

[0230] S36: The external data processing center receives the internal edge device connection through accept(), establishes socket handle mapping table, and supports multiple edge device concurrent communication.

[0231] ​S37: The external data processing center reads the buffer data by calling recv() in a loop, parses the data packet according to the protocol, and extracts the encoded data after verification.

[0232] S38: The external data processing center encapsulates the to-be-sent string according to the protocol (adds a start symbol, length, check bit, and end symbol), and sends it to the target edge device through send().

[0233] S39: The internal edge device creates a TCP socket and connects the external data processing center IP address and port through connect().

[0234] S310: The internal edge device encapsulates the to-be-sent data according to the protocol and uploads it to the external data processing center through send(). When receiving instructions from the external data processing center, it parses the data packet and performs corresponding operations.

[0235] S311: Set up a heartbeat packet mechanism (send PING string regularly) to detect network connection status and automatically reconnect in case of exception.

[0236] S312: The internal and external networks interwork, allowing the external data processing center to receive head-mounted camera video streams, hole probe device video streams, step difference gap detection device strings, and head-mounted earphone audio from the internal edge device in real time.

[0237] Step S4: The external data processing center performs scratch detection on the received video data and selectively records scratch detection key frame data, video stream data, and step difference gap detection data into the database.

[0238] Step S4 includes:

[0239] S41: The external data processing center classifies and performs different operations on the received head-mounted earphone video stream, hole probe device video stream, step difference gap detection device string, and head-mounted earphone audio.

[0240] S42: For head-mounted earphone video streams and hole probe device video streams, perform scratch detection and recognition based on the YOLOv8 network. By introducing the CBAM (Convolutional Block Attention Module) attention mechanism in the Neck part of YOLOv8 and using the SIoU (Scale-Invariant Intersection over Union) loss function for parameter update, the performance of scratch detection is significantly improved. (As shown in the improved YOLOv8 network model diagram) Figure 5

[0241] ​S43: YOLOv8, the overall architecture includes Backbone, Neck and Head three main parts. Backbone is responsible for extracting features from input images, Neck fuses and enhances features, and Head is used to predict the category and location of the target.

[0242] S44: The image to be detected is scaled to 640x640 and sent to the trained neural network model for detection;

[0243] S45: The image first comes to the Backbone part in the model, and after the P1 layer of Backbone, a standard 3x3 Conv module, the convolution kernel size is 32, that is, a feature map with 32 channels is obtained;

[0244] S46: The feature map passes through the standard 3x3 Conv module of P2 layer and the 2nd layer C2f module of the network, and the channel number of the feature map is 64. C2f uses more jump connections and feature splicing to make the features more rich;

[0245] S47: The feature map passes through the standard 3x3 Conv module of P3 layer and the 4th layer C2f module of the network, and the channel number is 128. C2f is used to further fuse the feature map;

[0246] S48: The feature map passes through the standard 3x3 Conv module of P4 layer and the 6th layer C2f module of the network, and the channel number is 256. C2f is used to further fuse the feature map;

[0247] S49: The feature map passes through the standard 3x3 Conv module of P5 layer and the 8th layer C2f module of the network, and the channel number is 512. C2f is used to further fuse the feature map;

[0248] S410: The feature map passes through the SPPF module, and the channel number is 512, which is used for spatial pyramid pooling to realize feature extraction of different scales and capture detailed information of the target at different scales;

[0249] S411: The feature map comes to the Neck module in the model, first passes through the upsampling operation to expand the feature map by one, and is spliced with the feature map passing through the 6th layer, then the spliced feature passes through the 12th layer C2f module of the network, and then passes through the CBAM attention mechanism module, which is used to improve the feature representation ability and strengthen the spatial information, and the output dimension size is 256 feature map;

[0250] S412: The feature map of the last layer is expanded by one through upsampling, and is spliced with the feature map passing through the 4th layer, then the spliced feature passes through the 16th layer C2f module of the network, and then passes through the CBAM attention mechanism module, and the output dimension size is 128 feature map;Figure 6 CBAM attention mechanism principle diagram

[0251] S413: To enhance the attention of YOLOv8 on scratch features, the present application introduces a CBAM attention mechanism in the Neck part. CBAM consists of a Channel Attention Module and a Spatial Attention Module.

[0252] S414: The Channel Attention Module operates on the channel dimension of the feature map, assigning different weights to each channel to highlight important channel features. Specifically, the Channel Attention Module first applies a 1x1 convolution to the input feature map F to reduce the channel dimension to 1, resulting in a feature map F Global Average Pooling (GAP) and Global Max Pooling (GMP), respectively, to obtain two global feature vectors and , whose calculation formulas are as follows:

[0253] ;

[0254] ;

[0255] where represents the value of all channels in the i-th row and j-th column of the feature map F; F is the height of the feature map in the spatial dimension; is the width of the feature map in the spatial dimension; and is used to describe the spatial dimensions of the input feature map F, where C is the number of channels.

[0256] These two global feature vectors are input into a shared Multilayer Perceptron (MLP), which is activated by a ReLU activation function and a Sigmoid activation function, to obtain a channel attention map :

[0257] ;

[0258] where σ represents the Sigmoid activation function. v avg is a global feature vector obtained by performing Global Average Pooling (GAP) on the input feature map F; v max is a global feature vector obtained by performing Global Max Pooling (GMP) on the input feature map F.

[0259] ​Finally, the channel attention map M c is multiplied with the input feature map F F channel by channel to obtain a feature map enhanced by channel attention :

[0260] ;

[0261] wherein M c is multiplied with F element by element.

[0262] S415: The spatial attention module then operates on the spatial dimension of the feature map, assigning different weights to each spatial position to highlight important spatial regions. The spatial attention module first performs channel attention enhancement on the feature map F performs channel dimension average pooling and maximum pooling respectively to obtain two spatial feature maps and , whose calculation formulas are as follows:

[0263] ;

[0264] ;

[0265] The two spatial feature maps are spliced in the channel dimension to obtain a new feature map , and a 7x7 convolution layer and a Sigmoid activation function are used to obtain a spatial attention map :

[0266] ;

[0267] wherein represents a splicing operation in the channel dimension;

[0268] Finally, the spatial attention map is multiplied with the feature map enhanced by channel attention element by element to obtain a final feature map enhanced by CBAM attention mechanism :

[0269] .

[0270] wherein is multiplied with element by element.

[0271] S416: The feature map of the previous layer is passed through a standard 3x3 Conv module, spliced with the feature map of the 13th layer, and then the spliced feature is passed through the 20th layer C2f module of the network, followed by the CBAM attention mechanism module, outputting a feature map with a dimension size of 256;

[0272] S417: The feature map of the previous layer is passed through a standard 3*3 Conv module, spliced with the feature map passed through the 9th layer, and then the spliced feature is passed through the 24th layer C2f module of the network, followed by a CBAM attention mechanism module, and a feature map with a dimension size of 512 is output;

[0273] S418: The feature map passed through the 17th layer is input into the Head module of the network for identification and location positioning of the scratch target;

[0274] S419: The feature map passed through the 21st layer is input into the Head module of the network for identification and location positioning of the scratch target;

[0275] S420: The feature map passed through the 25th layer is input into the Head module of the network for identification and location positioning of the scratch target;

[0276] S421: In order to more accurately update the model parameters, the present application adopts SIoU as the loss function. The SIoU loss function comprehensively considers the overlapping area, center point distance and width-height ratio between the predicted frame and the real frame, and can more effectively optimize the target detection model. (As shown in the SIoU loss function principle diagram Figure 7 , wherein B is the predicted frame, B GT is the real frame, σ represents the distance between the center of the predicted frame and the center of the real frame, C h is the vertical distance between the center of the predicted frame and the center of the real frame, C w is the horizontal distance between the center of the predicted frame and the center of the real frame, and α and β represent arcsin(C h / σ) and arcsin(C w / σ), respectively.)

[0277] The calculation formula of the SIoU loss function is as follows:

[0278] ;

[0279] Wherein, represents the intersection over union (Intersection over Union) between the predicted frame and the real frame, Δ represents the angle loss, and Ω represents the distance loss and the width-height ratio loss.

[0280] The calculation formula of the angle loss Δ is as follows:

[0281] ;

[0282] Wherein, represents the Euclidean distance between the center points of the predicted frame and the real frame, Diagonal length of the minimum bounding rectangle of the predicted box and the real box.

[0283] The calculation formula of the distance loss and the aspect ratio loss Ω is as follows:

[0284] ;

[0285] wherein, and respectively represent the distance of the predicted box and the real box in the width and height directions, and respectively represent the width and height of the real box.

[0286] In the training process, the parameters of the model are updated by minimizing the SIoU loss function, so that the model can more accurately detect scratches.

[0287] S422: The scratch key frames detected by the head-mounted camera video stream and the hole probe equipment video stream are automatically entered into the external data processing center database, and the historical data can be viewed and traced in real time in the visualization interface. (As shown in the database framework diagram of Figure 8

[0288] S423: For the step difference gap detection equipment string data, it is segmented into detection categories (such as step difference, gap, chamfer, round corner, etc.) and real values by the external data processing center, and the segmented data is automatically entered into the external data processing center database according to the detection time.

[0289] S424: The external data processing center is equipped with a professional database visualization interactive interface, and the operation personnel can perform data adding, deleting, modifying and querying operations on the database based on the interface. For different detection scenes, independent databases can be constructed, and seamless switching between databases is supported, thereby realizing the full-process traceability and precise controllability of detection data.

[0290] S425: For the historical data in the database, representative data can be selected to automatically generate personalized reports. This report function module has a perfect comment embedding mechanism, which allows the operator to enter professional evaluations for the results of a certain detection task, and seamlessly integrates the comments into the corresponding report system.

[0291] S426: The external data processing center and the internal edge equipment are both equipped with a headset device, which builds a real-time voice communication link between the inside and the outside, ensuring that the inside and the outside can realize instant and efficient voice information interaction.

[0292] ​The above is only the preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification or equivalent change of the above embodiment according to the technical essence of the present application falls within the protection scope of the present application.

Claims

1. A confined space quality detection method based on multimodal edge cloud transmission, characterized in that, Includes the following steps: Step S1: Internal workers carry edge detection equipment into the workpiece to be tested and start various data acquisition devices within the confined space; In step S1, the workpiece to be inspected is identified, and internal personnel carry various data acquisition devices and edge detection devices into the narrow space of the workpiece. Each data acquisition device includes a head-mounted camera, a borescope, a step gap detection device, and a head-mounted headset. The edge detection device acts as a data relay station, connected to each data acquisition device. The data acquisition devices scan the interior of the workpiece and transmit the data to the edge detection device. Each portable device is equipped with lighting equipment, which the personnel can turn on or off as needed to ensure the availability of the acquired data. Step S2: The wrist-worn display terminal worn by the internal workers displays the video footage from the head-mounted camera and the borehole inspection equipment in real time; Step S3: The video data, step gap detection data, and real-time voice interaction data of the head-mounted camera and borehole equipment inside the confined space are transmitted to the external data processing center via TCP / IP; Step S4: The external data processing center performs scratch detection on the received video data and selectively enters the scratch detection keyframe data, video stream data, and step gap detection data into the database; Step S4 includes the following steps: Step S41: The external data processing center performs different operations on the received headphone video stream, borehole equipment video stream, step gap detection equipment string, and headphone audio classification. Step S42: For the video streams of the headset and the borescope, scratch detection and recognition are performed on them based on the YOLOv8 network. The CBAM attention mechanism is introduced into the Neck part of YOLOv8, and the SIoU loss function is used for parameter update. Step S43: Scale the image to be detected to 640×640 and feed it into the trained neural network model for detection; Step S44: The image first comes to the Backbone part of the model. After passing through the P1 layer of the Backbone, a standard 3×3 Conv module with a convolution kernel size of 32, a feature map with 32 channels is obtained. Step S45: The feature map passes through the standard 3×3 Conv module of layer P2 and the C2f module of the second layer of the network. The number of channels in the feature map is 64. C2f uses more skip connections and feature concatenation to make the features richer. Step S46: The feature map passes through the standard 3×3 Conv module of layer P3 and the C2f module of layer 4 of the network, with 128 channels. C2f is used to further fuse the feature map. Step S47: The feature map passes through the standard 3×3 Conv module of layer P4 and the C2f module of layer 6 of the network, with 256 channels. C2f is used to further fuse the feature map. Step S48: The feature map passes through the standard 3×3 Conv module of layer P5 and the C2f module of layer 8 of the network, with 512 channels. C2f is used to further fuse the feature map. Step S49: The feature map passes through the SPPF module with 512 channels, which is used for spatial pyramid pooling to extract features at different scales and capture detailed information of the target at different scales. Step S410: The feature map comes to the Neck module in the model. First, it is upsampled to double the size of the feature map and then concatenated with the feature map from the 6th layer. The concatenated feature map is then passed through the C2f module of the 12th layer of the network, and then through the CBAM attention mechanism module to improve the feature representation ability and enhance spatial information, outputting a feature map with a dimension of 256. Step S411: Upsample the feature map of the previous layer to double its size, concatenate it with the feature map of the 4th layer, and then pass the concatenated feature map through the C2f module of the 16th layer of the network, followed by the CBAM attention mechanism module, and output a feature map with a dimension of 128. Step S412: The feature map from the previous layer is passed through a standard 3×3 Conv module and concatenated with the feature map from the 13th layer. The concatenated feature map is then passed through the C2f module of the 20th layer of the network, and then through the CBAM attention mechanism module to output a feature map with a dimension of 256. Step S413: The feature map from the previous layer is passed through the standard 3×3 Conv module and concatenated with the feature map from the 9th layer. The concatenated feature map is then passed through the C2f module of the 24th layer of the network, and then through the CBAM attention mechanism module to output a feature map with a dimension of 512. Step S414: Input the feature map after the 17th layer into the Head module of the network to identify and locate the scratch target; Step S415: Input the feature map after the 21st layer into the Head module of the network to identify and locate the scratch target; Step S416: Input the feature map after the 25th layer into the Head module of the network to identify and locate the scratch target; Step S417: The key frames of scratches detected by the head-mounted camera video stream and the borehole inspection equipment video stream are automatically entered into the external data processing center database, and historical data can be viewed and traced in real time in the visualization interface; Step S418: For the string data of the step gap detection device, it is divided into detection category and actual value by the external data processing center. The segmented data is automatically entered into the database of the external data processing center according to the detection time. Step S419: The external data processing center is equipped with a professional database visualization interface, which allows operators to perform data addition, deletion, modification and query operations on the database. Step S420: For historical data in the database, select representative data to automatically generate personalized reports. This report function module has a complete comment embedding mechanism. Step S421: Both the external data processing center and the internal edge devices are equipped with headsets to establish a real-time voice communication link between the internal and external systems, ensuring that the internal and external systems can achieve instant and efficient voice information interaction.

2. The confined space quality detection method based on multimodal edge cloud transmission according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: The edge detection device is equipped with a wrist-worn small LCD screen, allowing internal staff to view the video feeds from the head-mounted camera and the borescope in real time; Step S22: Use FFmpeg technology to capture the video data stream from the head-mounted high-resolution camera; Step S23: Open the input video stream using avformat_open_input and parse the video stream parameters contained in the container format; Step S24: Call avformat_find_stream_info to obtain stream metadata and determine the encoding context of each stream; Step S25: Repeatedly call av_read_frame to extract raw data packets from the input video stream, separate them according to stream type and store them in the corresponding queue; Step S26: Separate the audio packet sequence and video packet sequence, carrying timestamps and stream index information, create a codec context for each stream, and initialize the encoding parameters through avcodec_parameters_to_context; Step S27: Call avcodec_find_decoder to obtain the H.264 encoder, and open the decoder using avcodec_open2; Step S28: Loop through which Packet is sent to the decoder: Input compressed data via avcodec_send_packet and output raw data frames by calling avcodec_receive_frame; Step S29: Push the data frames decoded by FFmpeg to the wrist display terminal, and the internal operators can view the real-time video footage from the head-mounted high-resolution camera; Step S210: Push the data frames collected by the borehole exploration equipment to the wrist display terminal, so that the internal workers can view the real-time video footage inside the confined space; Step S211: The measurement results of the step gap detection device are temporarily stored in the memory of the edge detection device in the form of a string.

3. A confined space quality detection method based on multimodal edge cloud transmission according to claim 2, characterized in that: The borehole probe is used to inspect areas physically inaccessible to personnel within a confined space. It includes a flexible probe that can extend into the confined space and an image acquisition device integrated at the probe's tip. The probe's outer diameter is no greater than 10mm, and it integrates a flexible conduit and lighting components. The conduit can bend and rotate at ±90° to adapt to complex spatial structures. The lighting components use high-brightness LED light sources and are equipped with diffused lenses to achieve uniform illumination. The image acquisition device integrated at the probe's tip is equipped with a miniature camera with a resolution of 1920×1080, supporting dynamic video acquisition at 30 frames per second. The step difference / gap detection device is a handheld, high-precision, non-contact optical measurement component suitable for high-precision detection of step differences and gaps in various sizes, with a measurement error of less than or equal to ±25μm and a measurement range of 0.1mm-25mm. A headset is used for voice interaction between internal personnel and the external data center, enabling audio input and output via the Windows Multimedia API.

4. The confined space quality detection method based on multimodal edge cloud transmission according to claim 3, characterized in that: Step S3 includes the following steps: Step S31: Both the edge detection device and the external data processing center are equipped with 10 Gigabit Ethernet network cards, support the TCP / IP protocol stack, and implement network communication functions through C++; Step S32: The external data processing center integrates the WinSock network programming interface, including a user interface module, a data processing module, and a network communication module; Step S33: The edge detection device and the external data processing center interact with each other via a 10 Gigabit Ethernet cable, including data reception and transmission; Step S34: The type of data reception and transmission is determined according to the different types of data collected by the internal data acquisition device, including the transmission of video stream from the headset, the transmission of video stream from the borehole probe, the transmission of string from the step gap detection device, and the transmission of audio from the headset. Step S35: The external data processing center creates a TCP socket based on WinSock: socket(), binds it to a specified IP address and port: bind(), and listens for connection requests: listen(); Step S36: The external data processing center receives the connection from the edge detection device through accept(), establishes a Socket handle mapping table, and supports concurrent communication between multiple edge devices; Step S37: The external data processing center repeatedly calls recv() to read buffer data, parses data packets according to the protocol, and extracts encoded data after verification. Step S38: The external data processing center encapsulates the string to be sent according to the protocol, including adding a start character, length, checksum, and end character, and sends it to the target edge device via send(). Step S39: The edge detection device creates a TCP socket and connects to the external data processing center's IP address and port using connect(); Step S310: The edge detection device encapsulates the data to be sent according to the protocol and uploads it to the external data processing center via send(); when receiving instructions from the external data processing center, it parses the data packet and performs the corresponding operation; Step S311: Set up a heartbeat mechanism to periodically send PING strings to check network connection status and automatically reconnect when abnormal; Step S312: Internal and external network interconnection enables the external data processing center to receive in real time the video stream from the head-mounted camera, the video stream from the borescope, the string from the step gap detection device, and the audio from the head-mounted headphones sent by the edge detection device.

5. A confined space quality detection method based on multimodal edge cloud transmission according to claim 4, characterized in that: In step S42, the SIoU loss function is calculated using the following formula: ； in, The cross-over ratio (CROR) represents the intersection-over-union ratio between the predicted bounding box and the ground truth bounding box; Δ represents the angle loss; and Ω represents the distance loss and aspect ratio loss. The formula for calculating the angle loss Δ is as follows: ； in, This represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box. This represents the diagonal length of the smallest bounding rectangle between the predicted bounding box and the ground truth bounding box; The formulas for calculating distance loss and aspect ratio loss Ω are as follows: ； in, and These represent the distances between the predicted bounding box and the ground truth bounding box in the width and height directions, respectively. and These represent the width and height of the actual bounding box, respectively. During training, the model parameters are updated by minimizing the SIoU loss function.

6. A confined space quality detection method based on multimodal edge cloud transmission according to claim 5, characterized in that: In step S42, the overall architecture of YOLOv8 consists of three main parts: Backbone, Neck, and Head. Backbone is responsible for extracting features from the input image, Neck fuses and enhances the features, and Head is used to predict the category and location of the target.

7. A confined space quality detection method based on multimodal edge cloud transmission according to claim 6, characterized in that: The CBAM attention mechanism module includes a channel attention module and a spatial attention module.

8. A confined space quality detection method based on multimodal edge cloud transmission according to claim 7, characterized in that: The channel attention module operates on the channel dimension of the feature map, assigning different weights to each channel to highlight important channel features; firstly, it processes the input feature map... Perform global average pooling and global max pooling respectively to obtain two global feature vectors. and The calculation formula is as follows: ； ； in, Representation of feature map F The values ​​of all channels in the i-th row and j-th column; It is the height of the feature map in the spatial dimension; It is the width of the feature map in the spatial dimension; These two global feature vectors are input into a shared multilayer perceptron, and after passing through the ReLU activation function and the Sigmoid activation function, the channel attention map M is obtained. c ∈R C : ； Where σ represents the Sigmoid activation function; v avg It is the global feature vector obtained by performing global average pooling on the input feature map F; v max It is the global feature vector obtained by performing global max pooling (GMP) on the input feature map F; Finally, the channel attention map M c With input feature map F Perform channel-by-channel multiplication to obtain the feature map after channel attention enhancement. : ； Among them, M c and F Multiply element by element.

9. A confined space quality detection method based on multimodal edge cloud transmission according to claim 8, characterized in that: The spatial attention module operates on the spatial dimension of the feature map, assigning different weights to each spatial location to highlight important spatial regions; firstly, it processes the feature map after channel attention enhancement. Perform average pooling and max pooling along the channel dimension to obtain two spatial feature maps. and The calculation formula is as follows: ； ； These two spatial feature maps are concatenated along the channel dimension to obtain a new feature map. The spatial attention map is obtained by passing a 7×7 convolutional layer and a sigmoid activation function. : ； in, This indicates a splicing operation along the channel dimension; Finally, the spatial attention map Feature maps with channel attention enhancement Element-wise multiplication is performed to obtain the final feature map enhanced by the CBAM attention mechanism. : ； in, and Multiply element by element.

Images