Wind turbine generator maintenance operation system based on intelligent dynamic multimedia stream

Abstract

The invention discloses a wind turbine generator set maintenance operation system based on intelligent dynamic multimedia streams, and relates to the technical field of wind turbine generator set remote maintenance, and the wind turbine generator set maintenance operation system comprises the steps that an intelligent communication terminal collects on-site audio and video data in real time through a microphone and a camera, and captures an environment sound, a voice instruction and a picture of an equipment operation process; the intelligent communication terminal encodes the collected audio and video data and transmits the audio and video data to the maintenance operation server in a wireless communication mode; the intelligent command tablet personal computer decodes the received audio and video data and restores the compressed audio and video data into playable content; the intelligent command tablet computer plays decoded audio and video data through a loudspeaker and a high-definition screen, and command personnel communicate with operating personnel in real time by checking videos. According to the invention, artificial intelligence and edge computing technologies are introduced, and real-time identification, dynamic optimization and continuous tracking are carried out on the key area in the operation scene, so that the real-time performance and stability of multimedia data transmission are remarkably improved.

Description

Technical Field

[0001] The invention relates to the technical field of remote maintenance of wind turbine sets, in particular to a maintenance operation system of wind turbine sets based on intelligent dynamic multimedia stream. Background Art

[0002] With the transformation of the global energy structure, wind power has been widely used as a clean and renewable energy form. The continuous operation of wind turbines requires regular inspection and maintenance to ensure their normal operation. However, wind turbines face many challenges in carrying out inspection and maintenance operations under high altitude, remote and complex environmental conditions. Traditional wind turbine maintenance mainly relies on manual operation and experience judgment, which is inefficient and easily disturbed by the external environment. In addition, with the expansion of wind turbine scale and the increase of technical complexity, traditional maintenance methods are difficult to meet the increasing real-time and accuracy requirements. Therefore, in recent years, remote maintenance systems based on multimedia technology and artificial intelligence have gradually become a research hotspot. These systems can improve maintenance efficiency and accuracy to a certain extent through real-time audio and video transmission and intelligent analysis.

[0003] However, the existing remote maintenance system still has many shortcomings, especially in the transmission and processing of multimedia data. First, traditional remote maintenance systems often use fixed encoding and transmission methods, and cannot be dynamically adjusted according to fluctuations in network bandwidth and changes in operating scenarios, resulting in data loss or delays when the network status is poor, affecting the real-time and accuracy of maintenance operations. Secondly, the existing system lacks intelligent processing methods for audio and video data of the operating scene, and cannot effectively identify and locate key areas, resulting in the loss of key information or interference by noise. In addition, traditional systems often ignore the optimization of audio and video quality during the compression and transmission of multimedia data, resulting in problems such as reduced image quality and sound quality during decoding and playback, further affecting the decision-making ability of commanders. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a wind turbine maintenance operation system based on intelligent dynamic multimedia flow to solve the problems of key area identification and dynamic adjustment in multimedia data processing in the prior art.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, an embodiment of the present invention provides a wind turbine maintenance operation system based on intelligent dynamic multimedia streaming, which includes a data acquisition module: the intelligent communication terminal collects audio and video data on site in real time through a microphone and a camera, and captures ambient sound, voice commands and images of the equipment operation process; a data transmission module: the intelligent communication terminal encodes and processes the collected audio and video data, and transmits it to the maintenance operation server through wireless communication; an intelligent adjustment module: the maintenance operation server uses an artificial intelligence architecture to identify and locate key areas of the operation picture, and dynamically adjusts the audio and video data in the identified key areas; an intelligent optimization module: optimizes the key areas and non-key areas of the operation picture through an edge computing model; a data forwarding module: forwards the dynamically adjusted audio and video data from the maintenance operation server to an intelligent command tablet computer; a data presentation module: the intelligent command tablet computer decodes the received audio and video data, and restores the compressed audio and video data to playable content; a command and control module: the intelligent command tablet computer plays the decoded audio and video data through a speaker and a high-definition screen, and the commander views the video and communicates with the operator in real time.

[0008] As a preferred solution of the wind turbine maintenance operation system based on intelligent dynamic multimedia stream of the present invention, the real-time collection of on-site audio and video data, capturing the ambient sound, voice instructions and the screen of the equipment operation process, the specific steps are:

[0009] Use 4K HDR cameras and omnidirectional condenser microphones, deployed at high altitudes to cover the work area;

[0010] Set the camera frame rate to 30fps, the resolution to 3840x2160, the audio sampling rate to 48kHz, and the quantization bit to 24-bit;

[0011] Add timestamps to each frame of video and audio data, remove video noise using Gaussian filters, and optimize brightness and contrast through histogram equalization;

[0012] The audio is de-noised using spectral subtraction and the echo is eliminated using an adaptive filtering algorithm.

[0013] As a preferred solution of the wind turbine maintenance operation system based on intelligent dynamic multimedia stream described in the present invention, wherein: the intelligent communication terminal encodes the collected audio and video data and transmits it to the maintenance operation server via wireless communication. The specific steps are:

[0014] After data collection, ensure the synchronization of audio and video, and match audio and video frames through timestamps;

[0015] Use HEVC encoding to compress the synchronized audio and video data;

[0016] After the video is encoded, the audio data is compressed through AAC, using sub-band filtering and perceptual quantization to separate and compress the audio bands;

[0017] The encoded audio and video data stream is encrypted via AES-256, and the encrypted data is transmitted via the SRTP protocol, and the transmission rate is dynamically adjusted according to network conditions.

[0018] As a preferred solution of the wind turbine maintenance operation system based on intelligent dynamic multimedia stream of the present invention, wherein: the audio data is compressed by AAC, and the audio frequency band is separated and compressed by sub-band filtering and perceptual quantization. The specific steps are:

[0019] Sampling the collected audio data to convert the continuous audio signal into a discrete digital signal;

[0020] Divide the discrete digital signal into multiple frames, each frame contains a short audio sample;

[0021] After the frame processing, the audio signal of each frame is decomposed into multiple sub-bands by applying a polyphase filter bank, each sub-band corresponds to a different frequency range;

[0022] The sub-band filtered frequency band signals are input into the perception model, which accurately analyzes the masking effect of each frequency band signal based on the sensitivity of the human ear to different frequencies and volumes;

[0023] According to the analysis results of the perceptual model, perceptual quantization is performed on the audio data of each sub-band;

[0024] The quantized data is further compressed through Huffman coding;

[0025] Recombine and encapsulate the encoded sub-band data into an AAC frame;

[0026] Output the encapsulated AAC frames as a continuous audio data stream.

[0027] As a preferred solution of the wind turbine maintenance operation system based on intelligent dynamic multimedia stream of the present invention, the specific steps of identifying and locating the key area of ​​the operation screen are as follows:

[0028] The maintenance operation server synchronously processes the received audio and video data, extracts each frame of the image, and decodes it into an RGB matrix;

[0029] Based on the synchronized frames, the pre-trained Faster R-CNN deep learning model is loaded, and each frame image is input into the convolutional neural network for feature extraction;

[0030] The extracted feature vectors are analyzed by the scene recognition model to classify and identify the current operation scene;

[0031] Based on scene recognition, the region proposal network is used to generate candidate key regions, classify and locate them, and finally output key region objects and their locations.

[0032] Based on the results of scene recognition, prompt information related to the current scene is generated, and links to relevant documents and operation manuals are extracted from the knowledge base.

[0033] As a preferred solution of the wind turbine maintenance operation system based on intelligent dynamic multimedia stream of the present invention, the specific steps of dynamically adjusting the audio and video data in the identified key area are as follows:

[0034] Based on scene recognition and key area positioning, the encoding parameters are dynamically adjusted to optimize the use of network bandwidth through image similarity analysis based on perceptual hashing;

[0035] Use Kalman filter and optical flow method to continuously track key areas and adjust encoding parameters in real time;

[0036] The dynamically adjusted and optimized audio and video data are re-encoded and packaged, and the scene prompts and knowledge base links are transmitted to the smart command tablet together with the audio and video data through adaptive transmission rate adjustment.

[0037] As a preferred solution of the wind turbine maintenance operation system based on intelligent dynamic multimedia streaming described in the present invention, wherein: the key areas and non-key areas of the operation screen are optimized by the edge computing model, and the specific steps are as follows:

[0038] Segment each frame and use the pre-trained Mask R-CNN deep learning model to identify key areas;

[0039] The pre-trained deep learning model outputs a binary mask identifying the key region R k and non-critical area R n ;

[0040] The Sobel operator is used to detect the boundary of the key area, and the extracted boundary area is recorded as B k ;

[0041] For each key area B k , by calculating its motion vector in consecutive frames and marking its priority P k The expression is:

[0042]

[0043] Among them, u i and v i are the horizontal and vertical motion vectors of the key area in the i-th frame, N is the number of window frames, and i is the index number representing the frame;

[0044] Perform wavelet transform on each frame and decompose it into sub-bands of different frequencies;

[0045] Define the subband coefficient after wavelet transform as W j,k , where j represents the scale level and k represents the subband index;

[0046] For the key area R k Assign high weight α k , for non-critical area R n Assign low weight α n , the weight distribution expression is:

[0047]

[0048] Among them, σ 2 represents the variance of the motion vector in the key area;

[0049] Through the multi-objective optimization algorithm, the coding parameters of each subband are optimized under the premise of meeting the bandwidth and computing resource constraints. The objective function L of the optimization problem is expressed as:

[0050]

[0051] Where j represents the scale level, k represents the subband index, and W j，k is the original subband coefficient after wavelet transform, is the optimized subband coefficient, dA is the area element of the integration region, and λ is the trade-off parameter;

[0052] According to the optimized subband coefficients Dynamically adjust the quantization step size Q k and Q n , quantization step size Q k and Q n The expressions are:

[0053]

[0054] Among them, N k and N n are the number of pixels in the critical area and the non-critical area, respectively;

[0055] Perform entropy coding on the quantized subband coefficients to further compress the data;

[0056] During the transmission process, the transmission rate is adjusted in real time according to the network status;

[0057] After encoding and transmission, the quality of the received video data is evaluated using peak signal-to-noise ratio and structural similarity indicators;

[0058] Set the trade-off parameter λ to λ min ≤λ≤λ max ;

[0059] When the value of λ is large, the optimization process focuses more on the quality of the key area, which is suitable for scenes with key areas as the main body;

[0060] When the value of λ is small, the optimization process is more balanced and is used in scenarios where both critical areas and non-critical areas are important.

[0061] As a preferred solution of the wind turbine maintenance operation system based on intelligent dynamic multimedia stream of the present invention, the intelligent command tablet computer decodes the received audio and video data, and the specific steps are as follows:

[0062] Receive audio and video data streams and additional scene prompt information and knowledge base link data packets from the maintenance operation server through the SRTP protocol;

[0063] Using the AES-256 algorithm and the pre-shared key, the received encrypted audio and video data stream is decrypted to obtain the original compressed audio and video data;

[0064] Use the streaming media parser to analyze the decrypted data stream and extract the video stream information;

[0065] Call the hardware-accelerated H.265 decoder for initialization, read the compressed bit stream frame by frame, perform entropy decoding, inverse quantization, inverse wavelet transform and motion compensation, and restore the YUV image frame;

[0066] Convert the YUV image frame to RGB format by adaptive histogram equalization;

[0067] Use the streaming media parser to analyze the decrypted data stream and extract the audio stream information;

[0068] Call the hardware-accelerated AAC decoder for initialization and send the AAC compressed audio frame to the decoder to restore the uncompressed digital audio signal;

[0069] According to the configuration requirements of the playback device, the audio data is resampled and format converted, and a dynamic range compression algorithm is applied to enhance speech clarity and background sound layering.

[0070] As a preferred solution of the wind turbine maintenance operation system based on intelligent dynamic multimedia stream of the present invention, wherein: the compressed audio and video data is restored to playable content, the specific steps are:

[0071] After decoding, the timestamps of the audio and video frames are matched, and an adaptive buffer is generated based on network fluctuations and decoding delays;

[0072] Using GPU acceleration, the synchronized RGB video frames in the buffer are rendered to the tablet display through OpenGL;

[0073] The audio data in the buffer is converted into an analog signal through a digital-to-analog converter and outputted through an audio output terminal of the tablet computer;

[0074] Dynamically adjust the video playback frame rate according to the tablet screen refresh rate;

[0075] While the video is being played, the data packets sent from the maintenance operation server are parsed;

[0076] Extract scene prompt information, knowledge base link and associated timestamp, and when the timestamp of audio and video data matches the scene prompt information, a prompt box pops up.

[0077] As a preferred solution of the wind turbine maintenance operation system based on intelligent dynamic multimedia stream of the present invention, when the timestamp of the audio and video data matches the scene prompt information, a prompt box pops up, and the specific steps are:

[0078] The tablet receives the attached data packet from the maintenance operation server, decrypts and decompresses it, and obtains the original data including the scene prompt information, the knowledge base link and the timestamp;

[0079] Parse the data packet, extract and parse the timestamp, prompt information and link fields, and organize them into structured processable data;

[0080] Match the extracted timestamp with the current video playback time, and when the match is successful, mark the corresponding prompt information as to be displayed;

[0081] Queue the marked reminder messages in chronological order;

[0082] When the video playback reaches the corresponding time point, a prompt box is generated to display relevant prompt information and links.

[0083] The beneficial effects of the present invention are as follows: the present invention introduces artificial intelligence and edge computing technologies, performs real-time identification and dynamic optimization of key areas in the operating scene based on a deep learning model, combines Kalman filtering and optical flow method to continuously track the motion trajectory of key areas, and dynamically adjusts encoding parameters to optimize bandwidth usage, thereby significantly improving the real-time and reliability of audio and video data transmission, and ensuring high-quality maintenance operations in complex network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0085] Figure 1 This is a flow chart of the wind turbine maintenance system based on intelligent dynamic multimedia streaming in Example 1. DETAILED DESCRIPTION

[0087] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0088] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0089] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0090] Example 1, reference Figure 1 This is the first embodiment of the present invention, which provides a wind turbine maintenance operation system based on intelligent dynamic multimedia flow, including the following modules: data acquisition module, data transmission module, intelligent adjustment module, intelligent optimization module, data forwarding module, data presentation module and command and control module;

[0091] The data acquisition module is used by intelligent communication terminals to collect real-time audio and video data on site through microphones and cameras, capturing ambient sounds, voice commands and images of the equipment operation process.

[0092] Going a step further, choose a 4K HDR camera (such as Sony IMX586) and an omnidirectional condenser microphone (such as ShureMX391 / O) and deploy them at high altitudes to cover the work area;

[0093] Among them, the microphone needs to be installed near the camera to ensure synchronization of audio and video capture.

[0094] Set the camera frame rate to 30fps and the resolution to 3840x2160 to ensure smooth and clear video;

[0095] The audio sampling rate is set to 48kHz and the quantization bit is 24-bit to ensure the capture of high-quality ambient sound.

[0096] Add timestamps to each frame of video and audio data to ensure audio and video synchronization;

[0097] For example, timestamps can be added using a unified time base such as a system clock or a global time code.

[0098] Use Gaussian filter to remove video noise, and optimize brightness and contrast through histogram equalization;

[0099] Preferably, a Gaussian filter is used to smooth the image by weighted averaging the image pixels, which helps to reduce high-frequency noise (such as random noise) in the image. In video processing, this can significantly improve the visual quality of the image, making the picture clearer and smoother.

[0100] At the same time, compared with other simple average filters, the Gaussian filter can better retain the edge details of the image while removing noise. This is because the weighted average of the Gaussian filter is performed according to the Gaussian distribution based on the distance between the pixel and the center point. The closer the pixel is to the center, the greater the weight, so it can better retain the edge information in the image.

[0101] Preferably, histogram equalization is a global processing method that can automatically adapt to different scenes and lighting conditions without manually adjusting brightness and contrast settings. This makes it very effective when dealing with complex and changing environments; using histogram equalization adjusts the brightness distribution of pixels in the image so that the overall image has a wider brightness range, thereby improving the image contrast. This is particularly effective for scenes with uneven lighting or low contrast (such as shadow areas or environments with low brightness).

[0102] At the same time, through equalization processing, the dark and bright details in the image can be better displayed, especially those details that cannot be clearly seen in the original image due to insufficient light or excessive brightness. This is very important for capturing key picture information during the operation process.

[0103] The audio is de-noised using spectral subtraction and the echo is eliminated using an adaptive filtering algorithm.

[0104] Among them, the adaptive filtering algorithm for echo cancellation is used to cancel the echo fed back from the loudspeaker to the microphone.

[0105] The data transmission module is used by the intelligent communication terminal to encode the collected audio and video data and transmit it to the maintenance operation server via wireless communication.

[0106] Furthermore, after data collection, the synchronization of audio and video is ensured, and audio and video frames are matched through timestamps;

[0107] Use HEVC (H.265) encoding to compress the synchronized audio and video data;

[0108] HEVC encoding generates key frames and differential frames through inter-frame differences, motion compensation, and CABAC entropy coding, reducing inter-frame redundancy and further compressing the amount of data to ensure efficient encoding;

[0109] Preferably, motion compensation is an advanced technique based on frame differences that estimates the motion trajectory of objects in the video and predicts certain parts of the current frame to reduce redundant information. The encoder calculates motion vectors and uses these vectors to predict the content of the current frame.

[0110] Preferably, CABAC is an advanced entropy coding technique that achieves lossless compression by modeling the probability of coded symbols. CABAC dynamically adjusts the probability distribution of symbols based on context information, thereby achieving higher compression efficiency.

[0111] Preferably, inter-frame difference is a technology used for video compression. It reduces redundant information between frames by comparing the pixel differences between adjacent frames. Only the changed parts are encoded, and the unchanged parts are usually not re-encoded, thereby reducing the amount of data, which specifically refers to removing redundant information using an inter-frame difference algorithm and retaining only frames with significant changes to reduce the data volume.

[0112] Among them, "frames with significant changes" refer to those frames whose content changes exceed a preset threshold compared with the previous frame. This change is usually determined by calculating the pixel difference between the current frame and the previous frame.

[0113] For example, in video processing, each frame of an image can be represented as a matrix, where each element in the matrix represents the brightness or color value (such as grayscale value or RGB value) of a pixel;

[0114] For frame F t and the previous frame F t-1 , calculate their pixel difference matrix D t The expression is:

[0115] D t (x,y)=|F t (x,y)-F t-1 (x,y)|;

[0116] Where (x, y) represents the position of the pixel in the frame;

[0117] Calculate the frame difference matrix D t The sum or mean of the elements of to evaluate the degree of change of the entire frame is expressed as:

[0118]

[0119] Where N is the total number of pixels in the image;

[0120] Set a threshold T diff , used to determine whether the change is significant. If Δ t More than T diff , then the current frame F t It is a "significantly changed frame" that needs to be retained and encoded. The specific expression is as follows:

[0121] Frames with significant changes = if (Δt>Tdiff) then F t

[0122] It should also be noted that in some application scenarios, global change detection may not be accurate enough, so local change detection can be introduced. That is, the frame is divided into small blocks (such as 8x8 or 16x16 pixels), the change value of each small block is calculated separately, and it is determined whether there are small blocks with significant changes. If there are, the frame is considered to be a "significantly changed frame".

[0123] In summary, "significantly changed frames" can be clearly defined as those frames whose changes at the pixel level exceed the set threshold compared with the previous frame, which helps to reduce the encoding of redundant frames and thus improve the efficiency of video compression.

[0124] Furthermore, after the video encoding is completed, the audio data is compressed through AAC, and the audio frequency bands are separated and compressed using sub-band filtering and perceptual quantization to ensure that the sound quality is maintained at a high compression rate. The specific steps are as follows:

[0125] Sampling the collected audio data to convert the continuous audio signal into a discrete digital signal;

[0126] It should be noted that this process is implemented through analog-to-digital conversion (ADC), and the sampling result is an audio sample sequence with a fixed time interval.

[0127] Divide the discrete digital signal into multiple frames, each frame contains a short audio sample;

[0128] Preferably, the frame processing allows the audio data to be processed and encoded frame by frame, which facilitates the application of subsequent compression technology.

[0129] After the frame processing, the audio signal of each frame is decomposed into multiple sub-bands by applying a polyphase filter bank, each sub-band corresponds to a different frequency range;

[0130] It should be noted that the audio data is separated into a plurality of frequency band signals, which respectively represent the low-frequency, middle-frequency and high-frequency components of the audio.

[0131] The sub-band filtered frequency band signals are input into the perception model, which accurately analyzes the masking effect of each frequency band signal based on the sensitivity of the human ear to different frequencies and volumes;

[0132] Preferably, the perceptual model is used to utilize the auditory masking effect of the human ear to intelligently identify and discard those frequency components in the audio that have little impact on the sound quality, thereby optimizing the compression efficiency; by dynamically adjusting the quantization accuracy, the perceptual model significantly reduces the amount of audio data while retaining the sound quality to the maximum extent, making the compressed audio both compact and maintaining high-quality playback, which is particularly suitable for scenarios that require efficient transmission and storage.

[0133] According to the analysis results of the perceptual model, perceptual quantization is performed on the audio data of each sub-band;

[0134] It should be noted that the quantization process is to map continuous audio signal values ​​to discrete digital values. For those frequency components that the perception model determines can be ignored, lower precision quantization is used; for important frequency components, higher precision quantization is used. This step can effectively reduce the redundant information of audio data and reduce the data size.

[0135] The quantized data is further compressed through Huffman coding;

[0136] Preferably, Huffman coding is a lossless entropy coding method, which allocates short codes and long codes according to the probability of data occurrence, further reducing data redundancy.

[0137] The encoded sub-band data is reassembled and encapsulated into an AAC frame, where each frame contains the compressed data of all sub-bands and the necessary frame header information for recovering the audio signal during decoding and playback;

[0138] Output the encapsulated AAC frames as a continuous audio data stream.

[0139] It should be noted that this audio data stream can be encapsulated together with the video data or transmitted independently, ensuring efficient audio compression and good sound quality.

[0140] The encapsulated MP4 files are encrypted using AES-256 to ensure data security during transmission. The encrypted data is transmitted using the SRTP protocol, and the transmission rate is dynamically adjusted based on network conditions.

[0141] The intelligent adjustment module is used to maintain the operation server using artificial intelligence architecture to identify and locate key areas of the operation screen, and dynamically adjust the audio and video data in the identified key areas.

[0142] Furthermore, the maintenance operation server performs synchronization processing (such as processing through timestamps) on the received audio and video data to ensure the consistency of the audio and video frames on the timeline;

[0143] Preferably, during the audio and video data acquisition and encoding process, each audio frame and video frame will be attached with a timestamp. After receiving the data, the maintenance operation server compares the timestamps of the audio and video frames to ensure that they are aligned on the time axis.

[0144] Extract each frame of the image from the synchronized video stream (e.g., by using a decoder), and decode it into an RGB matrix;

[0145] It should be noted that the most commonly used decoding library is FFmpeg, which supports decoding of multiple video formats (such as H.264, HEVC, etc.). The maintenance job server can use the FFmpeg library or similar tools to process the video stream and extract frame-by-frame image data.

[0146] It should also be noted that the decoded frame image is usually stored in YUV format, but the deep learning model usually requires RGB format input. Therefore, the YUV format needs to be converted to an RGB matrix.

[0147] Based on the synchronized frames, the pre-trained Faster R-CNN deep learning model is loaded, and each frame image is input into the convolutional neural network (CNN) for feature extraction;

[0148] Among them, the extracted feature vector contains multi-scale information in the image, providing basic data for subsequent scene recognition and key area recognition.

[0149] Preferably, Faster R-CNN is an advanced target detection model with high detection accuracy and efficiency. It introduces the Region Proposal Network (RPN), which can quickly generate high-quality candidate regions. This enables Faster R-CNN to quickly locate key objects in the scene when processing complex operating scenes, reduce interference from irrelevant areas, and ensure efficient recognition of key areas.

[0150] It should be noted that the use of the pre-trained Faster R-CNN deep learning model can give full play to its advantages in target detection, feature extraction, real-time, versatility and scalability, thereby ensuring accurate identification and positioning of key areas. Combined with the scene recognition model, the system can generate prompt information related to the operation scene in real time, providing efficient and reliable technical support for intelligent maintenance.

[0151] Preferably, the core of CNN is the convolution layer, which scans the input image through the convolution kernel (filter / kernel) to extract local features. The convolution layer slides over the width and height of the image, calculates the weighted sum of each local area, and thus extracts a set of feature maps.

[0152] It should be noted that the specific steps of "inputting each frame of image into the convolutional neural network for feature extraction" are: first, load the pre-trained CNN model (such as ResNet, VGG) and set it to evaluation mode to avoid parameter updates. Next, adjust each frame of image to the fixed size required by the model (such as 224x224 pixels), and normalize the pixel values ​​to the mean and standard deviation range expected by the model. Then, input the preprocessed image into the CNN model, and extract multi-level image features through the convolutional layer and pooling layer of the model. Finally, further process the extracted feature map as needed, such as flattening it into a one-dimensional feature vector for scene recognition or target detection tasks.

[0153] Analyze the extracted feature vectors through a scene recognition model (such as a ResNet model) to classify and identify the current operation scene (such as identifying the type of a certain device or a specific operation step);

[0154] The advantages of ResNet in scene recognition are mainly reflected in its powerful feature extraction capabilities, efficient training of deep networks, good generalization capabilities, and extensive community support. By using ResNet, the maintenance operation server can more accurately identify complex operation scenarios and maintain efficient and stable performance in various application scenarios.

[0155] If the ResNet-50 model is used for scene recognition, the specific steps for classifying and identifying the current job scene are as follows: Input each frame of the extracted RGB image into the ResNet-50 model, and extract the high-dimensional features of the image through a series of convolutions and residual connections; the extracted feature vector is mapped to the output space of the scene classification through the fully connected layer of ResNet. The Softmax layer converts the output into a probability distribution to determine the most likely scene category; finally, the scene category is output and used for subsequent key area detection and operation guidance.

[0156] Based on scene recognition, the region proposal network (RPN) is used to generate candidate key regions, classify and locate them, and finally output key region objects and their locations.

[0157] It should be noted that the Region Proposal Network (RPN) is a convolutional neural network used to generate candidate regions (usually called "proposal boxes" or "candidate boxes") that may contain objects in an image. It was first introduced in the Faster R-CNN model and quickly became a core component in the field of object detection.

[0158] Preferably, RPN shares the feature extraction layer with the basic convolutional neural network, which means that the feature extraction process only needs to be performed once, greatly improving computational efficiency; RPN can be trained end-to-end with the target detection network, making the coupling between proposal box generation and target detection tighter, thereby improving detection accuracy; by generating anchor boxes of multiple scales and multiple aspect ratios, RPN can flexibly adapt to target objects of different sizes and shapes, improving its adaptability to complex scenes; RPN can not only generate candidate regions, but also accurately adjust the position and size of these regions through bounding box regression, thereby enhancing the accuracy of the detection frame.

[0159] It should also be noted that "classify and locate it, and finally output the key area object and its location". The specific steps are: on the feature map of the input image, each unit will generate multiple anchor boxes with different scales and aspect ratios to cover potential targets of different sizes and shapes; multiple anchor boxes will be generated at each feature map position. Assuming the feature map size is H×W, there are a total of H×W×k anchor boxes (where k is the number of anchor boxes generated at each position); each anchor box is classified into two categories, and RPN outputs a classification score to indicate whether the area contains the target object; the classification loss is usually calculated using the cross entropy loss function, and the goal is to maximize the classification score of the anchor box containing the target; for the anchor box classified as containing the target, RPN outputs a set of regression parameters to adjust the precise position and size of the anchor box to better fit the target object. These regression parameters represent the relative offset between the anchor box and the target object; the non-maximum suppression (NMS) algorithm is used to remove candidate boxes with too much overlap and retain the optimal candidate area. The candidate area finally output is sent to the subsequent classification network for further processing.

[0160] Based on the results of scene recognition, prompt information related to the current scene is generated, and links to relevant documents and operation manuals are extracted from the knowledge base.

[0161] It should be noted that these tips and links will be sent to the command tablet computer through the maintenance operation server so that the command personnel can instantly obtain relevant reference materials and operation suggestions.

[0162] Furthermore, based on scene recognition and key area positioning, the encoding parameters are dynamically adjusted through image similarity analysis based on perceptual hashing to optimize the use of network bandwidth and ensure the clarity of key areas;

[0163] Preferably, perceptual hashing can capture the overall characteristics of an image and help quickly compare the similarities between different frames.

[0164] It should be noted that by comparing the perceptual hash values ​​of consecutive frames, the similarity of the key areas between different frames is calculated. If the similarity is high (i.e., the change is not large), the allocation of encoding resources can be appropriately reduced; if the similarity is low (i.e., the change is large), the encoding resources need to be increased to ensure the clarity of the key areas. Finally, according to the results of the similarity analysis, the encoding parameters of the key areas are dynamically adjusted. For example, the encoding bit rate, frame rate or resolution is adjusted to optimize the use of network bandwidth.

[0165] It should also be noted that while dynamically adjusting encoding parameters to optimize the use of network bandwidth, using scene prompt information and knowledge base links as auxiliary information may affect the display priority of key areas;

[0166] Kalman filter and optical flow method are used to continuously track key areas and adjust encoding parameters in real time to ensure the continuity and stability of key areas in subsequent frames;

[0167] Preferably, a Kalman filter is used to predict the position of the key region in subsequent frames, and an optical flow method is used to capture the motion of pixels within the region.

[0168] It should be noted that the encoding parameters of the key areas are adjusted in real time according to the tracking results. For changes in the location of the area, the system will dynamically adjust the bit rate, frame rate or other encoding parameters during the encoding process to ensure the continuity and stability of the key areas in the video. Finally, the position information and motion data generated during the tracking process are fed back to the model to further optimize the accuracy of detection and tracking. The adaptive adjustment of the model can improve the robustness in complex scenes.

[0169] It should also be noted that the tracking data is fed back to the model to continuously optimize the detection and tracking accuracy. At the same time, the scene recognition results and prompt information serve as background information, which may trigger specific optimization measures or strategy adjustments.

[0170] Secondary encoding and packaging of dynamically adjusted and optimized audio and video data to ensure that key information is not lost during transmission;

[0171] Adaptive transmission rate adjustment ensures data transmission quality when the network environment is unstable. At the same time, scene prompts and knowledge base links are transmitted to the smart command tablet along with audio and video data to help commanders understand and make decisions.

[0172] The intelligent optimization module is used to optimize the key and non-key areas of the operation screen through the edge computing model.

[0173] Furthermore, each frame is segmented and the key areas are identified using the pre-trained Mask R-CNN deep learning model;

[0174] It should be noted that the Mask R-CNN model was chosen for image segmentation because it has good segmentation accuracy in complex scenes; the Sobel operator is used for boundary extraction to enhance the edge details of key areas, which is beneficial for subsequent encoding optimization. By calculating motion vectors to mark priorities, it can ensure that the quality of key areas remains optimal in dynamic scenes.

[0175] The pre-trained deep learning model outputs a binary mask identifying the key region R k and non-critical area R n ;

[0176] The Sobel operator is used to detect the boundaries of the key areas to improve the edge fidelity during subsequent compression. The extracted boundary area is recorded as B k ;

[0177] For each key area B k , by calculating its motion vector in consecutive frames and marking its priority P k The expression is:

[0178]

[0179] Among them, u i and v i are the horizontal and vertical motion vectors of the key area in the i-th frame, N is the number of window frames, and i is the index number representing the frame;

[0180] It should be noted that after segmenting and marking the areas, an edge computing model needs to be designed to optimize the processing of key areas and non-key areas. The core of the model is a dynamic optimization algorithm based on multi-scale analysis. The specific steps are as follows:

[0181] Perform wavelet transform on each frame and decompose it into sub-bands of different frequencies;

[0182] It should be noted that wavelet transform can effectively decompose the different frequency information of an image, allowing us to process the image in detail at different scales. The multi-objective optimization algorithm can find the optimal solution under multiple constraints, ensuring the high quality of key areas while saving bandwidth and computing resources as much as possible.

[0183] Define the subband coefficient after wavelet transform as W j,k , where j represents the scale level and k represents the subband index;

[0184] It should be noted that W j,k It represents the coefficients corresponding to the jth scale level (or resolution level) and the kth subband (usually related to the direction) after wavelet transform. These coefficients describe the detailed information of the image in different frequency subbands.

[0185] For the key area R k Assign high weight α k , for non-critical area R n Assign low weight α n , the weight distribution expression is:

[0186]

[0187] Among them, σ 2 represents the variance of the motion vector in the key area;

[0188] Through a multi-objective optimization algorithm (such as particle swarm optimization algorithm), the coding parameters of each subband are optimized under the premise of meeting the bandwidth and computing resource constraints. The objective function L of the optimization problem is expressed as:

[0189]

[0190] Where j represents the scale level, k represents the subband index, and W j,k is the original subband coefficient after wavelet transform, is the optimized subband coefficient, dA is the area element of the integration area, and λ is a trade-off parameter used to control the optimization weight of the key area and the non-key area;

[0191] It should be noted that Represents the adjustment coefficients at the jth scale level and the kth subband after optimization. These optimized coefficients are used in subsequent quantization and encoding steps to compress image data while retaining key visual information, especially in the key area R k middle.

[0192] It should also be noted that dA is the area element in the integral, which is usually used to represent the area of ​​a small region taken when performing integration in the image or space domain.

[0193] A more specific definition of dA is as follows:

[0194] In the context of 2D image processing, dA usually represents a tiny area element on the image plane. For discrete images, dA can be thought of as the area of ​​a single pixel. In this case, if the pixels of the image are square, then dA can be understood as the area of ​​each pixel, i.e., the area unit of 1 pixel.

[0195] In the formula of the objective function L, the integral symbol and In the key area R k and non-critical area R n This means that we accumulate all the pixels within these regions.

[0196] Specifically, dA is in region R k or R n The result of the integration is the sum of all pixels within a pixel area.

[0197] Since images are made up of discrete pixels, in actual calculations, integration is usually converted into summation. Therefore, dA in the integral formula can be regarded as the area of ​​a single pixel in the summation operation. In this case, the integral in the formula is usually expressed as a weighted summation of all pixel values.

[0198] The specific examples are explained as follows:

[0199] For an image region R k or R n , if you want to calculate the cumulative amount (such as error) in a region, the integral expression in the formula is:

[0200]

[0201] Where (x, y) represents the pixel coordinates in the image, and dA corresponds to the area of ​​each pixel after discretization (usually it can be taken as 1);

[0202] It should also be noted that for continuous images: If the image is continuous, then dA represents the area of ​​a small region on the image. For example, if the image is a continuous function I(x,y), then the integration is performed on the x and y coordinates, and dA corresponds to dx·dy - that is, the product of small increments in the x and y directions.

[0203] For discrete images: In discrete image processing, dA is usually simplified to 1, representing the area of ​​a single pixel. Therefore, the integration operation is often converted into the process of summing all pixel values.

[0204] According to the optimized subband coefficients Dynamically adjust the quantization step size Q k and Q n , quantization step size Q k and Q n The expressions are:

[0205]

[0206] Among them, N k and N n are the number of pixels in the critical area and the non-critical area, respectively;

[0207] Preferably, the formula flexibly controls the quantization accuracy of the image by adaptively adjusting the quantization step size according to the optimized sub-band coefficient distribution. Compared with the fixed step size quantization method, this adaptive quantization can better preserve image details while reducing the amount of data.

[0208] Perform entropy coding (such as arithmetic coding) on ​​the quantized subband coefficients to further compress the data;

[0209] It should be noted that when compressing the quantized sub-band coefficients, the priority of the key area is first considered to ensure that the visual quality of the key area is maximized under limited bandwidth and computing resources.

[0210] The specific steps are as follows: According to the previous priority calculation results, for the key area R k and non-critical area R n Different quantification strategies are used. For the key area R k , choose a smaller quantization step size Q k , in order to retain more detailed information. For non-critical areas R n , use a larger quantization step size Q n , to achieve a higher compression ratio; after priority quantization, the quantized subband coefficients are further compressed using entropy coding (such as arithmetic coding). Since the key area data has been finely quantized, its numerical distribution is more concentrated, so in the entropy coding process, a shorter code length can be used to represent the key area data, thereby improving compression efficiency; through the above-mentioned priority quantization and entropy coding strategies, it is ensured that the data in the key area occupies fewer bits while maintaining a high visual quality, and the data in the non-key area is compressed more efficiently without significantly affecting the overall visual effect.

[0211] During the transmission process, the transmission rate is adjusted in real time according to the network status to ensure that the video data can be transmitted smoothly and minimize packet loss or delay when the bandwidth fluctuates;

[0212] It should be noted that the transmission control adopts the TCP-friendly transmission control protocol (TFRC) to ensure that the transmission quality of key areas is prioritized when the bandwidth fluctuates.

[0213] Preferably, adaptive quantization can dynamically adjust the quantization step size according to the image content, thereby minimizing the amount of data while maintaining image quality. Entropy coding further compresses data, which helps improve transmission efficiency. The TFRC protocol performs well under network fluctuations and is suitable for real-time audio and video transmission.

[0214] After encoding and transmission are completed, in order to evaluate the video quality at the receiving end, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) indicators are used to evaluate the quality of the received video data, thereby verifying whether the optimization strategy for maintaining high quality in key areas during transmission is effective.

[0215] The PSNR expression is defined as:

[0216]

[0217] Among them, MAX is the maximum value of the image data, and MSE is the mean square error;

[0218] Set the trade-off parameter λ to λ min ≤λ≤λ max , which is used to control the optimization proportion of key areas and non-key areas. By adjusting the value of λ, flexible deployment can be achieved in different application scenarios.

[0219] Where: min = 0.1 represents the minimum value of the trade-off parameter, λ max =10 indicates the maximum value of the trade-off parameter.

[0220] When the value of λ is large, the optimization process focuses more on the quality of the key area, which is suitable for scenes with key areas as the main body;

[0221] When the value of λ is small, the optimization process is more balanced and is used in scenarios where both critical areas and non-critical areas are important.

[0222] The data presentation module is used to intelligently command the tablet computer to decode the received audio and video data and restore the compressed audio and video data into playable content.

[0223] Furthermore, the audio and video data stream and the additional scene prompt information and knowledge base link data packet are received from the maintenance operation server through the SRTP protocol;

[0224] Use the AES-256 algorithm and the pre-shared key to decrypt the received encrypted audio and video data stream to obtain the original compressed audio and video data;

[0225] Preferably, the advantage of using the AES-256 algorithm is that it provides a highly secure encryption standard that is widely used in government, business, and financial fields. The 256-bit key length of AES-256 makes it highly resistant to cracking, and can effectively protect the confidentiality of audio and video data during transmission and storage, ensuring that the data will not be accessed or tampered with by unauthorized third parties.

[0226] Use the streaming media parser to analyze the decrypted data stream and extract the video stream information (such as video encoding format, resolution, frame rate, etc.);

[0227] Call the hardware-accelerated H.265 decoder for initialization, read the compressed bit stream frame by frame, perform entropy decoding, inverse quantization, inverse wavelet transform and motion compensation, and restore the YUV image frame;

[0228] Preferably, the advantage of using the H.265 decoder is that it can process video data with higher compression efficiency. Compared with the previous generation H.264 standard, H.265 (also known as HEVC) can reduce the size of video files by about 50% while maintaining the same video quality. This means that in the case of limited bandwidth, the H.265 decoder can provide higher quality video streams while reducing storage and transmission costs.

[0229] Convert YUV image frames to RGB format (suitable for display device playback) and use adaptive histogram equalization to enhance the contrast and visual effects of the image;

[0230] Use the streaming media parser to analyze the decrypted data stream and extract the audio stream information (such as audio encoding format, sampling rate, number of channels, etc.);

[0231] Call the hardware-accelerated AAC decoder for initialization and send the AAC compressed audio frame to the decoder to restore the uncompressed digital audio signal;

[0232] Preferably, the advantage of using an AAC decoder is that the AAC (Advanced Audio Coding) format provides higher sound quality than traditional audio formats such as MP3, especially at lower bit rates. The AAC decoder can take full advantage of this advantage to provide a clearer and richer audio experience, suitable for streaming media, broadcasting, and various consumer electronic devices, achieving better audio compression and playback effects.

[0233] According to the configuration requirements of the playback device, the audio data is resampled and format converted, and a dynamic range compression algorithm is applied to enhance speech clarity and background sound layering.

[0234] Furthermore, after decoding, the timestamps of the audio and video frames are matched to ensure that the audio and video are synchronized during playback, while an adaptive buffer is generated based on network fluctuations and decoding delays;

[0235] It should be noted that the length of the buffer is dynamically adjusted according to the network conditions to ensure smooth playback.

[0236] In the case of a poor network environment or a long decoding time, if the video frame delay in the buffer is too large, some video frames or audio frames will be discarded appropriately to maintain synchronous playback and avoid freezes;

[0237] Using GPU acceleration, the synchronized RGB video frames in the buffer are rendered to the tablet display through OpenGL;

[0238] Convert the audio data in the buffer into an analog signal through a digital-to-analog converter (DAC), and output the analog signal through an audio output terminal of the tablet computer (such as a computer speaker or headphone output);

[0239] Dynamically adjust the video frame rate according to the tablet screen refresh rate to ensure that there is no tearing or freezing during audio and video playback, thus improving the user viewing experience.

[0240] During the video playback, the data packet attached to the maintenance operation server is parsed to extract the scene prompt information, knowledge base link and associated timestamp. The specific steps are as follows:

[0241] The tablet receives the attached data packet from the maintenance operation server, decrypts and decompresses it, and obtains the original data including the scene prompt information, the knowledge base link and the timestamp;

[0242] Parse the data packet, extract and parse the timestamp, prompt information and link fields, and organize them into structured processable data;

[0243] Match the extracted timestamp with the current video playback time, and when the match is successful, mark the corresponding prompt information as to be displayed;

[0244] Arrange the marked prompts in chronological order to ensure that they are displayed at the right time during the video playback and do not interfere with the main content of the video;

[0245] When the video playback reaches the corresponding time point, a prompt box is generated to display relevant prompt information and links.

[0246] It should be noted that users can click on the link to view related content, and if the configuration allows, the video playback may be automatically paused.

[0247] When the timestamp of the audio and video data matches the scene prompt information, a prompt box pops up to ensure that the prompt information and link are displayed synchronously with the current playback content at the appropriate time point;

[0248] It should be noted that a prompt box is dynamically generated above or on the side of the video playback interface, displaying prompt content and knowledge base links related to the current scene; the user can click on the link in the prompt box, and the tablet will call the built-in browser or PDF viewer to open the relevant document or operation manual; if the user clicks on the link during playback, the system can choose to automatically pause the audio and video playback, and continue playback after the user returns.

[0249] For example: When the video plays to the operating steps of a certain device, the system detects that the timestamp of the step matches the timestamp of a scene prompt information. Then the system will display a prompt box at this time point to inform the user what to pay attention to, or provide a knowledge base link that the user can click to read further operation guide.

[0250] The command and control module is used for the smart command tablet to play the decoded audio and video data through speakers and high-definition screens. Command personnel can view the video and communicate with operators in real time.

[0251] The tablet computer receives the audio and video data stream and the attached scene prompt information package from the maintenance operation server through the SRTP protocol, decrypts it using the AES-256 algorithm, and obtains the compressed audio and video data.

[0252] Use the streaming media parser to extract the encoding format, resolution, frame rate of the video stream, and the encoding format, sampling rate, number of channels and other information of the audio stream.

[0253] The hardware-accelerated H.265 decoder is called to decode the video, restore it to YUV image frames, and convert it to RGB format; at the same time, the AAC decoder is used to decode the audio and restore it to uncompressed digital audio signals.

[0254] The decoded audio and video frames are timestamped and adaptive buffers are generated to ensure synchronous playback. The buffer length is dynamically adjusted according to network conditions, and frames with excessive delay are discarded to avoid lags.

[0255] The GPU is used to accelerate the rendering of RGB video frames to the display, while the audio signal is output to the speaker or headphones through the DAC.

[0256] During playback, the attached data packets are parsed, timestamps, prompt information, and knowledge base links are extracted, and matched with the current video playback time.

[0257] When the match is successful, a prompt box is generated in chronological order, displaying relevant information and links. Users can click to view the content, and the system automatically pauses or continues playing according to the configuration.

[0258] While the video is playing and prompt information is displayed, the commander can use the tablet computer to communicate with the operators in real-time audio and video to ensure the accuracy of the operation.

[0259] Users can open related documents by clicking the link in the prompt box. Playback may be automatically paused and resumed after returning. The system dynamically adjusts playback and communication strategies based on network conditions and user interactions to ensure a smooth experience.

[0260] In summary, the present invention introduces artificial intelligence and edge computing technologies, performs real-time identification and dynamic optimization of key areas in the operation scene based on a deep learning model, combines Kalman filtering and optical flow method to continuously track the motion trajectory of key areas, and dynamically adjusts encoding parameters to optimize bandwidth usage, which significantly improves the real-time and reliability of audio and video data transmission, and ensures high-quality maintenance operations in complex network environments.

[0261] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of a wind turbine maintenance operation system based on intelligent dynamic multimedia streaming is provided.

[0262] The experimental environment includes intelligent communication terminals, maintenance operation servers, intelligent command tablet computers, and a network architecture that supports edge computing. The audio and video data used in the experiment comes from the simulation environment of the actual operation scene, which includes a variety of complex background sounds and dynamic operation pictures.

[0263] First, during the experimental preparation stage, the intelligent communication terminal collected audio and video data of the operation site in real time through microphones and cameras. During the experiment, the collected data included a variety of background noises (such as the sound of equipment operation) and audio data of operation instructions, while capturing video images of operators in different operation steps. Audio and video data were collected at a rate of 30 frames per second (FPS) and encoded in real time into H.265 format.

[0264] Next, the intelligent communication terminal transmits the encoded audio and video data to the maintenance operation server through the Wi-Fi 6 network. After receiving the data, the maintenance operation server first uses the pre-trained Mask R-CNN deep learning model to segment each frame of the image and identify key areas (such as the operator's hand movements, the equipment's operating interface) and non-key areas (such as background walls, etc.). For the identified key areas, the Sobel operator is further used to detect their boundaries, extract the boundary areas, and determine their priority P by calculating the motion vectors of the key areas in consecutive frames.k .

[0265] After determining the priority of the key area, the job server performs wavelet transform on each frame of the image and decomposes it into sub-bands of different frequencies. Different weights α are assigned to key areas and non-key areas according to their priorities. k and α n Then, through the multi-objective optimization algorithm, the coding parameters of each sub-band are optimized under the premise of meeting the network bandwidth and computing resource constraints, and the dynamic adjustment of the image is completed.

[0266] Finally, the dynamically adjusted audio and video data is forwarded to the intelligent command tablet. The command tablet decodes the received audio and video data and restores it to playable content. The decoded data is played through a high-definition screen and speakers for command personnel to view and analyze in real time. During the playback process, the experiment evaluates the quality of the video data through the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) indicators, and records the data transmission delay and bandwidth utilization.

[0267] The details are shown in Table 1 below:

[0268] Table 1 Video optimization experiment results

[0269]

[0270] Through the analysis of experimental data, it can be clearly seen that the edge computing model has significant advantages in audio and video data optimization. First, the two indicators of PSNR and SSIM show a significant improvement in video quality. When the PSNR value of the benchmark video is 35.2dB, the PSNR values ​​are increased to 38.5dB, 42.1dB and 45.6dB respectively through optimization processing of different λ values, indicating that the signal quality of the video has been significantly enhanced. At the same time, the SSIM value is increased from the baseline 0.86 to 0.90, 0.93 and 0.95, indicating that the structural similarity of the video is closer to the original scene after optimization.

[0271] Secondly, the performance of bandwidth utilization and transmission delay also demonstrates the effectiveness of optimization. Without optimization, the bandwidth utilization is as high as 20Mbps and the transmission delay is as high as 150ms. After high-λ optimization, the bandwidth utilization is reduced to 9.5Mbps and the transmission delay is reduced to 95ms, indicating that the optimization algorithm significantly reduces bandwidth usage and shortens delay while ensuring video quality, which helps to improve the real-time performance and efficiency of the system.

[0272] In addition, the frame rate comparison between key areas and non-key areas also reflects the accuracy of the optimization algorithm. Under high λ optimization conditions, the frame rate of key areas is maintained at 22FPS, while that of non-key areas is reduced to 18FPS. This differentiated processing of frame rates effectively allocates more resources to key areas, ensuring the clarity and smoothness of key operation scenes, while non-key areas appropriately reduce resource allocation, maximizing overall resource utilization efficiency.

[0273] Finally, the comparison of data packet loss rate further verifies the reliability of optimization. Without optimization, the packet loss rate is 1.0%, while after high λ optimization, the packet loss rate is reduced to 0.1%. This shows that the optimization algorithm also has obvious advantages in combating network instability and improves the stability of data transmission.

[0274] In summary, this experiment confirmed the effectiveness and innovation of the edge computing model in audio and video data optimization by comparing various parameters under different optimization conditions. Compared with the prior art, the present invention not only achieved significant improvements in video quality, but also showed excellent performance in bandwidth utilization, transmission delay, frame rate control, and data packet loss rate. These experimental results fully demonstrate the technical innovation and practical value of the present invention, and provide strong support for the efficient integration of intelligent communication and command systems.

[0275] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A wind turbine maintenance system based on intelligent dynamic multimedia streaming, characterized in that: include, Data acquisition module: The intelligent communication terminal collects audio and video data on site in real time through microphones and cameras, capturing ambient sound, voice commands, and images of the equipment operation process; Data transmission module: The intelligent communication terminal encodes the collected audio and video data and transmits them to the maintenance operation server via wireless communication; Intelligent adjustment module: The maintenance operation server uses artificial intelligence architecture to identify and locate key areas of the operation screen, and dynamically adjusts the audio and video data in the identified key areas; Intelligent optimization module: optimizes the key and non-key areas of the operation screen through edge computing models; Data forwarding module: forwards dynamically adjusted audio and video data from the maintenance operation server to the intelligent command tablet computer; Data presentation module: The intelligent command tablet decodes the received audio and video data and restores the compressed audio and video data into playable content; Command and control module: The intelligent command tablet plays decoded audio and video data through speakers and high-definition screens, and commanders view videos and communicate with operators in real time.

2. The wind turbine maintenance system based on intelligent dynamic multimedia stream according to claim 1, characterized in that: The real-time collection of on-site audio and video data, capturing environmental sounds, voice commands and images of the device operation process, is specifically performed as follows: Use 4K HDR cameras and omnidirectional condenser microphones, deployed at high altitudes to cover the work area; Set the camera frame rate to 30fps, the resolution to 3840x2160, the audio sampling rate to 48kHz, and the quantization bit to 24-bit; Add timestamps to each frame of video and audio data, remove video noise using Gaussian filters, and optimize brightness and contrast through histogram equalization; The audio is de-noised using spectral subtraction and the echo is eliminated using an adaptive filtering algorithm.

3. The wind turbine maintenance system based on intelligent dynamic multimedia stream according to claim 2, characterized in that: The intelligent communication terminal encodes the collected audio and video data and transmits them to the maintenance operation server via wireless communication. The specific steps are: After data collection, ensure the synchronization of audio and video, and match audio and video frames through timestamps; Use HEVC encoding to compress the synchronized audio and video data; After the video is encoded, the audio data is compressed through AAC, using sub-band filtering and perceptual quantization to separate and compress the audio bands; The encoded audio and video data stream is encrypted via AES-256, and the encrypted data is transmitted via the SRTP protocol, and the transmission rate is dynamically adjusted according to network conditions.

4. The wind turbine maintenance system based on intelligent dynamic multimedia stream according to claim 3 is characterized in that: The audio data is compressed by AAC, and the audio frequency band is separated and compressed by sub-band filtering and perceptual quantization. The specific steps are: Sampling the collected audio data to convert the continuous audio signal into a discrete digital signal; Divide the discrete digital signal into multiple frames, each frame contains a short audio sample; After the frame processing, the audio signal of each frame is decomposed into multiple sub-bands by applying a polyphase filter bank, each sub-band corresponds to a different frequency range; The sub-band filtered frequency band signals are input into the perception model, which accurately analyzes the masking effect of each frequency band signal based on the sensitivity of the human ear to different frequencies and volumes; According to the analysis results of the perceptual model, perceptual quantization is performed on the audio data of each sub-band; The quantized data is further compressed through Huffman coding; Recombine and encapsulate the encoded sub-band data into an AAC frame; Output the encapsulated AAC frames as a continuous audio data stream.

5. The wind turbine maintenance system based on intelligent dynamic multimedia stream according to claim 4, characterized in that: The specific steps of identifying and locating the key areas of the operation screen are as follows: The maintenance operation server synchronously processes the received audio and video data, extracts each frame of the image, and decodes it into an RGB matrix; Based on the synchronized frames, the pre-trained Faster R-CNN deep learning model is loaded, and each frame image is input into the convolutional neural network for feature extraction; The extracted feature vectors are analyzed by the scene recognition model to classify and identify the current operation scene; Based on scene recognition, the region proposal network is used to generate candidate key regions, classify and locate them, and finally output key region objects and their locations. Based on the results of scene recognition, prompt information related to the current scene is generated, and links to relevant documents and operation manuals are extracted from the knowledge base.

6. The wind turbine maintenance system based on intelligent dynamic multimedia stream according to claim 5, characterized in that: The specific steps of dynamically adjusting the audio and video data in the identified key area are as follows: Based on scene recognition and key area positioning, the encoding parameters are dynamically adjusted to optimize the use of network bandwidth through image similarity analysis based on perceptual hashing; Use Kalman filter and optical flow method to continuously track key areas and adjust encoding parameters in real time; The dynamically adjusted and optimized audio and video data are re-encoded and packaged, and the scene prompts and knowledge base links are transmitted to the smart command tablet together with the audio and video data through adaptive transmission rate adjustment.

7. The wind turbine maintenance system based on intelligent dynamic multimedia stream according to claim 6, characterized in that: The specific steps of optimizing the key and non-key areas of the operation screen by using the edge computing model are as follows: Segment each frame and use the pre-trained Mask R-CNN deep learning model to identify key areas; The pre-trained deep learning model outputs a binary mask identifying the key region R k and non-critical area R n ; The Sobel operator is used to detect the boundary of the key area, and the extracted boundary area is recorded as B k ; For each key area B k , by calculating its motion vector in consecutive frames and marking its priority P k The expression is: Among them, u i and v i are the horizontal and vertical motion vectors of the key area in the i-th frame, N is the number of window frames, and i is the index number representing the frame; Perform wavelet transform on each frame and decompose it into sub-bands of different frequencies; Define the subband coefficient after wavelet transform as W j,k , where j represents the scale level and k represents the subband index; For the key area R k Assign high weight α k , for non-critical area R n Assign low weight α n , the weight distribution expression is: Among them, σ 2 represents the variance of the motion vector in the key area; Through the multi-objective optimization algorithm, the coding parameters of each subband are optimized under the premise of meeting the bandwidth and computing resource constraints. The objective function L of the optimization problem is expressed as: Where j represents the scale level, k represents the subband index, and W j，k is the original subband system after wavelet transform The expressions are: Among them, N k and N n are the number of pixels in the critical area and the non-critical area, respectively; Perform entropy coding on the quantized subband coefficients to further compress the data; During the transmission process, the transmission rate is adjusted in real time according to the network status; After encoding and transmission, the quality of the received video data is evaluated using peak signal-to-noise ratio and structural similarity indicators; Set the trade-off parameter λ to λ min ≤λ≤λ max ; When the value of λ is large, the optimization process focuses more on the quality of the key area, which is suitable for scenes with key areas as the main body; When the value of λ is small, the optimization process is more balanced and is used in scenarios where both critical areas and non-critical areas are important.

8. The wind turbine maintenance system based on intelligent dynamic multimedia stream according to claim 7, characterized in that: The smart command tablet computer decodes the received audio and video data, and the specific steps are: Receive audio and video data streams and additional scene prompt information and knowledge base link data packets from the maintenance operation server through the SRTP protocol; Use the AES-256 algorithm and the pre-shared key to decrypt the received encrypted audio and video data stream to obtain the original compressed audio and video data; Use the streaming media parser to analyze the decrypted data stream and extract the video stream information; Call the hardware-accelerated H.265 decoder for initialization, read the compressed bit stream frame by frame, perform entropy decoding, inverse quantization, inverse wavelet transform and motion compensation, and restore the YUV image frame; Convert the YUV image frame to RGB format by adaptive histogram equalization; Use the streaming media parser to analyze the decrypted data stream and extract the audio stream information; Call the hardware-accelerated AAC decoder for initialization and send the AAC compressed audio frame to the decoder to restore the uncompressed digital audio signal; According to the configuration requirements of the playback device, the audio data is resampled and format converted, and a dynamic range compression algorithm is applied to enhance speech clarity and background sound layering.

9. The wind turbine maintenance system based on intelligent dynamic multimedia stream according to claim 8, characterized in that: The specific steps of restoring the compressed audio and video data to playable content are as follows: After decoding, the timestamps of the audio and video frames are matched, and an adaptive buffer is generated based on network fluctuations and decoding delays; Using GPU acceleration, the synchronized RGB video frames in the buffer are rendered to the tablet display through OpenGL; The audio data in the buffer is converted into an analog signal through a digital-to-analog converter and outputted through an audio output terminal of the tablet computer; Dynamically adjust the video playback frame rate according to the tablet screen refresh rate; While the video is being played, the data packets sent from the maintenance operation server are parsed; Extract scene prompt information, knowledge base link and associated timestamp, and when the timestamp of audio and video data matches the scene prompt information, a prompt box pops up.

10. The wind turbine maintenance system based on intelligent dynamic multimedia stream according to claim 9, characterized in that: When the timestamp of the audio and video data matches the scene prompt information, a prompt box pops up. The specific steps are: The tablet receives the attached data packet from the maintenance operation server, decrypts and decompresses it, and obtains the original data including the scene prompt information, the knowledge base link and the timestamp; Parse the data packet, extract and parse the timestamp, prompt information and link fields, and organize them into structured processable data; Match the extracted timestamp with the current video playback time, and when the match is successful, mark the corresponding prompt information as to be displayed; Queue the marked reminder messages in chronological order; When the video playback reaches the corresponding time point, a prompt box is generated to display relevant prompt information and links.

Images