Large model intention recognition method and system based on multi-modal perception

Abstract

The invention discloses a large model intention recognition method and system based on multi-modal perception, is applied to the technical field of artificial intelligence and communication, and aims at solving the problems that an existing system is poor in robustness and low in accuracy when processing multi-modal data which is from a complex physical environment and unstable in signal quality. The method comprises the following steps: acquiring multi-modal data such as field images, audios and texts related to communication network facility operation and maintenance; performing signal quality evaluation on each mode, and calculating a reliability score of each mode; when a low-reliability mode is identified, the characteristics of a high-reliability mode are enhanced through a cross-mode compensation model by using the characteristics of the high-reliability mode; and finally, according to the reliability score of each modal, dynamically determining a fusion weight, carrying out weighted fusion, and generating a high-robustness comprehensive feature vector for representing an operation and maintenance state, and through active evaluation, intelligent compensation and dynamic fusion, the data processing stability and analysis precision of the system in a complex environment are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a large model intention recognition method and system based on multi-modal perception. BACKGROUND

[0002] With the popularization of the fifth generation mobile communication technology 5G network and optical fiber network, the scale of communication network facilities such as mobile communication base stations and optical fiber transmission line equipment is increasingly large, which brings challenges to their operation and maintenance guarantee. In the field of operation and maintenance, the introduction of artificial intelligence AI technology has become a trend. For example, the AI system analyzes the image, audio information and electronic work order data provided by the on-site operation and maintenance personnel to assist in fault diagnosis, state evaluation and resource scheduling.

[0003] The existing AI operation and maintenance system usually adopts multi-modal fusion technology, that is, it comprehensively processes multiple data sources such as images, audios, texts and the like. However, in actual application, these systems have certain deficiencies. First, the existing system lacks the ability to distinguish data quality. The communication network operation and maintenance site environment is complex, and the image data may be of poor quality due to light, obstruction or jitter, the audio data may be difficult to identify due to noise, and the text work order may have problems such as non-standard description or information loss. The existing system often processes high-quality and low-quality data together, which causes low-quality data to interfere with model judgment. Secondly, most systems use fixed weights or simple splicing methods for multi-modal feature fusion. This static fusion strategy cannot adapt to the dynamic changes of input data quality. For example, when the image quality is high and the voice quality is poor, the system will still give the voice feature a fixed weight, affecting the stability and reliability of the final result. Thirdly, when the data quality of a certain modality is determined to be too low, some systems may directly discard the information of that modality. This approach avoids some noise interference, but also causes the loss of effective information, reducing the system's ability to conduct comprehensive analysis.

[0004] Therefore, when processing multi-modal data of unstable quality, the poor robustness and low reliability caused by the inability to adapt to changes in data quality are technical problems that restrict the application of AI technology.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] Therefore, the application provides a large model intention recognition method and system based on multi-modal perception to overcome the problem of low processing result accuracy in complex real environment caused by the inability of multi-modal artificial intelligence system to perceive and adapt to the dynamic changes of input data quality, thereby improving the robustness of the system.

[0007] The embodiment of the application provides a large model intention recognition method based on multi-modal perception, which is executed by a computer device and comprises the following steps: Obtaining multi-modal data related to communication network facility operation and maintenance, wherein the multi-modal data comprises field image data, field audio data and text work order data; For each modality in the multi-modal data, calculating a reliability score representing the signal quality of the modality data; Based on the reliability score, identifying a low-reliability modality whose reliability score is lower than a preset compensation threshold, and using the features of one or more high-reliability modalities to enhance the features of the low-reliability modality through a pre-trained cross-modal compensation model to obtain compensated features; Determining a fusion weight for subsequent processing according to the reliability score, and performing weighted fusion on the features of each modality and the compensated features based on the fusion weight to generate a comprehensive feature vector representing the operation and maintenance state of the communication network facility.

[0008] In some optional embodiments, after obtaining the multi-modal data, the method further comprises: Adjusting the resolution of the field image data to a first preset resolution; and Adjusting the sampling rate of the field audio data to a first preset sampling rate.

[0009] In some optional embodiments, the reliability score of the field image data is calculated by: Calculating the structural similarity index between the field image data and a standard reference image as part of the reliability score.

[0010] In some optional embodiments, the reliability score of the field image data is calculated by: Analyzing the brightness histogram of the field image data to generate a score representing the lighting condition, and combining the score with the structural similarity index to determine the reliability score of the field image data.

[0011] In some optional embodiments, the reliability score of the field audio data is calculated by: Using a perceptual speech quality evaluation algorithm to calculate the intelligibility of the field audio data as part of the reliability score.

[0012] In some optional embodiments, the calculating the reliability score of the live audio data further comprises: calculating a signal-to-noise ratio of the live audio data, and combining the signal-to-noise ratio with the intelligibility to determine the reliability score of the live audio data.

[0013] In some optional embodiments, the cross-modal compensation model is a neural network-based feature converter; the step of enhancing the feature of the low-reliability modality comprises, when the live image data is the low-reliability modality: extracting keywords representing the fault type from the text work order data, and inputting the keywords into the feature converter to generate compensation features for enhancing the color or texture aspect of the live image data.

[0014] In some optional embodiments, the method further comprises: inputting the comprehensive feature vector into a pre-trained fault classification model to output a classification result indicating the fault type of the communication network facility and a corresponding confidence score.

[0015] In some optional embodiments, the method further comprises: generating an electronic diagnostic report based on the classification result only when the confidence score is higher than a preset output threshold.

[0016] In some optional embodiments, the communication network facility is a mobile communication base station or an optical fiber transmission line device.

[0017] Embodiments of the present application provide a large model intent recognition system based on multi-modal perception, comprising: a data acquisition module configured to acquire multi-modal data related to communication network facility operation and maintenance, the multi-modal data including live image data, live audio data, and text work order data; a quality evaluation module configured to calculate, for each modality in the multi-modal data, a reliability score representing the signal quality of the modality data; a feature compensation module configured to identify, based on the reliability scores, a low-reliability modality whose reliability score is lower than a preset compensation threshold, and to enhance the feature of the low-reliability modality using the features of one or more high-reliability modalities through a pre-trained cross-modal compensation model to obtain a compensated feature; a dynamic fusion module configured to determine a fusion weight for subsequent processing according to the reliability scores, and to perform weighted fusion on the features of each modality and the compensated features based on the fusion weight to generate a comprehensive feature vector representing the operation and maintenance state of the communication network facility.

[0018] In some optional embodiments, a data preprocessing module is further included, configured to adjust the resolution of the live image data to a first preset resolution and adjust the sampling rate of the live audio data to a first preset sampling rate before the quality assessment module processes.

[0019] In some optional embodiments, in the feature compensation module, the cross-modal compensation model is a neural network-based feature converter configured to, when the live image data is of a low reliability modality, receive keywords representing the fault type extracted from the text ticket data, and output compensation features for enhancing the color or texture aspects of the live image data.

[0020] In some optional embodiments, a diagnosis output module is further included, configured to: receive the integrated feature vector and input it into a pre-trained fault classification model to output a classification result indicating the fault type of the communication network facility and a corresponding confidence score; and generate an electronic diagnosis report based on the classification result only when and if the confidence score is higher than a preset output threshold.

[0021] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application.

[0022] The multi-modal perception-based large model intent recognition method and system of the present application has the following beneficial effects: The present method quantitatively evaluates the quality of multi-modal data and dynamically adjusts the fusion weight, thereby improving the stability and accuracy of the system under the interference of low-quality data. The cross-modal feature compensation mechanism can repair the low-reliability modality using the high-reliability modality without discarding information, thereby minimizing information loss and improving the accuracy of subsequent analysis. It can adapt to the characteristics of uneven data quality in complex environments such as communication operation and maintenance, ensuring reliable operation of the system in actual scenarios. By adding confidence gating to the final output result, the system ensures that it only gives conclusions when it is confident enough, avoiding the risk of misjudgment. BRIEF DESCRIPTION OF DRAWINGS

[0023] Other features, objects, and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings.

[0024] Figure 1 is a flowchart of a multi-modal perception-based large model intent recognition method according to an embodiment of the present application; Figure 2 is a structural schematic diagram of a multi-modal perception-based large model intent recognition system according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined. Therefore, the actual execution order may change depending on the specific circumstances.

[0028] Multimodal data fusion aims to integrate information from various sources to achieve a more comprehensive and accurate understanding of a target object. In the field of communication network infrastructure operation and maintenance, multimodal data such as images, audio, and text contain complementary information about equipment status. However, in practical applications, each modality of data is often affected by noise, resulting in inconsistent quality. Directly fusing low-quality data will reduce the accuracy of the fusion results. Therefore, it is necessary to evaluate the quality of each modality of data and adjust the contribution of each modality in the fusion process based on the quality evaluation results. For low-quality data, feature compensation can be performed using information from high-quality modalities, thereby improving the usability of low-quality modalities before fusion. Weighted fusion is a multimodal fusion method that linearly combines features according to the weights of each modality, with the magnitude of the weights reflecting the properties of the corresponding modality. By reasonably setting the weights, high-quality modalities can play a greater role in the fusion results, thereby improving the robustness and accuracy of the fusion results.

[0029] like Figure 1 As shown, this embodiment of the invention provides a large-scale intent recognition method based on multimodal perception. This method is executed by a computer device and used for communication network infrastructure operation and maintenance. The method includes steps such as data acquisition, reliability score calculation, low-reliability modality identification and feature compensation, and dynamic weighted fusion, which can improve the accuracy of data processing in complex environments. It includes the following steps: S100、data acquisition step, acquire multi-modal data related to communication network facility operation and maintenance, the multi-modal data includes field image data, field audio data and text work order data. For example, the computer device receives the JPEG format image taken by the operation and maintenance personnel on the spot of the mobile communication base station, the recorded WAV format audio and the submitted JSON format text work order through the network interface. Among them, the image data records the running state of the base station equipment, such as the color of the equipment indicator light; the audio data contains the voice description of the operation and maintenance personnel on the spot and the environmental noise, such as the running sound of the equipment fan; the text work order data records the alarm code and the brief fault description information. Optionally, the acquisition of image data can be realized through real-time video stream connected to the base station camera, the audio data can be realized through real-time audio stream connected to the base station microphone, and the text work order data can be realized by reading the historical work order data stored in the database.

[0030] S200, a reliability score calculation step, for each modality in the multi-modal data, calculate a reliability score representing the quality of the data signal of this modality. For example, for the on-site image data, use image processing algorithms to calculate the sharpness, brightness, contrast, etc. of the image, which can reflect the quality of the image. Specifically, the Laplacian variance of the image can be used to evaluate the sharpness, the higher the variance, the clearer the image; the average pixel value of the image can be calculated to evaluate the brightness, and the average pixel value that is too high or too low represents overexposure or overdarkness of the image; the contrast parameter of the gray level co-occurrence matrix of the image can be calculated to evaluate the contrast of the image. After normalizing these indicators, the reliability score of the image is obtained by weighted summation, for example, the weights of sharpness, brightness, and contrast are 0.6, 0.2, and 0.2, respectively. For on-site audio data, speech quality evaluation algorithms can be used to calculate the signal-to-noise ratio, loudness, etc. of the audio. Specifically, the PESQ algorithm defined in the ITU-T P.862 standard can be used to evaluate the speech quality, which can output a PESQ score, the higher the score, the better the speech quality; the ratio of the energy of the audio signal to the noise energy can be calculated to evaluate the signal-to-noise ratio. After normalizing these indicators, the reliability score of the audio is obtained by weighted summation, for example, the weights of the PESQ score and the signal-to-noise ratio are 0.7 and 0.3, respectively. For text work order data, indicators such as format specification, word integrity, information accuracy, etc. of the work order can be analyzed, for example, whether the work order contains necessary fields such as alarm code, device ID, fault description, etc. can be checked; whether the words in the work order are complete, such as "power failure", "line interruption", etc. can be checked; the information in the work order can be compared with historical work order data to check whether the information is accurate. According to these indicators, a scoring rule is manually set to score the text work order data to obtain the reliability score of the text work order. Optionally, the sharpness of the image can be evaluated by calculating the information entropy of the image, the signal-to-noise ratio of the audio can be evaluated by calculating the spectral flatness of the audio, and the reliability score of the text work order can be automatically evaluated by training a text classification model.

[0031] S300, low-reliability modality identification and feature compensation step, based on the reliability score, identify low-reliability modalities whose reliability scores are lower than a preset compensation threshold, and use the features of one or more high-reliability modalities to enhance the features of the low-reliability modalities through a pre-trained cross-modality compensation model to obtain compensated features. For example, set the preset compensation threshold to 0.6, if the reliability score of the image is 0.5, the reliability score of the audio is 0.4, and the reliability score of the text work order is 0.8, then the image and the audio are identified as low-reliability modalities, and the text work order is identified as a high-reliability modality. Then, use the features of the text work order to enhance the features of the image and the audio through the cross-modality compensation model. The cross-modality compensation model can use a deep neural network model, such as a Transformer model. The model takes the features of the text work order as input and predicts the missing features of the image and the audio. Specifically, the text work order contains the word "power failure", and the model can predict that the image should contain the feature of the power indicator light being off, and the audio should contain the feature of the power device stopping running. Then, the model-predicted features are fused with the original image and audio features to obtain the compensated features. For example, for the image, the model-predicted feature map of the indicator light being off can be fused with the original image feature map; for the audio, the model-predicted acoustic feature spectrum of the device stopping running can be fused with the original audio acoustic feature spectrum. Optionally, the cross-modality compensation model can use a generative adversarial network model, which takes the features of the text work order as a condition to generate images and audios consistent with the text description, thereby achieving compensation for low-quality images and audios.

[0032] S400, a dynamic weighting fusion step, determines a fusion weight for subsequent processing according to the reliability score, and weights and fuses the features of each modality and the compensated features based on the fusion weight to generate a comprehensive feature vector for representing the operation and maintenance state of the communication network facility. For example, the reliability scores of each modality are normalized using a Softmax function to obtain the fusion weight of each modality. Specifically, assuming that the reliability scores of image, audio and text work order are 0.5, 0.4 and 0.8 respectively, the fusion weights obtained after normalization by the Softmax function are 0.2, 0.1 and 0.7 respectively. Then, the feature vectors of each modality are multiplied by the corresponding fusion weight, and the weighted feature vectors of all modalities are added to obtain a comprehensive feature vector. The comprehensive feature vector can fuse the information of each modality, and can dynamically adjust the influence of each modality on the final result according to the reliability degree of each modality. For example, if the reliability of the text work order is high, the features of the text work order will occupy a higher weight in the comprehensive feature vector, thereby improving the accuracy of the final result. Optionally, an attention mechanism can be used to dynamically adjust the fusion weight of each modality. The attention mechanism can automatically learn the weight of each modality according to the nature of the modality features, thereby achieving more accurate fusion.

[0033] Through the above steps, the embodiment can adaptively process multi-modal data, and can dynamically adjust the influence of each modality on the final result according to the quality of each modality data, improve the accuracy of data processing in complex environments, and thus can more accurately represent the operation and maintenance state of the communication network facility.

[0034] In some embodiments, after acquiring the multi-modal data, the on-site image data and the on-site audio data are standardized pre-processed to ensure that the input data of the subsequent data processing flow has a consistent format and range, reduce the computational complexity, and improve the model processing accuracy. For the on-site image data, the specific way to adjust the resolution to the first preset resolution is that no matter what the original resolution of the on-site captured image is, such as 1920x1080 pixels, 1280x720 pixels or other resolutions, the data pre-processing module will uniformly adjust the resolution to the first preset resolution through an image scaling algorithm. The first preset resolution can be set according to actual application requirements, for example, set to 512x512 pixels. The image scaling algorithm can be implemented in multiple ways, such as a bicubic interpolation algorithm, a nearest neighbor interpolation algorithm or a Lanczos resampling algorithm. The bicubic interpolation algorithm can provide good image quality by considering the weighted average of 16 pixels around the source pixel, reducing distortion in the image scaling process. The algorithm first determines the corresponding position of the target image pixel in the original image, then uses a cubic polynomial function to calculate the weight of the surrounding pixels, and finally takes the weighted average value of these pixels as the value of the target image pixel. Optionally, if the original image resolution is higher than the first preset resolution, a downsampling method is used for scaling, such as using an area sampling algorithm; if the original image resolution is lower than the first preset resolution, an upsampling method is used for scaling, such as using a deconvolution network for super-resolution reconstruction. The data pre-processing module can use an image processing library, such as OpenCV, to implement the image resolution adjustment function. OpenCV provides a resize function that can specify the interpolation algorithm type, thereby implementing image scaling with different quality and speed. After the image resolution adjustment is completed, all image data will have a uniform size, facilitating subsequent feature extraction and analysis. For the on-site audio data, the specific way to adjust the sampling rate to the first preset sampling rate is that no matter what the original sampling rate of the on-site recorded audio is, such as 16kHz, 48kHz or other sampling rates, the data pre-processing module will uniformly adjust the sampling rate to the first preset sampling rate through an audio resampling algorithm. The first preset sampling rate can be set according to actual application requirements, for example, set to 44.1kHz. There are also multiple ways to implement the audio resampling algorithm, such as linear interpolation, polyphase filter bank or sinc function interpolation. The polyphase filter bank resampling algorithm first decomposes the original audio signal into multiple sub-band signals, then performs interpolation or decimation on each sub-band signal, and finally combines the reconstructed sub-band signals into a new audio signal. This algorithm can effectively reduce the aliasing phenomenon in the resampling process and improve the quality of the resampled audio.Optionally, if the original audio sampling rate is higher than the first preset sampling rate, down-sampling is used for re-sampling; if the original audio sampling rate is lower than the first preset sampling rate, up-sampling is used for re-sampling. The data preprocessing module can use an audio processing library, such as Librosa, to implement the audio re-sampling function. Librosa provides a resample function that can specify the re-sampling algorithm type, thereby achieving audio re-sampling of different quality and speed. After the audio sampling rate adjustment is completed, all audio data will have a uniform sampling rate, facilitating subsequent feature extraction and analysis.

[0035] By adjusting the resolution of the live image data and the sampling rate of the live audio data to the preset values, the embodiment ensures that the input data of the multi-modal data processing flow has a consistent format, simplifies the subsequent data processing flow, reduces the computational complexity, and lays a foundation for subsequent quality evaluation, feature compensation, and dynamic fusion steps.

[0036] In some embodiments, to quantify the quality of the live image data, the structural similarity index SSIM is calculated. First, a standard reference image needs to be determined. This standard reference image can be one or more high-quality images pre-stored in the system, which represent the appearance that the communication network facilities, such as base station equipment or fiber line equipment, should present in ideal conditions. For example, for a certain type of indicator light panel in a base station cabinet, a clear photo under good lighting conditions can be taken as a reference image. Alternatively, a gallery containing multiple reference images can be established for different equipment types and states.

[0037] The specific process of calculating SSIM is as follows: 1. Preprocessing: Adjust both the live image data and the reference image to a uniform size. If the resolution of the live image is different from that of the reference image, a bicubic interpolation algorithm can be used to scale it to the same resolution as the reference image, e.g., 512x512 pixels. This step ensures that the subsequent SSIM calculation is performed on images of the same size. In addition, it is necessary to convert color images to grayscale images to simplify the calculation.

[0038] 2. Calculate local statistics: The idea of SSIM is to compare the structural similarity of two images in local regions, so it is necessary to calculate the mean, variance, and covariance of each local window. The size of the local window can be chosen according to the actual situation, for example, a window of 11x11 pixels can be chosen. When calculating the mean, add all the pixel values in the window and divide by the total number of pixels in the window. When calculating the variance, first calculate the square of the difference between each pixel value and the mean, then add all the squared differences, and divide by the total number of pixels in the window. When calculating the covariance, calculate the product of the difference between the pixel value of the two images in the corresponding window and the respective mean, then add all the products, and divide by the total number of pixels in the window.

[0039] 3. Calculate SSIM index: According to the local statistics, calculate the SSIM index of each local window. The calculation formula of SSIM index is as follows: where, and represent the mean of the live image and the reference image in the current window, is the average of all pixel values in the current local window (such as a window of 11x11 pixels) of the live image data, is the average of all pixel values in the local window of the same position corresponding to the live image of the standard reference image. and represent the variance of the live image and the reference image in the current window, represents the covariance of the live image and the reference image in the current window, and are two constants to prevent the denominator from being zero. and The value of can be adjusted according to the range of pixel values of the image.

[0040] 4. Average SSIM index: Average the SSIM index of all local windows to get the final SSIM index. This average SSIM index represents the overall structural similarity of the live image and the reference image. The value range of the SSIM index is [-1, 1], the higher the value, the higher the similarity. Generally, SSIM index above 0.8 is considered to have good similarity. The calculated SSIM index will be part of the reliability score of the live image data. For example, the SSIM index can be directly used as the reliability score, or it can be combined with other quality indicators to get the final reliability score.

[0041] Alternatively, different image similarity metrics can be used instead of SSIM. For example, peak signal-to-noise ratio (PSNR), mean squared error (MSE), or variants of structural similarity index (SSIM) such as multi-scale SSIM (MS-SSIM) can be used. MS-SSIM calculates the structural similarity of images at different scales, which can more comprehensively evaluate the quality of images. In some alternative embodiments, different reference images can be selected according to different device types and operation scenarios. For example, for fiber line devices, a clear and undamaged photo of an optical cable joint can be taken as a reference image. For mobile communication base stations, a photo of an antenna connector in good condition and without corrosion can be taken as a reference image.

[0042] By calculating the structural similarity index between the field image data and the standard reference image, the quality of the image can be effectively evaluated, and the index can be used as the basis for the reliability score of the image data. This helps the system to identify low-quality image data and take appropriate measures, such as enhancing image features through cross-modal compensation, to improve the robustness and accuracy of the system.

[0043] In some embodiments, calculating the reliability score of the field image data further comprises: analyzing the brightness histogram of the field image data to generate a score representing the lighting condition, and combining the score with the structural similarity index to determine the reliability score of the field image data.

[0044] First, after the on-site image data is received by the quality assessment module, the image processing unit uses an image processing library, such as OpenCV, to perform analysis of the brightness histogram. Specifically, the image is first converted to a grayscale image. The conversion formula can be: Gray = 0.299 * Red + 0.587 * Green + 0.114 * Blue. Then, the brightness histogram of the grayscale image is calculated, which counts the number of pixels in the image at each brightness level, with the brightness level range typically being 0 to 255. Then, the brightness histogram is analyzed to generate a score representing the lighting conditions. One implementation method is to calculate the brightness mean. The brightness mean is obtained by summing the brightness values of all pixels and dividing by the total number of pixels. If the brightness mean is below a preset lower limit, for example 50, it is considered that the overall image is dark and the lighting is insufficient; if the brightness mean is above a preset upper limit, for example 200, it is considered that the overall image is overexposed and the lighting is too strong. Based on the brightness mean, a lighting score can be generated, for example, the lighting score can be defined as a piecewise linear function: when the brightness mean is less than 50, the lighting score is equal to the brightness mean divided by 50; when the brightness mean is greater than 200, the lighting score is equal to 400 minus the brightness mean divided by 200; when the brightness mean is between 50 and 200, the lighting score is 1. Alternatively, a weighted average brightness value can also be used, with smaller weights assigned to brightness values at both ends of the histogram and larger weights assigned to brightness values in the middle to reduce the impact of extreme brightness values. In addition, the structural similarity index SSIM is also calculated by the image processing library. When calculating SSIM, a standard reference image needs to be provided. This reference image can be a photo of the same scene under ideal lighting conditions, or an average result of historical images of the same model device. The calculation formula of SSIM is as follows: wherein, and are the average brightness of the reference image and the image to be evaluated, and are the standard deviations of the reference image and the image to be evaluated, is the covariance of the reference image and the image to be evaluated, and are small constants used to stabilize the calculation to avoid a zero denominator. Finally, the lighting score and the structural similarity index are combined to determine the reliability score of the on-site image data. The combination method can be a weighted average, for example, wherein is a weight coefficient, with a value range of 0 to 1, used to adjust the contribution of SSIM and the lighting score to the final reliability score. and are the normalized values of SSIM and illumination score, normalized to the range of 0 to 1. The purpose of normalization is to ensure that SSIM and illumination score are on the same scale. Alternatively, a non-linear combination can also be used, for example, using a Sigmoid function to transform the illumination score to enhance its contribution to the final reliability score.

[0045] Through the above steps, the embodiment can comprehensively consider the structural similarity and illumination condition of the image, so as to more accurately evaluate the quality of the image and provide a more reliable basis for subsequent feature compensation and fusion. If the structure of the image is similar to the reference image and the illumination condition is good, the reliability score is higher; otherwise, if the image is unclear, or the illumination is too dark or too bright, the reliability score is lower. Such design enables the system to more intelligently process image data of different qualities, thereby improving the overall operation and maintenance efficiency and accuracy.

[0046] In some embodiments, for the calculation of the on-site audio data clarity, the quality evaluation module adopts a perceptual speech quality evaluation algorithm, referred to as PESQ algorithm, to quantify the speech quality in the audio signal.

[0047] The implementation process of the PESQ algorithm can include the following steps: First, the input live audio data needs to be pre-processed. Pre-processing can include sampling rate conversion of the audio data, unified to a sampling rate of, for example, 16 kHz. Pre-processing can also include amplitude normalization of the audio data, such as scaling the amplitude of the audio signal to the range of -1 to 1. Amplitude normalization helps to eliminate volume differences caused by different recording equipment or recording distances. Second, the pre-processed live audio data is input into the PESQ algorithm. The implementation of the PESQ algorithm relies on a reference signal, which is usually a clear speech signal with the same content as the live audio data. The reference signal can be obtained by recording the same content of speech in a quiet environment, or generated using speech synthesis technology. The PESQ algorithm evaluates the quality of speech by comparing the differences between the live audio data and the reference signal. The PESQ algorithm analyzes the noise, distortion and other impairments in the signal and gives a PESQ score. The PESQ score is usually between -0.5 and 4.5, and the higher the score, the better the quality of the speech. For example, if the live audio data contains strong background noise, the PESQ algorithm will detect these noises and reduce the PESQ score. Conversely, if the live audio data is very clear, the PESQ algorithm will give a higher score. In some embodiments, the PESQ algorithm defined in the ITU-T P.862 standard can be used. The standard describes in detail the steps and parameter settings of the PESQ algorithm. In addition, there are many open source PESQ algorithm implementation libraries available, such as the pypesq library in Python, which can be used to implement the PESQ algorithm. Optionally, other speech quality evaluation algorithms can be used instead of the PESQ algorithm. For example, the Perceptual Evaluation of Speech Quality, abbreviated as POLQA algorithm, can be used to evaluate the quality of speech. The POLQA algorithm is an improvement over the PESQ algorithm and can more accurately evaluate the quality of modern speech codecs. In addition, subjective auditory test methods can also be used to evaluate the quality of speech. Subjective auditory test methods require a group of listeners to listen to live audio data and score based on their subjective feelings.

[0048] By using the PESQ algorithm to calculate the intelligibility of live audio data, a quantitative indicator can be obtained to represent the quality of the audio signal. This indicator can be used as part of the reliability score of the audio data for subsequent feature compensation and dynamic fusion steps. This reliability score can reflect the quality of the audio data, so that the system can more intelligently process multi-modal data and improve the practicality of the system in real-world scenarios.

[0049] In some embodiments, calculating the reliability score of live audio data includes, in addition to calculating the intelligibility using a perceptual speech quality assessment algorithm, calculating the signal-to-noise ratio (SNR) of the live audio data and combining the SNR with the intelligibility to determine the reliability score of the live audio data.

[0050] Specifically, the signal-to-noise ratio (SNR) calculation is implemented as follows: First, it is necessary to distinguish between speech segments and noise segments in the audio data. One implementation method is to use a Voice Activity Detection (VAD) algorithm, such as an energy-based VAD algorithm or a machine learning-based VAD algorithm. An energy-based VAD algorithm sets an energy threshold; audio frames above this threshold are considered speech frames, and those below are considered noise frames. This threshold can be dynamically adjusted based on ambient noise. For example, in the initial stage, assuming the first second of audio data contains only noise, its average energy is calculated as the initial threshold, and subsequently updated based on the real-time noise level. A machine learning-based VAD algorithm uses a pre-trained classification model, such as a Gaussian Mixture Model (GMM) or a Recurrent Neural Network (RNN), to distinguish between speech frames and noise frames. After distinguishing between speech segments and noise segments, the average energy of the speech segments and the average energy of the noise segments are calculated separately. One implementation method for energy calculation is to calculate the root mean square (RMS) value of each audio frame, and then average the RMS values ​​of all frames in both the speech and noise segments. Signal-to-noise ratio (SNR) is defined as the ratio of the average energy of the speech segment to the average energy of the noise segment, usually expressed in decibels (dB). The formula is SNR = 10 * log10(average energy of speech segment / average energy of noise segment). After calculating intelligibility and SNR, they need to be combined to determine the final audio reliability score. One approach is to use a weighted average method. For example, assuming the intelligibility score (PESQ) is normalized to the range of 0 to 1, and the SNR is also normalized to the range of 0 to 1, then the final reliability score... It can be represented as in This is a weighting coefficient, ranging from 0 to 1. This weighting coefficient can be adjusted based on the actual application scenario, or it can be learned through machine learning methods. In some scenarios where higher clarity is desired, it can be set... Greater than 0.5; in other scenarios, the signal-to-noise ratio may be even lower, so it can be set to... Less than 0.5. This represents the normalized Perceptual Speech Quality Assessment (PESQ) score, a value between 0 and 1 used to quantify the clarity and fidelity of audio. The closer the value is to 1, the higher the audio quality and the clearer it is; the closer the value is to 0, the lower the audio quality and the more severe the distortion. representative of the normalized signal-to-noise ratio (SNR), which is a value between 0 and 1, quantifying the strength of the signal relative to the noise in the audio. The closer the value is to 1, the stronger the signal is compared to the noise, and the "cleaner" the audio is; the closer the value is to 0, the more severe the noise interference is. Optionally, a non-linear combination can also be used to determine the final audio reliability score. For example, a Sigmoid function can be used to map the intelligibility and the signal-to-noise ratio to the interval of 0 to 1, and then multiply or add the mapped values.

[0051] By considering both the intelligibility and the signal-to-noise ratio of the audio, the quality of the audio data can be more comprehensively evaluated, thereby improving the accuracy and robustness of subsequent data processing. For example, even if the intelligibility of the audio is high, but the signal-to-noise ratio is low, it indicates that there is a large amount of noise in the audio, which may interfere with subsequent speech recognition or speech analysis tasks. By reducing the reliability score of this audio data, the interference of noise can be reduced, and the overall performance of the system can be improved. Optionally, other audio quality indicators, such as jitter, harmonic distortion, etc., can also be added to further improve the accuracy of audio reliability evaluation.

[0052] In some embodiments, the cross-modal compensation model in the feature compensation module has an internal structure of a neural network-based feature converter, which can adopt a Transformer architecture, a convolutional neural network (CNN) architecture, or a recurrent neural network (RNN) architecture. Taking the Transformer architecture as an example, the model includes a text encoder and an image feature decoder. The text encoder is responsible for converting the words in the text ticket into vector representations, and the image feature decoder generates compensation features for enhancing the color or texture of the field image data according to the vectors. The step of enhancing the features of the low-reliability modality includes the following when the field image data is of a low-reliability modality: first, extracting words representing the fault type from the text ticket data. For example, in the case of an electronic ticket data in JSON format, with the content being {"ticket_id": "T20240315", "alarm_code": "0x00B2", "description": "OLT device optical power abnormal attenuation"}, the value corresponding to "alarm_code" "0x00B2" and the value corresponding to "description" "OLT device optical power abnormal attenuation" are extracted as words by a JSON parser. These words are input into a pre-trained text encoder, such as a BERT model, a RoBERTa model, or a T5 model. The text encoder converts the words into high-dimensional embedding vectors. For example, the BERT model can convert each word into a 768-dimensional vector. Then, the embedding vectors are input into the image feature decoder to generate compensation features. The image feature decoder can be a deconvolutional neural network or a Transformer-based decoder structure. Taking the deconvolutional neural network as an example, it receives the embedding vectors output by the text encoder as the initial state or context vector. The decoder includes multiple deconvolutional layers, up-sampling layers, and activation function layers. For example, the decoder can include 3 deconvolutional layers, each with a convolution kernel size of 3x3 and a step size of 1; the up-sampling layer uses a bilinear interpolation algorithm for up-sampling; and the activation function layer uses a ReLU function. Through training, the decoder can convert text information into visual features related to the fault type. For example, when the word is "optical power abnormal attenuation", the decoder may generate a red channel feature map with high activation values in the OLT device optical interface area in the image, indicating that there may be an abnormality in the optical interface. Or, it may generate a feature map with a specific texture corresponding to the optical fiber connector area, suggesting that there may be a loose or contaminated optical fiber connector. Finally, the generated compensation features are fused with the features extracted from the original field image. The fusion method can be vector addition, i.e., adding the pixel value of the compensation feature map to the corresponding pixel value of the original image feature map, or channel splicing, i.e., adding the compensation feature map as a new channel to the original image feature map.Through the fusion, the original image features are enhanced to contain more information related to the fault type. Optionally, the text encoder can be replaced by an LSTM recurrent neural network or a GRU gated recurrent unit network. These networks can also convert text words into vector representations and be used to guide the generation process of the image feature decoder. In some other optional embodiments, the image feature decoder can adopt the structure of a generative adversarial network (GAN). The generator generates compensation features according to the text embedding vector, and the discriminator is responsible for judging whether the generated compensation features are real, thereby improving the quality of the generated features.

[0053] By using the above technical means, the embodiment can use the information in the text work order to compensate for the features of low-quality on-site image data, thereby improving the diagnosis capability of the system for faults. For example, in the case of insufficient light causing image blur, the system can enhance the red indicator light feature of the optical interface area in the image through the "optical power abnormality" word in the text work order, so that the fault diagnosis module can accurately determine the fault type.

[0054] In some embodiments, after generating the comprehensive feature vector, the vector is further used for fault diagnosis. Specifically, the comprehensive feature vector is input into a pre-trained fault classification model, which is used to output a classification result indicating the fault type of the communication network facility according to the input comprehensive feature vector, and give a confidence score corresponding to the classification result, which indicates the reliability of the classification result. The pre-trained fault classification model can be a support vector machine (SVM) model, which uses a radial basis kernel function and has been trained on a sample set containing a large amount of historical operation and maintenance data. The historical operation and maintenance data sample set contains feature vectors of various fault types and corresponding fault labels. The SVM model learns these samples to establish a mapping relationship between the feature vector and the fault type. The training of the SVM model can be realized by using a machine learning library such as Scikit-learn. After the training is completed, the SVM model is deployed in the data processing system to perform fault classification on new comprehensive feature vectors.

[0055] In some embodiments, the pre-trained fault classification model can also be a small neural network, such as a fully connected neural network containing two hidden layers. The input layer of the neural network receives the comprehensive feature vector, and the output layer contains multiple neurons, each corresponding to a possible fault type. The neural network converts the output of each neuron into a probability value through a Softmax function, which can be interpreted as a confidence score for the fault type corresponding to the neuron. The training of the small neural network can be achieved by using a deep learning framework such as TensorFlow or PyTorch. By adjusting the structural parameters of the neural network, such as the number of hidden layers, the number of neurons in each layer, and the type of activation function, the performance of the model can be optimized. After training is complete, the neural network is deployed in the data processing system for fault classification of new comprehensive feature vectors.

[0056] In some embodiments, the fault classification model can also be other types of classifiers, such as decision trees, random forests, or k-nearest neighbor algorithms. The choice of classification model depends on the specific application scenario and the characteristics of the data.

[0057] In specific operations, when a communication network facility operation and maintenance event occurs, the data acquisition module first acquires relevant multi-modal data, and then the quality assessment module and the feature compensation module process these data to generate a comprehensive feature vector. Next, the comprehensive feature vector is input into the pre-trained fault classification model. The fault classification model analyzes the input comprehensive feature vector according to the mapping relationship learned internally, and finally outputs a classification result indicating the fault type and a confidence score representing the reliability of the classification result. For example, the fault classification model may output a classification result of "optical module damage" with a confidence score of 0.92, indicating that the model has a 92% confidence that the fault type is optical module damage.

[0058] Optionally, the comprehensive feature vector before inputting into the fault classification model can be processed for dimensionality reduction. For example, the principal component analysis (PCA) method can be used to extract the main components of the comprehensive feature vector, thereby reducing the computational load of the model and improving the generalization ability of the model.

[0059] Through the above embodiments, the method can convert the adaptively processed multi-modal data features into specific fault type judgments and confidence assessments, effectively transforming data into information and providing decision support for operation and maintenance personnel. The confidence score can help operation and maintenance personnel judge the reliability of the fault diagnosis result and avoid incorrect operations due to model misjudgment.

[0060] In some embodiments, after the fault classification model outputs the classification result and the confidence score, the system will evaluate the confidence score. Specifically, assuming that a pre-set output threshold The value is set to 0.92, which means that the system needs at least 92% confidence to consider the classification result reliable before proceeding to the subsequent electronic diagnostic report generation step. If the result is lower than this threshold, the result is considered unreliable and no report is generated.

[0061] The confidence score assessment is implemented as follows: the system compares the confidence score output by the fault classification model with a preset output threshold T_output. This comparison can be implemented using a simple "if" statement in the computer program. For example, if the confidence score output by the classification model is 0.95, the program will determine if 0.95 is greater than 0.92. If the result is true, the program will continue to generate the electronic diagnostic report. Conversely, if the confidence score output by the classification model is 0.88, the program will determine if 0.88 is greater than 0.92. If the result is false, the program will skip the step of generating the electronic diagnostic report. The electronic diagnostic report can be generated in various ways. For example, a report template can be predefined, containing fixed text paragraphs and variable data fields. When the confidence score is higher than the threshold, the system will fill the corresponding fields in the report template with the fault classification result, relevant multimodal data snapshots (e.g., photos and audio clips of the fault occurrence), and the confidence score, thereby generating a complete electronic diagnostic report. The report can be saved as a PDF document and sent to relevant maintenance personnel via email or system message. The data snapshot can be implemented by extracting a representative segment from the original image and audio data. For example, for image data, a partial image containing alarm indicator lights can be extracted; for audio data, a 5-second audio clip containing abnormal sounds can be extracted. These data segments can be added as attachments to the electronic diagnostic report for maintenance personnel to reference. Optionally, the output threshold T_output can be adjusted according to different fault types. For example, for some fault types that are easily misjudged, the output threshold can be set to a higher value, such as 0.95, to reduce the risk of false alarms; while for some fault types that are relatively easy to judge, the output threshold can be set to a lower value, such as 0.85, to improve the efficiency of report generation. The output threshold can be adjusted manually or through automatic learning based on historical data. Optionally, if the confidence score is lower than the preset output threshold, the system can take other measures instead of skipping the report generation step. For example, the system can automatically send a message to the field engineer, prompting them to retake photos of the single-board indicator lights or provide a more detailed sound description. Alternatively, the system can initiate a manual review process, submitting the fault classification results and related data to experts for further judgment.

[0062] This embodiment introduces a confidence gating mechanism to ensure that the system only gives a diagnosis result when it is confident enough, avoiding the risk of misjudgment due to model guessing. This is significant for mission fields such as communication support, and can significantly improve the reliability and security of the system. By setting appropriate output thresholds, a balance between report generation efficiency and diagnostic accuracy can be achieved.

[0063] In some embodiments, the adaptive multi-modal data processing method and system of the present application is applied to the operation and maintenance scene of a mobile communication base station. Specifically, assuming that a mobile communication base station appears to be offline, the operation and maintenance personnel arrive at the scene, use a handheld terminal to take a photo of the base station's feeder line interface, record an audio containing environmental noise and equipment working sound, and submit a text work order containing an alarm code through a work order system. After receiving the data of the three modalities, the quality assessment module assesses the quality of the image, audio and text data. For image data, the quality assessment module calls an image processing algorithm library, such as OpenCV, to calculate indicators such as image sharpness and brightness. Specifically, the sharpness can be evaluated by calculating the Laplacian variance of the image, and if the Laplacian variance is lower than a preset threshold, such as 50, the image is considered to be blurred. The brightness evaluation can be performed by calculating the mean value of the image gray histogram, and if the mean value is lower than a preset lower limit, such as 30, or higher than a preset upper limit, such as 220, the image is considered to be too dark or overexposed. The sharpness and brightness scores are weighted and averaged to obtain the reliability score of the image. For audio data, the quality assessment module uses a speech quality assessment algorithm, such as PESQ or STOI, to evaluate the sharpness of the audio. At the same time, the signal-to-noise ratio of the audio is calculated, which can be obtained by estimating the energy ratio of the speech segment and the noise segment. If the signal-to-noise ratio is lower than a preset threshold, such as 5dB, the audio quality is considered to be poor. The sharpness and signal-to-noise ratio scores are weighted and averaged to obtain the reliability score of the audio. For text work order data, the quality assessment module checks the format specification and word integrity of the work order. For example, check if the work order contains necessary fields such as alarm code, device ID, etc. If the work order format is standard and the information is complete, the text data is considered to be highly reliable and is assigned a high reliability score. Assuming that the reliability scores of the image and audio are lower than a preset compensation threshold, such as 0.6, the feature compensation module is triggered. The feature compensation module uses the high-reliability text data to enhance the features of the low-reliability modalities through a pre-trained cross-modal compensation model, such as a neural network based on Transformer. Specifically, the words, such as "feeder line" and "loose interface", are extracted from the text work order and input into the cross-modal compensation model. The model outputs image and audio features related to the words. For images, the model can generate visual features indicating a loose interface, such as loose connectors and rust. For audio, the model can generate sound features indicating a loose interface, such as the current sound produced by poor contact. Fuse these compensated features with the original image and audio features to obtain the compensated features. The dynamic fusion module determines the fusion weights according to the reliability scores of each modality. For example, directly use the normalized reliability scores as weights. Weighted fusion of the feature vectors of each modality generates a comprehensive feature vector. For example, the image and audio use the compensated feature vectors, and the text uses the original feature vectors.The integrated feature vector is input to the fault diagnosis module, and the diagnosis module outputs the fault type and a confidence score. For example, the output is a diagnosis result of "loose antenna feeder interface", with a confidence score of 0.95. If the confidence score is higher than a preset output threshold, for example 0.90, an electronic diagnosis report is generated.

[0064] Optionally, the communication network facility is an optical transmission line device, such as an optical distribution box or an OLT device. After the operation and maintenance personnel arrive at the optical distribution box site, they take a photo of the optical fiber connector, record the sound of the on-site environment, and submit a text work order containing the alarm code and device information. The subsequent processing process is similar to the mobile communication base station operation and maintenance scenario, with the difference being that the cross-modal compensation model needs to be trained for the characteristics of the optical transmission line device to generate corresponding compensation features. For example, when the text work order indicates "excessive optical fiber attenuation", the compensation model can generate image features indicating that the optical fiber connector is dirty or the optical cable is bent.

[0065] In the above manner, the embodiment can adaptively process images, audio and text data of different qualities, and enhance the features of low-quality modalities using the cross-modal compensation model, thereby improving the accuracy and robustness of fault diagnosis, so that the scheme can be applied to different communication network facility operation and maintenance scenarios such as mobile communication base stations and optical transmission line devices.

[0066] As shown in Figure 2 The embodiment of the application provides a large model intent recognition system based on multi-modal perception, which is applied to a communication network facility operation and maintenance scenario and aims to improve the accuracy and robustness of data processing in complex environments, and is used to implement the large model intent recognition method based on multi-modal perception of any of the above embodiments. The system comprises: The data acquisition module M100 is configured to acquire multi-modal data related to the operation and maintenance of the communication network facilities, including field image data, field audio data, and text work order data. In one specific implementation, the data acquisition module M100 can receive data stream from an operation and maintenance terminal through a network interface, which can be a smartphone equipped with a camera or a dedicated handheld device. The field image data can be a JPEG format photo taken by the terminal camera containing the device status indicator light, with a resolution of 1920x1080 pixels. The field audio data can be a WAV format audio file recorded by the terminal microphone describing the device abnormal sound, with a sampling rate of 48kHz. The text work order data can be a JSON format text filled by the operation and maintenance personnel through the terminal App, containing fault code, device model, etc. The data acquisition module M100 also contains data verification functions to verify whether the received data format conforms to the pre-defined specifications, for example, whether the image file is damaged, whether the audio file is complete, and whether the text work order contains necessary fields. Optionally, the data acquisition module M100 can also support reading data from local storage media such as SD cards, or receiving data from other devices through wireless networks such as Wi-Fi.

[0067] The quality assessment module M200 is configured to compute a reliability score representing the quality of each modality of the multi-modal data. For the on-site image data, the quality assessment module M200 can call an image processing library, such as OpenCV, to compute the sharpness, contrast, and brightness of the image. The sharpness can be obtained by computing the Laplacian variance of the image, a higher variance value indicates more details and a higher sharpness. The contrast can be obtained by computing the range of the gray levels of the image, a larger range of the gray levels indicates a higher contrast. The brightness can be obtained by computing the average pixel value of the image, an average pixel value close to the middle value, such as 128, indicates a moderate brightness. The quality assessment module M200 can normalize these indicators and use a weighted average method to compute the reliability score of the image. For the on-site audio data, the quality assessment module M200 can compute the signal-to-noise ratio SNR of the audio, a higher SNR indicates a better audio quality. The SNR can be obtained by estimating the energy ratio of the speech segment and the noise segment. The quality assessment module M200 can also use a speech quality assessment algorithm, such as PESQ, to compute the perceptual quality score of the audio. The quality assessment module M200 normalizes these indicators and uses a weighted average method to compute the reliability score of the audio. For the text ticket data, the quality assessment module M200 can check the completeness and the normalization of the text, for example, check if there are missing necessary fields, if there are invalid characters. If the text ticket is complete and normalized, it is considered to have a higher reliability, for example, assigned a value of 0.95. Optionally, the quality assessment module M200 can use a machine learning model, such as a convolutional neural network CNN, to directly learn the reliability score from the raw image or audio data.

[0068] The feature compensation module M300 is configured to identify a low-reliability modality whose reliability score is lower than a preset compensation threshold based on the reliability scores, and to enhance the features of the low-reliability modality by using the features of one or more high-reliability modalities through a pre-trained cross-modality compensation model to obtain compensated features. The preset compensation threshold can be adjusted according to the actual application scenario, for example, set to 0.6. If the reliability score of the image is 0.5, which is lower than the threshold, it is considered that the image quality is poor and needs to be compensated. The cross-modality compensation model can be a neural network based on Transformer, which is pre-trained on a large amount of multi-modal data to learn the correlation between different modalities. For example, if the text work order contains the description of “red alarm light” and the image quality is poor and cannot clearly identify the color of the alarm light, the feature compensation module M300 can use the text information to generate a color feature vector representing “red”, and fuse the vector with the original image feature vector to enhance the feature expression of the image. The compensation method of audio is similar, for example, the device model information in the text work order can be used to generate an acoustic feature spectrum representing the typical sound of the device to enhance the features of the original audio. The feature compensation module M300 can input the text of the text work order into a text encoder such as BERT to obtain the embedding vector of the text, and then input the vector into an image feature decoder such as a deconvolutional network to generate a compensation feature map. The feature map is fused with the original image feature map.

[0069] The dynamic fusion module M400 is configured to determine a fusion weight for subsequent processing according to the reliability score, and to perform weighted fusion on the features and compensated features of each modality based on the fusion weight to generate a comprehensive feature vector representing the operation and maintenance state of the communication network facility. The fusion weight is related to the reliability score, and the higher the reliability score, the greater the weight. For example, the normalized reliability score can be directly used as the weight, or the reliability score can be converted to obtain the weight using a Softmax function. The dynamic fusion module M400 extracts the feature vectors of each modality, uses the compensated feature vector for a low-reliability modality, and uses the original feature vector for a high-reliability modality. Then, the feature vectors of each modality are weighted and summed according to the corresponding weights to obtain a comprehensive feature vector. The comprehensive feature vector can fuse information from different modalities and adaptively adjust according to the quality of each modality, thereby improving the robustness and accuracy of data processing.

[0070] Through the cooperative work of the above modules, the embodiment of the present application provides a large model intention recognition system based on multi-modal perception. The system can dynamically adjust the processing strategy according to the quality of the input data, compensate for low-quality data, and weightedly fuse data of different modalities, so as to generate a high-quality comprehensive feature vector for subsequent fault diagnosis or other operation and maintenance tasks. The system can effectively deal with data quality problems in complex environments and improve operation and maintenance efficiency and accuracy.

[0071] In some embodiments, before the multi-modal data is processed by the quality assessment module M200, the data preprocessing module performs the following operations to ensure the consistency and effectiveness of subsequent processing.

[0072] For on-site image data, regardless of its original resolution, the data preprocessing module adjusts it to a first preset resolution. For example, if the resolution of the original image is 1920x1080 pixels or 1280x720 pixels, the data preprocessing module will uniformly use a bicubic interpolation algorithm to scale or enlarge it to 512x512 pixels. The bicubic interpolation algorithm is an image resampling algorithm that determines the value of a new pixel by calculating the weighted average of the 16 pixels around the pixel to be interpolated, which can reduce the sawtooth and mosaic phenomenon while preserving image details. The image processing module can use, for example, the OpenCV library to realize the scaling function of the image, and the bicubic interpolation algorithm is one of the default options of the resize function in the OpenCV library. Alternatively, other image interpolation algorithms such as bilinear interpolation or Lanczos interpolation can be used. The bilinear interpolation algorithm calculates the weighted average of the four pixels around the pixel to be interpolated, which has a smaller calculation amount but may cause image blurring. The Lanczos interpolation algorithm uses a more complex kernel function to better preserve image details, but has a larger calculation amount.

[0073] For on-site audio data, the data preprocessing module adjusts the sampling rate of the audio to a first preset sampling rate. For example, if the sampling rate of the original audio is 16kHz or 48kHz, it is uniformly converted to 44.1kHz through resampling. Resampling refers to the process of changing the sampling rate of digital audio, which can be achieved by interpolating or decimating the original audio samples. For example, the librosa library can be used to perform audio resampling operations by calling the librosa.resample function and specifying the original sampling rate and target sampling rate. Alternatively, other audio sampling rates such as 16kHz or 22.05kHz can be selected, and the specific selection depends on the application scenario and the requirement for audio quality.

[0074] Through the above preprocessing steps, all input images will have the same size, and all input audio will have the same sampling rate, which eliminates the influence brought by the difference of data sources, so that the subsequent quality evaluation and feature extraction modules can be processed on a unified basis, simplifying the design and training process of the model, and improving the processing efficiency and accuracy.

[0075] In some embodiments, the feature compensation module M300 comprises a cross-modal compensation model implemented using a neural network-based feature converter. The feature converter is configured to receive words representing fault types extracted from the text work order data and output compensation features for enhancing the color or texture aspects of the field image data when the field image data is determined by the quality evaluation module M200 to be a low reliability modality.

[0076] In particular, the feature converter can employ a Transformer architecture, which contains two main parts, an encoder and a decoder. The encoder is used to convert the words in the text work order into embedding vectors, and the decoder is used to generate compensation features based on the embedding vectors. The word extraction of the text work order data is implemented using natural language processing techniques. For example, a pre-trained BERT model can be used to analyze the text work order data and identify the words therein. For example, when the text work order data is "alarm code 0x00A1, user reports weak signal", the BERT model can identify the two words "0x00A1" and "weak signal". These words can represent the fault type and will be input into the feature converter. Optionally, the word extraction can also use a simple rule matching method, for example, a pre-defined word list is defined, and then the words are extracted by checking whether the text work order data contains these words. The word list can be adjusted and updated according to the actual application scenario to improve the accuracy of word extraction. When the on-site image data is determined to be a low reliability modality, for example, the image is blurred or the lighting is insufficient, the feature converter will be started. The word embedding vectors from the text work order are input into the decoder of the feature converter. The decoder is trained to generate compensation feature maps related to the semantics of the input words. For example, if the words are "0x00A1" and "weak signal", and according to historical data, this alarm code usually corresponds to "single board hardware failure", which is manifested as "red indicator light always on", then the decoder will generate a feature map with the same size as the original image feature map, in which the pixels corresponding to the indicator light region have a higher red channel activation value. This feature map is the compensation feature, which can enhance the expression of "red" in the original image feature. In order to fuse the compensation feature with the original image feature, a variety of methods can be used. One method is to directly add the compensation feature map and the original image feature map element by element. Another method is to splice the compensation feature map and the original image feature map, and then learn the fused features through a convolution layer. Optionally, an attention mechanism can also be used to implement feature fusion, for example, using the compensation feature map as the query, the original image feature map as the key and the value, and calculating the attention weight to dynamically fuse the two feature maps. The generated compensation feature can enhance the color or texture information in the on-site image data, thereby improving the quality of the image data. For example, in the case of insufficient lighting, the compensation feature can enhance the color information of the indicator light, making the color of the indicator light more obvious. In the case of image blur, the compensation feature can enhance the texture information of the image, making the details of the image more clear. Through the compensation feature, the usability of low-quality image data can be effectively improved, thereby improving the accuracy of the entire multi-modal data processing system.

[0077] Through the neural network-based feature converter, information in the text ticket data can be utilized to enhance the on-site image data in the case of poor image data quality, thereby avoiding information loss caused by poor image quality and improving the robustness and reliability of the multi-modal data processing system.

[0078] In some embodiments, the diagnostic output module receives the integrated feature vector generated by the dynamic fusion module M400 The vector is transmitted to a pre-trained fault classification model. The fault classification model can take various forms, such as a support vector machine (SVM), a neural network, or a decision tree. In this embodiment, the fault classification model is a three-layer fully connected neural network that includes an input layer, a hidden layer, and an output layer. The input layer receives the integrated feature vector as input, the hidden layer contains 128 neurons and uses a ReLU activation function for non-linear transformation. The output layer contains several neurons corresponding to different fault types. In this example, it is assumed that the communication network facility can have four types of faults: "optical module damage", "loose network cable", "power failure", and "software configuration error". Therefore, the output layer contains 4 neurons, and the Softmax function is used to convert the output to a probability distribution. The output value of each neuron represents the probability of the corresponding fault type, and the sum of these probability values is 1. The neural network is pre-trained on a large amount of labeled historical operation and maintenance data. The training data includes various combinations of multi-modal data and their corresponding true fault type labels. The training process uses the backpropagation algorithm to optimize the network parameters by minimizing the cross-entropy loss function, so that the network can accurately map the input integrated feature vector to the correct fault type. The fault classification model outputs a classification result, which indicates the fault type of the communication network facility, such as "optical module damage". At the same time, the model also outputs a confidence score, which represents the degree of certainty of the classification result. The confidence score is represented by the highest probability value output by the Softmax function. For example, if the output probabilities of the four neurons in the output layer are 0.1, 0.2, 0.6, and 0.1, respectively, the classification result is "power failure" and the corresponding confidence score is 0.6.

[0079] The diagnostic output module includes a pre-set output threshold to determine whether the classification result is sufficiently reliable. In this example, it is assumed that The output threshold is set to 0.85. The diagnostic output module compares the confidence score with the output threshold. If the confidence score is higher than the output threshold, the diagnostic output module determines that the classification result is reliable, and generates an electronic diagnostic report based on the classification result. The electronic diagnostic report contains the fault type, the multimodal data snapshots related to the fault, and the confidence score. For example, if the classification result is "power failure" and the confidence score is 0.92, the electronic diagnostic report contains the text description of "power failure", the photo of the device power indicator light, the audio of the power abnormal sound, and the confidence score of 0.92. The electronic diagnostic report is automatically sent to the maintenance personnel to assist them in troubleshooting. If the confidence score is lower than the output threshold, the diagnostic output module determines that the classification result is not reliable enough, and does not generate an electronic diagnostic report. As an alternative, the diagnostic output module can push a message to the terminal of the field engineer, requesting him to provide more information, such as "not enough information, please check the device connection status and upload the photos again". Optionally, the fault classification model can be an integrated classifier composed of multiple independent classifiers, each of which predicts the fault type, and the final classification result is determined by voting or weighted averaging. In other optional embodiments, the electronic diagnostic report can contain recommended solutions for the fault type, which are stored in a pre-established knowledge base and automatically retrieved according to the classification result.

[0080] By implementing the present embodiment, the diagnostic output module can automatically diagnose the fault type of the communication network facility according to the comprehensive feature vector and provide the corresponding confidence evaluation, ensuring that only the reliable diagnostic result is used to generate the electronic diagnostic report, avoiding the risk of misjudgment caused by model guessing, and improving the practicability of the system in the field of communication support and other tasks.

[0081] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, some simple deductions or substitutions can be made without departing from the concept of the present application, and all of them should be considered as falling within the protection scope of the present application.

Claims

1. A method for large-scale model intent recognition based on multimodal perception, characterized in that, This method is executed by a computer device and includes: Acquire multimodal data related to the operation and maintenance of communication network facilities, including on-site image data, on-site audio data, and text work order data; For each mode in the multimodal data, a reliability score characterizing the signal quality of that mode of data is calculated; Based on the reliability score, low reliability modes with reliability scores below a preset compensation threshold are identified. Then, using the features of one or more high reliability modes, the features of the low reliability modes are enhanced through a pre-trained cross-modal compensation model to obtain compensated features. The fusion weights for subsequent processing are determined based on the reliability score, and the features of each modality and the compensated features are weighted and fused based on the fusion weights to generate a comprehensive feature vector that characterizes the operation and maintenance status of the communication network facility.

2. The method according to claim 1, characterized in that, After acquiring the multimodal data, the process also includes: Adjust the resolution of the on-site image data to a first preset resolution; and The sampling rate of the on-site audio data is adjusted to the first preset sampling rate.

3. The method according to claim 1, characterized in that, Calculating the reliability score of the on-site image data includes: Calculate the structural similarity index between the field image data and a standard reference image, as part of the reliability score.

4. The method according to claim 3, characterized in that, Calculating the reliability score of the on-site image data further includes: The brightness histogram of the on-site image data is analyzed to generate a score characterizing the lighting conditions, and this score is combined with the structural similarity index to determine the reliability score of the on-site image data.

5. The method according to claim 1, characterized in that, Calculating the reliability score of the on-site audio data also includes: The clarity of the on-site audio data is calculated using a perceptual speech quality assessment algorithm.

6. The method according to claim 5, characterized in that, Calculating the reliability score of the on-site audio data also includes: Calculate the signal-to-noise ratio (SNR) of the on-site audio data, and determine the reliability score of the on-site audio data based on the combination of the SNR and the clarity.

7. The method according to claim 1, characterized in that, The method further includes: The integrated feature vector is input into a pre-trained fault classification model to output a classification result indicating the fault type of the communication network facility and the corresponding confidence score.

8. The method according to claim 7, characterized in that, The method further includes: An electronic diagnostic report is generated based on the classification result only when the confidence score is higher than a preset output threshold.

9. The method according to claim 1, characterized in that, The communication network facilities are mobile communication base stations or fiber optic transmission line equipment.

10. A large-scale model intent recognition system based on multimodal perception, characterized in that, include: The data acquisition module is configured to acquire multimodal data related to the operation and maintenance of communication network facilities. The multimodal data includes on-site image data, on-site audio data, and text work order data. The quality assessment module is configured to calculate a reliability score characterizing the signal quality of each mode in the multimodal data. The feature compensation module is configured to identify low reliability modes with reliability scores below a preset compensation threshold based on the reliability score, and enhance the features of the low reliability modes using features of one or more high reliability modes through a pre-trained cross-modal compensation model to obtain compensated features. The dynamic fusion module is configured to determine the fusion weights for subsequent processing based on the reliability score, and to perform weighted fusion of the features of each modality and the compensated features based on the fusion weights to generate a comprehensive feature vector characterizing the operation and maintenance status of the communication network facility.

Images