Cross-modal intelligent operation and maintenance system and method based on lightweight mixed reality glasses

Abstract

The invention discloses a cross-modal intelligent operation and maintenance system and method based on lightweight mixed reality glasses, and the system comprises an MR glasses module which collects a configuration interface image presented on a real computer module of a digital trunking base station in real time, synchronously collects the head motion posture data of an operator, and transmits the data to a server; the space-vision coupling original data comprises a configuration interface image and the visual direction of an operator; the method comprises the following steps: performing data complexity scoring on space-vision coupling original data through a data processing and analyzing module, monitoring a local computing resource occupancy rate in real time, and switching between a local computing mode and a data return mode according to the data complexity score and the local computing resource occupancy rate. And through a real-time assistance and guidance module, in a local calculation mode or a data return mode, according to an analysis result, displaying a function to be displayed in a real view of an operator in an overlapping manner through spatial audio and retina projection so as to guide the operator to complete base station configuration in real time.

Description

Technical Field

[0001] This application relates to the field of laser powder bed melting technology, and more specifically, to a cross-modal intelligent operation and maintenance system and method based on lightweight mixed reality glasses. Background Technology

[0002] In the operation and maintenance of digital trunking base stations, configuring the base station equipment is a crucial step. Currently, when configuring parameters on the computer modules equipped in actual base stations, maintenance personnel mainly rely on personal experience and paper operation manuals. This traditional method has many drawbacks.

[0003] In China, a major telecommunications operator, during a large-scale base station configuration project, encountered parameter errors in approximately 30% of the configuration tasks due to the use of traditional methods and the lack of real-time guidance and error reporting mechanisms. Each configuration session took an average of 4-5 hours. Internationally, a multinational telecommunications company, in a base station construction project in a remote area, relied on paper manuals for configuration. Because the manuals were not updated in a timely manner, base station parameter configurations were incorrect, resulting in signal coverage falling short of expectations. Re-debugging took two weeks, significantly increasing manpower and material costs.

[0004] Existing auxiliary tools also have limitations. For example, simple computer-based configuration guides cannot achieve real-time interaction with operators; some visual auxiliary software also struggles to provide accurate guidance based on actual on-site operating scenarios. For instance, traditional AR-based auxiliary software can only provide fixed operation steps and cannot provide dynamic guidance based on data changes during the actual configuration process; while some cloud-based analysis systems suffer from high data transmission latency and strong network dependence, failing to effectively assist configuration work when the network is poor. These problems lead to low base station configuration efficiency, high error rates, increased human and material costs, and affect the normal operation of base stations and signal coverage. Summary of the Invention

[0005] To address at least one deficiency or improvement need in existing technologies, this invention provides a cross-modal intelligent operation and maintenance system and method based on lightweight mixed reality glasses, solving the problems of low efficiency and high error rate in traditional configuration methods. By acquiring configuration images and data from digital trunking base stations through MR glasses, and utilizing local computing within the MR glasses or analysis by a backend server, real-time monitoring, error identification, and intelligent guidance of the base station configuration process are achieved.

[0006] To achieve the above objectives, according to one aspect of the present invention, a cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses is provided, the system comprising: The MR glasses module is used to acquire the configuration interface image displayed on the real computer module of the digital trunking base station in real time, and simultaneously acquire the head movement posture data of the operator, thereby obtaining the original spatial-visual coupling data containing the configuration interface image and the operator's gaze direction. The data processing and analysis module is communicatively connected to the MR glasses module and is used to score the data complexity of the spatial-visual coupling raw data and monitor the local computing resource utilization rate in real time. When the data complexity score is lower than the first threshold and the local computing resource utilization is lower than the second threshold, the local computing mode is activated. The MR glasses module processes the configuration interface image to extract key elements and parameter data, compares the parameter data with the basic configuration rules stored locally, and then performs preliminary analysis through a lightweight machine learning algorithm to obtain the first analysis result. When the data complexity score is higher than or equal to the first threshold or the local computing resource utilization rate is higher than or equal to the second threshold, the data backhaul mode is activated. The MR glasses module uploads the spatial-visual coupling raw data to the background processing module. The background processing module is used to call the intelligent analysis engine based on LSTM and SVM and combine it with the configuration rule knowledge base to perform in-depth analysis on the raw data to obtain the second analysis result. The real-time assistance and guidance module is communicatively connected to the data processing and analysis module and the MR glasses module. It is used to overlay error prompts, correct configuration suggestions and next operation procedures onto the operator's real field of vision through spatial audio and retinal projection, based on the first analysis result in local computing mode or the second analysis result in data backhaul mode, so as to guide the operator to complete the base station configuration in real time.

[0007] Furthermore, the MR glasses module utilizes a lightweight image recognition model to process the configuration interface image to extract key elements and parameter data; the lightweight image recognition model uses YOLOv8 as the baseline network and is obtained through the following methods: Replace the CSPDarknet structure in the backbone network with the MobileNet-v3 structure; Embed a CBAM attention mechanism module at the output of the MobileNet-v3 backbone network; A transfer learning strategy is adopted. In the first stage, the COCO dataset is used for pre-training. The pre-training process uses a stochastic gradient descent optimizer and sets the first model training hyperparameters. The first round of the pre-training process is called the first round. In the second stage, a local preset dataset is used for fine-tuning. The first round of the fine-tuning stage is called the first round. Furthermore, the output of the lightweight image recognition model is configured to output the position coordinates and category labels of the parameter input box, operation button, drop-down menu and status indicator icon in the base station configuration interface, and transmit the position coordinates and category labels to the data processing and analysis module for comparison with the standard information in the locally stored basic configuration rules or the background configuration rule knowledge base to generate a first analysis result or a second analysis result.

[0008] Furthermore, the MR glasses module includes: High-definition camera, used to capture images of the configuration interface displayed on the actual computer module of the digital trunking base station in real time; IMU sensor is used to detect the operator's head movement posture in real time; An edge computing unit is used to process the configuration interface image to extract key elements and parameter data, compare the parameter data with the basic configuration rules stored locally, and then perform preliminary analysis through a lightweight machine learning algorithm to obtain a first analysis result. The computing mode switching unit is used to upload the spatial-visual coupling raw data to the background processing module.

[0009] Furthermore, the background processing module includes: The intelligent analysis engine is built on a microservice architecture and uses a Long Short-Term Memory (LSTM) network to learn and predict operation sequences, and a Support Vector Machine (SVM) to classify and judge parameter data. A configuration rule knowledge base is configured and stored using MongoDB. This knowledge base stores the standard parameter ranges, configuration process rules, and common error cases for digital trunking base station configuration.

[0010] Furthermore, the formula for calculating the data complexity score in the data processing and analysis module is as follows: ; in, These are the weighting coefficients. Indicates the size of the data. This represents the computational logic complexity.

[0011] Furthermore, the data processing and analysis module includes: The data acquisition unit is used to score the data complexity of the spatial-visual coupled raw data and monitor the local computing resource utilization rate in real time. The computing mode switching unit is used to generate a local computing mode start command and send it to the MR glasses module when the data complexity score is lower than a first threshold and the local computing resource utilization rate is lower than a second threshold, and to generate a data feedback mode start command and send it to the MR glasses module when the data complexity score is higher than or equal to the first threshold or the local computing resource utilization rate is higher than or equal to the second threshold.

[0012] Furthermore, the real-time assistance and guidance module includes: An auxiliary computing unit is used to display error messages and configuration suggestions using spatial audio technology and retinal projection text. The guidance and support unit is used to retrieve the next operation process and generate control instructions that are overlaid and displayed in the form of virtual arrows, animation demonstrations, etc., so that the overlay display is in the operator's actual field of vision.

[0013] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: Flexible and efficient data processing: The system supports both local computation and background processing modes for MR glasses. Local computation mode, combined with lightweight algorithms and a local rule base, enables rapid response in scenarios with poor network connectivity or simple configurations, while reducing computational resource consumption by 40%. Data feedback mode employs optimized data compression and transmission protocols, combined with powerful background analysis capabilities, to handle complex configuration scenarios, improving analysis accuracy by 20%. A unique computation mode switching mechanism ensures efficient system operation in different scenarios, enhancing overall processing efficiency.

[0014] Real-time Precise Assistance: By capturing real-time configuration images and data from the actual computer module through MR glasses, and combining this with local or backend intelligent analysis, errors and anomalies during the configuration process can be accurately identified. This provides operators with timely and accurate guidance, effectively preventing configuration errors caused by human negligence. The collaborative optimization strategies between the MR glasses module and the backend processing module, such as dynamically adjusting data acquisition frequency and accuracy, and feedback optimization calculation rules, further improve the accuracy of the assistance.

[0015] Efficient Configuration Process: The system's intelligent guidance function helps operators quickly master the configuration steps, reducing time wasted due to unfamiliarity with the process. Actual testing shows that using this system reduces base station configuration time to an average of 2-2.5 hours, significantly improving configuration efficiency.

[0016] Reduced operation and maintenance costs: Significantly reduces the base station configuration error rate, reducing the extra work such as equipment debugging and troubleshooting caused by configuration errors. It is expected to reduce base station operation and maintenance costs by more than 40%, which has significant economic value. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is one of the structural schematic diagrams of a cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses, provided as an embodiment of this application.

[0019] Figure 2 This is the second schematic diagram of a cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses, provided as an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0021] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0022] This invention implements a cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses that integrates spatial location information. Figure 1 This embodiment provides one of the cross-modal intelligent operation and maintenance systems based on lightweight mixed reality glasses that optionally fuses spatial location information, such as... Figure 1 As shown, the system includes: an MR glasses module, used to acquire in real time the configuration interface image presented on the real computer module of the digital trunking base station, and simultaneously acquire the head movement posture data of the operator, thereby obtaining spatial-visual coupling raw data containing the configuration interface image and the operator's gaze direction; The data processing and analysis module is communicatively connected to the MR glasses module and is used to score the data complexity of the spatial-visual coupling raw data and monitor the local computing resource utilization rate in real time. When the data complexity score is lower than the first threshold and the local computing resource utilization is lower than the second threshold, the local computing mode is activated. The MR glasses module processes the configuration interface image to extract key elements and parameter data, compares the parameter data with the basic configuration rules stored locally, and then performs preliminary analysis through a lightweight machine learning algorithm to obtain the first analysis result. When the data complexity score is higher than or equal to the first threshold or the local computing resource utilization rate is higher than or equal to the second threshold, the data backhaul mode is activated. The MR glasses module uploads the spatial-visual coupling raw data to the background processing module. The background processing module is used to call the intelligent analysis engine based on LSTM and SVM and combine it with the configuration rule knowledge base to perform in-depth analysis on the raw data to obtain the second analysis result. The real-time assistance and guidance module is communicatively connected to the data processing and analysis module and the MR glasses module. It is used to overlay error prompts, correct configuration suggestions and next operation procedures onto the operator's real field of vision through spatial audio and retinal projection, based on the first analysis result in local computing mode or the second analysis result in data backhaul mode, so as to guide the operator to complete the base station configuration in real time.

[0023] In the operation and maintenance of digital trunking base stations, after the maintenance personnel wear MR glasses, the camera at the front of the glasses continuously captures the base station configuration software interface running on the real computer module, forming an image stream; at the same time, the sensor collects head posture angular velocity and acceleration data, and the two are merged into spatial-visual coupled raw data in the edge computing unit. When maintenance personnel input the transmit power, the data processing and analysis module immediately adjusts the calculation complexity. For example, if the data complexity score is <80 and the CPU utilization is <80%, the system maintains local calculation mode, directly extracts the coordinates and values ​​of the power input box, and compares them with the local rule base within the 20-40 W range. If an anomaly of 50 W is detected, the system immediately projects the text "Exceeding Limit" onto a red retina within a 120° field of view of the optical waveguide and provides a spatial audio prompt with ±15° precision indicating "Power Too High." If the data complexity score is ≥80 or the CPU utilization is ≥80%, the calculation mode switching unit uploads the spatial-visual coupled raw data to the backend processing module via a 5G-SA link using the HTTP / 3 compression protocol. The backend processing module then calls LSTM+SVM to perform in-depth analysis of the operation sequence and parameters, identifying the risk of insufficient frequency spacing with neighboring stations. Subsequently, the second analysis result is sent back, and the MR glasses project a virtual frequency distribution map in the same field of view and demonstrate the correction steps with animated arrows, thereby compressing the traditional configuration process.

[0024] Optionally, the data processing and analysis module includes: The data acquisition unit is used to score the data complexity of the spatial-visual coupled raw data and monitor the local computing resource utilization rate in real time. The computing mode switching unit is used to generate a local computing mode start command and send it to the MR glasses module when the data complexity score is lower than a first threshold and the local computing resource utilization rate is lower than a second threshold, and to generate a data feedback mode start command and send it to the MR glasses module when the data complexity score is higher than or equal to the first threshold or the local computing resource utilization rate is higher than or equal to the second threshold.

[0025] Based on the above embodiments, as an optional embodiment, in the cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses provided by the present invention, the MR glasses module uses a lightweight image recognition model to process the configuration interface image to extract key elements and parameter data; the lightweight image recognition model uses YOLOv8 as the baseline network and is obtained in the following way: Replace the CSPDarknet structure in the backbone network with the MobileNet-v3 structure; Embed a CBAM attention mechanism module at the output of the MobileNet-v3 backbone network; A transfer learning strategy is adopted. In the first stage, the COCO dataset is used for pre-training. The pre-training process uses a stochastic gradient descent optimizer and sets the first model training hyperparameters. The first round of the pre-training process is called the first round. In the second stage, a local preset dataset is used for fine-tuning. The first round of the fine-tuning stage is called the first round. In this embodiment, the lightweight image recognition model undergoes two-stage training before deployment: the first stage involves pre-training with 200,000 general images from 80 categories in the COCO dataset, using the SGD optimizer with a learning rate of 0.001 and 100 epochs; the second stage involves collecting 5,000 images of on-site configuration interfaces covering various base station types, time periods, and operating habits, fine-tuning them with a learning rate of 0.0001 for 50 epochs to improve detection speed and accuracy. The original CSPDarknet in the backbone network is replaced with MobileNet-v3, reducing the number of parameters, and a CBAM attention module is embedded at the output to weight the channel-spatial features of key elements such as the power input box and the PCI dropdown menu, ensuring high recognition accuracy even in high-temperature or high-humidity environments.

[0026] Based on the above embodiments, as an optional embodiment, in the cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses provided by the present invention, the output result of the lightweight image recognition model is configured as follows: outputting the position coordinates and category labels of the parameter input box, operation button, drop-down menu and status indicator icon in the base station configuration interface, and transmitting the position coordinates and category labels to the data processing and analysis module for comparison with the standard information in the locally stored basic configuration rules or the background configuration rule knowledge base to generate a first analysis result or a second analysis result.

[0027] In this embodiment, the model's output is parsed by the edge computing unit and output as a JSON object. The coordinates are normalized based on a 1920×1080 resolution, and the category labels can cover four categories: parameter input boxes, operation buttons, drop-down menus, and status indicator icons. Then, it is compared with the corresponding entries in the local SQLite rule table or the backend MongoDB knowledge base to generate the first or second analysis result, achieving millisecond-level error location.

[0028] Based on the above embodiments, as an optional embodiment, the MR glasses module in the cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses provided by the present invention includes: a high-definition camera for real-time acquisition of configuration interface images presented on the real computer module of the digital trunking base station; an IMU sensor for real-time detection of the operator's head movement posture; an edge computing unit for processing the configuration interface images to extract key elements and parameter data, comparing the parameter data with the locally stored basic configuration rules, and then performing preliminary analysis through a lightweight machine learning algorithm to obtain a first analysis result; and a computing mode switching unit for uploading the spatial-visual coupling raw data to the background processing module.

[0029] In this embodiment, the high-definition camera can employ dual 16-megapixel high-definition cameras to transmit 4K@30 fps YUV422 data to the edge computing unit via the HS-SPI bus; the IMU sensor can employ a nine-axis IMU to output attitude quaternions at 1kHz via the I2C interface; the computing mode switching unit has a built-in threshold register, which is set to trigger a GPIO high level to switch to data transmission when the CPU is >80% or the Score is >80 for 30 consecutive seconds; it also includes an optical waveguide display unit, which can adopt a nano-coating anti-strong light design, and can still maintain a brightness of 1000 nits under 100 klx direct sunlight, ensuring that the retinal projection text is clear and readable.

[0030] Based on the above embodiments, as an optional embodiment, in the cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses provided by the present invention, the background processing module includes: The intelligent analysis engine is built on a microservice architecture and uses a Long Short-Term Memory (LSTM) network to learn and predict operation sequences, and a Support Vector Machine (SVM) to classify and judge parameter data. A configuration rule knowledge base is configured and stored using MongoDB. This knowledge base stores the standard parameter ranges, configuration process rules, and common error cases for digital trunking base station configuration.

[0031] In this embodiment, the backend processing module runs on a Kubernetes microservice cluster. The intelligent analysis engine processes the temporal features of the operation sequence using an LSTM-128 network, classifies discrete parameters using SVM-RBF, and returns the fused results via a RESTful API. The configuration rule knowledge base is stored in a MongoDB replica set and supports hot updates, completing index reconstruction within 100 ms.

[0032] Based on the content of the above embodiments, as an optional embodiment, in the cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses provided by the present invention, the calculation formula for the data complexity score in the data processing and analysis module is as follows: ; in, , Indicates the size of the data. This represents the computational logic complexity.

[0033] Based on the above embodiments, as an optional embodiment, in the cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses provided by the present invention, the data processing and analysis module includes: The data acquisition unit is used to score the data complexity of the spatial-visual coupled raw data and monitor the local computing resource utilization rate in real time. The computing mode switching unit is used to generate a local computing mode start command and send it to the MR glasses module when the data complexity score is lower than a first threshold and the local computing resource utilization rate is lower than a second threshold, and to generate a data feedback mode start command and send it to the MR glasses module when the data complexity score is higher than or equal to the first threshold or the local computing resource utilization rate is higher than or equal to the second threshold.

[0034] Based on the above embodiments, as an optional embodiment, the real-time assistance and guidance module in the cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses provided by the present invention includes: The auxiliary computing unit is used to display error messages and configuration suggestions through spatial audio technology and retinal projection text; the guidance and support unit is used to retrieve the next operation process and generate control instructions that are overlaid and displayed in the form of virtual arrows, animation demonstrations, etc., so that the overlay display is in the operator's actual field of vision.

[0035] In this embodiment, the auxiliary computing unit of the real-time assistance and guidance module can integrate the HRTF spatial audio engine to render directional sound sources in real time based on head posture. The guidance support unit calls the Unity3D rendering pipeline to generate 4K virtual arrows on the optical waveguide. The vertex coordinates are aligned with the SLAM coordinate system and then superimposed on the screen of the real computer module at 30 fps. For example, if the maintenance personnel stare at the error input box for more than 500 ms without taking any action, the system will automatically play the next animation demonstration, achieving a total latency of <100 ms.

[0036] The following is a specific embodiment of the cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses of the present invention. Figure 2 This embodiment provides a second cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses, which optionally integrates spatial location information. Figure 2 As shown, the present invention implements a cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses that integrates spatial location information. The system includes an MR glasses module, a real computer module as a component of a digital trunking base station, and a back-end processing module.

[0037] When an operations and maintenance (O&M) technician is configuring a new digital trunking base station using this system, the technician first puts on MR glasses and begins operating the base station configuration software on the physical computer module. The dual 16-megapixel high-definition cameras of the MR glasses capture images of the configuration interface in real time at a frame rate of 60fps, while the nine-axis IMU sensor captures the operator's head movement posture data at a frequency of 1000Hz.

[0038] Secondly, the system initially operates in local computing mode. The lightweight image recognition model built into the MR glasses processes the acquired images and identifies key elements in the configuration interface, such as parameter input boxes and operation buttons. For example, the algorithm detects that maintenance personnel are entering base station transmit power parameters.

[0039] Then, when the system detects that the transmit power value entered by the maintenance personnel exceeds the normal range (e.g., 50W is entered, while the normal range should be 20-40W), the MR glasses immediately emit a warning sound through spatial audio technology, and mark the incorrect input box in red in the maintenance personnel's field of vision, while displaying the correct parameter range suggestion.

[0040] As the configuration process progressed, the system detected a data complexity score exceeding 80 points (according to the formula: (Based on calculations), it automatically switches to data feedback mode. At this time, the MR glasses transmit the acquired image and posture data to the backend processing module via the 5G-SA network.

[0041] In addition, the intelligent analysis engine in the backend processing module receives the data and performs in-depth analysis by combining it with the standard configuration information stored in the configuration rule knowledge base. For example, it discovers that when maintenance personnel configure frequency parameters, although the input values ​​are within the normal range, the frequency spacing with adjacent base stations is insufficient, which may lead to signal interference.

[0042] Based on this analysis, the MR glasses display a virtual frequency allocation map in the field of view of maintenance personnel, intuitively showing the frequency relationship between the current base station and neighboring base stations, and using animation to demonstrate how to adjust the frequency to avoid interference. Simultaneously, the system provides voice explanations, detailing the reasons and methods for the adjustments.

[0043] Once the configuration is complete, operations and maintenance personnel can review key operations throughout the configuration process via the MR glasses interface. The system also feeds back the configuration data to the backend for further optimization of the configuration rule knowledge base and analysis algorithms.

[0044] In this way, the system not only helps maintenance personnel quickly and accurately complete base station configuration, but also avoids potential parameter errors and performance issues, greatly improving configuration efficiency and accuracy. The entire configuration process has been shortened from the traditional 4-5 hours to 2-2.5 hours, and the configuration error rate has been reduced from 30% to below 10%.

[0045] The MR glasses module incorporates a lightweight image recognition model based on an improved YOLOv8 (achieving mAP@0.95 on the base station configuration interface element dataset), as well as lightweight edge computing units and computing mode switching units. The improved YOLOv8 algorithm employs a lightweight backbone network structure (such as replacing the CSPDarknet structure with the lighter MobileNet-v3 structure), combined with an attention mechanism (CBAM, Convolutional Block AttentionModule) to enhance the feature extraction capability of key elements of the base station configuration interface. During training, transfer learning techniques were employed, with pre-training using the publicly available COCO (Common Objects in Context) dataset. This dataset contains over 200,000 images covering 80 common object categories and is widely used in object detection. The pre-training phase used a stochastic gradient descent optimizer with a learning rate of 0.001 for 100 epochs. Fine-tuning was then performed using 5,000 specially collected base station configuration interface image data sets. These images were obtained through collaboration with multiple telecommunications operators in actual digital trunking base station configuration projects, using the MR glasses module described in this invention to capture images of base station configuration interfaces on real computer modules in different scenarios and time periods. To ensure data diversity and representativeness, the collection process covered various base station equipment models, different configuration interface layouts, and different operator habits. During the fine-tuning phase, the learning rate was adjusted to 0.0001, and training lasted for 50 epochs, resulting in a 30% improvement in detection speed and a 5% improvement in accuracy in base station configuration scenarios.

[0046] Spatial positioning is achieved using SLAM technology. The built-in interactive display unit utilizes optical waveguide technology, boasting a 120° ultra-wide field of view and 4K resolution, enabling real-time display of virtual guidance information, error messages, and operation flow instructions. It supports local computing mode and data feedback mode. The computing mode switching unit switches between modes based on data complexity scores and local computing resource utilization. The data complexity score is calculated using the following formula: ,in These are weighting coefficients, which can take values ​​of 0.6 and 0.4 respectively. Indicates the size of the data (in bytes). The computational complexity is represented by a comprehensive evaluation of factors such as the number of computational steps and conditional judgments for configuration parameters. Local computing resource utilization is determined by real-time monitoring of the CPU and memory usage of the edge computing unit. When the complexity of the configuration data exceeds the processing capacity threshold of the local edge computing unit (e.g., parameter computation complexity score > 80 points), or when the local computing resource utilization exceeds 80% for 30 consecutive seconds, the system automatically switches from local computing mode to data backhaul mode. When the network connection is restored and local computing resources are idle, the system can manually or automatically switch back to local computing mode. During the switching process, specific data caching and continuation processing mechanisms are employed, such as caching unfinished computational tasks locally at the moment of switching, and continuing processing or transmitting them to the background after the mode switch is complete. The MR glasses module runs on a Linux-based embedded operating system, while the image recognition module, local computing module, and other software are developed using C++, ensuring system stability and efficiency.

[0047] Regarding the MR glasses module: All hardware components are connected via a high-speed serial bus (HS-SPI) and an integrated circuit bus (I2C) to ensure stable data transmission. The dual 16-megapixel high-definition cameras (supporting 4K@30fps video capture) utilize the Sony IMX586 sensor, which boasts excellent low-light shooting performance and high dynamic range. Even in the complex lighting conditions of a base station equipment room, it can clearly capture images of the configuration interface and equipment, providing high-quality data for subsequent image recognition. The nine-axis IMU sensor (angular velocity measurement range ±2000° / s, acceleration measurement range ±16g) consists of a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, enabling real-time and accurate detection of the operator's head movements, including rotation, tilt, and movement.

[0048] The real computer module, as an inherent component of the digital trunking base station, connects to the base station's baseband unit and radio frequency unit via fiber optic and Ethernet interfaces, working together to complete the base station's signal processing and transmission functions. It runs professional base station configuration software, developed based on the Windows Server operating system, for operators to configure base station parameters. The real computer module's display output interface adopts the HDMI standard, facilitating image acquisition by the MR glasses module's camera without interfering with the normal operation of the computer.

[0049] The backend processing module is built on a microservice architecture, with each functional module communicating via a RESTful API. It includes a configuration rule knowledge base and an intelligent analysis engine. The configuration rule knowledge base uses MongoDB for storage, facilitating rapid querying and updating of configuration rule data. It stores a large amount of data, including standard parameter ranges, configuration process rules, and common error cases for digital trunking base station configurations. The intelligent analysis engine uses natural language processing and machine learning algorithms to receive configuration interface image data and related information transmitted from the MR glasses module, performing real-time analysis and judgment on the base station configuration process. The intelligent analysis engine uses a sequence model based on Long Short-Term Memory (LSTM) networks to learn and predict the operator's configuration operation sequences, combined with Support Vector Machine (SVM) for parameter data classification and judgment. Natural language processing technology is used to parse the text rules in the configuration rule knowledge base and transform them into executable judgment logic. The backend processing module is deployed on a Linux server cluster, and the intelligent analysis engine is developed using Python to fully leverage its advantages in data processing and machine learning.

[0050] Further, the real-time data acquisition steps are as follows: The MR glasses module acquires image data of the base station configuration interface on the real computer module at a frame rate of 60fps, while the IMU sensor acquires head movement posture data of the operator at a frequency of 1000Hz. The image data is stored in JPEG format, and the posture data is encapsulated in JSON format for easy subsequent processing and transmission.

[0051] Data processing and analysis steps: Local Computation Mode: The MR glasses module preprocesses the acquired image data, using a built-in improved YOLOv8 image recognition algorithm to extract key elements (such as parameter input boxes, operation buttons, etc.) and parameter data from the configuration interface. This data is compared with the basic configuration rules pre-stored within the MR glasses module. A lightweight machine learning algorithm (optimized using model pruning and quantization techniques, reducing the number of model parameters by 50% and converting data types from 32-bit floating-point numbers to 8-bit integers) performs a preliminary analysis of the operator's behavior patterns to determine if the current operation conforms to the normal configuration process. If obvious anomalies such as parameter input errors, missing operation steps, or incorrect sequence occur, the error type and risk level are directly calculated locally. The improved YOLOv8 algorithm, LSTM, SVM, and other technologies used in this invention are fundamentally different from commonly referred to large models (such as the GPT series, which are based on large-scale parameter training and have general language understanding and generation capabilities). The algorithm and model of this invention are designed for the specific field of base station configuration, focusing on specific tasks such as image recognition, operation sequence analysis and parameter judgment. Through training and optimization on a specific dataset, it achieves high-precision base station configuration assistance functions, rather than pursuing general language or task processing capabilities.

[0052] Data Backhaul Mode: The MR glasses module transmits the collected raw image data, posture data, and necessary preprocessing information to the backend processing module via a 5G-SA network. A mature HTTP / 3-based compressed transmission protocol is used for data transmission. This protocol achieves efficient data compression and stable transmission in practical applications, meeting the data backhaul requirements of this invention. The intelligent analysis engine uses image recognition algorithms to extract key elements and parameter data from the configuration interface. It then performs in-depth comparison and analysis based on standard configuration information in the configuration rule knowledge base. Through machine learning algorithms based on LSTM and SVM, it comprehensively analyzes the operator's operational behavior patterns, accurately determining whether the current operation conforms to the normal configuration process and calculating the type and risk level of various error situations. The URL of the data upload interface is https: / / backendserver.com / data-upload, the request method is POST, the parameter list includes image data, posture data, device identifier, etc., and the response format is JSON, containing processing status and result information.

[0053] Real-time assistance and guidance: Regardless of whether in local computing mode or data feedback mode, when an anomaly is detected during the configuration process, the MR glasses module uses spatial audio technology (positioning accuracy ±15°) and retinal projection text to highlight the error location and cause within the operator's field of vision, and provides correct configuration suggestions. If the operator stays in the configuration process for more than a preset threshold (e.g., 30 seconds without effective operation), the MR glasses module automatically retrieves the next operation flow from local storage or the backend processing module, and displays it overlaid in the real configuration scene in the form of virtual arrows, animation demonstrations, etc., to guide the operator to complete the subsequent configuration tasks. Simultaneously, the system records key operation data throughout the configuration process at 500ms intervals, allowing operators to review the operation backtracking at any time on the MR glasses interface. The MR glasses module can also update and optimize the local configuration rule base and calculation model based on optimized local computing rules or model parameters fed back by the backend processing module.

[0054] Practical supplement Network Environment Adaptability: When the system is in a scenario without 5G-SA network coverage or with poor network conditions (such as weak signal or high latency), the MR glasses module automatically enters a low-bandwidth mode. In this mode, the image acquisition frame rate is reduced to 30fps, and a more efficient compression algorithm is used (such as WebP format replacing JPEG format, further increasing the image data compression rate to 80%) to ensure stable data transmission. If the network connection is interrupted, the MR glasses module will prioritize local computing mode for operation assistance; when local computing resources are insufficient, the system will temporarily store important configuration data to the local storage device and automatically upload it after the network is restored.

[0055] Reliability Verification in Complex Environments: The MR glasses module underwent a 72-hour continuous stability test under high temperature (e.g., 50℃) and high humidity (e.g., 90% RH) conditions. Test results showed that the dual 16-megapixel high-definition cameras maintained clear imaging with no significant decrease in image resolution; the measurement error of the nine-axis IMU sensor was controlled within ±3%. In environments with strong electromagnetic interference (e.g., electromagnetic intensity reaching 100V / m), through shielding design and optimized filtering circuits, the MR glasses module's SLAM positioning technology still achieved accurate positioning with an error of less than 5cm. Simultaneously, the stability of the image recognition algorithm in complex environments was tested. Even under conditions of drastic changes in lighting and partial occlusion of interface elements, the algorithm's accuracy remained above 90%.

[0056] This invention also provides a cross-modal intelligent operation and maintenance method based on lightweight mixed reality glasses, implemented based on the aforementioned cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses, including: The configuration interface image displayed on the actual computer module of the digital trunking base station is acquired in real time, and the head movement posture data of the operator is acquired simultaneously, so as to obtain the original spatial-visual coupling data containing the configuration interface image and the operator's line of sight. The data complexity of the original spatial-visual coupling data is scored, and the local computing resource utilization is monitored in real time. When the data complexity score is lower than the first threshold and the local computing resource utilization rate is lower than the second threshold, the local computing mode is started to process the configuration interface image to extract key elements and parameter data, and the parameter data is compared with the basic configuration rules stored locally. Then, a preliminary analysis is performed through a lightweight machine learning algorithm to obtain the first analysis result. When the data complexity score is higher than or equal to the first threshold or the local computing resource utilization rate is higher than or equal to the second threshold, the data feedback mode is activated, and the intelligent analysis engine based on LSTM and SVM is called and combined with the configuration rule knowledge base to perform in-depth analysis on the original data to obtain the second analysis result. In local computing mode, based on the first analysis result, or in data backhaul mode, based on the second analysis result, error prompts, correct configuration suggestions, and next operation procedures are superimposed and displayed in the operator's real field of vision through spatial audio and retinal projection, so as to guide the operator to complete the base station configuration in real time.

[0057] It should be noted that the cross-modal intelligent operation and maintenance method based on lightweight mixed reality glasses provided in this embodiment of the invention can be implemented based on the cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses described in any of the above embodiments during specific operation, and will not be elaborated on in this embodiment.

[0058] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses, characterized in that, The system includes: The MR glasses module is used to acquire the configuration interface image displayed on the real computer module of the digital trunking base station in real time, and simultaneously acquire the head movement posture data of the operator, thereby obtaining the original spatial-visual coupling data containing the configuration interface image and the operator's gaze direction. The data processing and analysis module is communicatively connected to the MR glasses module and is used to score the data complexity of the spatial-visual coupling raw data and monitor the local computing resource utilization rate in real time. When the data complexity score is lower than the first threshold and the local computing resource utilization is lower than the second threshold, the local computing mode is activated. The MR glasses module processes the configuration interface image to extract key elements and parameter data, compares the parameter data with the basic configuration rules stored locally, and then performs preliminary analysis through a lightweight machine learning algorithm to obtain the first analysis result. When the data complexity score is higher than or equal to the first threshold or the local computing resource utilization rate is higher than or equal to the second threshold, the data backhaul mode is activated. The MR glasses module uploads the spatial-visual coupling raw data to the background processing module. The background processing module is used to call the intelligent analysis engine based on LSTM and SVM and combine it with the configuration rule knowledge base to perform in-depth analysis on the raw data to obtain the second analysis result. The real-time assistance and guidance module is communicatively connected to the data processing and analysis module and the MR glasses module. It is used to overlay error prompts, correct configuration suggestions and next operation procedures onto the operator's real field of vision through spatial audio and retinal projection, based on the first analysis result in local computing mode or the second analysis result in data backhaul mode, so as to guide the operator to complete the base station configuration in real time.

2. The cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses according to claim 1, characterized in that, The MR glasses module uses a lightweight image recognition model to process the configuration interface image to extract key elements and parameter data; the lightweight image recognition model uses YOLOv8 as the baseline network and is obtained through the following methods: Replace the CSPDarknet structure in the backbone network with the MobileNet-v3 structure; Embed a CBAM attention mechanism module at the output of the MobileNet-v3 backbone network; A transfer learning strategy is adopted. In the first stage, the COCO dataset is used for pre-training. The pre-training process uses a stochastic gradient descent optimizer and sets the training hyperparameters of the first model for pre-training. The first round is the first round of the pre-training process. In the second stage, a local preset dataset is used for fine-tuning, and the round of the fine-tuning stage is the first round.

3. The cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses according to claim 2, characterized in that, The output of the lightweight image recognition model is configured to output the position coordinates and category labels of the parameter input box, operation button, drop-down menu and status indicator icon in the base station configuration interface, and transmit the position coordinates and category labels to the data processing and analysis module for comparison with the standard information in the locally stored basic configuration rules or the background configuration rule knowledge base to generate a first analysis result or a second analysis result.

4. The cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses according to claim 3, characterized in that, MR glasses module, including: High-definition camera, used to capture images of the configuration interface displayed on the actual computer module of the digital trunking base station in real time; IMU sensor is used to detect the operator's head movement posture in real time; An edge computing unit is used to process the configuration interface image to extract key elements and parameter data, compare the parameter data with the basic configuration rules stored locally, and then perform preliminary analysis through a lightweight machine learning algorithm to obtain a first analysis result. The computing mode switching unit is used to upload the spatial-visual coupling raw data to the background processing module.

5. The cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses according to claim 1, characterized in that, The background processing module includes: The intelligent analysis engine is built on a microservice architecture and uses a Long Short-Term Memory (LSTM) network to learn and predict operation sequences, and a Support Vector Machine (SVM) to classify and judge parameter data. A configuration rule knowledge base is configured, which is stored using MongoDB. This knowledge base stores the standard parameter ranges, configuration process rules, and common error cases for digital trunking base station configuration.

6. The cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses according to claim 1, characterized in that, The formula for calculating the data complexity score in the data processing and analysis module is as follows: ; in, These are the weighting coefficients. Indicates the size of the data. This represents the computational logic complexity.

7. The cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses according to claim 1, characterized in that, The data processing and analysis module includes: The data acquisition unit is used to score the data complexity of the spatial-visual coupled raw data and monitor the local computing resource utilization rate in real time. The computing mode switching unit is used to generate a local computing mode start command and send it to the MR glasses module when the data complexity score is lower than a first threshold and the local computing resource utilization rate is lower than a second threshold, and to generate a data feedback mode start command and send it to the MR glasses module when the data complexity score is higher than or equal to the first threshold or the local computing resource utilization rate is higher than or equal to the second threshold.

8. The cross-modal intelligent operation and maintenance system based on lightweight mixed reality glasses according to claim 1, characterized in that, The real-time assistance and guidance module includes: An auxiliary computing unit is used to display error messages and configuration suggestions using spatial audio technology and retinal projection text. The guidance and support unit is used to retrieve the next operation process and generate control instructions that are overlaid and displayed in the form of virtual arrows, animation demonstrations, etc., so that the overlay display is in the operator's actual field of vision.

9. A cross-modal intelligent operation and maintenance method based on lightweight mixed reality glasses, implemented based on the system described in any one of claims 1-8, characterized in that, include: The configuration interface image displayed on the actual computer module of the digital trunking base station is acquired in real time, and the head movement posture data of the operator is acquired simultaneously, so as to obtain the original spatial-visual coupling data containing the configuration interface image and the operator's line of sight. The data complexity of the original spatial-visual coupling data is scored, and the local computing resource utilization is monitored in real time. When the data complexity score is lower than the first threshold and the local computing resource utilization rate is lower than the second threshold, the local computing mode is started to process the configuration interface image to extract key elements and parameter data, and the parameter data is compared with the basic configuration rules stored locally. Then, a preliminary analysis is performed through a lightweight machine learning algorithm to obtain the first analysis result. When the data complexity score is higher than or equal to the first threshold or the local computing resource utilization rate is higher than or equal to the second threshold, the data feedback mode is activated, and the intelligent analysis engine based on LSTM and SVM is called and combined with the configuration rule knowledge base to perform in-depth analysis on the original data to obtain the second analysis result. In local computing mode, based on the first analysis result, or in data backhaul mode, based on the second analysis result, error prompts, correct configuration suggestions, and next operation procedures are superimposed and displayed in the operator's real field of vision through spatial audio and retinal projection, so as to guide the operator to complete the base station configuration in real time.

Images