Mixed reality intelligent inspection method and system based on multi-modal fusion

Abstract

The invention relates to the technical field of mixed reality and artificial intelligence fusion application, in particular to a mixed reality intelligent inspection method and system based on multi-modal fusion. According to the technical scheme, the method comprises the following steps: synchronously acquiring a user view image and voice stream data through mixed reality equipment; performing target detection on the view image by adopting an improved YOLOv10 model, and outputting a candidate target area and a confidence score # imgabs0 # thereof; voice input is transferred through a Whisper voice recognition model, and a semantic vector # imgabs1 # is generated by using a BERT semantic embedding model. According to the invention, through a multi-mode cooperative sensing mechanism of fusing image, voice and space gazing information, in combination with context semantic optimization and lightweight terminal deployment, intelligent target identification and natural interaction with high precision, real-time performance and strong robustness are realized in a complex industrial inspection scene.

Description

Technical Field

[0001] The present invention relates to the technical field of application of the fusion of mixed reality and artificial intelligence, and in particular to a mixed reality intelligent inspection method and system based on multimodal fusion. Background Art

[0002] Traditional industrial inspection methods mainly rely on manual observation and single-modal detection, and have problems such as low recognition accuracy, poor spatial understanding, and low interaction efficiency. Although mixed reality devices are introduced in some scenarios to achieve visualization enhancement, most current systems only have image enhancement or simple voice command recognition capabilities, and lack the fusion analysis and linkage response mechanism for multi-source information such as images, voice, and space. At the same time, existing solutions generally rely on cloud processing, which is subject to communication delays and data privacy requirements, and it is difficult to meet the needs of real-time intelligent interaction in complex environments. Therefore, this application proposes a mixed reality intelligent inspection method and system based on multimodal fusion. Summary of the Invention

[0003] The purpose of the present invention is to address the problems of low recognition accuracy, poor interaction efficiency and insufficient spatial perception in existing industrial inspection technologies, and to propose a mixed reality intelligent inspection method and system based on multimodal fusion.

[0004] In a first aspect, the present application provides a mixed reality intelligent inspection method based on multimodal fusion, comprising the following steps: S1, synchronously collecting user field of view images and voice stream data through a mixed reality device; S2. Use the improved YOLOv10 model to perform target detection on the field of view image and output the candidate target area and its confidence score ; S3. Transcribe the speech input using the Whisper speech recognition model and generate semantic vectors using the BERT semantic embedding model. , and compare it with the prior semantic vector of the candidate target Perform matching; S4. Calculate the user's gaze direction vector based on the IMU information and depth image provided by the mixed reality device , and the candidate target center direction vector Calculate spatial directional consistency; S5. Build contextual semantic memory by combining multiple rounds of speech input and optimize the semantic matching process through the attention mechanism. S6. Calculate the final score of the candidate target through the multimodal fusion function; S7. Based on the fusion scoring results, the target is highlighted on the mixed reality interface, and voice feedback is triggered to complete the intelligent inspection interactive closed loop.

[0005] Optionally, in step S6, the final scoring formula is:

[0006]

[0007] in, is the final score of the candidate target calculated by the multimodal fusion function, 、 、 is the weight coefficient used in the fusion scoring decision, The user's gaze direction vector Direction vector to the candidate target center The calculated spatial direction consistency score is To pass the current speech semantic vector and the prior semantic vector of the candidate target The semantic matching score obtained by matching.

[0008] Optionally, the improved YOLOv10 model integrates a multi-scale feature pyramid structure and a spatial attention module for real-time target detection on edge devices.

[0009] Optionally, in step S3, semantic matching uses the semantic vector cosine similarity calculation formula:

[0010]

[0011] in, To represent the current speech semantic vector and the semantic vector of the candidate target semantic matching degree.

[0012] Optionally, in step S4, the spatial direction consistency is calculated using the direction cosine formula:

[0013]

[0014] in, is the user's gaze direction vector, is the direction vector of the candidate target center.

[0015] Optional, weight coefficient used in ensemble scoring decision , , It can be dynamically adjusted based on the context, and the optimization strategy is based on the following objective function:

[0016]

[0017] in, is the fusion loss function, including It is the image detection loss, used to optimize the target detection accuracy; It is the semantic matching loss, which is used to optimize the correspondence between speech and image semantics; is the spatial direction consistency error term, which is used to optimize the consistency between the user's gaze direction and the target direction; 、 、 They are image detection losses , semantic matching loss , spatial direction consistency error term The weighting coefficients can be automatically optimized through the training process or set empirically to balance the contributions of each mode.

[0018] In a second aspect, the present application provides a mixed reality intelligent inspection system based on multimodal fusion for executing the mixed reality intelligent inspection method based on multimodal fusion described in the first aspect, comprising the following modules integrated into an MR head-mounted device:

[0019] Image perception module: used for field of view image acquisition and target detection;

[0020] Speech recognition module: used for transcribing speech stream data;

[0021] Semantic reasoning module: used to generate semantic vectors and perform matching;

[0022] Spatial direction calculation module: used for gaze direction and target direction calculation;

[0023] Fusion decision module: used for multimodal rating fusion;

[0024] Mixed reality augmented display module: used for target highlighting and voice feedback.

[0025] Optionally, the fusion decision module adopts a Transformer architecture to model image features, semantic vectors and spatial direction vectors respectively and then performs cross-modal weighted fusion decision.

[0026] Compared with the prior art, this application has at least one of the following beneficial technical effects:

[0027] The present invention forms a multimodal collaborative perception mechanism by fusing image detection, speech semantic recognition and spatial gaze information, which can effectively improve the accuracy and stability of target recognition in complex inspection environments.

[0028] The present invention combines contextual semantic memory with the attention mechanism to optimize the intention recognition process in multi-round voice interaction and enhance the continuity and natural fluency of human-computer interaction.

[0029] The present invention adopts a lightweight model design, and all algorithm modules can be independently deployed locally on the terminal, with field application advantages such as high real-time performance, high computing efficiency, and low network dependence.

[0030] The system structure of the present invention is universal and applicable to various high-security inspection scenarios such as nuclear facilities, power equipment, and industrial workshops, and has good engineering applicability and promotion value.

[0031] The present invention achieves high-precision, real-time, and highly robust intelligent target recognition and natural interaction in complex industrial inspection scenarios through a multimodal collaborative perception mechanism that integrates image, voice, and spatial gaze information, combined with contextual semantic optimization and lightweight terminal deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of a mixed reality intelligent inspection method based on multimodal fusion proposed by the present invention.

[0033] Figure 2 This is the network structure diagram of the improved YOLOv10 model proposed in this invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below with reference to the accompanying drawings and specific embodiments.

[0035] Example

[0036] The present invention proposes a mixed reality intelligent inspection method based on multimodal fusion, the method flow is as follows Figure 1 As shown in the figure, the terminal device that implements the mixed reality intelligent inspection method based on multimodal fusion is equipped with an image acquisition module, a voice acquisition module, a spatial perception module, a data fusion module, and an interaction enhancement module. These modules are integrated and deployed locally on the terminal to collaboratively complete intelligent inspection tasks. The entire system does not rely on external server resources and has the ability to operate independently and in real time on the terminal side.

[0037] Image Capture and Object Detection Module: The mixed reality terminal uses a front-facing HD camera to capture real-time video streams from the inspection site. The captured images are processed through a pre-processing module for noise reduction, illumination compensation, and normalization to adapt to the complex lighting and occlusion environments that may exist in industrial sites.

[0038] After image preprocessing, the image is input into the improved YOLOv10 target detection model. This model introduces the following optimization design based on the traditional YOLOv10 structure. Figure 2 As shown:

[0039] First, we design the C2f_RFAConv module, embedding the RFA attention mechanism within the C2f architecture to enhance feature extraction and improve the network's perception of multi-scale objects. Secondly, we replace the original feature fusion module with a BiFPN architecture using channel-weighted fusion to achieve efficient interaction and fusion of multi-layer feature information. Finally, we introduce Shape-IoU into the loss function to further improve the model's accuracy in object shape matching.

[0040] The model outputs a set of candidate targets, recording the location information (center coordinates and bounding box), category label and confidence score of each detected target .

[0041] Voice Capture and Semantic Understanding Module: The mixed reality terminal's built-in microphone captures patrol personnel's natural speech input in real time. Taking into account the diverse dialects, colloquial expressions, and ambient noise present on-site, the system incorporates adaptive voice noise reduction and dynamic sound source localization modules to improve the signal-to-noise ratio (SNR) of speech recognition and the accuracy of speech segmentation.

[0042] The speech signal is input into the Whisper model for on-device speech transcription, obtaining preliminary text transcription results. The transcription results are input into the BERT semantic embedding model to generate high-dimensional semantic vectors. The system builds a multi-level semantic dictionary of target category labels in advance and generates semantic embedding vectors for each category. .

[0043] The cosine similarity is used to calculate the matching degree between the current speech semantic vector and each detection target category label vector:

[0044]

[0045] in, To represent the current speech semantic vector and the semantic vector of the candidate target semantic matching degree.

[0046] The system also introduces semantic extension dictionaries (such as industry synonym lists and hyponym maps) to enhance the robustness of semantic matching and adapt to the differences in semantic expression among inspection personnel.

[0047] Spatial direction analysis module: used to obtain the gaze direction information of the patrol personnel in real time to guide target screening. The IMU module collects the head posture angular velocity, acceleration and Euler angle data in real time, combines the three-dimensional scene point cloud data obtained by the depth camera, and calculates the current gaze direction vector through the IMU and visual fusion algorithm. For each candidate target, calculate its spatial center direction vector , and calculate the spatial direction consistency score:

[0048]

[0049] in, is the user's gaze direction vector, is the direction vector of the candidate target center.

[0050] The system further introduces a dynamic field of view width correction mechanism:

[0051] The spatial angle tolerance range for small targets at a distance is relaxed; the judgment threshold for large targets at a close distance is tightened; and the spatial matching sensitivity is dynamically adjusted in combination with the depth perception model to ensure that the gaze direction score has a reasonable judgment effect at all viewing distances.

[0052] Contextual semantic memory mechanism: During multiple rounds of continuous voice interaction, the system dynamically constructs a semantic context graph. Each round of voice semantic vector Store it in the semantic memory library, and introduce a time decay mechanism to control the historical semantic contribution.

[0053] In the current semantic matching process, the system introduces historical reinforcement items:

[0054]

[0055] in, : Enhanced semantic matching score, taking into account the similarity between current semantics and historical semantics; : The semantic vector of the current speech and candidate target semantic vector The basic similarity score (such as using cosine similarity); :To the front Accumulate the historical speech semantic vectors of each round; Indicates the The historical semantic contribution coefficient of each round. This mechanism effectively mitigates the critical jump phenomenon caused by slight fluctuations in the user's expression during continuous voice interaction, ensuring the coherence and stability of the overall interaction intention; : Current speech semantic vector With the Turn history speech semantic vector similarity;

[0056] Multimodal fusion decision module: The fusion decision module integrates the image detection confidence, spatial consistency score and semantic matching score to perform weighted fusion decision:

[0057]

[0058] in, is the final score of the candidate target calculated by the multimodal fusion function, 、 、 is the weight coefficient used in the fusion scoring decision, The user's gaze direction vector Direction vector with the candidate target center The calculated spatial direction consistency score is To pass the current speech semantic vector and the prior semantic vector of the candidate target The semantic matching score obtained by matching. The weight coefficient is dynamically and adaptively adjusted through the system learning process, taking into account factors such as the complexity of the current inspection task, the degree of target occlusion, the degree of speech ambiguity, and the credibility of the spatial gaze direction to comprehensively optimize the weight distribution.

[0059] The fusion decision module further integrates a lightweight Transformer structure and performs cross-modal feature interactive learning on the three types of modal feature streams before fusion calculation, enhancing the system's fusion stability and anomaly resolution capabilities in complex interference environments.

[0060] Model training and overall optimization mechanism: The overall system training process uses a joint multimodal loss function for end-to-end optimization:

[0061]

[0062] in, is the fusion loss function, including It is the image detection loss, used to optimize the target detection accuracy; It is the semantic matching loss, which is used to optimize the correspondence between speech and image semantics; is the spatial direction consistency error term, which is used to optimize the consistency between the user's gaze direction and the target direction; 、 、 They are image detection losses , semantic matching loss , spatial direction consistency error term The weighting coefficients can be automatically optimized through the training process or set empirically to balance the contributions of each mode.

[0063] Enhanced display and interactive feedback module: when integrated scoring After exceeding the trust threshold, the system determines that the current target is the object of user attention. The mixed reality interface generates enhanced display effects in real time, including:

[0064] The three-dimensional highlighted border is superimposed with the target label;

[0065] Guidance arrows point to high-risk components or key inspection areas;

[0066] Floating text prompts the current equipment status and inspection points;

[0067] Synchronous voice broadcast of recognition results, risk warnings and follow-up inspection suggestions.

[0068] The system supports users to complete confirmation through gaze maintenance and simple voice commands, avoiding the burden of manual operation and adapting to natural human-computer collaborative interaction in complex on-site environments without manual control.

[0069] In summary, the present invention provides a mixed reality intelligent inspection method based on multimodal fusion. By locally integrating three types of information sources, namely image detection, speech semantic understanding and spatial direction perception, in the mixed reality terminal, a dynamic adaptive multimodal fusion decision-making mechanism is constructed, which realizes high-accuracy, high-real-time and high-naturalness human-machine collaborative inspection in complex industrial environments. The system proposes targeted improvements in terms of lightweight image detection algorithm, continuous optimization of semantic intent, spatial gaze direction consistency modeling and cross-modal joint feature learning, breaking through the technical bottlenecks of existing inspection systems in terms of single-modal recognition capability, speech interaction stability and spatial scene adaptability, and has broad engineering application value and industrial promotion prospects. The technical solution is particularly suitable for nuclear facility inspection, high-voltage power detection, intelligent manufacturing equipment maintenance and remote complex scene intelligent inspection tasks, and provides effective technical support for multimodal human-computer interaction technology in the field of industrial production safety.

[0070] Example 2

[0071] This embodiment provides a mixed reality intelligent inspection system based on multimodal fusion for executing the mixed reality intelligent inspection method based on multimodal fusion of embodiment 1, including the following modules integrated into an MR head-mounted device:

[0072] Image perception module: used for field of view image acquisition and target detection;

[0073] Speech recognition module: used for transcribing speech stream data;

[0074] Semantic reasoning module: used to generate semantic vectors and perform matching;

[0075] Spatial direction calculation module: used for gaze direction and target direction calculation;

[0076] Fusion decision module: used for multimodal rating fusion;

[0077] Mixed reality augmented display module: used for target highlighting and voice feedback.

[0078] The fusion decision module adopts the Transformer architecture, models the image features, semantic vectors and spatial direction vectors respectively, and then performs cross-modal weighted fusion decision.

[0079] The system, integrated into an MR headset, uses multiple specialized modules working together to simultaneously process multimodal information, including images and speech. The image perception module utilizes an improved YOLOv10 model for real-time target detection, adapting to edge devices and ensuring no noticeable delays during inspections. The speech recognition and semantic reasoning modules combine to transcribe speech into semantic vectors and match them with the target's prior semantic vector. This, combined with contextual semantic memory built from multiple rounds of speech, optimizes matching through an attention mechanism, improving the accuracy of understanding user intent. The spatial direction calculation module uses device IMU information and depth images to calculate the consistency between the user's gaze direction and the target's center direction, enhancing the accuracy of determining the user's focus.

[0080] The fusion decision module utilizes a Transformer architecture to perform a cross-modal weighted fusion of image, semantic, and spatial orientation information. Weights are dynamically adjusted based on context, adapting to diverse inspection scenarios and optimizing final scoring results. The mixed reality augmented display module creates a complete interactive closed loop by highlighting targets and providing voice feedback. This allows users to access information without manual intervention, making it particularly suitable for scenarios where both hands may be occupied.

[0081] In addition, the modular design of the system makes each functional module independent and clear, which is convenient for individual upgrades and maintenance. The modules work together to improve the efficiency and accuracy of inspections, making intelligent inspections more convenient and efficient.

[0082] The above specific embodiments are merely several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant inspirations of the above embodiments, those skilled in the art may make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A mixed reality intelligent inspection method based on multimodal fusion, characterized in that: The following steps are involved: S1. Synchronously collect user visual field images and voice stream data through mixed reality devices; S2. Use the improved YOLOv10 model to perform target detection on the field of view image and output the candidate target area and its confidence score ; S3. Transcribe the speech input using the Whisper speech recognition model and generate semantic vectors using the BERT semantic embedding model. , and compare it with the prior semantic vector of the candidate target Perform matching; S4. Calculate the user's gaze direction vector based on the IMU information and depth image provided by the mixed reality device , and the candidate target center direction vector Calculate spatial directional consistency; S5. Build contextual semantic memory by combining multiple rounds of speech input and optimize the semantic matching process through the attention mechanism. S6. Calculate the final score of the candidate target through the multimodal fusion function; S7. Based on the fusion scoring results, the target is highlighted on the mixed reality interface, and voice feedback is triggered to complete the intelligent inspection interactive closed loop.

2. The mixed reality intelligent inspection method based on multimodal fusion according to claim 1 is characterized in that: In step S6, the final scoring formula is: in, is the final score of the candidate target calculated by the multimodal fusion function, 、 、 is the weight coefficient used in the fusion scoring decision, The user's gaze direction vector Direction vector with the candidate target center The calculated spatial direction consistency score is To pass the current speech semantic vector and the prior semantic vector of the candidate target The semantic matching score obtained by matching.

3. The mixed reality intelligent inspection method based on multimodal fusion according to claim 1 is characterized in that: The improved YOLOv10 model integrates a multi-scale feature pyramid structure and a spatial attention module for real-time object detection on edge devices.

4. The mixed reality intelligent inspection method based on multimodal fusion according to claim 1 is characterized in that: In step S3, semantic matching is performed using the semantic vector cosine similarity calculation formula: in, To represent the current speech semantic vector and the semantic vector of the candidate target semantic matching degree.

5. The mixed reality intelligent inspection method based on multimodal fusion according to claim 1 is characterized in that: In step S4, the spatial direction consistency is calculated using the direction cosine formula: in, is the user's gaze direction vector, is the direction vector of the candidate target center.

6. The mixed reality intelligent inspection method based on multimodal fusion according to claim 1 is characterized in that: Weight coefficient used in fusion scoring decision , , It can be dynamically adjusted based on the context, and the optimization strategy is based on the following objective function: in, is the fusion loss function, including It is the image detection loss, used to optimize the target detection accuracy; It is the semantic matching loss, which is used to optimize the correspondence between speech and image semantics; is the spatial direction consistency error term, which is used to optimize the consistency between the user's gaze direction and the target direction; 、 、 They are image detection losses , semantic matching loss , spatial direction consistency error term The weighting coefficients can be automatically optimized through the training process or set empirically to balance the contributions of each mode.

7. A mixed reality intelligent inspection system based on multimodal fusion for executing the mixed reality intelligent inspection method based on multimodal fusion according to any one of claims 1 to 6, characterized in that: Includes the following modules integrated into MR headsets: Image perception module: used for field of view image acquisition and target detection; Speech recognition module: used for transcribing speech stream data; Semantic reasoning module: used to generate semantic vectors and perform matching; Spatial direction calculation module: used for gaze direction and target direction calculation; Fusion decision module: used for multimodal rating fusion; Mixed reality augmented display module: used for target highlighting and voice feedback.

8. The mixed reality intelligent inspection system based on multimodal fusion according to claim 7 is characterized in that: The fusion decision module adopts the Transformer architecture, models the image features, semantic vectors and spatial direction vectors respectively, and then performs cross-modal weighted fusion decision.

Images