A mixed reality intelligent inspection method and system based on multi-modal fusion

Abstract

The present application relates to the technical field of mixed reality and artificial intelligence fusion application, and particularly relates to a mixed reality intelligent inspection method and system based on multi-modal fusion. The technical scheme comprises the following steps: synchronously collecting user visual field image and voice stream data through a mixed reality device; using an improved YOLOv10 model to detect the visual field image, outputting candidate target regions and their confidence scores; and using a Whisper voice recognition model to transcribe voice input and using a BERT semantic embedding model to generate semantic vectors. Through the multi-modal collaborative perception mechanism of fusing image, voice and spatial gaze information, combined with context semantic optimization and lightweight terminal deployment, the present application realizes high-precision, real-time and robust intelligent target recognition and natural interaction in complex industrial inspection scenarios.

Description

Technical Field

[0001] This invention relates to the field of mixed reality and artificial intelligence fusion application technology, and in particular to a mixed reality intelligent inspection method and system based on multimodal fusion. Background Technology

[0002] Traditional industrial inspection methods primarily rely on manual observation and single-modal detection, resulting in low recognition accuracy, poor spatial understanding, and low interaction efficiency. While mixed reality devices are introduced in some scenarios to enhance visualization, current systems mostly only possess image enhancement or simple voice command recognition capabilities, lacking mechanisms for the fusion and analysis of multi-source information such as images, voice, and spatial data, as well as for coordinated response. Furthermore, existing solutions generally depend on cloud processing, which, constrained by communication latency and data privacy requirements, makes it difficult to meet the real-time intelligent interaction needs in complex environments. Therefore, this application proposes a mixed reality intelligent inspection method and system based on multimodal fusion. Summary of the Invention

[0003] The purpose of this invention is to address the problems of low accuracy, poor interaction efficiency, and insufficient spatial perception in existing industrial inspection and identification technologies, and to propose a mixed reality intelligent inspection method and system based on multimodal fusion.

[0004] In a first aspect, this application provides a mixed reality intelligent inspection method based on multimodal fusion, including the following steps: S1, synchronously acquiring user field-view images and voice stream data through a mixed reality device;

[0005] S2. The improved YOLOv10 model is used to perform target detection on the field of view image, and candidate target regions and their confidence scores are output. ;

[0006] S3. Transcribe the speech input using the Whisper speech recognition model and generate semantic vectors using the BERT semantic embedding model. This is compared with the prior semantic vector of the candidate target. Perform a match;

[0007] S4. Calculate the user's gaze direction vector based on the IMU information and depth image provided by the mixed reality device. and the direction vector of the candidate target center Consistency of computational space orientation;

[0008] S5. Combine multi-turn voice input to build contextual semantic memory, and optimize the semantic matching process through an attention mechanism;

[0009] S6. Calculate the final score of the candidate target using the multimodal fusion function;

[0010] S7. Based on the fusion scoring results, the target is highlighted on the mixed reality interface and voice feedback is triggered to complete the intelligent inspection interaction loop.

[0011] Optionally, in step S6, the final scoring formula is:

[0012]

[0013] in, The final score of the candidate targets is calculated using the multimodal fusion function. , , To integrate the weighting coefficients used in the scoring decision, To use the user's gaze direction vector With the direction vector of the candidate target center The calculated spatial orientation consistency score, To use the current speech semantic vector Prior semantic vectors of candidate targets The semantic matching score obtained from the match.

[0014] Optionally, the improved YOLOv10 model integrates a multi-scale feature pyramid structure and a spatial attention module for real-time target detection on edge devices.

[0015] Optionally, in step S3, semantic matching uses the semantic vector cosine similarity calculation formula:

[0016]

[0017] in, To represent the current speech semantic vector semantic vectors of candidate targets The semantic matching degree.

[0018] Optionally, in step S4, the spatial orientation consistency calculation uses the direction cosine formula:

[0019]

[0020] in, For the user's gaze direction vector, The direction vector of the candidate target center.

[0021] Optional, weighting coefficients used in fusion scoring decisions , , The optimization strategy can be dynamically adjusted based on the context, and is based on the following objective function:

[0022]

[0023] in, The fusion loss function includes, respectively, This is the image detection loss, used to optimize target detection accuracy; This is the semantic matching loss, used to optimize the correspondence between speech and image semantics; This is the spatial orientation consistency error term, used to optimize the consistency between the user's gaze direction and the target direction; , , Image detection loss Semantic matching loss Spatial orientation consistency error term The weighting coefficients can be automatically optimized during the training process or set empirically to balance the contributions of each modality.

[0024] Secondly, this application provides a mixed reality intelligent inspection system based on multimodal fusion for implementing the mixed reality intelligent inspection method based on multimodal fusion described in the first aspect, comprising the following modules integrated into an MR head-mounted device:

[0025] Image perception module: used for field-of-view image acquisition and target detection;

[0026] Speech recognition module: used for transcribing voice stream data;

[0027] Semantic reasoning module: used to generate semantic vectors and perform matching;

[0028] Spatial orientation calculation module: used for calculating the gaze direction and target orientation;

[0029] Fusion Decision Module: Used for multimodal score fusion;

[0030] Mixed Reality Augmented Display Module: Used for target highlighting and voice feedback.

[0031] Optionally, the fusion decision module adopts the Transformer architecture, and performs cross-modal weighted fusion decision after modeling image features, semantic vectors and spatial direction vectors respectively.

[0032] Compared with the prior art, this application includes at least one of the following beneficial technical effects:

[0033] This invention integrates image detection, speech semantic recognition, and spatial gaze information to form a multimodal collaborative perception mechanism, which can effectively improve the accuracy and stability of target recognition in complex inspection environments.

[0034] This invention combines contextual semantic memory and attention mechanisms to optimize the intent recognition process in multi-turn voice interaction, thereby enhancing the continuity and natural fluency of human-computer interaction.

[0035] This invention adopts a lightweight model design, and all algorithm modules can be deployed independently on the terminal, which has advantages in field applications such as high real-time performance, high computing efficiency, and low network dependence.

[0036] The system structure of this invention is universal and applicable to various high-safety inspection scenarios such as nuclear facilities, power equipment, and industrial workshops, and has good engineering applicability and promotion value.

[0037] This invention achieves high-precision, real-time, and robust intelligent target recognition and natural interaction in complex industrial inspection scenarios by integrating a multimodal collaborative perception mechanism that combines image, voice, and spatial gaze information, along with contextual semantic optimization and lightweight terminal deployment. Attached Figure Description

[0038] Figure 1 This is a flowchart of a mixed reality intelligent inspection method based on multimodal fusion proposed in this invention.

[0039] Figure 2 This is a diagram of the improved YOLOv10 model network structure proposed in this invention. Detailed Implementation

[0040] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0041] Example

[0042] This invention proposes a mixed reality intelligent inspection method based on multimodal fusion, the method flow is as follows: Figure 1 As shown, the terminal device implementing the mixed reality intelligent inspection method based on multimodal fusion is equipped with an image acquisition module, a voice acquisition module, a spatial perception module, a data fusion module, and an interaction enhancement module. Each module is integrated and deployed locally on the terminal, working together to complete the intelligent inspection task. The system as a whole does not rely on external server resources and has the ability to run independently in real time on the terminal side.

[0043] Image acquisition and target detection module: The mixed reality terminal acquires real-time video streams of the inspection site via a front-facing high-definition camera. The acquired images undergo noise reduction, illumination compensation, and normalization processing by a preprocessing module to adapt to the complex lighting and occlusion environments that may exist in industrial sites.

[0044] After image preprocessing, the image is input into the improved YOLOv10 object detection model. This model introduces the following optimized design based on the traditional YOLOv10 structure, as shown in the following diagram. Figure 2 As shown:

[0045] First, the C2f_RFAConv module is designed, embedding the RFA attention mechanism into the C2f structure to enhance feature extraction capabilities and improve the network's perception of multi-scale targets. Second, a channel-weighted fusion BiFPN structure is adopted to replace the original feature fusion module, achieving efficient interaction and fusion of multi-layer feature information. Finally, Shape-IoU is introduced into the loss function to further improve the model's accuracy in target shape matching.

[0046] The model outputs a set of candidate targets, recording the location information (center coordinates and bounding box), class label, and confidence score of each detected target. .

[0047] Voice Acquisition and Semantic Understanding Module: The mixed reality terminal's built-in microphone collects the natural speech input of inspection personnel in real time. Considering the presence of multiple dialects, colloquial expressions, and environmental noise interference on site, the system introduces an adaptive speech noise reduction and dynamic sound source localization module to improve the signal-to-noise ratio of speech recognition and the accuracy of speech segmentation.

[0048] The speech signal is input into the Whisper model for edge speech transcription, yielding preliminary text transcription results. These transcription results are then input into the BERT semantic embedding model to generate high-dimensional semantic vectors. The system pre-constructs a multi-level semantic dictionary of target category labels and generates semantic embedding vectors for each category. .

[0049] The matching degree between the current speech semantic vector and the category label vector of each detected target is calculated using cosine similarity:

[0050]

[0051] in, To represent the current speech semantic vector semantic vectors of candidate targets The semantic matching degree.

[0052] The system also introduces semantic extended dictionaries (such as industry thesaurus and hyponym graphs) to enhance the robustness of semantic matching and adapt to the differences in semantic expression among inspection personnel.

[0053] Spatial orientation analysis module: Used to acquire real-time gaze direction information of inspection personnel to guide target selection. The IMU module collects real-time head posture angular velocity, acceleration, and Euler angle data, and combines this with 3D scene point cloud data acquired by the depth camera. An IMU-visual fusion algorithm is then used to calculate the current gaze direction vector. For each candidate target, calculate its spatial center direction vector. And calculate the spatial orientation consistency score:

[0054]

[0055] in, For the user's gaze direction vector, The direction vector of the candidate target center.

[0056] The system further introduces a dynamic field-of-view width correction mechanism:

[0057] The tolerance range for spatial angles of small targets at long distances is widened; the threshold for judging large targets at close distances is tightened; and the spatial matching sensitivity is dynamically adjusted in conjunction with the depth perception model to ensure that the gaze direction score has a reasonable judgment effect at all viewing distances.

[0058] Contextual semantic memory mechanism: During multiple rounds of continuous voice interaction, the system dynamically constructs a semantic time series graph. Each round of voice semantic vector... Store it in a semantic memory bank and introduce a time decay mechanism to control the contribution of historical semantics.

[0059] In the current semantic matching process, the system introduces historical reinforcement terms:

[0060]

[0061] in, The enhanced semantic matching score takes into account the similarity between the current semantics and historical semantics. : Current round speech semantic vector With candidate target semantic vector The basic similarity score (e.g., using cosine similarity); : To the front Accumulate the semantic vectors of historical speech; Indicates the first The historical semantic contribution coefficient is used. This mechanism effectively buffers the critical jump phenomenon caused by slight fluctuations in the user's expression during continuous voice interaction, ensuring the overall interaction intent remains consistent and stable. : Current speech semantic vector With the Round historical speech semantic vector Similarity;

[0062] Multimodal fusion decision module: The fusion decision module integrates image detection confidence, spatial consistency score, and semantic matching score to perform a weighted fusion decision.

[0063]

[0064] in, The final score of the candidate targets is calculated using the multimodal fusion function. , , To integrate the weighting coefficients used in the scoring decision, To use the user's gaze direction vector With the direction vector of the candidate target center The calculated spatial orientation consistency score, To use the current speech semantic vector Prior semantic vectors of candidate targets The semantic matching score obtained from the matching. The weight coefficients are dynamically and adaptively adjusted through the system learning process, taking into account factors such as the complexity of the current inspection task, the degree of target occlusion, the degree of speech ambiguity, and the credibility of spatial gaze direction to comprehensively optimize the weight distribution.

[0065] The fusion decision module further integrates a lightweight Transformer structure, which performs cross-modal feature interaction learning on the three types of modal feature streams before fusion computation, thereby enhancing the system's fusion stability and anomaly discrimination capability in complex interference environments.

[0066] Model training and overall optimization mechanism: The overall system training process employs a joint multimodal loss function for end-to-end optimization.

[0067]

[0068] in, The fusion loss function includes, respectively, This is the image detection loss, used to optimize target detection accuracy; This is the semantic matching loss, used to optimize the correspondence between speech and image semantics; This is the spatial orientation consistency error term, used to optimize the consistency between the user's gaze direction and the target direction; , , Image detection loss Semantic matching loss Spatial orientation consistency error term The weighting coefficients can be automatically optimized during the training process or set empirically to balance the contributions of each modality.

[0069] Enhanced display and interactive feedback module: When merging scores Once the target exceeds a certain threshold, the system determines that the current target is the user's focus. Enhanced display effects are generated in real-time within the mixed reality interface, including:

[0070] The highlighted border in 3D space is overlaid with the target label;

[0071] The guide arrows point to high-risk components or key inspection areas;

[0072] Floating text prompts the current equipment status and key inspection points;

[0073] The system provides synchronized voice broadcasts of recognition results, risk warnings, and subsequent inspection recommendations.

[0074] The system supports users to complete confirmation through gaze-holding and concise voice commands, avoiding the burden of manual operation and adapting to natural human-computer collaborative interaction in complex on-site environments without manual control.

[0075] In summary, this invention provides a mixed reality intelligent inspection method based on multimodal fusion. By integrating three information sources—image detection, speech semantic understanding, and spatial orientation perception—locally on a mixed reality terminal, a dynamic adaptive multimodal fusion decision-making mechanism is constructed, enabling highly accurate, real-time, and natural human-machine collaborative inspection in complex industrial environments. The system proposes targeted improvements in areas such as lightweight image detection algorithms, continuous optimization of semantic intent, consistent modeling of spatial gaze direction, and cross-modal joint feature learning. It overcomes the technical bottlenecks of existing inspection systems in terms of single-modal recognition capabilities, speech interaction stability, and spatial scene adaptability, possessing broad engineering application value and promising industrial application prospects. This technical solution is particularly suitable for nuclear facility inspection, high-voltage power detection, intelligent manufacturing equipment maintenance, and remote complex scene intelligent inspection tasks, providing effective technical support for multimodal human-machine interaction technology in the field of industrial safety production.

[0076] Example 2

[0077] This embodiment provides a mixed reality intelligent inspection system based on multimodal fusion for executing the mixed reality intelligent inspection method based on multimodal fusion of Embodiment 1, including the following modules integrated into an MR head-mounted device:

[0078] Image perception module: used for field-of-view image acquisition and target detection;

[0079] Speech recognition module: used for transcribing voice stream data;

[0080] Semantic reasoning module: used to generate semantic vectors and perform matching;

[0081] Spatial orientation calculation module: used for calculating the gaze direction and target orientation;

[0082] Fusion Decision Module: Used for multimodal score fusion;

[0083] Mixed Reality Augmented Display Module: Used for target highlighting and voice feedback.

[0084] The fusion decision module adopts the Transformer architecture, and performs cross-modal weighted fusion decision after modeling image features, semantic vectors and spatial direction vectors respectively.

[0085] The system is integrated into the MR headset and works collaboratively through multiple specialized modules to simultaneously process multimodal information such as images and speech. The image perception module uses an improved YOLOv10 model to achieve real-time target detection, adapting to edge devices and ensuring no significant delay in the inspection process. The speech recognition and semantic reasoning module combines speech to transcribe speech into semantic vectors and match them with the target's prior semantic vectors. It also incorporates contextual semantic memory built from multiple rounds of speech and optimizes the matching through an attention mechanism to improve the accuracy of understanding user intent. The spatial orientation calculation module calculates the consistency between the user's gaze direction and the target's center direction based on the device's IMU information and depth image, enhancing the accuracy of judging the user's focus on the target.

[0086] The fusion decision-making module adopts the Transformer architecture, which performs cross-modal weighted fusion of information such as images, semantics, and spatial orientation. The weights can be dynamically adjusted based on context, adapting to different inspection scenarios and optimizing the final scoring results. The mixed reality augmented display module forms a complete interactive loop by highlighting targets and providing voice feedback. Users can obtain information without manual operation, which is especially suitable for scenarios where their hands may be occupied.

[0087] In addition, the modular design of the system makes each functional module independent and clear, which facilitates individual upgrades and maintenance. The collaboration of each module improves the efficiency and accuracy of inspection, making intelligent inspection more convenient and efficient.

[0088] The above specific embodiments are merely several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant teachings of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A mixed reality intelligent inspection method based on multimodal fusion, characterized in that, Includes the following steps: S1. Simultaneously collect user's visual field image and audio stream data through mixed reality devices; S2. The improved YOLOv10 model is used to perform target detection on the field of view image, and candidate target regions and their confidence scores are output. ; S3. Transcribe the speech input using the Whisper speech recognition model and generate semantic vectors using the BERT semantic embedding model. This is compared with the prior semantic vector of the candidate target. Perform a match; S4. Calculate the user's gaze direction vector based on the IMU information and depth image provided by the mixed reality device. and the direction vector of the candidate target center Consistency of computational space orientation; S5. Combine multi-turn voice input to build contextual semantic memory, and optimize the semantic matching process through an attention mechanism; S6. Calculate the final score of the candidate target using a multimodal fusion function; in step S6, the final score formula is: in, The final score of the candidate targets is calculated using the multimodal fusion function. , , To integrate the weighting coefficients used in the scoring decision, To use the user's gaze direction vector With the direction vector of the candidate target center The calculated spatial orientation consistency score, To use the current speech semantic vector Prior semantic vectors of candidate targets The semantic matching score obtained from the matching; S7. Based on the fusion scoring results, the target is highlighted on the mixed reality interface and voice feedback is triggered to complete the intelligent inspection interaction loop.

2. The mixed reality intelligent inspection method based on multimodal fusion according to claim 1, characterized in that, The improved YOLOv10 model integrates a multi-scale feature pyramid structure and a spatial attention module for real-time target detection on edge devices.

3. The mixed reality intelligent inspection method based on multimodal fusion according to claim 1, characterized in that, In step S3, semantic matching uses the semantic vector cosine similarity calculation formula: in, To represent the current speech semantic vector semantic vectors of candidate targets The semantic matching degree.

4. The mixed reality intelligent inspection method based on multimodal fusion according to claim 1, characterized in that, In step S4, the spatial orientation consistency calculation uses the direction cosine formula: in, For the user's gaze direction vector, is the direction vector of the candidate target center.

5. The mixed reality intelligent inspection method based on multimodal fusion according to claim 1, characterized in that, Weighting coefficients used in fusion scoring decisions , , The optimization strategy can be dynamically adjusted based on the context, and is based on the following objective function: in, The fusion loss function includes, respectively, This is the image detection loss, used to optimize target detection accuracy; This is the semantic matching loss, used to optimize the correspondence between speech and image semantics; This is the spatial orientation consistency error term, used to optimize the consistency between the user's gaze direction and the target direction; , , Image detection loss Semantic matching loss Spatial orientation consistency error term The weighting coefficients can be automatically optimized during the training process or set empirically to balance the contributions of each modality.

6. A mixed reality intelligent inspection system based on multimodal fusion for executing the mixed reality intelligent inspection method based on multimodal fusion as described in any one of claims 1-5, characterized in that, Including the following modules integrated into the MR headset: Image perception module: used for field-of-view image acquisition and target detection; Speech recognition module: used for transcribing voice stream data; Semantic reasoning module: used to generate semantic vectors and perform matching; Spatial orientation calculation module: used for calculating the gaze direction and target orientation; Fusion Decision Module: Used for multimodal score fusion; Mixed Reality Augmented Display Module: Used for target highlighting and voice feedback.

7. The mixed reality intelligent inspection system based on multimodal fusion according to claim 6, characterized in that, The fusion decision module adopts the Transformer architecture, and performs cross-modal weighted fusion decision after modeling image features, semantic vectors and spatial direction vectors respectively.

Images