AR-oriented 3D and multimedia content fusion image analysis method and system

Abstract

The application relates to the technical field of augmented reality, in particular to an AR-oriented 3D and multimedia content fusion image analysis method and system. The method comprises the following steps: acquiring multi-modal intention information and component semantic information; generating and displaying a virtual highlight / arrow guide for indicating a component in a real scene; determining a first component pointed to by the guide based on visual positioning; fusing the multi-modal intention and the component semantic to infer a real operation target to obtain a second component; comparing the first component and the second component; if there is a deviation, adjusting the virtual guide to point to the second component according to the comparison result, and synchronously updating the display position / posture of the three-dimensional model and the multimedia content. The method solves the technical problem that, when a virtual content high-precision fusion is implemented by using an existing AR image analysis method, the virtual information positioning is inaccurate, shakes or is misplaced due to factors such as visual obstruction, device structure repetition and light change, which seriously affects user experience and operation accuracy.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and more specifically, to a method and system for image analysis that fuses 3D and multimedia content for AR. Background Technology

[0002] The animation and cultural creative industry is promoting the integration and innovation of "content + technology," with a key direction being the integration of industrial knowledge dissemination and skills training into maintenance training for real-world equipment. These applications typically rely on augmented reality (AR) devices to overlay and display 3D models and multimedia content corresponding to the target equipment within a real-world scene. Virtual guidance then guides learners through the maintenance steps, thereby enhancing immersion while lowering the learning threshold and increasing the fun of training.

[0003] However, in the actual operation of maintenance training for real-world equipment, achieving high-precision integration of virtual content with real-world scenarios still faces challenges. Especially when interference factors such as obstructed view, repetitive equipment structures, and drastic changes in lighting conditions occur simultaneously or consecutively, the visual positioning results of augmented reality devices may become unstable. This can lead to inaccurate positioning, jitter, or even misalignment of virtual guidance, 3D models, and multimedia content, affecting learners' understanding and accuracy in operating key objects.

[0004] For example, in a training program focused on the maintenance of compressor units, learners wear augmented reality (AR) devices for task training. The system first overlays a perfectly aligned 3D model of the actual compressor unit onto environmental information captured by a camera, and then projects a multimedia playback window to the right of the model to display a guiding video on "replacing a specific valve." Initially, the 3D model and multimedia content display are stable, and the relative position of the virtual content and the real-world scene remains consistent even when the learner turns their head or moves slightly.

[0005] When personnel, props, or moving objects (such as a pushed cart or a walking companion) briefly obstruct the field of vision during training, some key visual feature points are momentarily lost. After the obstruction is removed, if there are many similar-looking, repetitively arranged heat dissipation fins and fastening bolts on the compressor unit, the system may mismatch during relocalization, causing the entire virtual information layer to "jump" to the wrong position. At this time, if the multimedia content plays to interactive nodes such as "loosen the third fixing bolt," the virtual highlighted area and virtual indicator arrow may point to the wrong bolt, causing learners to perform incorrect interactions, resulting in incorrect training step judgments, biased scoring logic, reduced training effectiveness, and damage to experience consistency.

[0006] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0007] This application discloses a 3D and multimedia content fusion image analysis method and system for AR, aiming to solve the technical problem that existing AR image analysis methods, when achieving high-precision fusion of virtual content, suffer from inaccurate positioning, jitter, or misalignment of virtual information due to factors such as field-of-view occlusion, repetitive device structure, and changes in lighting, which seriously affect user experience and operational accuracy.

[0008] The technical solution of this application is as follows:

[0009] In a first aspect, this application discloses an image analysis method for fusing 3D and multimedia content for AR, applied to augmented reality devices. The augmented reality devices are used to overlay and display virtual content in a real-world scene. The virtual content includes at least a 3D model corresponding to the target device in the real-world scene and multimedia content. The method includes:

[0010] Acquire the user's multimodal intent information, which includes at least body movement information, gaze information, and voice information;

[0011] Obtain semantic information of the components of the target device. The semantic information of the components is provided by a pre-stored 3D model corresponding to the target device, and includes the identity of each component of the target device and its spatial position range in the device coordinate system, as well as the functional description and geometric feature information of the components.

[0012] Virtual guides are generated and displayed in real-world scenarios. These virtual guides are virtual highlighted areas and / or virtual arrows used to indicate target device components.

[0013] Based on the visual positioning results of the augmented reality device, the component that the virtual guide points to in the real scene is determined, and the first component is obtained;

[0014] Based on multimodal intent information and component semantic information, the operator's actual operational goal is inferred, and the second component is obtained;

[0015] Compare the identification marks and / or spatial location ranges of the first component and the second component to obtain the comparison results;

[0016] Adjust the position of the virtual guide according to the comparison results, so that the virtual guide points to the second component;

[0017] The display method of virtual content is updated according to the adjusted virtual guidance, including updating the display position and / or posture of 3D models and multimedia content in the real scene, and maintaining the preset relative positional relationship between 3D models and multimedia content.

[0018] Furthermore, based on multimodal intent information and component semantic information, the operator's actual operational goal is inferred, resulting in a second component, including:

[0019] Obtain the pre-stored industrial maintenance task flowchart. The industrial maintenance task flowchart is the maintenance task flowchart corresponding to the target equipment. The industrial maintenance task flowchart includes the preconditions, postconditions, corresponding expected operation components, and expected logical state of the expected operation components after the operation is completed for each task step.

[0020] Obtain the physical status information of the corresponding expected operation component. The physical status information includes the operator's confirmation information, the physical status of key components related to the expected operation component obtained based on image recognition, and the sensor data inside the target device.

[0021] Update the current task status based on physical status information and the industrial maintenance task flowchart;

[0022] Based on the updated task status, determine the next task step in the industrial maintenance task flowchart, and obtain the expected operation parts corresponding to the next task step to form a list of the next expected operation parts.

[0023] Based on the next list of expected operation components, obtain the identity, functional description and spatial location range of each expected operation component from the component semantic information;

[0024] Based on the updated task status in the industrial maintenance task flowchart, the corresponding task steps and / or activated task paths are used to calculate the task-driven intent prediction score for each expected operation component in the next expected operation component list.

[0025] Based on multimodal intent information and the identity, functional description and spatial location range of each expected operation component, the intent matching score of each expected operation component is calculated.

[0026] The task-driven intent prediction score and intent matching score are then fused to determine the second component.

[0027] Furthermore, based on the updated task status and the corresponding task steps and / or activated task paths in the industrial maintenance task flowchart, a task-driven intent prediction score is calculated for each expected operation component in the next expected operation component list, including:

[0028] Based on the updated task status, identify whether there are conditional branches or parallel paths in the industrial maintenance task flowchart;

[0029] When a conditional branch is detected in the industrial maintenance task flowchart, the physical state and / or logical state of the equipment component is determined based on the physical state information, and the condition of the conditional branch is judged to activate the corresponding task path; and within the activated task path, the task-driven intent prediction score is calculated for the expected operation component in the next expected operation component list.

[0030] When parallel paths are identified in the industrial maintenance task flowchart, basic task-driven intent prediction scores are calculated for the expected operation components on the parallel paths. The basic task-driven intent prediction scores for the expected operation components on the parallel paths are then adjusted based on the operator's current area of ​​focus to obtain the task-driven intent prediction score. The current area of ​​focus is determined by limb movement information and / or line-of-sight information.

[0031] Furthermore, when parallel paths are identified in the industrial maintenance task flowchart, and the current area of ​​interest simultaneously covers multiple adjacent expected operating components, adjustments are made to the prediction score of the basic task-driven intent, including:

[0032] Continuously monitor the frequency of the operator's gaze shifting between multiple adjacent expected operating parts and the percentage of time spent on each adjacent expected operating part;

[0033] Acquire the degree of matching between the operator's hand pointing and / or touch gestures and the gestures of multiple adjacent expected operating parts;

[0034] When the frequency of the gaze focus jumping between multiple adjacent expected operation parts exceeds the preset jumping threshold and the posture matching degree is lower than the preset matching threshold, the micro-geometric features and functional difference descriptions of multiple adjacent expected operation parts are extracted from the semantic information of the parts, and the part discrimination score is generated based on the micro-geometric features and functional difference descriptions.

[0035] The attention weight of each adjacent expected operation component is calculated based on the dwell time ratio, posture matching degree, and component differentiation score.

[0036] Based on the attention weight, the basic task-driven intent prediction scores of multiple adjacent expected operation components are adjusted to obtain the task-driven intent prediction score.

[0037] Furthermore, based on multimodal intent information and component semantic information, the operator's actual operational goal is inferred, resulting in a second component, including:

[0038] For multiple components in the semantic information of components that correspond to the areas of interest indicated by body movement information and / or gaze information, calculate the matching scores of multiple components with body movement information, the matching scores of multiple components with gaze information, and the matching scores of multiple components with speech information respectively.

[0039] The matching scores of body movement information, eye gaze information, and voice information are fused according to preset fusion weights to obtain the intention confidence scores of each component.

[0040] The component with the highest intent confidence score is identified as the second component.

[0041] Furthermore, the attention weights of each adjacent expected operating component are calculated, including:

[0042] Assess the stability of the operator's line-of-sight data stream and calculate the real-time reliability score of the line-of-sight based on the frequency of fluctuations and the percentage of dwell time.

[0043] The stability of the hand pointing posture data stream is continuously evaluated, and the real-time reliability score of the hand pointing posture is calculated by combining the displacement variance of the key points of the hand in the hand pointing posture and the continuity of the pointing vector.

[0044] The reliability score of component discrimination information is evaluated based on the uniformity of the component discrimination score distribution.

[0045] Based on the real-time reliability score of the gaze focus, the real-time reliability score of the hand pointing posture, and the reliability score of the component differentiation information, the contribution ratio of dwell time, posture matching degree, and component differentiation score to the attention weight calculation is dynamically adjusted to obtain the attention weight of each adjacent expected operation component.

[0046] Furthermore, the stability of the operator's gaze focus data stream is evaluated, and a real-time reliability score for gaze focus is calculated by combining the fluctuation frequency and dwell time percentage, including:

[0047] Acquire the operator's head posture data;

[0048] Based on head posture data, identify the jitter or drift components in the gaze focus data stream caused by head shaking or unstable device wearing;

[0049] Acquire ambient lighting data;

[0050] Based on ambient lighting data, identify abnormal components in the gaze focus data stream caused by pupil recognition instability due to changes in lighting.

[0051] Remove jitter or drift components and abnormal components from the gaze focus data stream to obtain the corrected gaze focus data stream;

[0052] The jump frequency and dwell time percentage are recalculated based on the corrected gaze focus data stream;

[0053] Assess the stability of the operator's line-of-sight data stream, and calculate the real-time reliability score of the line-of-sight focus by combining the recalculated bounce frequency and dwell time percentage.

[0054] Furthermore, the jitter frequency is recalculated based on the corrected gaze focus data stream, including:

[0055] Acquire the operator's detailed operating instructions and / or the magnified display status of the operating area;

[0056] Based on the fine operation instructions and / or the local magnified display status, identify whether the current state is in the preset fine operation mode;

[0057] In the fine operation mode, the calculation threshold for the jumping frequency is dynamically adjusted;

[0058] Frequency domain analysis was performed on the corrected line-of-sight focus data stream to identify high-frequency, low-amplitude oscillation components;

[0059] Based on the degree of matching between the oscillation component and the fine operation mode, it is determined whether the oscillation component is a real jitter caused by fine operation. Data processing is performed on the corrected line-of-sight focus data stream: when it is determined that the oscillation component is a real jitter caused by fine operation, the oscillation component is included in the calculation of the jitter frequency; when it is determined that the oscillation component is sensor noise or residual jitter, the oscillation component is excluded from the calculation of the jitter frequency.

[0060] The jumping frequency is calculated based on the line-of-sight focus data stream after data processing.

[0061] Furthermore, based on the degree of matching between the oscillation component and the fine-tuning mode, it is determined whether the oscillation component is a genuine fluctuation caused by fine-tuning, including:

[0062] Continuously monitor the spatial distribution and temporal evolution of the oscillation components across multiple adjacent expected operating components;

[0063] Obtain spatial proximity and functional association information between multiple adjacent expected operating components;

[0064] Based on spatial distribution, temporal evolution, spatial proximity, and functional correlation information, identify whether the oscillation component exhibits a continuous, focused activity pattern on a single expected operating component;

[0065] When the oscillation component exhibits a continuous, focused activity pattern on a single expected operating component, the oscillation component is determined to be a genuine fluctuation caused by fine operation.

[0066] When the oscillation component exhibits a dispersed and irregular activity pattern among multiple adjacent expected operating components, the oscillation component is determined to be sensor noise or residual jitter.

[0067] Secondly, this application also discloses a 3D and multimedia content fusion image analysis system for AR, comprising:

[0068] The information acquisition module is used to acquire the user's multimodal intent information, which includes at least body movement information, gaze information, and voice information.

[0069] The semantic information acquisition module is used to acquire the semantic information of the components of the target device. The semantic information of the components is provided by the pre-stored 3D model corresponding to the target device, and includes the identity of each component of the target device and its spatial position range in the device coordinate system, as well as the functional description and geometric feature information of the components.

[0070] The guidance determination module is used to generate and display virtual guidance in a real-world scenario. The virtual guidance consists of a virtual highlighted area and / or a virtual indicator arrow used to indicate the target device component.

[0071] The target localization module is used to determine the component that the virtual guide points to in the real scene based on the visual localization results of the augmented reality device, and obtain the first component;

[0072] The target inference module is used to infer the operator's actual operational target based on multimodal intent information and component semantic information, and obtain the second component;

[0073] The comparison module is used to compare the identity and / or spatial location range of the first component and the second component to obtain the comparison result;

[0074] The guidance adjustment module is used to adjust the position of the virtual guide according to the comparison results, so that the virtual guide points to the second component;

[0075] The content update module is used to update the display method of virtual content according to the adjusted virtual guidance, including updating the display position and / or posture of the 3D model and multimedia content in the real scene, and maintaining the preset relative positional relationship between the 3D model and multimedia content.

[0076] Beneficial effects: This application can significantly improve the robustness and accuracy of AR systems in complex industrial scenarios, avoiding operational errors or even safety accidents caused by virtual information misalignment. Technicians can obtain more stable and accurate virtual operation guidance, thereby improving work efficiency and operational safety, and providing a more reliable augmented reality solution for industrial maintenance, training, and other fields. Attached Figure Description

[0077] Figure 1 This application provides a flowchart illustrating a method for fusion image analysis of 3D and multimedia content for AR.

[0078] Figure 2 This is one of the flowcharts of an AR-oriented 3D and multimedia content fusion image analysis method provided in another embodiment of this application.

[0079] Figure 3 This is a second flowchart illustrating an AR-oriented 3D and multimedia content fusion image analysis method, provided as another embodiment of this application.

[0080] Figure 4 This is the third flowchart illustrating an AR-oriented 3D and multimedia content fusion image analysis method, provided as another embodiment of this application.

[0081] Figure 5 This is the fourth flowchart illustrating an AR-oriented 3D and multimedia content fusion image analysis method, provided as another embodiment of this application.

[0082] Figure 6 The fifth flowchart illustrates a method for fusion image analysis of 3D and multimedia content for AR, provided as another embodiment of this application.

[0083] Figure 7 This is a sixth flowchart illustrating an AR-oriented 3D and multimedia content fusion image analysis method, provided as another embodiment of this application.

[0084] Figure 8 This is the seventh flowchart illustrating an AR-oriented 3D and multimedia content fusion image analysis method, provided as another embodiment of this application.

[0085] Figure 9 This is the eighth flowchart illustrating an AR-oriented 3D and multimedia content fusion image analysis method, provided as another embodiment of this application.

[0086] Figure 10 A flowchart of an AR-oriented 3D and multimedia content fusion image analysis system is provided in this application.

[0087] In the diagram: 1. Information acquisition module; 2. Semantic information acquisition module; 3. Guidance determination module; 4. Target positioning module; 5. Target inference module; 6. Comparison module; 7. Guidance adjustment module; 8. Content update module. Detailed Implementation

[0088] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0089] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0090] While traditional augmented reality (AR) technology, combined with animation, has shown great potential in applications such as virtual industrial maintenance training, accurately overlaying virtual 3D models and multimedia content onto the real world to provide intuitive operational guidance for learners / trainees (operators), existing image analysis methods face numerous challenges in achieving high-precision fusion of virtual content in real / semi-realistic training scenarios due to the complexity and variability of the environment. Particularly when multiple interference factors occur simultaneously or consecutively, such as obstructed viewpoints, repetitive equipment structures, and drastic changes in lighting conditions, AR systems often struggle to maintain stable environmental perception, leading to inaccurate virtual information positioning, jitter, or even misalignment, severely impacting training guidance effectiveness and operational accuracy.

[0091] Reference Figure 1 In response, this application proposes an image analysis method for fusing 3D and multimedia content for AR, applied to augmented reality devices. The augmented reality devices are used to overlay and display virtual content on a real-world scene. The virtual content includes at least a 3D model corresponding to the target device in the real-world scene and multimedia content. The method includes:

[0092] S1000: Acquires the user's multimodal intent information, which includes at least body movement information, gaze information, and voice information;

[0093] S2000: Obtain semantic information of the components of the target device. The semantic information of the components is provided by the pre-stored 3D model corresponding to the target device, and includes the identity of each component of the target device and its spatial position range in the device coordinate system, as well as the functional description and geometric feature information of the components.

[0094] S3000: Generates and displays virtual guides in a real-world scene. The virtual guides are virtual highlighted areas and / or virtual arrows used to indicate target device components.

[0095] S4000: Based on the visual positioning results of the augmented reality device, determine the component that the virtual guide points to in the real scene, and obtain the first component;

[0096] S5000: Based on multimodal intent information and component semantic information, infer the operator's true operational goal and obtain the second component;

[0097] S6000: Compare the identification and / or spatial location range of the first component with that of the second component to obtain the comparison result;

[0098] S7000: Adjust the position of the virtual guide according to the comparison results, so that the virtual guide points to the second component;

[0099] S8000: Updates the display method of virtual content according to the adjusted virtual guide, including updating the display position and / or posture of 3D models and multimedia content in the real scene, and maintaining the preset relative positional relationship between 3D models and multimedia content.

[0100] Specifically, augmented reality devices are those that can overlay virtual information onto the real world, such as AR glasses and AR helmets. These devices are typically equipped with components such as cameras, sensors, and displays, enabling them to perceive the environment in real time, perform visual positioning, and display virtual content to meet the interactive presentation needs of virtual inspection and training.

[0101] Virtual content refers to digital information generated in augmented reality devices and overlaid on real-world scenes. It includes at least a 3D model corresponding to the target device and multimedia content. The 3D model is a digital representation of the target device, providing information such as its structure and components. Multimedia content can be video, images, text, etc., used to provide operational guidance or auxiliary information, and can carry training prompts, task descriptions, or training evaluation information.

[0102] Multimodal intent information refers to various forms of information obtained from operators (such as trainers / learners) to infer their operational intentions. It includes at least body movement information (e.g., gestures, body posture), gaze information (e.g., gaze focus, area of ​​fixation), and speech information (e.g., voice commands, verbal descriptions). This information can be acquired through sensors built into augmented reality devices or external sensors, reflecting the trainee's actual operational goals during training tasks.

[0103] Component semantic information refers to detailed information describing each component of the target device, provided by a pre-stored 3D model corresponding to the target device. This information includes the component's identifier (e.g., component name, number), its spatial location range in the device coordinate system (e.g., 3D coordinates, dimensions), as well as the component's functional description (e.g., the component's function, purpose) and geometric feature information (e.g., shape, texture), to support component-level prompts and error correction during the training task process.

[0104] Virtual guidance refers to virtual prompts generated and displayed in a real-world scenario to indicate specific components of a target device. It can be a virtual highlighted area (e.g., displaying a glowing area around a component) and / or a virtual arrow (e.g., an arrow pointing to the component), and can be integrated with mechanisms such as training progress and assessment feedback.

[0105] The method of this application is applied to an augmented reality device that can overlay and display virtual content on a real-world scene. The virtual content includes at least a 3D model corresponding to the target device in the real-world scene and multimedia content. The method flow is as follows:

[0106] First, acquire the user's multimodal intent information, including at least body movement information, gaze information, and voice information. Body movement information can be acquired by the depth sensor or inertial measurement unit (IMU) built into the augmented reality device, or by an external motion capture system, such as capturing and analyzing when an operator points a finger in a certain direction; gaze information can be acquired by eye-tracking sensors, such as recording when the gaze stays on a certain area for a long time; voice information can be acquired by a microphone, such as recognizing and converting the operator's voice when they say "this valve" to support voice interaction or command triggering during training.

[0107] Secondly, semantic information about the components of the target device is acquired. This semantic information is provided by a pre-stored 3D model corresponding to the target device, and includes the identification of each component, its spatial location range in the device coordinate system, as well as the functional description and geometric feature information of the components. Component semantic information can be obtained by importing CAD models or manually annotating them after 3D scanning and modeling. For example, for a compressor, its 3D model can include the identification of components such as the intake valve, exhaust valve, and heat sink, their precise 3D location and size range in the compressor's overall coordinate system, and functional descriptions (e.g., the intake valve controls the gas intake) and geometric feature information (e.g., the heat sink has a regular fin structure), thus providing semantic support for the "identify components—complete operations—obtain feedback" chain in the learning game.

[0108] Next, virtual guidance is generated and displayed in the real-world scene. This virtual guidance consists of virtual highlighted areas and / or virtual arrows used to indicate target device components. The virtual guidance can be generated based on a preset maintenance procedure or initial instructions from the operator. For example, the system locates the component corresponding to the current task in a 3D model and overlays a virtual highlighted area or virtual arrow at the physical location of that component in the real-world scene. Its display position and orientation are determined based on the visual positioning results of the augmented reality device. For instance, when the intake valve needs to be checked, a flashing virtual highlighted area is displayed at its physical location as a training step prompt.

[0109] Then, based on the visual localization results of the augmented reality device, the component that the virtual guide points to in the real-world scene is determined, resulting in the first component. The visual localization results are typically achieved through SLAM (Simultaneous Localization and Mapping): the device perceives the environment in real time through a camera, constructs an environmental map, and determines its own position and orientation; the system then calculates the three-dimensional position of the virtual guide in the real-world scene and compares it with the semantic information of the component to determine the physical component it points to. For example, if the center point of the virtual guide falls within the spatial range of the air intake valve, then the air intake valve is the first component, used to characterize the landing point of the guide presented by the current system.

[0110] Subsequently, based on multimodal intent information and component semantic information, the system infers the operator's actual operational target and obtains the second component. The system integrates body movement information, gaze information, and voice information, combined with component semantic information, for determination. For example, when the operator's gaze is focused on the exhaust valve for a long time, while pointing at the exhaust valve with their hand and saying the valve aloud, even if the virtual guidance is currently pointing to the intake valve, the system can still infer that the actual operational target is the exhaust valve and identify it as the second component, serving as a basis for correcting guidance deviations.

[0111] Next, the identification and / or spatial location range of the first and second components are compared to obtain the comparison result. When the two are inconsistent, the position of the virtual guide is adjusted according to the comparison result so that the virtual guide points to the second component. For example, if the first component is an intake valve and the second component is an exhaust valve, the virtual guide is moved from the position of the intake valve to the position of the exhaust valve to avoid misleading operations during training guidance.

[0112] Simultaneously, the display method of virtual content is updated according to the adjusted virtual guidance, including updating the display position and / or posture of 3D models and multimedia content in the real scene, while maintaining the preset relative positional relationship between the 3D models and multimedia content. For example, after the virtual guidance is adjusted from the intake valve to the exhaust valve, the parts of the 3D model related to the exhaust valve can be highlighted, and the multimedia content can switch to the operation steps related to the exhaust valve; the preset relative positional relationship is maintained during the update (e.g., the video window always floats on the right side of the 3D model), thereby maintaining the consistency and readability of the training interface.

[0113] In the context of this application, the necessity of this application can be further explained: In traditional augmented reality (AR) applications, systems typically rely primarily on visual positioning results overlaid with virtual guidance (e.g., highlighting a bolt). When obstructions cause the loss of visual feature points, or when there are many similar-looking components on the device, visual positioning is prone to errors, leading to virtual guidance pointing incorrectly to the wrong component. This can cause problems such as incorrect training steps, inconsistent assessment judgments, and interruptions in the continuity of immersive training. Taking a compressor maintenance scenario as an example, when obstructions cause the AR device to mismatch during relocalization, resulting in the virtual information being transmitted to the wrong location on the compressor unit and pointing to the wrong bolt, this manifests as a "jump" message, guidance drift, or an abnormal task failure judgment. For example, when visual positioning incorrectly places the virtual guide on bolt A, but the operator's gaze is focused on bolt B for a long time, their hand points to bolt B, and they say "loosen this bolt," this application can use multimodal intent information combined with component semantic information to infer that the real target is bolt B. Then, through comparison and adjustment of the first and second components, the virtual highlighted area or virtual indicator arrow is moved from bolt A to bolt B, and the display of the 3D model and multimedia content is updated simultaneously, thereby achieving stable and reliable interactive guidance in maintenance training combining AR and animation.

[0114] Reference Figure 2 In another embodiment of this application, S5000 further includes:

[0115] S5100: Obtain the pre-stored industrial maintenance task flowchart. The industrial maintenance task flowchart is the maintenance task flowchart corresponding to the target equipment. The industrial maintenance task flowchart includes the preconditions, postconditions, corresponding expected operation components, and expected logical state of the expected operation components after the operation is completed for each task step.

[0116] S5200: Acquire the physical status information of the corresponding expected operation component. The physical status information includes the operator's confirmation information, the physical status of key components related to the expected operation component obtained based on image recognition, and sensor data inside the target device.

[0117] S5300: Updates the current task status based on physical status information and industrial maintenance task flowchart;

[0118] S5400: Based on the updated task status, determine the next task step in the industrial maintenance task flowchart, obtain the expected operation parts corresponding to the next task step, and form a list of the next expected operation parts.

[0119] S5500: Based on the next list of expected operation components, obtain the identity, functional description and spatial location range of each expected operation component from the component semantic information;

[0120] S5600: Based on the updated task status in the industrial maintenance task flowchart, the corresponding task steps and / or activated task paths are used to calculate the task-driven intent prediction score for each expected operation component in the next expected operation component list.

[0121] S5700: Based on multimodal intent information and the identity, functional description and spatial location range of each expected operation component, calculate the intent matching score of each expected operation component;

[0122] S5800: The task-driven intent prediction score and intent matching score are fused to determine the second component.

[0123] Specifically, an industrial maintenance task flowchart can be understood as a predefined series of operational steps and their interrelationships, aiming to provide structured task guidance for operators during virtual industrial maintenance training. This flowchart details the preconditions (i.e., the states that must be met before completing the step), postconditions (i.e., the states reached after completing the step), the expected operating components directly related to the step, and the expected logical states of these expected operating components after the operation is completed. For example, in the task step of "replacing the filter element," the precondition might be "the old filter element has been removed," the postcondition is "the new filter element has been installed," the expected operating component is "the new filter element," and its expected logical state is "installed and sealed."

[0124] Physical status information refers to real-time or near-real-time data related to the components to be operated, reflecting the actual condition of the operating environment and equipment components. This information may include confirmation information from operators via voice commands or gestures, the physical status of key components related to the components to be operated (e.g., whether the component has been disassembled or installed) obtained through image recognition by the camera on the augmented reality device, and real-time data obtained from internal sensors (such as temperature sensors, pressure sensors, vibration sensors, etc.). This physical status information collectively provides the system with objective evidence of the current task execution progress.

[0125] In practical applications, updating the current task status based on physical state information and the industrial maintenance task flowchart means that the system compares the collected physical state information with the preconditions and postconditions in the industrial maintenance task flowchart to determine whether the current task step has been completed and updates the current progress of the entire task flow. For example, when the image recognition result shows "old filter element removed" and the operator confirms "preparing to install new filter element," the system will update the task status from "removing old filter element" to "installing new filter element."

[0126] Furthermore, based on the updated task status, the next task step is determined in the industrial maintenance task flowchart, and the expected operating parts corresponding to the next task step are obtained to form the next expected operating parts list. This means that the system searches for all expected operating parts associated with the next task step to be executed in the industrial maintenance task flowchart based on the currently updated task status. For example, if the current task status is "Install new filter element", the next expected operating parts list may include "new filter element" and "filter element cover".

[0127] Building upon this, obtaining the identity, functional description, and spatial location range of each expected operational component from the component semantic information means that for each component in the next list of expected operational components, the system extracts its unique identity, detailed functional description (e.g., "for filtering oil", "for sealing interfaces"), and precise spatial location range in the device coordinate system from a pre-stored component semantic information database. This information provides the foundation for subsequent intent prediction and matching.

[0128] The solution proposed in this application introduces industrial maintenance task flowcharts and real-time physical status information, so that the operator's inference of the actual operation target no longer depends solely on instantaneous multimodal intent information, but incorporates the context and progress of the task, thus making it more suitable for the interactive closed loop of "step-by-step training - status verification - instant feedback" in virtual industrial maintenance training.

[0129] In some preferred embodiments, assuming an operator is using augmented reality equipment to train for the maintenance of an industrial pump, the next primary task indicated by the current task flowchart is "inspect and tighten the pump body connection bolts". First, the system retrieves a pre-stored industrial maintenance task flowchart, which is a maintenance task flowchart corresponding to the target device and includes the task step "inspect and tighten the pump body connection bolts". The expected operable component is the "pump body connection bolts", and the expected logical state of the bolts after the operation is completed (e.g., "tightened").

[0130] Next, the system acquires the physical status information of the corresponding expected operation components. For example, if the operator confirms "pump body inspection completed" via voice command, the augmented reality device's image recognition function detects slight signs of looseness on the surface of the pump body connecting bolts, and the device's internal sensor data shows that the pump body vibration is slightly higher than normal. Based on this physical status information, the system updates the current task status to "pump body connecting bolts to be tightened," and obtains the next expected operation component list based on the updated task status, which includes "pump body connecting bolts."

[0131] Subsequently, the system obtains the identity, functional description (e.g., "used to fix pump body components"), and spatial location range of the pump body connecting bolt from the component semantic information; calculates the task-driven intent prediction score for the pump body connecting bolt based on the industrial maintenance task flowchart (higher because the task clearly points to tightening the bolt); and calculates the intent matching score based on multimodal intent information (higher because the gaze focus lingers for a long time, the hand reaches towards the bolt, and the voice contains instructions such as "tighten this bolt").

[0132] Finally, the system merges the task-driven intent prediction score with the intent matching score. Since the pump body connecting bolt has high values ​​in both scores, the system ultimately determines that the pump body connecting bolt is the operator's actual operating target, i.e., the second component. If the virtual guide initially points to other components on the pump body (e.g., an adjacent sensor), the system will adjust the position of the virtual guide according to the comparison results so that it accurately points to the pump body connecting bolt, thereby providing the operator with accurate visual guidance.

[0133] Reference Figure 3 In another embodiment of this application, S5600 further includes:

[0134] S5610: Identify whether there are conditional branches or parallel paths in the industrial maintenance task flowchart based on the updated task status;

[0135] S5620: When a conditional branch is detected in the industrial maintenance task flowchart, the physical state and / or logical state of the equipment component is determined based on the physical state information, and the condition of the conditional branch is determined to activate the corresponding task path; and within the activated task path, the task-driven intent prediction score is calculated for the expected operation component in the next expected operation component list.

[0136] S5630: When parallel paths are detected in the industrial maintenance task flowchart, the basic task-driven intent prediction score is calculated for each expected operation component on the parallel path; and the basic task-driven intent prediction score of the expected operation component on the parallel path is adjusted according to the operator's current area of ​​focus to obtain the task-driven intent prediction score. The current area of ​​focus is determined by limb movement information and / or line of sight information.

[0137] Specifically, identifying whether conditional branches or parallel paths exist in an industrial maintenance task flowchart based on the updated task status means parsing the structure of a pre-stored industrial maintenance task flowchart and determining whether there are task nodes (conditional branches) that require selecting different paths based on specific conditions, or nodes that can perform multiple tasks simultaneously (parallel paths), based on the updated task status. This can be achieved by parsing the graphical structure of the flowchart or its corresponding logical description language.

[0138] When a conditional branch is detected in the industrial maintenance task flowchart, the current physical and / or logical state of the equipment component needs to be determined based on physical state information (such as internal sensor data and image recognition results). For example, whether a valve is open or closed, or whether an indicator light is on or off. Then, based on the preset conditions of the conditional branch (e.g., "If the valve is closed, execute task A; otherwise, execute task B"), it is determined whether the current state is met, thereby activating the corresponding task path. Once a task path is activated, the subsequent task-driven intent prediction score is only calculated within the activated task path, avoiding invalid predictions for components on inactive paths.

[0139] In practical applications, when parallel paths are identified in an industrial maintenance task flowchart, a basic task-driven intent prediction score is first calculated for each expected operational component on the parallel path. This basic score can be initially determined based on factors such as the inherent order of the task flow and the importance of the components. Furthermore, to improve prediction accuracy, this application adjusts these basic scores based on the operator's current area of ​​focus. The current area of ​​focus can be determined by the operator's body language information (e.g., hand pointing) and / or gaze information (e.g., gaze focus). For example, if the operator's gaze lingers on a component of a parallel task for an extended period, the task-driven intent prediction score for that component will be increased accordingly to reflect the operator's immediate intent, thus better aligning with the "learner focuses first, then operates" behavioral characteristic in maintenance training for real-world equipment.

[0140] The proposed solution improves the accuracy of task-driven intent prediction scores by refining the conditional branches and parallel paths in the industrial maintenance task flowchart, thereby reducing the risk of virtual guidance misleading learners in maintenance training for real-world equipment.

[0141] In some preferred embodiments, the following specific example illustrates the situation:

[0142] Suppose a flowchart for maintaining industrial equipment includes a conditional branch: after checking the "power module," if the "power indicator light" is on, proceed to "troubleshooting path A"; if the "power indicator light" is off, proceed to "troubleshooting path B." Furthermore, within "troubleshooting path A," there are two tasks that can be executed in parallel: "check wiring connections" and "replace fuses."

[0143] First, the system will identify conditional branches and parallel paths in the task flowchart.

[0144] After the operator completes the inspection of the "power module", the system obtains the physical status information of the "power indicator light" through image recognition or equipment sensor data. If the "power indicator light" is not lit, the system will activate "troubleshooting path B" based on conditions, and calculate the task-driven intent prediction score for the next expected operation component only within this path, such as "check the power cord".

[0145] If the "Power Indicator" lights up, the system activates "Troubleshooting Path A". At this time, for the parallel tasks "Check Line Connections" and "Replace Fuse", the system will first calculate a basic task-driven intent prediction score for each. For example, the basic score for "Check Line Connections" is 0.6, and the basic score for "Replace Fuse" is 0.4.

[0146] Furthermore, the system continuously monitors the operator's body language and gaze. If the operator's gaze remains focused on the area corresponding to "Check wiring connections" for an extended period, and their hand gestures also point towards that area, the system identifies "Check wiring connections" as the current area of ​​interest. Based on this, the system adjusts the task-driven intent prediction scores for these two components; for example, adjusting the score for "Check wiring connections" to 0.8 and the score for "Replace fuse" to 0.2. In this way, the system can more accurately predict the operator's actual operational goals, providing precise virtual guidance even in complex parallel tasks.

[0147] Reference Figure 4 In another embodiment of this application, a method for adjusting the prediction score of the basic task-driven intent is further proposed when parallel paths are identified in the industrial maintenance task flowchart and the current area of ​​interest simultaneously covers multiple adjacent expected operating components. This method includes:

[0148] S5631: Continuously monitor the frequency of the operator's gaze shifting between multiple adjacent expected operating parts and the percentage of time spent on each adjacent expected operating part.

[0149] S5632: Obtain the degree of matching between the operator's hand pointing posture and / or touch posture and the posture of multiple adjacent expected operating parts;

[0150] S5633: When the frequency of the gaze focus jumping between multiple adjacent expected operation parts exceeds the preset jumping threshold and the posture matching degree is lower than the preset matching threshold, extract the micro-geometric features and functional difference descriptions of multiple adjacent expected operation parts from the semantic information of the parts, and generate the part discrimination score based on the micro-geometric features and functional difference descriptions.

[0151] S5634: Calculate the attention weight of each adjacent expected operation component based on the dwell time ratio, posture matching degree, and component differentiation score;

[0152] S5635: Based on the attention weight, the basic task-driven intent prediction scores of multiple adjacent expected operation components are adjusted to obtain the task-driven intent prediction score.

[0153] Specifically, continuously monitoring the frequency of an operator's gaze shifting between multiple adjacent expected operational components and the percentage of time spent on each adjacent expected operational component involves capturing the operator's eye movement data in real time using eye-tracking sensors built into augmented reality devices or external cameras, and then analyzing the movement trajectory and dwell time of the gaze focus between different expected operational components using image processing and pattern recognition algorithms. The frequency of shifting can be understood as the number of times the gaze focus quickly moves from one component to another adjacent component within a short period, while the percentage of dwell time indicates the proportion of time the gaze focus remains on a specific component out of the total observation time. These indicators are used to quantify the operator's hesitation or comparison state when faced with multiple similar or adjacent components, to adapt to the learner's behavior of comparing and confirming adjacent components in maintenance training.

[0154] Acquiring the degree of pose matching between an operator's hand pointing and / or touching postures and multiple adjacent expected manipulation parts refers to acquiring the operator's three-dimensional pose information of their hand through hand tracking sensors (such as depth cameras or inertial measurement units, IMUs) and calculating the degree of agreement between the hand pointing vector or touch area and the spatial position and pose of each adjacent expected manipulation part. The pose matching degree can be a normalized score reflecting the directional nature or explicitness of the operator's hand movements or contact intentions. For example, a high matching degree indicates a precise pointing or touching of a part; a low matching degree indicates a vague movement of the hand over multiple parts, thus reflecting the learner's operational certainty during training steps.

[0155] When the operator's gaze jumps between multiple adjacent expected operating components more frequently than a preset jump threshold and the posture matching degree is lower than a preset matching threshold, it indicates that the operator's intention is uncertain or ambiguous. At this point, the system needs deeper information to assist in the judgment. The preset jump threshold and preset matching threshold are determined based on empirical data or user testing, and are used to define the critical state of ambiguous intention, thereby reducing the probability of interacting with incorrect components during maintenance training.

[0156] This process extracts microscopic geometric features and functional differences from the semantic information of multiple adjacent expected operational components, and generates component discrimination scores based on these features and functional differences. This means that when multimodal intent information (gaze and hand gestures) is insufficient to clearly distinguish components of the target device, the system delves deeper into pre-stored component semantic information. Microscopic geometric features can include fine textures, edge details, surface irregularities, and connection methods. These features may not be obvious macroscopically but are discriminative when observed up close. Functional differences refer to the specific role of a component in the overall device, its relationship with other components, and its operation method. For example, two adjacent bolts, one used to secure electrical wiring and the other to secure hydraulic pipes, have different functional descriptions. The component discrimination score is calculated based on these microscopic geometric features and functional differences using a machine learning model or expert rule system, and is used to quantify the distinguishability between different components. A higher score indicates that the component is more easily distinguished, thus providing a basis for fine-grained error correction of virtual guidance in the training scenario.

[0157] Calculating the attention weight of each adjacent expected operation component based on dwell time percentage, posture matching degree, and component discrimination score involves fusing multiple pieces of information. Attention weight is a comprehensive indicator reflecting the operator's true level of attention to specific adjacent components. For example, weighted summation, neural networks, or other fusion algorithms can be used, taking dwell time percentage, posture matching degree, and component discrimination score as input, and outputting the attention weight of each adjacent component. Dwell time percentage and posture matching degree reflect the operator's immediate behavioral intent, while the component discrimination score provides information on the inherent distinguishability of the component itself.

[0158] Based on attention weights, the basic task-driven intent prediction scores of multiple adjacent expected operation components are adjusted to obtain the task-driven intent prediction score. This means applying the calculated attention weights to the basic task-driven intent prediction score. For example, the basic task-driven intent prediction score can be multiplied by the corresponding attention weight, or adjusted through other functional relationships, so that components with higher attention weights receive higher task-driven intent prediction scores. This more accurately reflects the operator's actual operational goals and reduces frequent changes in virtual guidance caused by misjudgments during maintenance training.

[0159] The solution proposed in this application introduces more refined multimodal information and component semantic information to assist in judgment when the operator's area of ​​focus is simultaneously covered by multiple adjacent expected operating components, resulting in ambiguity of intent. This improves the stability of guidance and consistency of judgment during the maintenance training process.

[0160] In some preferred embodiments, the following specific example illustrates the situation:

[0161] Suppose that during an industrial equipment maintenance training task, an operator needs to tighten a specific bolt located in a complex valve assembly. This valve assembly contains two adjacent bolts, A and B, that look very similar, and their virtual guidance areas displayed in the AR device may overlap or be closely adjacent.

[0162] First, the AR system continuously monitors the operator's gaze focus. The system detects that the operator's gaze frequently jumps between bolt A and bolt B, exceeding a preset jumping threshold, and the time spent on any single bolt is consistently low. Simultaneously, the system uses hand-tracking sensors to acquire the operator's hand posture, finding that while the hand is generally pointing towards the area, it is not precisely pointing towards bolt A or bolt B, and the degree of match between the hand's pointing posture and the posture of any bolt is below a preset matching threshold.

[0163] Due to the high frequency of eye movement shifts and low hand gesture matching, the system's judgment of the operator's intentions is ambiguous. In this case, the system extracts detailed information about bolts A and B from pre-stored component semantic information. For example, bolt A might be described as an "M6 bolt for securing a pressure sensor," and its micro-geometric features might include a specific batch code on the head; while bolt B might be described as an "M6 bolt for securing a hydraulic line," and its micro-geometric features might include an anti-loosening washer on the head. These micro-geometric features and functional difference descriptions are used to generate component discrimination scores; for example, bolt A's discrimination score might be slightly higher than bolt B's because its batch code provides additional identification information.

[0164] Next, the system calculates the attention weights for bolt A and bolt B based on the proportion of gaze dwell time, the degree of hand posture matching, and the component discrimination score. For example, if the operator's gaze jumps around but lingers slightly longer on bolt A, and bolt A has a higher component discrimination score, then the attention weight for bolt A will be calculated to be higher.

[0165] Finally, the system adjusts the prediction scores of the basic task-driven intent for bolts A and B based on these attention weights. For example, if the attention weight for bolt A is 0.7 and for bolt B it is 0.3, then the final prediction score of the task-driven intent for bolt A will be significantly higher than that for bolt B. Thus, the system can accurately infer that the operator's actual operational target is bolt A and adjust the virtual guidance accordingly to precisely point to bolt A, thereby guiding the operator to complete the correct maintenance operation.

[0166] Reference Figure 5 In another embodiment of this application, S5000 further includes:

[0167] S5001: For multiple components in the semantic information of components that correspond to the areas of interest indicated by body movement information and / or gaze information, calculate the matching scores of multiple components with body movement information, the matching scores of multiple components with gaze information, and the matching scores of multiple components with speech information, respectively.

[0168] S5002: The matching scores of body movement information, eye gaze information, and voice information are fused according to the preset fusion weights to obtain the intention confidence scores of each component.

[0169] S5003: The component with the highest intention confidence score is identified as the second component.

[0170] Specifically, when inferring the operator's actual operational target, it is first necessary to identify multiple potential components corresponding to the area of ​​interest indicated by limb movement information and / or gaze information. The area of ​​interest can be determined by augmented reality devices by analyzing data such as the operator's head posture, eye movement trajectory, and hand posture, and candidate selection is completed within the spatial location range of the components defined by the component semantic information, so that the set of candidate components is consistent with the interaction range of the maintenance training task.

[0171] Subsequently, the system calculates a matching score for each potential component based on its physical movement, gaze, and voice information. The score for physical movement assesses the degree of conformity between the operator's hand or body posture and the component's spatial position and posture. The score for gaze measures the duration, frequency, and coverage of the operator's gaze on the component. The score for voice analysis examines the semantic relevance between keywords in the operator's verbal commands and the component's functional description or identifier. For example, if the operator's voice command contains verbs such as "start" or "stop," the system evaluates the matching degree between these verbs and the component's functional description, thus adapting to the interaction method of triggering steps or confirming operations via commands during maintenance training.

[0172] Furthermore, to comprehensively consider the contributions of different modalities, the matching scores of body movement information, gaze information, and voice information are fused according to preset fusion weights to obtain the intent confidence score for each potential component. The preset fusion weights can be dynamically adjusted or pre-configured based on the reliability and importance of different modalities or the characteristics of the current task scenario. For example, the weight of gaze information can be increased in scenarios requiring precise operation, and the weight of voice information can be increased in scenarios with clear voice commands. Finally, the potential component with the highest intent confidence score is determined as the second component, i.e., the operator's actual operational target, and is used to correct and adjust the direction of virtual guidance.

[0173] The above process refines the inference of the real operation target into a closed loop of "candidate recognition - multimodal matching - fusion decision": first, the potential component set is limited by the attention area, then the intention direction is independently quantified from three dimensions: limb action information, gaze information, and voice information, and finally, the multi-source clues are unified into the intention confidence score by the preset fusion weight, thereby reducing the risk of misjudgment caused by insufficient or ambiguous single modal information, and thus improving the consistency of step judgment and guidance feedback in the maintenance training process.

[0174] Reference Figure 6 In another embodiment of this application, a method for calculating the attention weight of each adjacent expected operating component is further proposed, which includes:

[0175] S5634-1: Assess the stability of the operator's line-of-sight data stream and calculate the real-time reliability score of the line-of-sight based on the bounce frequency and dwell time percentage.

[0176] S5634-2: Continuously evaluate the stability of the hand pointing posture data stream, and calculate the real-time reliability score of the hand pointing posture by combining the displacement variance of the hand key points and the continuity of the pointing vector.

[0177] S5634-3: Evaluate the reliability score of component discrimination information based on the uniformity of the distribution of component discrimination scores;

[0178] S5634-4: Based on the real-time reliability score of the gaze focus, the real-time reliability score of the hand pointing posture, and the reliability score of the component differentiation information, dynamically adjust the contribution ratio of dwell time, posture matching degree, and component differentiation score to the attention weight calculation to obtain the attention weight of each adjacent expected operation component.

[0179] Specifically, when calculating the attention weights of adjacent expected operational components, the reliability of each input data used for calculating the attention weights must first be assessed. The stability of the gaze focus data stream refers to the degree of fluctuation and consistency of the operator's gaze focus data over a period of time. Its assessment can be combined with the frequency of fluctuations and the proportion of dwell time to calculate the real-time reliability score of the gaze focus. This score reflects the confidence level of the gaze focus data as a basis for intention judgment. When the gaze focus data stream fluctuates significantly or exhibits anomalies, the real-time reliability score of the gaze focus decreases accordingly to avoid amplifying errors caused by obstructions, changes in lighting, or gaze tracking errors during maintenance training.

[0180] Furthermore, the stability of the hand pointing posture data stream refers to the stationarity and consistency of the operator's hand pointing posture data in a continuous time series. Its evaluation can be performed by combining the displacement variance of hand keypoints and the continuity of the pointing vector in the hand pointing posture to calculate the real-time reliability score of the hand pointing posture. When the displacement variance of hand keypoints is large or the pointing vector changes frequently, it indicates that the hand posture data is unstable, and the real-time reliability score of the hand pointing posture decreases, thus reducing misjudgments caused by hand tremors, movement hesitation, or interactive posture drift during training.

[0181] Furthermore, the reliability score of component discrimination information can be evaluated based on the uniformity of the component discrimination score distribution. The uniformity of distribution is used to characterize the degree of differentiation formed by the micro-geometric features and functional differences between multiple adjacent expected operating components. When the component discrimination score distribution is too concentrated or the differences are not obvious, it indicates that the component discrimination information is weak in distinguishing different components, and the reliability score of the component discrimination information is correspondingly reduced, thus adapting to the typical scenario of densely distributed visually similar components in maintenance training.

[0182] Therefore, after obtaining the real-time reliability scores of the gaze focus, hand pointing posture, and component discrimination information, the system dynamically adjusts the contribution ratios of dwell time, posture matching degree, and component discrimination score in the attention weight calculation based on these reliability scores. For example, when the real-time reliability score of the gaze focus is low, the contribution of the dwell time ratio in the attention weight calculation is reduced; when the real-time reliability score of the hand pointing posture is high, the contribution of posture matching degree is increased; and when the reliability score of component discrimination information is low, the contribution of component discrimination score is reduced. Through this dynamic adjustment mechanism, the system can more accurately integrate various input information as the input data quality changes with the environment and behavior, thereby obtaining the attention weight of each adjacent expected operation component, thus improving the consistency and stability of maintenance training step determination and virtual guidance correction.

[0183] The solution proposed in this application reduces the risk of inaccurate attention weight calculation due to unstable or unreliable input data by introducing a reliability assessment of the gaze focus data stream, hand pointing posture data stream, and component discrimination information, and dynamically adjusting the contribution ratio of these information in the attention weight calculation based on the assessment results. This reduces the phenomenon of frequent shaking or repeated jumping of virtual guidance during training.

[0184] Reference Figure 7 In another embodiment of this application, S5634-1 further includes the following steps:

[0185] S5634-11: Acquire the operator's head posture data;

[0186] S5634-12: Based on head posture data, identify the jitter or drift components in the gaze focus data stream caused by head shaking or unstable device wearing;

[0187] S5634-13: Acquire ambient lighting data;

[0188] S5634-14: Based on ambient lighting data, identify abnormal components in the line-of-sight focus data stream caused by pupil recognition instability due to changes in lighting.

[0189] S5634-15: Remove jitter or drift components and abnormal components from the gaze focus data stream to obtain a corrected gaze focus data stream;

[0190] S5634-16: Recalculate the bounce frequency and dwell time percentage based on the corrected gaze focus data stream;

[0191] S5634-17: Assess the stability of the operator’s line-of-sight data stream, and calculate the real-time reliability score of the line-of-sight focus by combining the recalculated bounce frequency and dwell time percentage.

[0192] Specifically, acquiring operator head posture data can be achieved by using the inertial measurement unit (IMU) built into the augmented reality device or an external tracking system to monitor the three-dimensional position and posture changes of the operator's head in real time, reflecting minute movements or swaying of the head in space. Identifying jitter or drift components based on head posture data involves analyzing the change patterns of the head posture data, such as detecting high-frequency vibrations or slow displacement trends, to determine whether there is any unrealistic movement in the gaze focus data stream caused by head swaying or unstable device wearing. For example, when the head posture data changes drastically in a short period of time, the corresponding change in the gaze focus data stream can be considered a jitter or drift component, to accommodate gaze jitter caused by learners turning their heads, looking down to align with components, or changes in the tightness of the device during maintenance training. Ambient lighting data can be acquired using ambient light sensors integrated into augmented reality devices to measure parameters such as brightness and color temperature of the surrounding environment in real time. Identifying anomalous components based on ambient lighting data refers to the potential temporary instability of the pupil recognition algorithm and abnormal gaze focus data when ambient lighting changes rapidly. By comparing changes in lighting with abnormal fluctuations in gaze focus data, anomalous components caused by pupil recognition instability due to lighting changes can be identified. This adapts to drastic lighting changes in inspection and training scenarios, such as hand occlusion, proximity to equipment shadows, or lighting switching in the training area. Therefore, jitter, drift, and anomalous components can be removed from the gaze focus data stream using techniques such as filtering, smoothing, or model-based prediction. For example, a Kalman filter can be used to filter out noise and unstable components, resulting in a corrected gaze focus data stream, thereby reducing the interference of occlusion and lighting changes on gaze stability assessment. Based on this, the bounce frequency and dwell time ratio are recalculated according to the corrected gaze focus data stream, so that the subsequent reliability assessment is based on purer and more realistic gaze behavior data. Finally, the stability of the gaze focus data stream is evaluated and combined with the recalculated bounce frequency and dwell time ratio to calculate the real-time reliability score of the gaze focus, thereby providing a more accurate and robust gaze reliability measure for the subsequent attention weight calculation.

[0193] This application's solution mitigates the inaccuracy of gaze focus data caused by external interference factors (such as head movement, unstable equipment wearing, and changes in ambient lighting) by preprocessing the original gaze focus data stream. By acquiring head posture data and ambient lighting data as auxiliary information, it identifies and quantifies jitter or drift components and abnormal components in the gaze focus data stream, removing them from the original stream to obtain a corrected gaze focus data stream that more accurately reflects the operator's gaze intent. Furthermore, it recalculates the frequency of jitter and the proportion of dwell time, improving the accuracy of key indicators and making the calculation of real-time reliability scores more reliable and effective. This fundamentally improves the quality of gaze focus data and supports stable inferences about real operational targets, thereby enhancing the stability of guidance and consistency of procedure judgment in maintenance training.

[0194] Reference Figure 8 In another embodiment of this application, it is further proposed to recalculate the jitter frequency based on the corrected gaze focus data stream, including:

[0195] S5634-161: Acquire the operator's detailed operating instructions and / or the magnified display status of the operating area;

[0196] Among these, precise operation instructions can be understood as operators explicitly instructing the system to enter a mode requiring high-precision interaction through voice, gestures, or interface input, such as "zoom in here" or "fine-tune." The local magnification display of the operating area refers to the augmented reality device magnifying a specific area currently of focus for the operator when displaying virtual content, assisting in more detailed observation or operation. For example, during training, a local magnified display of bolt, valve scale, or indicator light status can be used. All this information can serve as a basis for determining whether the operator is in precise operation mode.

[0197] S5634-162: Based on the fine operation instructions and / or the local magnified display status, identify whether the current state is in the preset fine operation mode;

[0198] Specifically, the system determines whether the current operating environment meets the preset conditions for a fine-grained operation mode based on the type of fine-grained operation command received or the activation status of a local magnified display. For example, when the system detects that the operator has issued a voice command for "fine-grained mode," or when a certain area on the augmented reality device's display screen is magnified to a preset threshold, it can be identified as being in fine-grained operation mode, thus distinguishing between the different interaction stages of "coarse positioning browsing" and "fine-grained alignment operation" during maintenance training.

[0199] S5634-163: In fine operation mode, dynamically adjust the calculation threshold of the jumping frequency;

[0200] Based on this, when the system recognizes that it is in a fine-grained operation mode, the threshold used to calculate the jitter frequency will be dynamically adjusted. For example, the threshold for calculating the jitter frequency can be appropriately relaxed or tightened to adapt to the characteristics of the gaze focus in fine-grained operation mode. This adjustment aims to enable the calculation of the jitter frequency to more sensitively capture the small-amplitude but high-frequency gaze movements unique to fine-grained operation, while avoiding misjudging these real jitters as instability, thereby reducing unnecessary error corrections or prompt jumps triggered by misjudging "fine-grained gaze micro-movements" during training.

[0201] S5634-164: Perform frequency domain analysis on the corrected line-of-sight focus data stream to identify high-frequency, low-amplitude oscillation components;

[0202] Specifically, frequency domain analysis is a mathematical method for converting time-domain signals into frequency-domain signals, such as through Fast Fourier Transform (FFT). By performing frequency domain analysis on the corrected line-of-sight focus data stream, different frequency components can be identified. Oscillatory components refer to periodic or quasi-periodic fluctuations exhibited in the line-of-sight focus data stream. They are characterized by high frequency but relatively small amplitude. This is usually related to the operator's subtle and rapid scanning or adjustment of the line of sight within the target area during fine observation or operation, such as the subtle scanning behavior of a learner aligning bolt holes or reading scales during training.

[0203] S5634-165: Based on the degree of matching between the oscillation component and the fine operation mode, determine whether the oscillation component is a real jitter caused by fine operation, and perform data processing on the corrected line-of-sight focus data stream: when it is determined that the oscillation component is a real jitter caused by fine operation, include the oscillation component in the calculation of the jitter frequency; when it is determined that the oscillation component is sensor noise or residual jitter, exclude the oscillation component from the calculation of the jitter frequency.

[0204] The degree of matching can be determined by comparing the frequency range, amplitude characteristics, and spatial distribution pattern of the oscillation component with the expected gaze behavior pattern under the fine-operation mode. For example, if the oscillation component exhibits characteristics of being highly concentrated in a specific small area and within a specific frequency range as expected under the fine-operation mode, the degree of matching is considered high, and it is judged as a genuine jitter. Conversely, if the oscillation component exhibits randomness, dispersion, or characteristics inconsistent with the fine-operation mode, the degree of matching is considered low, and it is judged as sensor noise or residual jitter. Based on the judgment results, selective data processing is performed on the gaze focus data stream to ensure that only jitter that truly reflects the operator's intention is included in the calculation, so as to ensure that the "fine-operation gaze" in maintenance training is not treated as unstable noise and weakens its intention indication value.

[0205] S5634-166: And calculate the jitter frequency based on the line-of-sight focus data stream after data processing.

[0206] Therefore, after selective data processing, the operator's jumping frequency is recalculated using a gaze focus data stream that includes real jumps and excludes noise, so as to provide input that is closer to the training interaction characteristics for subsequent calculation of gaze focus real-time reliability score and attention weight.

[0207] The solution proposed in this application solves the problem of distinguishing between real fluctuations and noise in the gaze focus data stream in augmented reality environments, especially when performing high-precision tasks, by introducing a mechanism for recognizing and processing fine operation modes.

[0208] Reference Figure 9 In another embodiment of this application, specifically, determining whether the oscillation component is a genuine fluctuation caused by fine operation based on the degree of matching between the oscillation component and the fine operation mode includes:

[0209] S5634-1651: Continuously monitor the spatial distribution and temporal evolution of oscillatory components across multiple adjacent expected operating components;

[0210] S5634-1652: Obtain spatial proximity and functional association information between multiple adjacent expected operating components;

[0211] S5634-1653: Based on spatial distribution, temporal evolution, spatial proximity, and functional correlation information, identify whether the oscillation component exhibits a continuous, focused activity pattern on a single expected operating component;

[0212] S5634-1654: When the oscillating component exhibits a continuous, focused activity pattern on a single expected operating component, the oscillating component is determined to be a genuine fluctuation caused by fine operation.

[0213] S5634-1655: When the oscillation component exhibits a dispersed and irregular activity pattern among multiple adjacent expected operating components, the oscillation component is determined to be sensor noise or residual jitter.

[0214] Specifically, continuous monitoring of the spatial distribution and temporal evolution of oscillation components on multiple adjacent expected operating components refers to real-time analysis of the corrected gaze focus data stream, tracking the corresponding positions of high-frequency, low-amplitude oscillation components in the real-world scenario and their dynamic trajectories over time, and extracting features such as instantaneous position, movement speed, and duration. This monitoring can be implemented using a space-time filter or pattern recognition algorithm to adapt to the micro-motion characteristics of gaze during fine interactions such as alignment, tightening, and turning in a small area during maintenance training.

[0215] Furthermore, acquiring spatial proximity and functional association information between multiple adjacent expected operating components refers to extracting the relative positions and dimensional relationships of these expected operating components in the equipment coordinate system, as well as their logical connections or operational sequences in the industrial maintenance task process, from pre-stored component semantic information. For example, two adjacent bolts have spatial proximity, while a valve and its corresponding control lever have functional association information, thereby providing constraints for the target attribution of fine attention in training levels by combining "sequential steps / linked components".

[0216] Based on this, the system identifies whether the oscillation component exhibits a continuous and focused activity pattern on a single expected operating component, according to spatial distribution, temporal evolution, spatial proximity, and functional association information. In other words, it determines whether the spatial trajectory of the oscillation component is concentrated in a local area of ​​a certain expected operating component for a long time and remains relatively stable. For example, if the oscillation component jumps around a screw in a small range for a long time, it may indicate that the operator is trying to tighten the screw.

[0217] When the oscillation component exhibits a continuous, focused activity pattern on a single expected operating component, it is determined to be a genuine jitter caused by fine manipulation. When the oscillation component exhibits a dispersed, irregular activity pattern among multiple adjacent expected operating components, it is determined to be sensor noise or residual jitter. This technical solution effectively distinguishes between genuine line-of-sight jitter and invalid jitter, avoiding the inclusion of unintentional jitter in the jitter frequency calculation. This improves the accuracy and reliability of jitter frequency calculation and provides more reliable input for subsequent virtual guidance adjustments and virtual content updates, enhancing the interaction accuracy and user experience of augmented reality devices in fine manipulation scenarios such as industrial maintenance.

[0218] Reference Figure 10 The specific embodiments of this application also disclose a 3D and multimedia content fusion image analysis system for AR, including:

[0219] Information acquisition module 1 is used to acquire the user's multimodal intent information, which includes at least body movement information, gaze information, and voice information;

[0220] Semantic information acquisition module 2 is used to acquire the semantic information of the components of the target device. The semantic information of the components is provided by the pre-stored three-dimensional model corresponding to the target device, and includes the identity of each component of the target device and its spatial position range in the device coordinate system, as well as the functional description and geometric feature information of the components.

[0221] The guidance determination module 3 is used to generate and display virtual guidance in a real-world scenario. The virtual guidance consists of a virtual highlighted area and / or a virtual indicator arrow used to indicate the target device component.

[0222] The target localization module 4 is used to determine the component that the virtual guide points to in the real scene based on the visual localization results of the augmented reality device, and obtain the first component;

[0223] The target inference module 5 is used to infer the operator's actual operation target based on multimodal intent information and component semantic information, and obtain the second component;

[0224] Comparison module 6 is used to compare the identification and / or spatial location range of the first component and the second component to obtain the comparison result;

[0225] The guidance adjustment module 7 is used to adjust the position of the virtual guide according to the comparison results, so that the virtual guide points to the second component;

[0226] The content update module 8 is used to update the display method of virtual content according to the adjusted virtual guidance, including updating the display position and / or posture of the 3D model and multimedia content in the real scene, and maintaining the preset relative positional relationship between the 3D model and multimedia content.

[0227] This system, through the collaborative work of its modules, effectively solves the problems of inaccurate positioning, jitter, or misalignment of virtual information in traditional augmented reality (AR) systems in complex industrial environments. In virtual industrial maintenance training applications, accurate target inference and virtual guidance adjustments help learners complete tasks, enhancing immersion and operational precision during the learning process.

[0228] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for image analysis that fuses 3D and multimedia content for AR, characterized in that, An augmented reality device is used to overlay and display virtual content onto a real-world scene. The virtual content includes at least a 3D model corresponding to a target device in the real-world scene and multimedia content. The method includes: The user's multimodal intent information is obtained, which includes at least body movement information, gaze information, and voice information; Obtain semantic information of the components of the target device. The semantic information of the components is provided by a pre-stored three-dimensional model corresponding to the target device, and includes the identity of each component of the target device and its spatial position range in the device coordinate system, as well as the functional description and geometric feature information of the components. In a real-world scenario, virtual guides are generated and displayed, wherein the virtual guides are virtual highlighted areas and / or virtual indicator arrows used to indicate target device components; Based on the visual positioning results of the augmented reality device, the component that the virtual guide points to in the real scene is determined, and the first component is obtained; Based on the multimodal intent information and the component semantic information, the operator's actual operation target is inferred, and the second component is obtained; Compare the identification and / or spatial location range of the first component with those of the second component to obtain the comparison result; Adjust the position of the virtual guide according to the comparison results, so that the virtual guide points to the second component; The display method of the virtual content is updated according to the adjusted virtual guidance, including updating the display position and / or posture of the 3D model and the multimedia content in the real scene, and maintaining the preset relative positional relationship between the 3D model and the multimedia content; The second component is obtained by inferring the operator's true operational goal based on the multimodal intent information and the component semantic information, including: For multiple components in the semantic information of the components that correspond to the areas of interest indicated by the body movement information and / or gaze information, the matching scores of the multiple components with the body movement information, the matching scores of the multiple components with the gaze information, and the matching scores of the multiple components with the speech information are calculated respectively. The matching scores of the body movement information, the matching scores of the gaze information, and the matching scores of the voice information are fused according to the preset fusion weights to obtain the intention confidence scores of each component. The component with the highest intent confidence score is identified as the second component.

2. The AR-oriented 3D and multimedia content fusion image analysis method according to claim 1, characterized in that, Based on the multimodal intent information and the component semantic information, the operator's actual operational goal is inferred, resulting in a second component, including: Obtain a pre-stored industrial maintenance task flowchart, which is a maintenance task flowchart corresponding to the target equipment, and the industrial maintenance task flowchart includes the preconditions, postconditions, corresponding expected operation components, and expected logical state of the expected operation components after the operation is completed for each task step. Obtain the physical state information of the corresponding expected operation component. The physical state information includes the operator's confirmation information, the physical state of key components related to the expected operation component obtained based on image recognition, and sensor data inside the target device. Update the current task status based on the physical state information and the industrial maintenance task flowchart; Based on the updated task status, determine the next task step in the industrial maintenance task flowchart, and obtain the expected operation components corresponding to the next task step to form a list of the next expected operation components. Based on the list of next expected operation components, obtain the identity identifier, functional description, and spatial location range of each expected operation component from the component semantic information; Based on the updated task status and / or the corresponding task steps and / or activated task paths in the industrial maintenance task flowchart, a task-driven intent prediction score is calculated for each expected operation component in the next expected operation component list. Based on the multimodal intent information and the identity, functional description and spatial location range of each expected operation component, the intent matching score of each expected operation component is calculated. The task-driven intent prediction score is then fused with the intent matching score to determine the second component.

3. The AR-oriented 3D and multimedia content fusion image analysis method according to claim 2, characterized in that, Based on the updated task status and the corresponding task steps and / or activated task paths in the industrial maintenance task flowchart, a task-driven intent prediction score is calculated for each expected operation component in the next expected operation component list, including: Based on the updated task status, identify whether there are conditional branches or parallel paths in the industrial maintenance task flowchart; When a conditional branch is detected in the industrial maintenance task flowchart, the physical state and / or logical state of the equipment component is determined based on the physical state information, and the condition of the conditional branch is judged to activate the corresponding task path; and within the activated task path, the task-driven intent prediction score is calculated for the expected operation component in the next expected operation component list. When parallel paths are identified in the industrial maintenance task flowchart, basic task-driven intent prediction scores are calculated for the expected operation components on the parallel paths. The basic task-driven intent prediction scores of the expected operation components on the parallel paths are adjusted according to the operator's current area of ​​focus to obtain the task-driven intent prediction score. The current area of ​​focus is determined by limb movement information and / or line of sight information.

4. The AR-oriented 3D and multimedia content fusion image analysis method according to claim 3, characterized in that, When parallel paths are identified in the industrial maintenance task flowchart, and the current area of ​​interest simultaneously covers multiple adjacent expected operation components, the adjustment of the prediction score for the basic task-driven intent includes: Continuously monitor the frequency of the operator's gaze shifting between multiple adjacent expected operating parts and the percentage of time spent on each adjacent expected operating part; Acquire the degree of matching between the operator's hand pointing and / or touch gestures and the gestures of multiple adjacent expected operating parts; When the frequency of the gaze focus jumping between multiple adjacent expected operation components exceeds a preset jumping threshold and the posture matching degree is lower than a preset matching threshold, the micro-geometric features and functional difference descriptions of multiple adjacent expected operation components are extracted from the component semantic information, and a component discrimination score is generated based on the micro-geometric features and the functional difference descriptions. Based on the dwell time percentage, the posture matching degree, and the component discrimination score, calculate the attention weight of each adjacent expected operation component; Based on the attention weight, the basic task-driven intent prediction scores of multiple adjacent expected operation components are adjusted to obtain the task-driven intent prediction scores.

5. The AR-oriented 3D and multimedia content fusion image analysis method according to claim 4, characterized in that, Calculate the attention weight of each adjacent expected operating component, including: Assess the stability of the operator's gaze focus data stream, and calculate the real-time reliability score of the gaze focus by combining the bounce frequency and the dwell time percentage; The stability of the hand pointing posture data stream is continuously evaluated, and the real-time reliability score of the hand pointing posture is calculated by combining the displacement variance of the key points of the hand in the hand pointing posture and the continuity of the pointing vector. The reliability score of the component discrimination score is evaluated based on the uniformity of the component discrimination score distribution. Based on the real-time reliability score of the gaze focus, the real-time reliability score of the hand pointing posture, and the reliability score of the component differentiation information, the contribution ratio of the dwell time, the posture matching degree, and the component differentiation score to the attention weight calculation is dynamically adjusted to obtain the attention weight of each adjacent expected operation component.

6. The AR-oriented 3D and multimedia content fusion image analysis method according to claim 5, characterized in that, Assess the stability of the operator's gaze focus data stream, and calculate a real-time reliability score for the gaze focus by combining the fluctuation frequency and the percentage of dwell time, including: Acquire the operator's head posture data; Based on the head posture data, identify the jitter or drift components in the gaze focus data stream caused by head shaking or unstable device wearing; Acquire ambient lighting data; Based on the ambient lighting data, identify the abnormal components in the gaze focus data stream caused by pupil recognition instability due to changes in lighting. Remove the jitter or drift component and the abnormal component from the gaze focus data stream to obtain the corrected gaze focus data stream; The jump frequency and the dwell time ratio are recalculated based on the corrected gaze focus data stream; The stability of the operator's gaze focus data stream is assessed, and a real-time reliability score for the gaze focus is calculated by combining the recalculated bounce frequency and the percentage of dwell time.

7. The AR-oriented 3D and multimedia content fusion image analysis method according to claim 6, characterized in that, The jitter frequency is recalculated based on the corrected gaze focus data stream, including: Acquire the operator's detailed operating instructions and / or the magnified display status of the operating area; Based on the fine operation instructions and / or the local magnified display status, identify whether the current state is in the preset fine operation mode; In the fine operation mode, the calculation threshold for the jumping frequency is dynamically adjusted; Frequency domain analysis was performed on the corrected line-of-sight focus data stream to identify high-frequency, low-amplitude oscillation components; Based on the degree of matching between the oscillation component and the fine operation mode, it is determined whether the oscillation component is a real jitter caused by fine operation, and the corrected gaze focus data stream is processed: when it is determined that the oscillation component is a real jitter caused by fine operation, the oscillation component is included in the calculation of the jitter frequency; when it is determined that the oscillation component is sensor noise or residual jitter, the oscillation component is excluded from the calculation of the jitter frequency. The jumping frequency is calculated based on the gaze focus data stream after data processing.

8. The AR-oriented 3D and multimedia content fusion image analysis method according to claim 7, characterized in that, Based on the degree of matching between the oscillation component and the fine-tuning mode, determine whether the oscillation component is a genuine fluctuation caused by fine-tuning, including: Continuously monitor the spatial distribution and temporal evolution of oscillation components across multiple adjacent expected operating components; Obtain spatial proximity and functional association information between multiple adjacent expected operating components; Based on the spatial distribution, the temporal evolution, the spatial proximity, and the functional association information, identify whether the oscillation component exhibits a continuous, focused activity pattern on a single intended operating component; When the oscillation component exhibits a continuous, focused activity pattern on a single intended operating component, the oscillation component is determined to be a genuine fluctuation caused by fine operation. When the oscillation component exhibits a dispersed and irregular activity pattern among multiple adjacent expected operating components, the oscillation component is determined to be sensor noise or residual jitter.

9. A 3D and multimedia content fusion image analysis system for AR, characterized in that, An augmented reality device is used to overlay and display virtual content onto a real-world scene. The virtual content includes at least a 3D model corresponding to a target device in the real-world scene and multimedia content, including: The information acquisition module is used to acquire the user's multimodal intent information, which includes at least body movement information, gaze information, and voice information; The semantic information acquisition module is used to acquire the semantic information of the components of the target device. The semantic information of the components is provided by a pre-stored three-dimensional model corresponding to the target device, and includes the identity identifier of each component of the target device and its spatial position range in the device coordinate system, as well as the functional description and geometric feature information of the components. The guidance determination module is used to generate and display virtual guidance in a real-world scenario. The virtual guidance is a virtual highlighted area and / or a virtual indicator arrow used to indicate a target device component. The target localization module is used to determine the component that the virtual guide points to in the real scene based on the visual localization results of the augmented reality device, and obtain the first component; The target inference module is used to infer the operator's actual operation target based on the multimodal intent information and the component semantic information, and obtain the second component; The second component is obtained by inferring the operator's true operational goal based on the multimodal intent information and the component semantic information, including: For multiple components in the semantic information of the components that correspond to the areas of interest indicated by the body movement information and / or gaze information, the matching scores of the multiple components with the body movement information, the matching scores of the multiple components with the gaze information, and the matching scores of the multiple components with the speech information are calculated respectively. The matching scores of the body movement information, the matching scores of the gaze information, and the matching scores of the voice information are fused according to the preset fusion weights to obtain the intention confidence scores of each component. The component with the highest intent confidence score is identified as the second component; The comparison module is used to compare the identity and / or spatial location range of the first component and the second component to obtain the comparison result; The guidance adjustment module is used to adjust the position of the virtual guide according to the comparison result, so that the virtual guide points to the second component; The content update module is used to update the display mode of the virtual content according to the adjusted virtual guidance, including updating the display position and / or posture of the 3D model and the multimedia content in the real scene, and maintaining the preset relative positional relationship between the 3D model and the multimedia content.

Images