Backpack virtual-real fusion imaging interaction method and system based on AR

Abstract

The invention relates to the technical field of augmented reality interaction, discloses an AR-based knapsack virtual-real fusion imaging interaction method and system, and solves the problems that traditional AR knapsack virtual-real fusion is split in vision, high in interaction false touch rate, unbalanced in computing power load adaptation and disjointed in method and system. The method comprises the steps of multi-modal data acquisition, dynamic weight virtual-real fusion, AR imaging output, multi-modal interaction response and load-computing power adaptive optimization. The system is correspondingly provided with a data acquisition module, a virtual-real fusion processing module, an AR imaging output module, an interactive response module and a self-adaptive optimization module. According to the method, the virtual-real fusion consistency and the interaction reliability are improved, the equipment performance and the endurance are balanced, multiple types of mobile AR scenes can be adapted without changing core hardware, and the method has relatively high industrial practicability.

Description

Technical Field

[0001] This invention relates to the field of augmented reality (AR) interaction technology, specifically to an AR-based backpack virtual-real fusion imaging interaction method and system. Background Technology

[0002] Backpack-based mobile AR devices have become the mainstream platform for outdoor AR applications due to their high portability and long battery life. However, existing technologies have the following key drawbacks:

[0003] Poor virtual-real fusion effect: Traditional solutions use fixed weight fusion algorithms that do not take into account real-time factors such as ambient light intensity and object distance, resulting in a visual disconnect between the virtual model and the real scene and a noticeable "floating" feeling;

[0004] Low reliability of interaction: It relies heavily on a single command to trigger the interaction, is easily affected by environmental interference causing accidental touches, or fails to trigger due to insufficient movement.

[0005] Mismatch between computing power and load: The computing power allocation is not dynamically adjusted according to the backpack load. When the load is large, the power consumption is too high and the battery life is drastically reduced. When the load is small, the computing power is redundant and resources are wasted.

[0006] The aforementioned shortcomings limit the widespread application of AR backpacks in high-precision interactive scenarios, thus necessitating a technical solution with simplified algorithms and highly coupled methodological systems. Summary of the Invention

[0007] This invention provides an AR-based backpack virtual-real fusion imaging interaction method and system, which solves the technical defects pointed out in the background art.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] In a first aspect, the present invention proposes an AR-based backpack virtual-real fusion imaging interaction method, comprising the following steps:

[0010] S1. Multimodal data acquisition: Spatial positioning data, depth data, ambient light data, and user interaction action data of the real scene are collected through the sensor components mounted on the backpack, and filtered and noise-reducing preprocessing is performed;

[0011] S2. Virtual-Real Fusion Processing: The pre-processed real scene data is registered with the pre-stored virtual model data, and a visually consistent virtual-real fusion scene data is generated through a dynamic weight fusion algorithm;

[0012] S3.AR Imaging Output: Transmits the fused scene data to the AR display terminal for real-time rendering and imaging, and feeds back the terminal's rendering status parameters;

[0013] S4. Interactive Response: Receives multimodal interactive commands from users for the integrated scenario, identifies command features, and triggers corresponding scenario updates or function execution operations;

[0014] S5. Adaptive Optimization: Dynamically adjusts the computing resource allocation strategy based on the backpack's current load weight and the power consumption status of each module to balance device performance and battery life.

[0015] As a further improvement to the technical solution of the present invention, the sensor components in step S1 include an inertial measurement unit, a depth camera, an ambient light sensor, a gesture recognition sensor, and a voice pickup module; the spatial positioning data is user attitude angle and displacement data collected by the inertial measurement unit, the depth data is real scene 3D point cloud data collected by the depth camera, and the preprocessed data format is a standardized binary stream adapted for subsequent calculations.

[0016] As a further improvement to the technical solution of the present invention, the calculation of the fusion weight in the dynamic weight fusion algorithm in step S2 satisfies the following formula:

[0017]

[0018] In the formula, This represents the display weight of the virtual model in the fused scene, with a value range of [0,1]. This is the normalized value of ambient light intensity; The perpendicular distance between the virtual model and the reference plane of the real scene; Render the frame rate normalization value for the AR terminal; The weighting coefficients are and satisfy the following conditions: ; The higher the value, the lower the transparency of the virtual model and the higher its fit with the real scene.

[0019] As a further improvement to the technical solution of the present invention, in step S2, the virtual-real registration adopts the iterative nearest point algorithm, and the registration accuracy is controlled within the range of 0.5mm-2mm to ensure that the spatial position of the virtual model matches that of the real scene.

[0020] As a further improvement to the technical solution of the present invention, the multimodal interaction commands in step S4 include gesture commands, voice commands, and posture commands; during command recognition, feature parameters of each command are extracted, including the magnitude of the gesture action feature vector, the confidence level of voice recognition, and the amplitude of posture change, and whether to trigger interaction is determined based on the feature parameters.

[0021] As a further improvement to the technical solution of the present invention, after the interaction is triggered in step S4, the operations performed include translation, rotation, scaling of the virtual model, annotation of virtual information, issuance of device control commands, and real-time feedback of the operation results to the AR display terminal to update the fused scene.

[0022] As a further improvement to the technical solution of the present invention, the resource allocation coefficient of the computing power resource allocation strategy in step S5 is... Satisfy the following formula:

[0023]

[0024] In the formula, This represents the current load weight of the backpack. This refers to the backpack's maximum rated load weight. A baseline coefficient is assigned to the computing power, with a value range of [0.6, 0.9]; according to Adjust the CPU core utilization and GPU rendering priority of the computing module.

[0025] Secondly, this invention proposes an AR-based backpack virtual-real fusion imaging interaction system for executing the method, the system comprising:

[0026] The data acquisition module is used to collect spatial positioning data, depth data, ambient light data, and user interaction data from real-world scenes, and to perform filtering and noise reduction preprocessing.

[0027] The virtual-real fusion processing module is used to register the pre-processed real scene data with the pre-stored virtual model data, and generate visually consistent virtual-real fusion scene data through a dynamic weight fusion algorithm;

[0028] The AR imaging output module is used to transmit the fused scene data to the AR display terminal for real-time rendering and imaging, and to feed back the terminal rendering status parameters.

[0029] The interaction response module is used to receive multimodal interaction commands from users for the integrated scenario, identify command features, and trigger corresponding scenario updates or function execution operations.

[0030] The adaptive optimization module is used to dynamically adjust the computing resource allocation strategy based on the current load weight of the backpack and the power consumption status of each module, so as to balance device performance and battery life.

[0031] As a further improvement to the technical solution of the present invention, the virtual-real fusion processing module, the interactive response module, and the adaptive optimization module are integrated into the edge computing unit. The edge computing unit includes an FPGA preprocessing subunit and an ARM computing subunit. The FPGA preprocessing subunit is used to filter, reduce noise, and convert the format of the multimodal data output by the data acquisition module. The ARM computing subunit pre-stores dynamic weight fusion algorithm and computing power allocation algorithm for virtual-real fusion computing, interactive command recognition, and dynamic control of computing power.

[0032] As a further improvement to the technical solution of the present invention, the system also includes an energy storage management module and a backpack carrier; the energy storage management module is electrically connected to each functional module, including a lithium battery pack and a power consumption monitoring unit, and when the total power consumption exceeds a preset threshold, it sends a signal to the adaptive optimization module to trigger a computing power reduction strategy; the backpack carrier has a built-in load sensor for collecting the current load weight of the backpack. And transmit to the adaptive optimization module; the AR imaging output module is adapted to AR display terminals such as AR glasses or AR helmets, and the rendering frame rate and virtual-real fusion processing module... Parameters are linked in real time.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] This invention achieves high-precision alignment between virtual models and real-world scenes, effectively eliminating the floating sensation in virtual-real fusion; relying on a multimodal interactive command comprehensive judgment mechanism, it improves the smoothness and accuracy of interactive operations, reducing false triggering issues; through load-computing power adaptive allocation, it achieves dynamic matching between computing resources and backpack load, balancing device performance and battery life, avoiding the problems of high power consumption under high load and redundant computing power under low load; it can be flexibly adapted to various mobile AR application scenarios such as industrial inspection, cultural tourism guidance, and emergency rescue. Attached Figure Description

[0035] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0036] Figure 1 This is a flowchart illustrating the steps of an AR-based backpack virtual-real fusion imaging interaction method according to the present invention.

[0037] Figure 2 This is a schematic diagram of a backpack virtual-real fusion imaging interactive system based on AR according to the present invention. Detailed Implementation

[0038] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention.

[0039] The present invention will be further described in detail below with reference to Figures 1-2.

[0040] As shown in the figure, this invention proposes an AR-based backpack virtual-real fusion imaging interaction method and system, which consists of two parts: an interaction method and a supporting execution system. The two are deeply coupled to form a complete technology system of data acquisition, fusion processing, imaging interaction, and computing power optimization, as detailed below:

[0041] (I) AR-based backpack virtual-real fusion imaging interaction method

[0042] This method is a modular closed-loop process, comprising 5 core steps. Each step is dynamically adjusted through real-time parameter feedback, as detailed below:

[0043] S1. Multimodal data acquisition and preprocessing

[0044] The backpack is equipped with multiple sensor components to simultaneously collect spatial positioning data, depth point cloud data, ambient light intensity data, and interactive data such as user gestures and voice commands from real-world scenes. Environmental interference signals are eliminated using filtering and noise reduction algorithms, and the collected data is standardized into a unified format before being output to the subsequent processing unit, providing a high-quality data source for virtual-real fusion and interactive recognition.

[0045] S2. Dynamic Weighted Real-World Fusion and High-Precision Registration

[0046] First, the Iterative Closest Point (ICP) algorithm is used to complete the spatial registration of the point cloud of the real scene and the point cloud of the virtual model, strictly controlling the registration error within the range of 0.5mm-2mm to ensure accurate spatial matching between the virtual model and the real scene. Then, a dynamic weight fusion formula is introduced, which combines three key variables: ambient light intensity, vertical distance between the virtual model and the real object, and real-time rendering frame rate of the AR terminal, to dynamically calculate the display weight of the virtual model and adjust the model transparency, so as to achieve seamless visual integration between virtual information and the real scene and solve the floating effect problem caused by fixed weight fusion.

[0047] The dynamic weight fusion formula is as follows:

[0048]

[0049] In the formula, This represents the display weight of the virtual model in the fused scene, with a value range of [0,1]. The normalized value of ambient light intensity collected by the ambient light sensor; The perpendicular distance between the virtual model and the reference plane of the real scene; This is the normalized value of the real-time rendering frame rate for the AR display terminal. Let be the weight coefficient, and satisfy... ; The higher the value, the lower the transparency of the virtual model and the higher its fit with the real scene.

[0050] S3.AR Imaging Output and Status Feedback

[0051] The fused scene data is transmitted to the AR display terminal and rendered in real time at a frame rate of no less than 90Hz to provide users with an immersive visual experience. At the same time, the terminal feeds back the rendering frame rate, display latency and other status parameters to the calculation module in real time as the parameter input for the dynamic weight fusion formula, forming a closed-loop optimization mechanism of fusion-rendering-feedback-re-fusion.

[0052] S4. Multimodal Interaction Response and Precise Triggering

[0053] The system extracts feature parameters from three types of user interaction commands: gestures, voice, and posture. It uses multi-dimensional features to comprehensively determine the validity of commands, avoiding accidental touches triggered by a single command. When a command is determined to be valid, it immediately executes operations such as virtual model scaling, translation, information annotation, or device control, and updates the results to the AR display scene in real time, achieving low-latency and highly reliable interactive response.

[0054] S5. Load-Computing Power Adaptive Optimization and Power Consumption Control

[0055] The backpack obtains the current load weight through its built-in load sensor, calculates the resource allocation coefficient based on the computing power allocation coefficient formula, and dynamically adjusts the CPU core utilization and GPU rendering priority of the computing module. When the backpack load exceeds 70% of the rated value, the non-core module computing power reduction strategy is automatically triggered to prioritize the core functions of virtual-real fusion and interactive response, so as to achieve the optimal balance between device performance and battery life.

[0056] The formula for the computing power allocation coefficient is as follows:

[0057]

[0058] In the formula, This represents the current load weight of the backpack. This refers to the backpack's maximum rated load weight. A baseline coefficient is assigned to the computing power, with a value range of [0.6, 0.9]; according to The calculation results can be used to linearly adjust the hardware resource usage ratio of the computing module, thereby achieving dynamic matching between computing power and load.

[0059] (II) AR-based backpack virtual-real fusion imaging interaction system

[0060] This system serves as a dedicated hardware platform for implementing the aforementioned methods. It adopts a modular, layered architecture, specifically comprising five core functional modules and supporting carriers and energy storage units, as follows:

[0061] Data acquisition module

[0062] The corresponding execution method step S1 integrates an inertial measurement unit (IMU), a depth camera, an ambient light sensor, a gesture recognition sensor, and a voice pickup module. It has a built-in filtering and preprocessing unit that can perform real-time noise reduction and standardized format conversion on the acquired multimodal data. The output spatial positioning data, depth data, etc. can be directly used as the input data source for the virtual-real fusion processing module.

[0063] Virtual-Real Fusion Processing Module

[0064] The corresponding execution method step S2 includes a built-in database of pre-stored virtual models, an ICP registration algorithm, and a dynamic weight fusion algorithm program. This enables high-precision registration between real scene point clouds and virtual model point clouds, and is based on the input... , , Parameter calculation of fusion weights Generate visually consistent and matched virtual-real fusion scene data.

[0065] AR imaging output module

[0066] In step S3 of the corresponding execution method, as an interactive unit connecting the virtual-real fusion processing module and the AR display terminal, it is responsible for the transmission and rendering control of the fusion scene data, and at the same time collects the real-time rendering frame rate and other status parameters of the AR display terminal and feeds them back to the virtual-real fusion processing module to realize the dynamic adjustment of the fusion weight.

[0067] Interactive response module

[0068] In the corresponding execution method step S4, a multimodal command recognition algorithm is integrated, which can extract feature parameters of gesture, voice and posture commands, identify valid commands through a comprehensive judgment mechanism, and send scene update commands to the AR imaging output module or send control commands to external devices.

[0069] Adaptive optimization module

[0070] In corresponding execution step S5, the backpack carrier load sensor transmits the data. The power consumption data of the data and energy storage management module is calculated based on the computing power allocation coefficient formula. The value dynamically adjusts the CPU core utilization and GPU rendering priority of the edge computing unit; when the load or power consumption exceeds the standard, a computing power reduction strategy is triggered to ensure the stable operation of core functions.

[0071] Supporting units

[0072] Backpack Carrier: Featuring a lightweight carbon fiber design, a built-in high-precision load sensor to collect the current load weight of the backpack, and a fixed mounting position for each functional module. It is ergonomically designed to ensure comfort during long-term wear.

[0073] Energy storage management module: Includes a large-capacity lithium battery pack and a power consumption monitoring unit, which provides stable power supply to each module, collects current and voltage data of each module in real time, and sends a frequency reduction trigger signal to the adaptive optimization module when the total power consumption exceeds the preset threshold.

[0074] The following section provides a detailed explanation of the specific implementation process of the method of this invention, using an industrial inspection scenario as an example.

[0075] The implementation process of the AR-based backpack virtual-real fusion imaging interaction method is as follows:

[0076] S1. Multimodal data acquisition and preprocessing

[0077] Inspection personnel wear AR backpacks equipped with the system of this invention when entering the workshop. The IMU of the data acquisition module collects personnel posture angle and displacement data at a sampling rate of 100Hz. The depth camera collects 3D point cloud data of the equipment. The ambient light sensor collects workshop light intensity data. The gesture sensor and voice pickup module collect personnel interaction command data. All data is converted into a standardized binary stream after being denoised by the module's built-in filtering unit and transmitted to the virtual-real fusion processing module.

[0078] S2. Dynamic Weighting and Real-World Fusion Processing

[0079] The virtual-real fusion processing module retrieves pre-stored virtual device model data and uses the ICP algorithm to complete the registration with the real device point cloud, with a registration accuracy set to 1mm; subsequently, it calculates the dynamic fusion weights and sets... =0.3、 =0.5、 =0.2, normalized value of ambient light intensity in the workshop =0.6, vertical distance between the virtual model and the device surface =0.8m, normalized value of AR terminal rendering frame rate =0.9, the calculation shows that the virtual model is superimposed on the surface of the real device with high fit. The calculation process is as follows:

[0080] .

[0081] S3.AR Imaging Output and Status Feedback

[0082] The virtual-real fusion processing module transmits the fused data to the AR imaging output module, which then sends it to the AR glasses. The AR glasses render the image at a frame rate of 100Hz, allowing inspectors to clearly see the superimposed scene of the equipment's real appearance and virtual internal structure. Simultaneously, the AR imaging output module collects the rendering frame rate of the AR glasses and feeds it back to the virtual-real fusion processing module for real-time adjustment of dynamic weights.

[0083] S4. Multimodal Interactive Response

[0084] The inspector makes a zoom-in gesture and issues a voice command to view the fault area. The interactive response module extracts the gesture feature vector with a magnitude of 0.8 and the voice recognition confidence level of 0.95. Based on the comprehensive judgment, the command is valid and a zoom-in command is sent to the AR imaging output module, triggering the virtual fault area zoom-in operation. The AR glasses update the displayed content in real time.

[0085] S5. Load-Computing Adaptive Optimization

[0086] The backpack's load sensor collects the current load. =5kg, and transmit to the adaptive optimization module, the backpack's maximum rated load. =10kg, computing power allocation baseline coefficient =0.8, substituting into the formula, we get:

[0087] ;

[0088] The adaptive optimization module is based on The value was adjusted to set the CPU core utilization of the edge computing unit to 50% and the GPU rendering priority to medium, balancing performance and battery life.

[0089] This invention enables functions such as virtual and real equipment display, fault labeling, and remote collaboration, significantly improving inspection efficiency and accuracy. It is applicable to multiple fields such as industrial inspection and cultural tourism guidance, and has significant economic and social value.

[0090] This invention achieves high-precision alignment between virtual models and real-world scenes, effectively eliminating the floating sensation in virtual-real fusion; relying on a multimodal interactive command comprehensive judgment mechanism, it improves the smoothness and accuracy of interactive operations, reducing false triggering issues; through load-computing power adaptive allocation, it achieves dynamic matching between computing resources and backpack load, balancing device performance and battery life, avoiding the problems of high power consumption under high load and redundant computing power under low load; it can be flexibly adapted to various mobile AR application scenarios such as industrial inspection, cultural tourism guidance, and emergency rescue.

[0091] The technical solutions provided by the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the embodiments of the present invention. The descriptions of the embodiments above are only for helping to understand the principles of the embodiments of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An AR-based backpack virtual-real fusion imaging interaction method, characterized in that, Includes the following steps: S1. Collect spatial positioning data, depth data, ambient light data, and user interaction action data of the real scene through the sensor components mounted on the backpack, and perform filtering and noise reduction preprocessing; S2. Register the pre-processed real scene data with the pre-stored virtual model data, and generate visually consistent virtual-real fusion scene data through a dynamic weight fusion algorithm; S3. Transmit the fused scene data to the AR display terminal for real-time rendering and imaging, and provide feedback on the terminal rendering status parameters; S4. Receive multimodal interaction commands from users for the integrated scenario, identify command features, and trigger corresponding scenario updates or function execution operations; S5. Based on the current load weight of the backpack and the power consumption status of each module, dynamically adjust the computing resource allocation strategy to balance device performance and battery life.

2. The method according to claim 1, characterized in that, The sensor components mentioned in step S1 include an inertial measurement unit, a depth camera, an ambient light sensor, a gesture recognition sensor, and a voice pickup module; the spatial positioning data is user attitude angle and displacement data collected by the inertial measurement unit, the depth data is real scene 3D point cloud data collected by the depth camera, and the preprocessed data format is a standardized binary stream adapted for subsequent calculations.

3. The method according to claim 1, characterized in that, The fusion weight calculation of the dynamic weight fusion algorithm described in step S2 satisfies the following formula: In the formula, This represents the display weight of the virtual model in the fused scene, with a value range of [0,1]. This is the normalized value of ambient light intensity; The perpendicular distance between the virtual model and the reference plane of the real scene; Render the frame rate normalization value for the AR terminal; The weighting coefficients are satisfied. ; The higher the value, the lower the transparency of the virtual model and the higher its fit with the real scene.

4. The method according to claim 3, characterized in that, In step S2, the virtual-real registration adopts the iterative nearest point algorithm, and the registration accuracy is controlled within the range of 0.5mm-2mm to ensure that the spatial position of the virtual model matches that of the real scene.

5. The method according to claim 1, characterized in that, The multimodal interaction commands mentioned in step S4 include gesture commands, voice commands, and posture commands; during command recognition, feature parameters of each command are extracted, including the magnitude of the gesture action feature vector, the confidence level of voice recognition, and the amplitude of posture change, and the interaction is triggered based on the feature parameters.

6. The method according to claim 5, characterized in that, After the interaction is triggered in step S4, the operations performed include translation, rotation, scaling of the virtual model, annotation of virtual information, issuance of device control commands, and real-time feedback of the operation results to the AR display terminal to update the fused scene.

7. The method according to claim 1, characterized in that, The resource allocation coefficient of the computing power resource allocation strategy described in step S5 Satisfy the following formula: In the formula, This represents the current load weight of the backpack. This refers to the backpack's maximum rated load weight. A baseline coefficient is assigned to the computing power, with a value range of [0.6, 0.9]; according to Adjust the CPU core utilization and GPU rendering priority of the computing module.

8. An AR-based backpack virtual-real fusion imaging interaction system, used to perform the method according to any one of claims 1-7, characterized in that, The system includes: The data acquisition module is used to collect spatial positioning data, depth data, ambient light data, and user interaction data from real-world scenes, and to perform filtering and noise reduction preprocessing. The virtual-real fusion processing module is used to register the pre-processed real scene data with the pre-stored virtual model data, and generate visually consistent virtual-real fusion scene data through a dynamic weight fusion algorithm; The AR imaging output module is used to transmit the fused scene data to the AR display terminal for real-time rendering and imaging, and to feed back the terminal rendering status parameters. The interaction response module is used to receive multimodal interaction commands from users for the integrated scenario, identify command features, and trigger corresponding scenario updates or function execution operations. The adaptive optimization module is used to dynamically adjust the computing resource allocation strategy based on the current load weight of the backpack and the power consumption status of each module, so as to balance device performance and battery life.

9. The system according to claim 8, characterized in that, The virtual-real fusion processing module, interactive response module, and adaptive optimization module are integrated within the edge computing unit. The edge computing unit includes an FPGA preprocessing subunit and an ARM computing subunit. The FPGA preprocessing subunit is used to filter, reduce noise, and convert the format of the multimodal data output by the data acquisition module. The ARM computing subunit pre-stores dynamic weight fusion algorithms and computing power allocation algorithms for virtual-real fusion computing, interactive command recognition, and dynamic control of computing power.

10. The system according to claim 8, characterized in that, The system also includes an energy storage management module and a backpack carrier; the energy storage management module is electrically connected to each functional module, including a lithium battery pack and a power consumption monitoring unit. When the total power consumption exceeds a preset threshold, it sends a signal to the adaptive optimization module to trigger a computing power reduction strategy; the backpack carrier has a built-in load sensor for collecting the current load weight of the backpack. And transmit to the adaptive optimization module; the AR imaging output module is adapted to AR display terminals such as AR glasses or AR helmets, and the rendering frame rate and virtual-real fusion processing module... Parameters are linked in real time.

Images