An augmented reality interaction method, device, system and storage medium

Abstract

This invention provides an augmented reality interaction method, device, system, and storage medium. The method includes: simultaneously acquiring first and second image information; dividing the two images into first and second multiple location regions, and marking the corresponding first and second target regions; calculating depth information or motion information as first output information based on the disparity or pixel difference between the two target regions, with the local processing result including the first output information; determining whether the first output information or the local processing result meets a trigger condition; if so, sending a third or fourth target region unused in the local processing to a cloud server; receiving the cloud processing result; and generating augmented reality interaction output based on the local processing result and the cloud processing result. This invention improves response speed and semantic understanding accuracy in complex 3D environments by using local binocular vision for rapid perception and selective cloud augmentation, while conserving wearable device resources.

Description

Technical Field

[0001] This invention relates to the fields of augmented reality and artificial intelligence, and in particular to an augmented reality interaction method, device, system and storage medium. Background Technology

[0002] Artificial intelligence, through methods such as machine learning and deep learning, endows machines with the abilities of perception, reasoning, and decision-making, and is becoming a core force driving the digital transformation of society. Wearable devices, such as smart headphones, glasses, watches, and brooches, have become important interfaces for individuals to access AI capabilities. By combining sensors to collect multimodal data (such as sound, light, temperature, location, and motion) in real time, wearable devices can provide users with a range of suggestions and answers to their questions.

[0003] However, AI functionality places far higher demands on the performance and energy consumption of wearable devices than ever before. Current technologies either rely entirely on local processing, leading to insufficient battery life and slow response times, or frequently call upon cloud AI, resulting in strong network dependence and uncontrollable latency. How to respond to changes in the external three-dimensional environment with the highest possible response speed and accuracy within the performance and battery life constraints of wearable scenarios is a pressing problem that needs to be solved. Summary of the Invention

[0004] The present invention provides an augmented reality interaction method, device, system and storage medium, which aims to solve or at least partially alleviate the above-mentioned problems existing in the prior art.

[0005] According to one aspect of the present invention, an augmented reality interaction method is provided, applied to a wearable device, comprising: simultaneously acquiring first image information and second image information through a first camera and a second camera on the wearable device; dividing the first image information and the second image information into a first plurality of location regions and a second plurality of location regions, respectively; marking a first target region and a second target region according to the degree of matching between at least one attribute of texture, brightness, and color of the first plurality of location regions and the second plurality of location regions and a preset template; calculating depth information or motion information of an environmental target based on the parallax or pixel difference between the first target region and the second target region to generate a local processing result; determining whether the depth information or the local processing result satisfies a preset trigger condition; in response to satisfying the trigger condition, extracting a third target region that does not overlap with the first target region from the first plurality of location regions, or extracting a fourth target region that does not overlap with the second target region from the second plurality of location regions, and sending the third target region or the fourth target region to a cloud server; receiving a cloud processing result from the cloud server; and generating an augmented reality interaction output based on the local processing result and the cloud processing result.

[0006] According to one embodiment of the present invention, the first camera and the second camera have the same optical parameters and are positioned on the wearable device at a fixed baseline distance.

[0007] According to one embodiment of the present invention, the division into a first plurality of location regions and a second plurality of location regions includes: dividing the first image information into the first plurality of location regions according to a preset division rule, and dividing the second image information into the second plurality of location regions according to the preset division rule; wherein, the preset division rule includes grid division, quadtree division, or adaptive division based on image content.

[0008] According to one embodiment of the present invention, the step of marking the first target region and the second target region further includes: based on a locally deployed first artificial intelligence model, performing feature extraction and matching on the first plurality of location regions and the second plurality of location regions respectively, to determine the correspondence between the first target region and the second target region; wherein the first target region and the second target region correspond to the same environmental target in the first image information and the second image information.

[0009] According to one embodiment of the present invention, the local processing result includes depth information or motion information calculated by a binocular vision algorithm, and semantic annotations generated by a first artificial intelligence model deployed on the wearable device; the cloud processing result is generated by a second artificial intelligence model deployed on the cloud server; and the second artificial intelligence model has a higher number of parameters or higher inference accuracy than the first artificial intelligence model.

[0010] According to one embodiment of the present invention, determining whether a preset triggering condition is met includes at least one of the following: determining whether the distance between the environmental target and the wearable device is less than a first threshold; determining whether the confidence level of the local processing result is lower than a second threshold; determining whether a user's attention instruction to the first environmental target is received.

[0011] According to one embodiment of the present invention, the first threshold is dynamically adjusted based on the current network status or the remaining power of the device.

[0012] According to one embodiment of the present invention, the third target area includes all or part of the first plurality of location areas excluding the first target area, and the fourth target area includes all or part of the second plurality of location areas excluding the second target area; wherein, the third target area or the fourth target area is used for the cloud server to identify supplementary environmental targets in the surrounding environment of the wearable device.

[0013] According to one embodiment of the present invention, the third target region or the fourth target region is sent to the cloud server in the form of an original image block, a compressed image block, or a feature vector extracted from an image block.

[0014] According to one embodiment of the present invention, before sending the third target region or the fourth target region, the method further includes: encoding the image data of the third target region or the fourth target region into a feature vector; and generating a model identifier corresponding to a second artificial intelligence model to be invoked in the cloud, wherein the model identifier is used to instruct the cloud server to select a target artificial intelligence model that matches the current environmental scene.

[0015] According to one embodiment of the present invention, the wearable device calculates depth information or motion information in real time using a binocular vision algorithm, and continuously runs a first artificial intelligence model deployed on the wearable device to generate semantic annotations in real time, which together constitute the local processing result; in response to the fulfillment of the triggering condition, the wearable device initiates a communication session with the cloud server, and controls the first artificial intelligence model and the second artificial intelligence model to enter a parallel running state, wherein the first artificial intelligence model continues to process subsequent frame images to generate subsequent semantic annotations, and the second artificial intelligence model processes the third target region or the fourth target region; and, based on the processing result of the first artificial intelligence model on subsequent frames and the cloud processing result returned by the cloud server, a time-series aggregated augmented reality interactive output is generated.

[0016] According to one embodiment of the present invention, the local processing result includes lightweight semantic annotations of environmental targets corresponding to the first target region and the second target region; the cloud processing result includes deep knowledge annotations of supplementary environmental targets corresponding to the third target region or the fourth target region; and the augmented reality interactive output includes aligning and aggregating the lightweight semantic annotations and the deep knowledge annotations in the spatiotemporal dimension.

[0017] According to another aspect of the present invention, a wearable device is provided, comprising: a first camera and a second camera having the same optical parameters and set at a fixed baseline distance, for synchronously acquiring first image information and second image information; a communication module for data interaction with a cloud server, including sending a third target region or a fourth target region to the cloud server in response to a trigger condition, wherein the third target region is a portion of the first plurality of location regions that does not overlap with the first target region, and the fourth target region is a portion of the second plurality of location regions that does not overlap with the second target region; an output module for presenting information to a user; a memory storing a computer program, the computer program including a locally deployed first artificial intelligence model; and a processor, wherein the processor, when executing the computer program, implements the method described in any of the preceding embodiments.

[0018] According to one embodiment of the present invention, the output module includes a display or a speaker.

[0019] According to another aspect of the present invention, an augmented reality interaction system is provided, characterized in that it includes: the aforementioned wearable device; and a cloud server, the cloud server being configured to: receive a third target region or a fourth target region from the wearable device, wherein the third target region is a first image information portion unused in the local processing of the wearable device, and the fourth target region is a second image information portion unused in the local processing of the wearable device; select a corresponding second artificial intelligence model according to a model identifier; perform deep analysis on the third target region or the fourth target region based on the second artificial intelligence model to generate a cloud processing result; and return the cloud processing result to the wearable device.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described in any of the preceding claims.

[0021] This invention divides binocular images into corresponding regions and rapidly acquires depth / motion information locally based on the disparity / pixel difference between the first and second target regions, achieving basic environmental perception without relying on complex AI models. Only when preset trigger conditions are met are unused third or fourth target regions (complementary regions) sent to the cloud for depth analysis. This significantly improves response speed and semantic understanding accuracy in complex environments while maximizing the conservation of wearable device computing resources, power, and uplink bandwidth. The temporal aggregation of local lightweight annotations and cloud-based deep knowledge annotations provides users with a real-time, rich, and low-latency augmented reality interactive experience. Attached Figure Description

[0022] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts wherein:

[0023] Figure 1 This is a flowchart of an augmented reality interaction method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a wearable device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a wearable device as a headphone in one embodiment of the present invention; Figure 4 This is a schematic diagram of the wearable device as an in-ear headphone in one embodiment of the present invention; Figure 5 This is a schematic diagram of the wearable device as smart glasses in one embodiment of the present invention; Figure 6 This is a schematic diagram illustrating an application scenario for building a three-dimensional point cloud map according to one embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this invention are used to distinguish different objects, not to describe a specific order. "First," "second," "third," and "fourth," etc., also do not necessarily refer to only one, but may refer to multiple. The terms "comprising" and "including" used in the specification and claims of this invention indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0026] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0027] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0028] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] This embodiment provides an augmented reality interaction method applied to wearable devices. For example... Figure 1 As shown, the method includes the following steps: Step S101: Synchronously acquire the first image information and the second image information.

[0030] Data is captured synchronously via a first and second camera on a wearable device. Both cameras have identical optical parameters (focal length, field of view, aperture, etc.) and are positioned on the device at a fixed baseline distance (e.g., ...). Figure 5 The left and right sides of the eyeglass frame shown, or as Figure 3 (The front part of the earphone body is shown). Through pre-calibrated corrections, the positions of the two parts in the relative coordinate system are determined, providing a foundation for subsequent binocular vision algorithms.

[0031] Step S102: Generate local processing results based on binocular vision algorithm.

[0032] First, the first image information and the second image information are respectively divided into a first plurality of location regions and a second plurality of location regions (e.g., according to the same grid division rule) to ensure that the regions in the two images have a corresponding relationship.

[0033] The first multiple location regions refer to multiple location regions formed after the first image information is divided. The first is used to identify its correspondence with the first image information, and the multiple represents the quantity characteristics of the regions. Similarly, the second multiple location regions refer to multiple location regions formed after the second image information is divided.

[0034] Then, based on the degree of matching between at least one attribute of texture, brightness, and color of the first plurality of location regions and the second plurality of location regions and a preset template (such as road surface, vehicle, or building texture), the first target region and the second target region are marked respectively, wherein the first target region and the second target region correspond to the same environmental target.

[0035] By utilizing binocular vision algorithms (such as semi-global matching, image patch matching, etc.), the depth information (i.e., distance to the user) or motion information of environmental targets is calculated based on the disparity or pixel difference between the first target region and the second target region. In this invention, this depth information or motion information is the first output information. This process does not require calling complex AI models, has low computational cost, and fast response. Specifically, by matching corresponding feature points in the first target region and the second target region, the disparity value is calculated, and combined with the camera focal length and baseline length, the depth value of each image patch in the scene can be obtained.

[0036] Furthermore, the first AI model deployed on wearable devices (such as an on-device offline large model, SLM, instruction fine-tuning model, etc.) can be used to perform preliminary classification of these target regions and generate lightweight semantic annotations. For example, the local AI model can classify image blocks into "restaurants", "vehicles", "sidewalks", etc.

[0037] According to one embodiment of the present invention, the locally deployed AI model includes an input layer that receives and initially processes sensor data, an encoding layer that transforms the input data into feature vectors and extracts several key features therefrom, a series of stacked transformer layers that assign weights to the feature vectors and perform reasoning and understanding, and an output layer. The large model stored locally in memory is preferably trained and deployed offline, and accelerated through methods such as customized inference engines and model accuracy control, making it more suitable for scenarios with limited hardware performance and battery life.

[0038] Step S103: Determine whether the preset triggering conditions are met.

[0039] Triggering conditions include, but are not limited to, one or a combination of the following: Determine whether the distance between the environmental target and the wearable device is less than a first threshold (e.g., when a user approaches a building within 50 meters). Determine whether the confidence level of the local processing result is lower than the second threshold (e.g., the local AI model cannot determine the object category, confidence level < 0.6). Determine whether the user has received a command to pay attention to the target in the first environment (such as the user's voice "What restaurant is that?" or by eye tracking or gesture pointing to the target).

[0040] The first threshold corresponds to the judgment of depth information and can be set according to different scenarios, such as 10 meters or 100 meters; the second threshold and the user attention command correspond to the judgment of the local processing result and can also be set according to different scenarios, such as 0.6 or 0.7.

[0041] According to one embodiment of the present invention, the first threshold can be dynamically adjusted according to the current network status (e.g., the first threshold can be increased when the 5G signal is strong) or the remaining power of the device (the first threshold can be reduced when the power is low to save communication energy consumption).

[0042] Step S104: In response to the fulfillment of the triggering condition, send partial image information to the cloud server.

[0043] Specifically, a third target region that does not overlap with the first target region is extracted from the first plurality of location regions, or a fourth target region that does not overlap with the second target region is extracted from the second plurality of location regions. The third and fourth target regions include environmental areas that the wearable device has not analyzed in detail during local processing, and are used by the cloud server to identify supplementary environmental targets.

[0044] Instead of sending the entire image, the third or fourth target region is sent to the cloud server as raw image blocks, compressed image blocks, or feature vectors. Simultaneously, a model identifier corresponding to the AI ​​model to be invoked in the cloud can be sent. This model identifier instructs the cloud server to select a target AI model from multiple candidate models that matches the current environmental scenario (e.g., dining, fitness, or meeting). By sending the third or fourth target region instead of all image information, uplink bandwidth and cloud computing resources can be significantly saved, while avoiding redundant calculations with locally processed content.

[0045] Step S105: Receive the cloud processing result from the cloud server.

[0046] The cloud server selects a corresponding second artificial intelligence model (with more parameters and higher inference accuracy) based on the received model identifier, and performs in-depth analysis on the third or fourth target region to identify supplementary environmental targets. For example, it analyzes restaurant opening hours, user reviews, recommended dishes, and discount information; or it analyzes gym equipment usage methods and training plans. The cloud processing results are returned to the wearable device via the network.

[0047] According to one embodiment of the present invention, a wearable device controls a first artificial intelligence model and a second artificial intelligence model to run in parallel: the wearable device calculates depth or motion information in real time using a binocular vision algorithm, and continuously runs the first artificial intelligence model to process subsequent frame images in real time and generate semantic annotations, which together constitute the local processing result; in response to the fulfillment of the trigger condition, the wearable device initiates a communication session with the cloud server, and the cloud-based second artificial intelligence model begins to process the third target region or the fourth target region. At this time, the first artificial intelligence model and the second artificial intelligence model enter a parallel running state. The results of the two models are superimposed and output in time, avoiding delays caused by serial waiting.

[0048] Step S106: Generate augmented reality interactive output based on local processing results and cloud processing results.

[0049] In this embodiment, the user wears the wearable device of the present invention (taking smart glasses as an example, e.g.) Figure 5 (As shown) Walking through an unfamiliar city street. The smart glasses are equipped with two front-facing cameras, located on the left and right sides of the frame, respectively. They have the same focal length (e.g., 35mm equivalent focal length), field of view (e.g., 120 degrees diagonally), and aperture (e.g., f / 2.0). The baseline distance between the two cameras is fixed at 65mm to simulate the binocular parallax of the human eye.

[0050] After the user activates navigation mode, the first and second cameras synchronously acquire first and second image information at a rate of 30 frames per second. In step S102, the region segmentation module divides each frame of image into a first plurality of location regions and a second plurality of location regions, corresponding to... Figure 6 The established street view cloud map. This embodiment adopts an adaptive grid partitioning algorithm: first, edge detection is performed on the image, and a 32×32 pixel fine grid is used in texture-rich areas such as building facades and street signs, while a 128×128 pixel coarse grid is used in areas with simple textures such as the sky and flat road surfaces, thereby reducing the amount of computation while ensuring the resolution of key areas.

[0051] The target region labeling module matches each location region against a pre-set template library. This library includes texture features, color histograms, and shape descriptors for categories such as restaurant signs, pedestrian crossings, traffic lights, and building entrances. The system calculates the similarity score between each grid region and each type of template. When a region's similarity to the "restaurant sign" template exceeds a third threshold (e.g., set to 0.75), it is labeled as a candidate region. Figure 6 The restaurant 210 in the first image is the first target area, and its corresponding matching area in the second image is the second target area.

[0052] Subsequently, the binocular vision computing module performs stereo matching on the candidate region of restaurant 210 in the first image and the corresponding region in the second image. Using the semi-global block matching (SGBM) algorithm, it searches for the best matching block in the corresponding row of the second image, centering on the candidate region. It calculates the pixel horizontal displacement (parallax d) between the first and second target regions, and combines this with the camera focal length f and baseline distance B to calculate the depth value Z using the formula Z=(f×B) / d. For example, when f=800 pixels, B=65mm, and d=50 pixels, the depth Z=1040mm (approximately 1.04 meters). For distant targets around 50 meters, the parallax is only a few pixels; the system uses a sub-pixel level matching algorithm to improve accuracy.

[0053] After the above processing, the local system completes the construction of the street view cloud map and classifies the image blocks into: restaurants 210, sidewalks 220, vehicles 230, other buildings 240, and roads 250, and quickly provides users with feedback on the location, distance and basic type of the restaurants 210 in front of them, meeting the real-time information needs in the tourism dining scenario.

[0054] In step S103, the system determines whether the target distance meets the corresponding target threshold. In this embodiment, the threshold is dynamically adjusted according to the network status: the device detects that the current 5G network signal is good (RSRP > -85dBm), and sets the first threshold to 100 meters. Since the restaurant 210 is only 50 meters away from the user, which is less than the first threshold, the triggering condition is met.

[0055] In step S104, the system extracts local data to be uploaded to the cloud: The restaurant 210, used for local depth calculation and recognition, is designated as the first target area; other surrounding buildings 240 associated with the first target area but not analyzed locally are extracted from the first plurality of location areas as the third target area; pedestrian walkways 220 and vehicles 230 are only provided with safety warnings by the local AI; roads 250 are not sent to the cloud due to their low relevance to dining needs. The system sends image blocks of the restaurant 210 and other buildings 240 to the cloud server. Simultaneously, the image blocks can be encoded locally and include model identifiers to instruct the cloud to select the most suitable large model to accelerate processing.

[0056] The sending distance range is set in conjunction with network status and cloud processing speed to ensure that the cloud results are loaded when the user approaches the restaurant 210; if the user is outside the range, communication with the cloud will not be established, thus saving device resources and power.

[0057] In step S105, after receiving the image block, the cloud server performs in-depth analysis of the restaurant 210's business scope, historical evaluations, and discount activities through a remote large model, and determines whether there are other dining options in the surrounding area based on other buildings 240, generating high-precision cloud processing results.

[0058] In step S106, the system aggregates the local processing results and the cloud processing results to generate augmented reality interactive output: Local AI output: "Restaurant 210 is 100 meters ahead, proceed straight ahead. Be aware of oncoming vehicle 230 and traffic signals." The cloud-based AI outputs: "You can choose a currently discounted set meal; if you are not interested in restaurant 210 and there are no other restaurants in the straight direction, it is recommended to turn left."

[0059] The system uses a miniature LED display on smart glasses to overlay AR tags in the user's field of vision, highlighting restaurant 210 and displaying its distance, while simultaneously playing voice prompts through bone conduction speakers.

[0060] Furthermore, the system employs a parallel operation strategy of a local first AI model and a cloud-based second AI model: the local model continuously processes subsequent frames, updating the relative position of the user and restaurant 210 in real time; the cloud model continues to monitor changes in the surrounding area after completing its analysis. When the user approaches the entrance of restaurant 210 (within 5 meters), the local model triggers another cloud query to retrieve the restaurant's daily special menu, waiting information, and other details, aggregating them with the previous cloud results in a time-series fashion to provide the user with continuous, real-time augmented reality interactive services.

[0061] Example 2: Indoor Gym Training Instruction Scenario In this embodiment, the user wears the wearable device of the present invention (taking a headset as an example, e.g.) Figure 3 (As shown) Entering the gym for strength training. The headset has a front-facing camera on each side with a baseline distance of 80mm, suitable for high-precision depth sensing at close to medium distances (0.5-10 meters).

[0062] After the user begins training, the device simultaneously acquires first and second image information. Due to the complex lighting conditions in the gym (shadows caused by overhead lights and interference from specular reflections), the region segmentation module adopts adaptive segmentation based on image content: first, illumination normalization processing (histogram equalization) is performed, and then a superpixel segmentation algorithm (such as SLIC, simple linear iterative clustering) is used to divide the image into irregular regions with similar colors and textures (first multiple location regions and second multiple location regions), which is more adaptable to the irregular shapes of the equipment than a fixed grid.

[0063] The target region marking module identifies multiple candidate regions: bench press area, dumbbell rack area, barbell area, and specular reflection area (interference). Through preset template matching, the first target region (the bench press in the first image) and the second target region (the corresponding bench press in the second image) are marked. The binocular vision calculation module calculates the parallax, determining that the bench press is approximately 1.2 meters away from the user.

[0064] The locally deployed first AI model (lightweight pose estimation model) analyzes the first and second target areas and identifies that another user is already using the bench press bench (local processing result: someone is in this area, confidence level 0.88). The system determines that the confidence level is higher than the second threshold (0.6), and temporarily does not trigger a cloud query, but only provides a voice prompt: "The bench press bench is currently occupied. We suggest waiting or choosing other equipment."

[0065] The user turns to the dumbbell rack, and the local system, using binocular matching, identifies a distance of 1.5 meters to the rack. The posture estimation model then identifies an empty space on the rack (confidence 0.92). At this point, the user issues a voice command: "See what else is in this area," indicating a focus on the surrounding environment (additional environmental targets). The trigger condition is met.

[0066] The system extracts the fourth target region: excluding the already identified dumbbell rack area (second target region) from the second multiple location regions, the wall area behind the dumbbell rack and the floor mat area to the right are selected as the fourth target region. These regions are encoded as feature vectors, accompanied by the model identifier "gym_equipment_v3".

[0067] The cloud server uses a second artificial intelligence model to analyze the fourth target area, identifying a training plan and muscle anatomy diagram on the wall, and the floor mat area as a stretching area. The cloud-based fitness knowledge base queries the system and returns: "There is a weekly press training plan on the wall behind you, recommended 4 sets x 12 reps; the right side is the stretching area, and it is recommended to stretch the pectoralis major muscles after training."

[0068] The system aggregates output: local results provide information on available dumbbell racks, while cloud-based results provide training resources and suggestions within the environment. The system announces via headphones: "There are available dumbbell racks; we recommend using 15kg dumbbells. Today's training plan is on the wall behind you; we recommend 4 sets of 12 reps. Please relax in the stretching area on your right after your workout." Simultaneously, since the headphones lack a display screen, the system provides directional guidance via voice: "The training plan poster is 2 meters directly behind you, at a height of 1.6 meters."

[0069] During training, the local model continuously monitors the user's movements using binocular vision: the first and second cameras capture stereo images of the user lifting the dumbbells, and the lifting height and speed are calculated based on the parallax changes in the first target area (left arm) and the second target area (right arm). When the system detects that the movement speed is too fast (possibly due to inertia), it immediately provides voice correction: "Please slow down and control the eccentric phase." This real-time feedback is processed entirely locally with a latency of less than 50ms and requires no network transmission. Only when the user asks, "Is my movement correct?" is cloud analysis triggered, uploading a fourth target area (containing the user's full-body posture) for professional evaluation.

[0070] Example 3: Intelligent Assistance Scenarios for Business Meetings In this embodiment, the user wears the wearable device (in the form of smart glasses) of the present invention to participate in a business meeting. The meeting room environment is complex, with multiple participants, and the background includes a projection screen, whiteboard, windows, etc.

[0071] After the device starts the meeting mode, the first and second cameras capture images at a frame rate of 15fps (the frame rate is reduced to save power because the meeting scene changes slowly). The region segmentation module divides the first and second image information into a first plurality of location regions and a second plurality of location regions, respectively, and uses face detection for pre-screening: firstly, a local lightweight face detector (such as a lightweight version of RetinaFace) is used to quickly locate face regions, and these regions are designated as the high-priority first plurality of location regions and second plurality of location regions.

[0072] The target region labeling module extracts features from the detected face regions, generating a 128-dimensional face feature vector. Since the lighting in the conference room may be uneven (brighter on one side near the window, darker on the other), the binocular vision computing module not only calculates depth but also performs illumination compensation: by comparing the brightness histograms of the first and second target regions, it estimates the direction of ambient light to assist in subsequent face recognition.

[0073] As an attendee approaches a user, the local system uses binocular parallax to calculate the distance, gradually decreasing from 3 meters to 1 meter. The first AI model (a local facial recognition model containing 5,000 internal employee features) attempts to identify the attendee, but with a confidence level of only 0.45 (below the second threshold of 0.6), it cannot confirm the identity. The trigger condition is met.

[0074] The system extracts the third target region: excluding the face region (first target region) from the first plurality of location regions, and selecting the shoulder region surrounding the face, the conference table region in the background, and the projection screen region as the third target region. These regions are sent to the cloud as raw image patches (uncompressed to ensure text recognition accuracy), along with the model identifier "meeting_assist_v1".

[0075] The cloud server loads a second artificial intelligence model to perform multi-dimensional analysis on the third target area: it identifies the employee badge in the shoulder area (although it was not recognized locally due to the angle, the high-resolution model in the cloud recognized the blurry text), it identifies the nameplate "Manager Zhang" in front of the person in the conference table area, and it identifies the title of the current presentation "Q3 Marketing Strategy" in the projection screen area. Combining this with the company database, it confirms that the person is Manager Zhang from the Marketing Department, who sent an email about the strategy to the user last week.

[0076] The cloud-based system returned a detailed, informative comment: "Zhang XX, Marketing Director, we sent you a draft of the Q3 marketing strategy last week, and we are currently presenting the relevant plan. We suggest you include your feedback on the draft."

[0077] The system generates aggregated output: local results provide basic information "someone is approaching," while cloud results provide identity and context. The AR glasses overlay a label on the person's head in the field of vision: "Manager Zhang, Marketing Director, email sender from last week," and whisper a prompt in the user's ear: "Manager Zhang is approaching. Remember to mention your feedback on the Q3 strategy draft."

[0078] During the meeting, the local model continuously monitors the first and second target areas of all participants, constructing a heatmap of the meeting room's personnel locations. When a change in content (page turning) is detected in the projection screen area (third target area), cloud-based OCR recognition is triggered to extract key points and generate a meeting minutes summary locally. This local monitoring + cloud supplementation model ensures both real-time performance (local tracking of personnel locations) and provides in-depth information (cloud-based recognition of screen content).

[0079] Example 4: Outdoor Running Safety Navigation Scenario In this embodiment, the user wears the wearable device of the present invention (in the form of a headset, such as...). Figure 3 (As shown) Running in a city park. This scenario has extremely high real-time requirements; safety-related decisions must be made within 100ms.

[0080] The device acquires first and second image information at a high frame rate of 60fps to ensure the capture of fast-moving vehicles and pedestrians. The region segmentation module employs epipolar constraint optimization: since the binocular cameras are fixed, corresponding points in the first and second images lie on the same horizontal line (epochal line). Therefore, during region segmentation, strips are divided horizontally (first multiple location regions and second multiple location regions), reducing the search range during stereo matching and improving processing speed.

[0081] The target region marking module employs motion saliency detection: it compares the optical flow changes of a first set of multiple location regions and a second set of multiple location regions across consecutive frames to mark regions whose motion speed differs from the background. For example, if a rapidly approaching region (possibly a bicycle or electric vehicle) is detected on the right, it is marked as the first target region and the second target region.

[0082] The binocular vision computing module quickly calculates the depth and relative velocity of the moving area: it calculates the radial velocity by measuring the parallax changes over multiple consecutive frames (e.g., 3 frames, 16.7ms interval). If an area is 5 meters away and approaches at a speed of 5 m / s, it is expected to reach the user's position in 1 second.

[0083] In step S103, the system determines that the distance (5 meters) is less than the first safety threshold (10 meters) and the relative speed is dangerous, and immediately triggers a local alarm (without waiting for the cloud), playing a high-frequency prompt tone through the headphones: "Watch out for vehicles coming from the right, keep to the left!" This is purely local processing with a delay of less than 30ms.

[0084] Simultaneously, the system extracts a fourth target area: excluding the processed right-side oncoming traffic area (second target area) from the second multiple location areas, and selecting the area of ​​the running track ahead (potentially containing water or obstacles) and the green belt area on the left as the fourth target area. Since network instability may occur during running (weak signal in some areas of the park), the system adjusts its transmission strategy based on network status (4G signal detected, moderate strength): reducing image resolution and sending only key feature points instead of complete image patches.

[0085] The cloud server analyzed the fourth target area and identified water accumulation on the track 20 meters ahead, and a hidden path leading to the restroom in the green belt on the left. The cloud server then reported: "There is water accumulation on the left side of the track 20 meters ahead, it is recommended to stay to the right; there is a restroom entrance in the green belt on your left, please turn left if you need to use it."

[0086] The system aggregates outputs: local real-time safety warnings (vehicle approaching from the right) and cloud-based environmental information (road conditions ahead, facility locations). Since the visual focus is on the distance when running, the headphones use spatial audio technology to simulate sound direction: "Water ahead is on your left" (sound comes from the left ear), "Restaurant entrance is 2 meters to your left" (sound comes from directly left), helping users perceive the spatial layout without looking at the device.

[0087] Furthermore, the system utilizes parallel processing to optimize the user experience: the local model continuously monitors all moving objects (continuous tracking of the first and second target areas), while the cloud model analyzes the static environment (the fourth target area). When a user runs out of the current area, the local model detects new potential hazards (such as a group of people walking dogs ahead) and immediately updates the local processing results, while the cloud simultaneously updates its analysis of the area ahead, with the two seamlessly connected in terms of timing.

[0088] Example 5: Smart Supermarket Shopping Assistance Scenarios In this embodiment, the user wears the wearable device of the present invention (in-ear headphone form, such as...). Figure 4 As shown, a device equipped with a miniature camera is used for shopping in a supermarket. This design offers good concealment and does not interfere with the user's normal product selection.

[0089] When a user picks up an item (such as a box of cereal), the first and second cameras capture first and second image information from a downward angle. Due to the special viewing angle (short-angle) and the potential for reflection from the product packaging (plastic packaging), the region segmentation module employs depth-based adaptive segmentation: first, an initial depth map is obtained through fast stereo matching (block matching algorithm), and then region segmentation is performed based on depth discontinuities (product edges) to separate the product from the background shelf.

[0090] The target area marking module identifies the first target area and the second target area (product packaging area). The first artificial intelligence model (lightweight product recognition model) identifies this as "XX brand oatmeal" with a confidence level of 0.85, which is higher than the second threshold (e.g., 0.6). The local processing result is generated as "XX high fiber oatmeal, 500g pack".

[0091] If a user gazes at the product for more than 2 seconds (detected via eye tracking or head posture estimation), a query for supplementary information is triggered. The system extracts a third target area: excluding the product area from the first multiple location areas, it selects the shelf area behind the product (displaying other products on the same shelf) and the price tag area below the product as the third target area.

[0092] The cloud server analyzes the third target area: identifying other flavors (honey and fruit) of the same brand on the shelf and comparing their prices; identifying the current price on the price tag using OCR and querying the historical price database. The cloud returns: "The same honey flavor is available on the same shelf below, 5 yuan more expensive but with a higher rating; the current price is 25 yuan, a 10% decrease from last week, the lowest price ever; you purchased this brand of nuts three months ago, and there is a discount when purchasing together."

[0093] The system announced through the earpiece: "This is XX high-fiber oatmeal, currently priced at 25 yuan (a 10% discount). The same brand's honey flavor is on the shelf below, 5 yuan more expensive but with better reviews. Considering you have purchased the same brand's nuts, you can get a 10 yuan discount when you buy them together."

[0094] Meanwhile, the local system continues to monitor: when a user puts down the oatmeal and picks up the honey-flavored oatmeal next to it, the local model updates the first and second target areas to identify the new product, and the cloud synchronously updates the analysis of the third target area, providing a detailed comparison of the product's nutritional components (cloud knowledge base query).

[0095] This scenario demonstrates the advantages of local-cloud collaboration: local rapid identification of basic product categories (avoiding uploading all shelf images, protecting privacy and saving bandwidth), and cloud analysis of shelf context (requiring a broader view and knowledge base, suitable for cloud processing). The selection of the third target area ensures that only necessary background information is transmitted, rather than the user's entire body or other personal items in the shopping cart.

[0096] Example 6: Industrial Equipment Inspection Scenario In this embodiment, technicians wear the wearable device of the present invention (in the form of smart glasses, equipped with a protective cover) to conduct equipment inspections in the factory workshop. The environment is noisy and the lighting is complex (there are areas with strong light for welding and areas with shadows).

[0097] The equipment acquires first and second image information. The region segmentation module uses multi-scale segmentation: fine segmentation (16x16 pixel grid) is performed on areas where equipment or instruments may exist, while coarse segmentation (256x256 pixels) is performed on distant backgrounds. The target region marking module identifies the first and second target regions (pressure gauge dial area) through template matching.

[0098] The binocular vision computing module calculates that the pressure gauge is about 0.8 meters away from the technician (a suitable close distance for reading). The first artificial intelligence model (lightweight OCR model) attempts to read the dial value, but due to the tilted angle (the technician is observing from the side), the local recognition confidence is only 0.5 (below the second threshold of 0.7), and it cannot read the value accurately.

[0099] Once the trigger condition is met, the system extracts the fourth target region: excluding the pressure gauge region from the second set of multiple location regions, it selects the pipe region surrounding the pressure gauge, the valve region, and the control cabinet region in the background as the fourth target region. These regions are sent as high-resolution image patches (high precision is required for industrial inspection), accompanied by the model identifier "industrial_inspection_v4".

[0100] The cloud server uses a highly complex model to analyze the fourth target area: in the pipe area, it identifies slight rust and color anomalies (which may be early signs of leakage); in the valve area, it identifies the valve handle position (which deviates from the standard position); and in the control cabinet area, it identifies a flashing alarm light (the flashing frequency may not have been captured locally due to frame rate issues).

[0101] The cloud-based in-depth analysis returned the following: "The pressure gauge reading is blurry; it is recommended to check from the front. There is early corrosion on the right-side pipeline; maintenance is recommended. The valve is not fully closed. The control cabinet has intermittent alarms, fault code E-302, which may be due to sensor drift."

[0102] The system aggregates outputs: basic local positioning (pressure gauge location) and comprehensive cloud-based analysis (equipment status, maintenance recommendations). AR glasses overlay annotations in the field of view: pressure gauges are marked with a yellow box indicating "reading from the front," right-side pipes are marked with a red box indicating "rust warning," valves are marked with an orange box indicating "not fully closed," and control cabinets are marked with a flashing red icon indicating "fault E-302."

[0103] Simultaneously, the system records the analysis results of all first and third / fourth target areas and generates inspection reports. The local model continuously tracks the movement of the technician's gaze, and when the gaze shifts to the next device, the first and second target areas are immediately updated. The new fourth target area is processed in parallel in the cloud, achieving a seamless inspection process.

[0104] This scenario fully leverages the combination of local low latency (real-time tracking of technicians' line of sight) and cloud-based powerful analysis capabilities (defect detection, fault diagnosis). The selection of the third / fourth target area ensures a comprehensive understanding of the environmental context, rather than isolated device identification.

[0105] Example 7: Guided tour of museum artifacts In this embodiment, visitors wear the wearable device of this invention (in the form of smart glasses, with a lightweight design) while touring the museum. The ambient lighting is relatively dim (to protect the cultural relics), and a high degree of quietness is required (loud audio cannot be played).

[0106] The device acquires first and second image information. Due to the low lighting, the region segmentation module uses brightness adaptive enhancement (local histogram equalization) before segmentation. The target region marking module identifies the first target region and the second target region (a bronze artifact exhibit).

[0107] The binocular vision computing module calculates the distance to the exhibit to be approximately 1.5 meters. The first artificial intelligence model identifies "bronze ware, Han Dynasty" (based on shape features) with a confidence level of 0.75, which is higher than the second threshold (e.g., 0.6) but lower than the fourth threshold (e.g., 0.85). The system determines that more detailed information needs to be supplemented from the cloud, and the triggering condition is met.

[0108] The system extracts the third target area: Excluding the exhibit itself from the first multiple location areas (to protect the privacy of cultural relics, no high-resolution images of the relics are uploaded, in accordance with museum regulations), the area of ​​the explanatory plaque below the exhibit, the reflective area of ​​the display case glass (which may show the crowding of visitors), and the background exhibition hall area are selected as the third target area. The explanatory plaque area is sent at high resolution (requiring OCR text recognition), while other areas are sent at medium resolution.

[0109] The cloud server analyzes the third target area: in the explanatory plaque area, it identifies the exhibit name "Western Han Dynasty Gilt Bronze Boshan Incense Burner" and extracts the explanatory text; in the exhibition hall area, it identifies the current visitor density (by counting people in the background); and in conjunction with the museum's knowledge base, it obtains the historical background, production process, and location of similar exhibits of the artifact.

[0110] Cloud feed: "This is a Western Han Dynasty gilt bronze Boshan censer, used for incense burning, unearthed from Maoling in 1981. Craftsmanship features: gilt inlay decoration, and a mountain-shaped lid symbolizing a mythical island in the sea. Similar exhibits are on display in Hall 3 on the second floor. The current visitor flow in the hall is moderate; we recommend spending 5 minutes to view it in detail."

[0111] The system generates aggregated output: local identification provides instant categorization (bronze artifacts), while the cloud provides detailed knowledge. Information cards are overlaid next to exhibits using AR glasses, displaying the name, era, and brief description, and a whispered explanation is provided via bone conduction headphones (audible only to the wearer): "What you see before you is a Western Han Dynasty gilt bronze Boshan incense burner..." Because explanatory plaques are uploaded rather than the artifacts themselves, both the digital copyright of the artifacts is protected, and rich information is provided.

[0112] In addition, when the local binocular vision detects that the user's head posture has been focused on a certain detail (a sub-region of the first target area) for a long time, it can trigger local magnification analysis, and the cloud provides professional interpretation of the detail (such as the specific techniques of gold inlay), so as to realize detailed guidance.

[0113] Example 8: Agricultural Planting Monitoring Scenario In this embodiment, farmers wear the wearable device of this invention (headphone-like, dustproof and waterproof) to inspect crop growth inside the greenhouse. The ambient light varies greatly (alternating between shadows and direct sunlight inside the greenhouse), and the crops are densely packed and have similar shapes.

[0114] The device acquires first and second image information. The region segmentation module uses a combination of color and depth for segmentation: first, regions are initially screened based on vegetation color (dominantly green), and then depth information is combined to distinguish foreground and background crops. The target region marking module identifies the first target region and the second target region (a tomato plant).

[0115] The binocular vision computing module calculates that the plant is about 0.6 meters away. The first artificial intelligence model (lightweight plant health recognition model) analyzes the leaf color and initially judges that "there may be mild nutrient deficiency symptoms, confidence level 0.65", which is lower than the fifth threshold (e.g., set to 0.8), triggering in-depth analysis in the cloud.

[0116] The system extracts the third target region: Excluding the main plant from the first multiple location regions, it selects the soil area surrounding the plant, the area of ​​adjacent plants, and the greenhouse roof area (showing light conditions) as the third target region. These regions are encoded as feature vectors and sent along with the model identifier "agriculture_diag_v2".

[0117] The cloud server used a high-precision agricultural model to analyze the third target area: identifying surface compaction and uneven moisture in the soil area; identifying that the plant was shorter than its neighbors in the adjacent plant area; and analyzing light intensity and uniformity in the top area. Combining the meteorological database and crop growth model, the cloud diagnosed: "This plant is deficient in nitrogen because the soil on the left side is compacted, hindering root absorption. It is recommended to loosen the soil on the left side and apply nitrogen fertilizer. The current sunlight in the greenhouse is too strong; it is recommended to turn on the shade net."

[0118] The system announced via earpiece: "This tomato plant may be lacking nitrogen. The soil on the left side looks a bit compacted. I suggest loosening the soil and applying fertilizer. The light inside the greenhouse is too strong right now, remember to turn on the shade net."

[0119] The local model continues to monitor: After farmers follow the recommendations, the local system re-collects data from the first and second target areas, analyzes leaf color changes, verifies the treatment effect, and continuously monitors environmental changes (climate data from the third target area) in the cloud, providing dynamic management suggestions.

[0120] This scenario demonstrates the synergy between rapid local screening (identifying suspicious plants) and professional cloud-based diagnosis (combining soil, climate, and neighboring plants for comprehensive judgment) in an agricultural environment. The selection of the third target area ensures a comprehensive understanding of the crop's growth environment, rather than isolated plant observation.

[0121] Example 9: Emergency Rescue Scene Assistance Scenarios In this embodiment, rescuers wear the wearable device (rugged smart glasses) of this invention to enter the post-earthquake rubble for search and rescue. The environment is extremely dangerous, with dim lighting and limited communication bandwidth (emergency communication network).

[0122] The device acquires first and second image information. The region segmentation module uses infrared assistance (if equipped) and visible light fusion to segment regions even in dark environments. The target region marking module identifies the first target region and the second target region (the space suspected to be where the trapped person is located).

[0123] The binocular vision computing module calculated the space to be approximately 2 meters deep, but the entrance was narrow. The first artificial intelligence model analyzed the structural stability with a confidence level of 0.55 (it's uncertain whether it's safe), triggering cloud-based structural analysis.

[0124] The system extracts the fourth target region: excluding the entrance region from the second multiple location regions, it selects the surrounding stacked building ruins, load-bearing structure areas, and ground crack areas as the fourth target region. Due to bandwidth limitations, these regions are compressed at a high ratio (JPEG quality factor 60) and key structural features are extracted before being sent, along with the model identifier "rescue_struct_v1".

[0125] The cloud server used a structural engineering model to analyze the fourth target area: it identified a secondary collapse risk level (medium), suggested support point locations, and detected a hidden load-bearing column on the right side (not identified locally). The cloud returned: "The entrance is relatively stable, but there is a risk of sliding in the ruins on the right side. It is recommended to support the area before entering; the internal space is 2.3 meters deep, and there may be trapped individuals; a load-bearing column was detected on the right side, which can be used as an anchor point."

[0126] The system displays a structural risk heat map through AR glasses (red indicates high-risk areas, and green indicates relatively stable areas) and provides voice guidance: "The entrance is temporarily safe, but there is a risk of landslide on the right. Secure it with support poles first. There is a 2-meter-deep space inside, and there may be people there. There is a pillar on the right where you can tie a safety rope."

[0127] Local model real-time monitoring: When rescuers enter, the first and second cameras continuously monitor minute changes in the first and second target areas (entrance structure) (visual deformation detection). Once the deformation exceeds the third threshold (local fast calculation), an alarm is immediately triggered locally (no cloud required): "Structural change, evacuate immediately!" At the same time, the cloud continuously analyzes environmental changes in the fourth target area (such as new collapse risks caused by aftershocks) and provides dynamic safety assessment.

[0128] This scenario fully leverages the advantages of local ultra-low latency (security alerts) and cloud-based professional analysis (structural assessment). The selection of the fourth target area helps the cloud understand the overall environment and provide safe rescue route suggestions.

[0129] Example 10: Smart Home Interaction Scenarios In this embodiment, the user wears the wearable device (in-ear headphone form) of the present invention while moving around at home. The environment is familiar but contains numerous devices (smart home appliances, furniture), and the lighting changes (day / night).

[0130] The device acquires first and second image information, and the region segmentation module uses object-level segmentation (instance-based segmentation). The target region marking module identifies the first target region and the second target region (smart air conditioner panel).

[0131] The binocular vision computing module calculates the panel at a distance of about 3 meters. The first artificial intelligence model recognizes "air conditioner, current set temperature 24 degrees" with a confidence level of 0.9, which is higher than the second threshold, and generates a local processing result.

[0132] The user's voice command, "See what else can be adjusted in this room," triggers a query for other devices in the room. The system extracts a third target area: excluding the air conditioner panel from the first multiple location areas, it selects the curtain area on the left side of the room, the lighting area on the right side, and the air purifier area in the background as the third target area.

[0133] The cloud server analyzes the third target area: it identifies the curtains as smart curtains (electrically controllable), the lights as dimmable smart lights, and the air purifier as a networked device with a current Air Quality Index (AQI) of 85. The cloud server queries user habits (historical data) and returns: "The left-side curtains can be closed to block the afternoon sun, the right-side lights can be adjusted to reading mode, and the air purifier is recommended to be set to high power mode (current AQI 85, this is the value you usually use for high power mode)."

[0134] The system announced via earpiece: "The air conditioner is currently set to 24 degrees Celsius. You can draw the curtains on the left to block the sun, turn up the lights on the right, and turn the air purifier to high speed (the air quality index is currently 85)."

[0135] The user says, "Then draw the curtains." The local system uses binocular vision to confirm the curtain's location (updating the first target area to the curtain), calculates the distance, and as the user approaches, the local infrared transmitter (if equipped) or Bluetooth controls the curtains to close. The entire process is primarily handled locally, with the cloud providing environmental device discovery and suggestions.

[0136] like Figure 2 As shown, the wearable device 100 includes: The first camera 110 and the second camera 120 employ a synchronized triggering mechanism to ensure that the exposure start time difference is less than 1ms, avoiding asynchrony between the two cameras due to motion. The sensor uses a stacked CMOS sensor, supports HDR (High Dynamic Range) mode, and adapts to different indoor and outdoor lighting conditions. The optical lens uses a six-element plastic lens to effectively eliminate distortion and ensure geometric consistency between the first and second images.

[0137] Region segmentation module, target region marking module, and binocular vision computing module: These modules are integrated into the SoC (System-on-a-Chip) as hardware accelerators or implemented as firmware of the DSP (Digital Signal Processor). The binocular vision computing module supports FP16 (half-precision floating-point) operations, improving computing speed while reducing power consumption.

[0138] Memory 130: Includes LPDDR5 memory (for running the first AI model) and UFS flash memory (for storing programs, model weights, and temporary data). The first AI model is stored in quantized form (INT8 precision) to reduce memory usage.

[0139] Controller 140: It adopts a heterogeneous multi-core architecture, including a high-performance core (processing complex logic), an energy-efficient core (continuously running basic perception), and a dedicated NPU (neural network processing unit to accelerate AI inference).

[0140] Transceiver 160: Supports Wi-Fi 6E, Bluetooth 5.3, and 5G NR. It features intelligent bandwidth management, automatically selecting transmission strategies based on network quality: sending raw image blocks when 5G network is strong, and sending highly compressed feature vectors when Wi-Fi signal is weak.

[0141] Local processing employs Dynamic Voltage Frequency Scaling (DVFS), reducing camera frame rate and processor frequency when the scene is static, and boosting to full performance when fast motion is detected. Cloud communication uses a batch transmission strategy, accumulating multiple third / fourth target areas before transmitting them all at once, reducing the number of RF module start-ups and shutdowns.

[0142] When network latency exceeds 500ms or packet loss rate is higher than 10%, the system switches to offline mode, relying solely on local processing. Simultaneously, it caches the third / fourth target regions, uploading them in batches once the network recovers. The local model automatically adjusts the second threshold, lowering the trigger conditions and providing more local functionality.

[0143] The selection of the third / fourth target area inherently protects privacy by excluding the first / second target areas (which could be faces or sensitive text) that users are primarily concerned with. Furthermore, all uploaded data is end-to-end encrypted, and the original data is deleted immediately after cloud processing, retaining only the aggregated results.

[0144] When the cloud-returned result severely conflicts with the local result (e.g., the local system identifies it as "car" while the cloud system identifies it as "building"), the system initiates a consistency check: resends the overlapping area for confirmation, or performs a spatiotemporal consistency check (based on the continuity of the previous frame). If the issue cannot be resolved, the local result (due to its low latency) is reported to the user first, and the cloud result is marked as pending verification.

[0145] To verify the technical effectiveness of this invention, we conducted tests on standard datasets and in real-world scenarios: Response latency comparison: The traditional all-cloud solution (uploading a full frame of 1080p image) has an average latency of 850ms (5G network), while the solution of this invention has an average latency of 120ms (80ms for local processing + 40ms for cloud transmission), which is a reduction of 85.9%.

[0146] Bandwidth consumption comparison: In a continuous 30-minute navigation task, the traditional solution consumes 1.2GB of uplink traffic, while the solution of this invention (only sending to the third / fourth target area) consumes 85MB, saving 92.9% of the traffic.

[0147] Accuracy comparison: In complex scenarios (occlusion, changes in lighting), the accuracy of the pure local solution is 72%, the pure cloud solution is 94%, and the solution of this invention (local + cloud aggregation) is 91%, which is close to the level of the pure cloud solution, but the response speed is significantly improved.

[0148] Battery life comparison: In continuous AR mode, the traditional local real-time AI solution lasts for 2.5 hours, while the solution of this invention lasts for 6.8 hours (because most of the calculations are lightweight binocular vision, and the cloud is only called when necessary).

[0149] This invention achieves efficient augmented reality interaction for wearable devices in resource-constrained environments through an innovative binocular vision region segmentation and complement transmission mechanism. The core of this mechanism lies in: local processing of the first and second target regions (binocular matching regions) to quickly acquire depth and basic semantics; and cloud processing of the third and fourth target regions (complementary regions) to provide depth knowledge and environmental context. This division of labor ensures both real-time performance (low latency locally) and intelligence (powerful cloud capabilities), while significantly saving bandwidth and energy consumption, making it suitable for a wide range of scenarios from daily life to professional industries.

[0150] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An augmented reality interaction method applied to wearable devices, characterized in that, include: First image information and second image information are simultaneously acquired through the first camera and second camera on the wearable device; The first image information and the second image information are respectively divided into a first plurality of location regions and a second plurality of location regions; Based on the degree of matching between at least one attribute of texture, brightness, and color of the first plurality of location regions and the second plurality of location regions and the preset template, the first target region and the second target region are marked respectively; Based on the disparity or pixel difference between the first target region and the second target region, the depth information or motion information of the environmental target is calculated as the first output information, and the local processing result includes the first output information. Determine whether the depth information or the local processing result meets the preset triggering conditions; In response to the fulfillment of the triggering condition, a third target region that does not overlap with the first target region is extracted from the first plurality of location regions, or a fourth target region that does not overlap with the second target region is extracted from the second plurality of location regions, and the third target region or the fourth target region is sent to the cloud server. Receive cloud processing results from the cloud server; and, Based on the local processing results and the cloud processing results, an augmented reality interactive output is generated.

2. The method according to claim 1, characterized in that, The first and second cameras have the same optical parameters and are positioned on the wearable device at a fixed baseline distance.

3. The method according to claim 1, characterized in that, The division into a first plurality of location regions and a second plurality of location regions includes: The first image information is divided into the first plurality of location regions according to a preset division rule, and the second image information is divided into the second plurality of location regions according to the preset division rule; The preset partitioning rules include grid partitioning, quadtree partitioning, or adaptive partitioning based on image content.

4. The method according to claim 3, characterized in that, The step of marking the first target region and the second target region respectively also includes: Based on the locally deployed first artificial intelligence model, feature extraction and matching are performed on the first plurality of location regions and the second plurality of location regions respectively to determine the correspondence between the first target region and the second target region; The first target region and the second target region correspond to the same environmental target in the first image information and the second image information.

5. The method according to claim 1, characterized in that, The local processing results include depth or motion information calculated by a binocular vision algorithm, and semantic annotations generated by a first artificial intelligence model deployed on the wearable device; the cloud processing results are generated by a second artificial intelligence model deployed on the cloud server; and the second artificial intelligence model has a higher number of parameters or higher inference accuracy than the first artificial intelligence model.

6. The method according to claim 5, characterized in that, Determining whether a preset trigger condition is met includes at least one of the following: Determine whether the distance between the environmental target and the wearable device is less than a first threshold; Determine whether the confidence level of the semantic annotations generated by the first artificial intelligence model deployed on the wearable device in the local processing result is lower than a second threshold; Determine whether a user's attention instruction for the first environmental target has been received.

7. The method according to claim 6, characterized in that, The first threshold is dynamically adjusted based on the current network status or the remaining battery power of the device.

8. The method according to claim 1, characterized in that, The third target region includes all or part of the regions in the first plurality of location regions except the first target region, and the fourth target region includes all or part of the regions in the second plurality of location regions except the second target region; The third target area or the fourth target area is used by the cloud server to identify supplementary environmental targets in the surrounding environment of the wearable device.

9. The method according to claim 8, characterized in that, The third or fourth target region is sent to the cloud server in the form of an original image block, a compressed image block, or a feature vector extracted from an image block.

10. The method according to claim 1, characterized in that, Before sending the third target area or the fourth target area, the method further includes: Encode the image data of the third target region or the fourth target region into a feature vector; and, Generate a model identifier corresponding to a second artificial intelligence model to be invoked in the cloud, wherein the model identifier is used to instruct the cloud server to select a target artificial intelligence model that matches the current environmental scenario.

11. The method according to claim 5, characterized in that, The wearable device calculates depth or motion information in real time using a binocular vision algorithm and continuously runs a first artificial intelligence model deployed on the wearable device to generate semantic annotations in real time, which together constitute the local processing result. In response to the fulfillment of the triggering condition, the wearable device initiates a communication session with the cloud server and controls the first artificial intelligence model and the second artificial intelligence model to enter a parallel running state, wherein the first artificial intelligence model continues to process subsequent frame images to generate subsequent semantic annotations, and the second artificial intelligence model processes the third target region or the fourth target region; Furthermore, based on the processing results of the first artificial intelligence model on subsequent frames and the cloud processing results returned by the cloud server, a time-series aggregated augmented reality interactive output is generated.

12. The method according to claim 1, characterized in that, The local processing results include lightweight semantic annotations for the environmental targets corresponding to the first target region and the second target region; the cloud processing results include deep knowledge annotations for the supplementary environmental targets corresponding to the third target region or the fourth target region; and the augmented reality interactive output includes aligning and aggregating the lightweight semantic annotations and the deep knowledge annotations in the spatiotemporal dimension.

13. A wearable device, characterized in that, include: The first camera and the second camera have the same optical parameters and are set at a fixed baseline distance to synchronously acquire first image information and second image information; A communication module is used to interact with a cloud server, including sending a third target area or a fourth target area to the cloud server in response to a trigger condition, wherein the third target area is a portion of the first plurality of location areas that does not overlap with the first target area, and the fourth target area is a portion of the second plurality of location areas that does not overlap with the second target area; The output module is used to present information to the user; A memory storing a computer program, the computer program including a locally deployed first artificial intelligence model; and, A processor that, when executing the computer program, implements the method as described in any one of claims 1 to 12.

14. The wearable device according to claim 13, characterized in that, The output module includes a display or a speaker.

15. An augmented reality interactive system, characterized in that, include: The wearable device as described in claim 13 or 14; as well as, The cloud server is configured as follows: Receive a third target region or a fourth target region from the wearable device, wherein the third target region is a portion of first image information that is not used in the local processing of the wearable device, and the fourth target region is a portion of second image information that is not used in the local processing of the wearable device; Select the corresponding second artificial intelligence model based on the model identifier, perform in-depth analysis on the third or fourth target region based on the second artificial intelligence model, and generate cloud processing results; and The cloud processing results are then returned to the wearable device.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 12.

Images