Object identification and explanation method and system based on intelligent wearable device

Abstract

The embodiment of the invention provides an object identification and explanation method based on intelligent wearable equipment, and the method comprises the steps: constructing a knowledge base of all space objects in a target region, and distributing a content identifier for each space object; constructing a point cloud map of the target area, dividing the point cloud map into a plurality of local point cloud maps, calibrating the position and range of each space object in each local point cloud map, and distributing a space identifier for each space object; establishing a binding relationship between the content identifier and the space identifier; positioning information and a real-time image are obtained through the intelligent wearable device, and a corresponding local point cloud map is determined according to the positioning information; performing feature matching on the real-time image and the local point cloud map to obtain a pose of the intelligent wearable device in the local point cloud map; determining a target object based on the pose; and querying a corresponding content identifier according to the space identifier of the target object, calling the commentary content and the three-dimensional model associated with the content identifier from the knowledge base, and outputting the commentary content and the three-dimensional model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of augmented reality technology, in particular to an object recognition and explanation method and system based on a smart wearable device. BACKGROUND

[0002] With the continuous progress of augmented reality (AR) technology and wearable computing devices, smart travel guide systems based on smart wearable devices (such as AR glasses, smart helmets, etc.) are gradually replacing traditional tour guide interpretation and handheld voice guides. Such systems perceive the spatial position of the user in real time, retrieve associated information from a pre-constructed multimedia database, and present it in a virtual-real integrated manner in the user's field of view, providing an immersive guide experience.

[0003] However, in actual deployment, to achieve natural and smooth interaction of "what you see is what you get", the existing technical solutions are difficult to achieve an effective balance between spatial perception accuracy, system real-time performance and device power consumption, and there are significant technical bottlenecks: In museum, art gallery, historical site and other scenarios, exhibits or scenic spots are extremely dense. To achieve accurate content triggering and superimposition, the system needs to have centimeter-level or even higher accuracy positioning to accurately distinguish whether the user is looking at the "left sculpture" or the "right mural". Currently, existing positioning and recognition solutions are difficult to meet this accuracy requirement in complex indoor and outdoor environments. While methods that rely on pre-deployed two-dimensional codes or specific markers (Markers) can improve accuracy, they disrupt the original look of the environment, have high deployment and maintenance costs, and are limited in applicable scenarios.

[0004] To improve recognition accuracy, the system needs to run continuously at high load to maintain high-frequency image full-matching or complex deep learning inference. This not only quickly depletes the battery power of the device, but also causes the device to heat up, resulting in a decrease in frame rate and an increase in rendering delay. The lack of accuracy and real-time performance will ultimately directly result in poor user experience.

[0005] Therefore, there is an urgent need for an information recognition and explanation method and system to optimize spatial perception accuracy, system response delay and terminal power consumption in large-scale, complex and diverse real scenarios. SUMMARY

[0006] The present application aims to overcome the defects of existing guide technology based on smart wearable devices in terms of positioning accuracy, real-time performance, deployment and maintenance costs, and provides an object recognition and explanation method and system based on a smart wearable device to simultaneously achieve high-precision positioning, low-latency response and low-power operation, thereby being applicable to large-scale, complex and diverse real scenarios and improving user experience.

[0007] In a first aspect, the embodiments of the present application provide a method for object recognition and explanation based on a smart wearable device, comprising: constructing a knowledge base of all spatial objects in a target area, assigning a content identifier to each spatial object, the knowledge base comprising explanation content and a three-dimensional model associated with the content identifier; constructing a point cloud map of the target area, dividing the point cloud map into a plurality of local point cloud maps, calibrating the position and range of each spatial object in each local point cloud map, and assigning a spatial identifier to each spatial object; the spatial identifier contains the position coordinates and boundary information of the spatial object; establishing a binding relationship between the content identifier and the spatial identifier; acquiring positioning information and real-time images through the smart wearable device, determining the corresponding local point cloud map according to the positioning information; performing feature matching between the real-time images and the local point cloud map to obtain the pose of the smart wearable device in the local point cloud map; determining a target object based on the pose; the target object is a spatial object that needs to be recognized and explained; querying the corresponding content identifier according to the spatial identifier of the target object, retrieving the explanation content and the three-dimensional model associated with the content identifier from the knowledge base, and outputting through the smart wearable device.

[0008] In some embodiments, determining the target object based on the pose comprises: determining a detection ray emitted from the smart wearable device based on the pose; judging whether the detection ray intersects with the spatial identifier of a spatial object in the local point cloud map; if the intersection lasts for a preset threshold, the spatial object is determined as the target object.

[0009] In some embodiments, judging whether the detection ray intersects with the spatial identifier of a spatial object in the local point cloud map further comprises a priority determination step: if the detection ray intersects with the spatial identifier of at least two spatial objects, a target object is determined from the at least two spatial objects according to a predetermined priority rule; wherein the priority rule is configured to calculate a priority score for each intersecting spatial object, and select the spatial object with the highest score as the target object; the priority score is calculated by quantifying and weighting at least one of the following factors: the projection area of the spatial object in the current field of view of the smart wearable device, the larger the projection area, the higher the score; the spatial straight-line distance between the spatial object and the smart wearable device, the closer the distance, the higher the score; The center position of the spatial object is offset from the center of the current field of view of the smart wearable device. The smaller the offset, the higher the score.

[0010] In some embodiments, the point cloud map is divided into a plurality of local point cloud maps according to the following division methods: According to the division of the geofence, the division of the building boundary, or the division of the grid with a predetermined size.

[0011] In some embodiments, the real-time image is feature-matched with the local point cloud map to obtain the pose of the smart wearable device in the local point cloud map, specifically including: extracting image feature points of the real-time image; matching the image feature points with map feature points in the local point cloud map; based on the successfully matched feature point pairs, calculating the pose of the smart wearable device in the local point cloud map through a pose solving algorithm, the pose including a three-dimensional position coordinate and a three-dimensional rotation angle.

[0012] In some embodiments, the explanation content and the three-dimensional model are output by the smart wearable device, specifically including: playing an explanation audio corresponding to the target object, and processing the explanation audio through spatial audio technology to associate the sound image direction of the explanation audio with the real spatial position of the target object; superimposing and displaying the three-dimensional model in the display interface of the smart wearable device, and the display position of the three-dimensional model corresponds to the real spatial position of the target object.

[0013] In some embodiments, the point cloud map of the target region is constructed, including: scanning the target region through a simultaneous localization and mapping technology, extracting spatial feature points, and generating a point cloud map based on the spatial feature points.

[0014] In some embodiments, the positioning information is obtained by the smart wearable device, specifically including: The smart wearable device obtains the positioning information through at least one of GPS, wireless network, or Bluetooth.

[0015] In a second aspect, the embodiments of the present application provide a smart wearable device for implementing the above-mentioned object recognition and explanation method based on a smart wearable device, including: an image sensor for collecting a real-time image; a positioning module for obtaining positioning information through at least one of GPS, wireless network, or Bluetooth; a communication module for receiving the positioning information and the local point cloud map, and communicating with a knowledge base server to obtain explanation content and a three-dimensional model; a processor configured to perform the following operations: The corresponding local point cloud map is determined based on the positioning information; the real-time image is matched with the local point cloud map to obtain the pose of the smart wearable device in the local point cloud map. Based on the pose, the target object is determined; the target object is the spatial object that needs to be identified and explained; the corresponding content identifier is queried according to the spatial identifier of the target object, and the explanatory content and 3D model associated with the content identifier are retrieved from the knowledge base; The display module is used to output the narration content and 3D models in an augmented reality format.

[0016] Thirdly, embodiments of this application provide a map-based object recognition and explanation system, including: The knowledge base server is used to build a knowledge base for all spatial objects within the target area. Each spatial object is assigned a content identifier. The knowledge base includes explanatory content and 3D models associated with the content identifier. The map building and subdivision server is used to build a point cloud map of the target area and divide the point cloud map into multiple local point cloud maps. In each local point cloud map, the location and extent of each spatial object are marked, and a spatial identifier is assigned to each spatial object. The spatial identifier contains the location coordinates and boundary information of the spatial object. Among them, the knowledge base server communicates with the map construction and partitioning server to establish the binding relationship between content identifiers and spatial identifiers; Based on the aforementioned smart wearable devices; The map building and segmentation server is also used to respond to requests from smart wearable devices and provide local point cloud maps corresponding to the location information; the knowledge base server is used to respond to requests from smart wearable devices and provide explanatory content and 3D models corresponding to the target object.

[0017] This application reduces the search range for feature matching by dividing the local point cloud map, effectively lowering computational complexity and latency, thus enabling real-time pose estimation and target recognition with immediacy and smoothness on resource-constrained wearable devices. By assigning spatial identifiers to spatial objects marked in the local point cloud map and combining this with geometric intersection judgment based on detection rays generated from device pose, centimeter-level spatial relationship determination is achieved, significantly improving recognition accuracy and enhancing the interactive experience. Furthermore, by introducing priority rules, false triggers can be reduced, further improving recognition accuracy and user experience. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an object recognition and explanation method based on a smart wearable device, as described in an embodiment of this application.

[0019] Figure 2 This is a flowchart illustrating the operation of the smart glasses according to an embodiment of this application. Detailed Implementation

[0020] The following specific embodiments illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Although the description of this application will be presented in conjunction with some embodiments, this does not mean that the features of this application are limited to this embodiment. On the contrary, the purpose of describing the application in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of this application. To provide a thorough understanding of this application, many specific details will be included in the following description. This application may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0021] It should be noted that in this specification, similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0022] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0023] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0024] In the description of this application, it should be understood that "electrical connection" can be understood as physical contact and electrical conduction between components; it can also be understood as the form of connection between different components in a circuit structure through physical lines that can transmit electrical signals, such as copper foil or wires on a printed circuit board (PCB). "Coupled through..." can be understood as electrical conduction through indirect coupling. Indirect coupling can be understood as contactless coupling. Those skilled in the art will understand that coupling refers to the phenomenon where there is a close cooperation and mutual influence between the inputs and outputs of two or more circuit elements or electrical networks, and energy is transferred from one side to the other through interaction. To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, in a first aspect, embodiments of this application provide a method for object recognition and explanation based on a smart wearable device, which includes: Step S1: Construct a knowledge base for all spatial objects within the target area, assign a content identifier to each spatial object, and the knowledge base includes explanatory content and 3D models associated with the content identifier. Construct a point cloud map of the target area and divide the point cloud map into multiple local point cloud maps. In each local point cloud map, mark the position and extent of each spatial object and assign a spatial identifier to each spatial object. The spatial identifier contains the position coordinates and boundary information of the spatial object. Establish the binding relationship between content identifiers and spatial identifiers; Step S2: Obtain positioning information and real-time images through the smart wearable device, determine the corresponding local point cloud map based on the positioning information; perform feature matching between the real-time image and the local point cloud map to obtain the pose of the smart wearable device in the local point cloud map. Step S3: Based on the pose, determine the target object; the target object is the spatial object that needs to be identified and explained. Step S4: Query the corresponding content identifier based on the spatial identifier of the target object, retrieve the explanatory content and 3D model associated with the content identifier from the knowledge base, and output them through the smart wearable device.

[0026] Specifically, in step S1, the target area can be a museum exhibition hall, a historical building complex, a city park, an exhibition hall, or other indoor or outdoor spaces that require intelligent explanations for visitors. First, all spatial objects within the target area undergo systematic information organization and digitization to form a complete knowledge base. A spatial object refers to a specific entity within the target area that has independent explanatory significance, such as exhibits, sculptures, paintings, and artifact display cases in a museum, or ancient trees, sculptures, and monuments in a park. Each spatial object is assigned a unique content identifier, which typically uses numbers, letters, or a combination of both, such as "OBJ-001" or "EXHIBIT-A01," facilitating rapid subsequent retrieval.

[0027] The knowledge base construction process includes collecting multimedia explanatory content related to each spatial object. This content includes text descriptions, audio explanations, video clips, and detailed 3D models. The 3D models are reconstructed by using high-precision 3D scanning equipment to perform a comprehensive scan of the spatial object, fully showcasing its shape, texture, and details.

[0028] In one implementation, the 3D model is stored in a mesh model format and associated with corresponding material maps for presenting a realistic 3D visualization effect on smart wearable devices. Narrative content is stored in a knowledge base server, with each piece of narration uniquely associated with a content identifier, and the narration content is configured to provide accurate and educational information related to spatial objects.

[0029] Simultaneously, the scenic area is scanned using SLAM (Simultaneous Localization and Mapping) equipment to extract spatial feature points, generating a point cloud map of the target area. This map contains a large number of 3D feature points acquired through dense camera scanning. This 3D feature point data comprehensively records the environmental structure of the area, including the ground, walls, and pillars, as well as the surface geometry of various spatial objects. To reduce computational resource consumption and avoid efficiency degradation caused by loading the complete point cloud map, the point cloud map is divided into multiple local point cloud maps according to preset rules.

[0030] In this embodiment of the application, the point cloud map is divided into multiple local point cloud maps, and the division is based on the following criteria: division based on geofences, division based on building boundaries, or division based on grids of predetermined sizes.

[0031] In one implementation, local point cloud maps are divided based on geofencing. Specifically, the overall boundary of the target area is first determined, and then the area is divided into several sub-areas according to functional zoning, natural dividing lines, or visitor flow paths. For example, a museum exhibition hall is used as a local point cloud map, and a theme garden in a park is used as a local point cloud map. The size of each local point cloud map is controlled within a reasonable range to ensure that the smart wearable device can complete loading and processing in a short time.

[0032] In each local point cloud map, the location and extent of spatial objects need to be accurately calibrated. The calibration process involves identifying a subset of the point cloud data for each spatial object and determining its bounding box or bounding volume. Typically, 3D bounding boxes, axis-aligned bounding boxes, or oriented bounding boxes are used to represent the extent of spatial objects. Spatial identifiers consist of location coordinates and boundary information; for example, the location coordinates are the 3D coordinates of the object's center point, and the boundary information includes the length, width, height, and orientation angle of the bounding box. Calibration is completed through a combination of manual annotation and semi-automatic bounding box selection, ensuring calibration accuracy down to the centimeter level.

[0033] Finally, a binding relationship is established between content identifiers and spatial identifiers. This binding relationship is typically stored in the form of a database table, with one record corresponding to one spatial object, containing both the content identifier and the spatial identifier. Through this binding, once a spatial object is identified, the corresponding explanatory content and 3D model can be directly retrieved. This dual-identifier binding mechanism keeps the knowledge base consistent with the map data; even if the exhibit's location is subsequently adjusted, only the binding relationship needs to be updated, avoiding the need to repeatedly rebuild the knowledge base.

[0034] like Figure 2 As shown, in step S2, the smart wearable device can be smart glasses, a smart helmet, or other wearable devices with display, camera, and computing capabilities. In one embodiment, the smart wearable device has a built-in global navigation satellite system receiver module, an inertial measurement unit, and a wireless communication module for acquiring real-time positioning information. The positioning information includes the device's geographic coordinates, altitude, and approximate orientation.

[0035] Once the user enters the target area, the smart wearable device first obtains a rough location via the Global Navigation Satellite System, or, in an indoor environment, combines Wi-Fi fingerprint, Bluetooth beacon, or UWBS positioning technology to obtain a more precise initial location. Real-time images are continuously captured by the device's front-facing camera, maintaining a frame rate of over 30 frames per second to ensure a smooth visual experience.

[0036] Based on the acquired location information, the smart wearable device sends a request to the map building and subdivision server. The server determines the user's current sub-region based on the location information and returns the corresponding local point cloud map. The local point cloud map only contains the environmental point cloud of that sub-region and the position coordinates and boundary information of the labeled spatial objects. This on-demand loading method greatly reduces data transmission volume and device memory usage, effectively reducing computational complexity and latency, enabling rapid map acquisition even in environments with limited network conditions.

[0037] It should be noted that when the positioning information is not accurate enough, multiple candidate local point cloud maps that are closest to the current location can be loaded first, and then the most accurate one can be identified through preliminary matching of image features, thereby improving the robustness of the system.

[0038] By performing feature matching between real-time images and local point cloud maps, the pose of the smart wearable device within the local point cloud map is obtained. Pose refers to the three-dimensional position coordinates and three-dimensional rotation angle of the smart wearable device in the local point cloud map coordinate system, determining the user's current precise viewpoint and position. Feature matching between real-time images and local point cloud maps is a core step in achieving high-precision positioning.

[0039] Specifically, image feature points are first extracted from real-time images. Then, these extracted feature points are matched against pre-stored map feature points in a local point cloud map. The local point cloud map, during its construction phase, has already extracted and stored stable 3D feature points in the environment, such as salient structures like corners, pillar edges, and exhibit outlines. Initial matching is completed through nearest neighbor search and ratio testing, eliminating incorrect matches.

[0040] Based on successfully matched feature point pairs, the precise pose of the smart wearable device is calculated using a pose calculation algorithm. Commonly used pose calculation algorithms include the perspective n-point algorithm combined with random sampling consistency for optimization, resulting in a pose matrix containing three translational components and three rotational components. This algorithm can still converge to the correct pose even with some mismatches.

[0041] For example, when a user stands in the center of an exhibition hall facing a large mural, real-time images capture features such as the mural's edges, surrounding pillars, and floor textures. These features are then matched with corresponding points in a local point cloud map, allowing for the precise calculation of the user's exact distance to the mural and their line of sight. This high-precision pose acquisition provides a reliable foundation for subsequent target object identification, effectively avoiding recognition errors caused by positioning drift.

[0042] Furthermore, inter-frame tracking techniques can be used to continuously track feature points between consecutive frames, and pose fusion optimization can be performed by combining data from the inertial measurement unit, making pose estimation smoother and more stable. Even when the user turns their head quickly or experiences brief occlusion, the system can maintain continuous positioning capabilities.

[0043] Step S3: Determine the target object based on pose. After obtaining the precise pose of the smart wearable device in the local point cloud map, the next step is to determine the specific spatial object that the user is currently intending to focus on or is observing, i.e., the target object. This step is crucial for achieving accurate interpretation, and its core lies in associating the user's real-time gaze or device orientation with known spatial objects in the local point cloud map.

[0044] Specifically, the process of determining the target object based on pose can be broken down into the following sub-steps.

[0045] Step S31: Determine the detection ray emitted from the smart wearable device based on the pose. The detection ray is a virtual ray that originates from the optical center of the device's camera and extends along the device's current orientation. In three-dimensional space, the detection ray represents the direction of the user's gaze. Its starting point is determined by the three-dimensional position coordinates in the pose, and its direction is determined by the three-dimensional rotation angle in the pose, represented as a forward vector determined by the rotation matrix from the device coordinate system to the world coordinate system. Optionally, the direction of the detection ray can also be aligned based on the user's gaze point direction determined by eye tracking.

[0046] Step S32: Determine whether the detection ray intersects with the spatial identifier of a spatial object within the local point cloud map. If they intersect and the duration of the intersection reaches a preset threshold, the intersecting spatial object is identified as the target object. The spatial identifier contains the position coordinates and boundary information of the spatial object and exists in the form of a 3D bounding box. If the detection ray passes through the 3D bounding box, it is considered to intersect with the spatial object.

[0047] In real-world scenarios, a user's gaze may quickly sweep across multiple objects or linger briefly on a particular object. To avoid triggering incorrect interpretations due to fleeting glances, a duration threshold mechanism is introduced. The system continuously tracks and times the overlap between the ray and the bounding box of each spatial object. Only when the duration of the intersection between the ray and the bounding box of a spatial object exceeds a preset threshold, such as 1.5 or 2 seconds, is that object ultimately identified as the target object. This mechanism simulates the "gaze" behavior of humans when observing objects, effectively filtering out unintentional gaze movements and improving recognition accuracy and the naturalness of the user experience.

[0048] In one implementation, the preset threshold can be dynamically configured and adaptively adjusted according to the characteristics of the actual application scenario to achieve an optimal balance between response sensitivity and interaction stability. For example, in indoor venues with dense exhibits (such as museums), to support users in quickly and continuously browsing different exhibits, the system uses a relatively short trigger threshold (e.g., 1 to 3 seconds) to provide a more agile interactive response. Conversely, in outdoor scenes with open views and large object spacing (such as parks and squares), to effectively avoid accidental triggering caused by distant objects entering the field of view, the system will automatically use a longer trigger threshold (e.g., 3 to 6 seconds), thereby improving the accuracy of target recognition and the clarity of the interaction intent. This dynamic adjustment mechanism can be automatically triggered based on scene point cloud density, spatial object distribution characteristics, or pre-set scene geofencing information.

[0049] Step S32 further includes determining whether the detection ray intersects with the spatial identifiers of spatial objects within the local point cloud map, and also includes a priority determination step: if the detection ray intersects with the spatial identifiers of at least two spatial objects, then one of the at least two spatial objects is determined as the target object according to a predetermined priority rule. Step S42 also involves a processing strategy when the detection ray intersects with the spatial identifiers of multiple spatial objects simultaneously, resolving the ambiguity problem of recognizing overlapping or adjacent objects.

[0050] The priority rule is configured as follows: calculate a priority score for each intersecting spatial object, and select the spatial object with the highest score as the target object; the priority score is obtained by quantifying and weighting at least one of the following factors: The projected area of ​​a spatial object in the current field of view of a smart wearable device; the larger the projected area, the higher the score. The straight-line distance between the spatial object and the smart wearable device; the closer the distance, the higher the score. The degree of offset of the center position of a spatial object relative to the current field of view of the smart wearable device; the smaller the offset, the higher the score.

[0051] The specific calculation process is as follows: First, for each intersecting spatial object, based on its bounding box size and current pose, it is projected onto the virtual imaging plane of the smart wearable device, and the proportion of pixel area it occupies in the current field of view is calculated, i.e., the projection area factor. Generally, the larger the area occupied in the field of view, the more likely the object is to be the center of the user's attention. Second, the Euclidean distance from the device position to the center point of the spatial object is calculated, i.e., the straight-line distance in space. The closer the object is, the greater its attractiveness. Finally, the pixel distance between the projection point of the spatial object's center point on the imaging plane and the center point of the screen is calculated, i.e., the degree of field of view center offset. The closer an object is to the center of the field of view, the more likely it is the object the user is looking at directly.

[0052] The three factors are normalized so that their values ​​range from 0 to 1. Then, a weight is assigned to each factor; for example, projected area is weighted at 0.5, distance at 0.3, and center offset at 0.2. The priority score is the weighted sum of all factors. After calculating scores for all intersecting spatial objects, the system selects the spatial object with the highest score as the final target object.

[0053] For example, in a museum setting, a user stands in front of a display case containing a bronze ding (tripod cauldron) and a jade bi (disc). The user's line of sight causes the detection ray to pass through the enclosure of both artifacts simultaneously. The system then calculates a priority score. Assuming the bronze ding is larger, occupying approximately 40% of the current field of view, while the jade bi occupies only 15%; the bronze ding is 3 meters away from the user, and the jade bi is 3.2 meters away; the center point projection of the bronze ding is slightly off-center from the screen by 50 pixels, while the jade bi's offset is 120 pixels. After weighted calculation, the bronze ding's priority score is significantly higher than the jade bi's. Therefore, the system identifies the bronze ding as the target object and prepares to retrieve its explanatory content. This decision-making mechanism, based on multi-factor quantification, can intelligently discern the user's true intentions and make a reasonable choice even when objects are closely arranged.

[0054] In another implementation, priority rules incorporate historical records or user preferences. For example, if a user has listened to an explanation of an exhibit within the past few minutes, the system appropriately lowers the exhibit's priority score, thereby reducing the probability of the user encountering the same content. Alternatively, objects marked as "key exhibits" are given a base bonus to ensure they are easier to identify.

[0055] Step S4: Query the corresponding content identifier based on the spatial identifier of the target object, retrieve the explanatory content and 3D model associated with the content identifier from the knowledge base, and output them through the smart wearable device.

[0056] Once the target object is identified, the system enters the content retrieval and presentation phase. First, based on the target object's spatial identifier, the system queries the previously established binding relationship database for its corresponding content identifier. After obtaining the content identifier, the smart wearable device sends a request to the knowledge base server via a wireless network, carrying the content identifier in the request.

[0057] Upon receiving a request, the knowledge base server retrieves all multimedia content associated with the content identifier from the storage system, primarily including narration and 3D models. The narration includes various formats, such as structured text descriptions, pre-recorded professional audio files, and relevant historical video clips. The 3D model files are stored in a common 3D file format.

[0058] This content data is packaged and transmitted over the network to the smart wearable device. Upon receiving the data, the device displays it through its display module. Steps S41 and S42 describe the specific output method in detail.

[0059] Step S41: Play the explanatory audio corresponding to the target object, and process the audio using spatial audio technology to associate the sound image location of the audio with the spatial position of the target object. When playing the explanatory audio, the display module of the smart wearable device applies a spatial audio algorithm to calculate and generate an audio signal with spatial orientation in real time based on the precise position of the target object in three-dimensional space.

[0060] Specifically, the system acquires the spatial coordinates of the target object and the user's real-time pose. By calculating the direction vector of the target object relative to the user's head, the theoretical azimuth and elevation angles of the sound source are determined. Then, the original mono or stereo audio is filtered using a head-related transfer function to generate a binaural audio signal. For example, when the user identifies a statue on the left, the narration comes from the left; if the user turns their head, the perceived direction of the sound changes accordingly, maintaining a spatial correspondence with the real world. This immersive audio experience greatly enhances the sense of presence and realism, making the narration and the observed object feel integrated, thus improving the user experience.

[0061] Step S42: The 3D model is overlaid and displayed on the display interface of the smart wearable device. The display position of the 3D model corresponds to the spatial position of the target object. The display interface is usually a semi-transparent or fully transparent display screen located in front of the user, rendering the 3D model in real time and overlaying it onto the real-world image seen by the user.

[0062] In one implementation, a complete restored model of a damaged cultural relic is overlaid and displayed; a cross-sectional view of its internal structure or a dynamic animation of its working principle is overlaid and displayed; and the evolution of the appearance of an ancient building in different historical periods is overlaid and displayed. The display position of the 3D model is perfectly aligned with the real object, remaining stable as the user moves without drifting or misalignment.

[0063] In one implementation, the output of the narration is coordinated. When the audio narration begins to play, relevant key text information or prompt icons are also displayed as floating labels next to the 3D model. Audio, 3D visualization, and text prompts together form a three-dimensional information presentation system, satisfying the user's cognitive needs at different levels.

[0064] Secondly, embodiments of this application provide a smart wearable device. This smart wearable device is the core hardware carrier for implementing the aforementioned method. Specifically, it includes an image sensor, a communication module, a processor, and a display module.

[0065] Image sensors are used to acquire real-time images. A high-resolution color camera, supplemented by a depth camera or infrared camera, is employed to obtain clear images and depth information under varying lighting conditions, providing a richer data source for feature matching.

[0066] The positioning module is used to obtain positioning information through at least one of GPS, wireless network or Bluetooth; the positioning module integrates multiple communication protocols, such as a global navigation satellite system receiver for obtaining satellite positioning signals, a wireless local area network module for connecting to the Internet and exchanging data with a server, and a Bluetooth module for connecting to indoor positioning beacons.

[0067] The communication module receives location information and a local point cloud map, and communicates with the knowledge base. Based on the location information, this module obtains the local point cloud map from the map building and subdivision server, and requests and receives explanatory content and 3D model data from the knowledge base server.

[0068] The processor is configured to perform a series of critical operations. These operations include: determining the corresponding local point cloud map based on the positioning information; performing feature matching between the real-time image and the local point cloud map to obtain the pose; determining the target object based on the pose; querying the corresponding content identifier based on the spatial identifier of the target object and retrieving the associated explanatory content and 3D model from the knowledge base. The processor can be a high-performance mobile system chip, integrating a central processing unit, a graphics processing unit, and a neural network processing unit. The graphics processing unit is responsible for real-time rendering of the 3D model and some spatial audio calculations, the neural network processing unit can accelerate deep learning tasks such as image feature extraction, and the central processing unit is responsible for overall process control and logical judgment.

[0069] The display module is used to output the narration content and 3D models in augmented reality.

[0070] In addition, smart wearable devices typically include an inertial measurement unit (IMU) to provide angular velocity and acceleration data, aiding in pose estimation and improving tracking stability when image features are temporarily missing. The device also includes essential components such as a battery, storage unit, and audio input / output unit.

[0071] Thirdly, embodiments of this application provide a map-based object recognition and explanation system. This system employs a cloud-based collaborative architecture, primarily comprising a knowledge base server, a map building and partitioning server, and the aforementioned smart wearable device.

[0072] The knowledge base server is used to build a knowledge base for spatial objects and assign content identifiers to each spatial object. It is a backend data management and service system that provides database management, content storage, retrieval, and streaming media services. Administrators can upload and manage narration content and 3D models through a dedicated interface, and maintain the binding relationship between content identifiers and spatial identifiers. When smart wearable devices request content, it can respond quickly and provide a data stream.

[0073] The map building and partitioning server is used to construct point cloud maps and partition them into multiple local point cloud maps. Within each local point cloud map, it identifies the location and extent of spatial objects, assigns spatial identifiers, and establishes binding relationships between content identifiers and spatial identifiers. This server is responsible for processing raw point cloud scan data and running offline computational tasks such as point cloud segmentation, object calibration, and map partitioning. It maintains complete map data for the entire target area as well as a database of partitioned local maps. Simultaneously, it stores and manages a binding relationship table between spatial identifiers and content identifiers. This table serves as the source or copy of the binding relationships in the knowledge base server, ensuring data consistency between the two ends.

[0074] The map building and subdivision server is used to provide local point cloud maps corresponding to location information in response to requests from smart wearable devices. When a smart wearable device sends a request containing coarse location information, the server queries its database based on the location information, finds one or more local point cloud map files covering that location, and sends them to the device.

[0075] The knowledge base server is used to respond to requests from smart wearable devices by providing explanatory content and 3D models corresponding to the target object. Once the device identifies the target object and retrieves the content identifier, it sends a request to the knowledge base server. The server then retrieves the corresponding audio, video, text, and 3D model files based on the content identifier and transmits them to the device over the network.

[0076] In the entire system's workflow, a large amount of offline computing and data storage is handled by cloud servers, while real-time image processing, pose calculation, rendering, and interaction are completed on local devices. This architecture ensures both the system's powerful functionality and scalability, as well as the real-time and smooth user experience. Even in areas with poor network signal, smart wearable devices can pre-cache local point cloud maps of the current area and explanations of popular objects, enabling offline explanation functionality.

[0077] In one implementation, some functions of the map building and partitioning server are integrated into edge computing nodes deployed locally within the venue, reducing reliance on cloud networks. Smart wearable devices communicate directly with edge nodes within the venue to obtain maps and content, achieving lower data transmission latency and higher system reliability, making it suitable for applications with high real-time requirements.

[0078] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for object recognition and explanation based on a smart wearable device, characterized in that, include: Construct a knowledge base for all spatial objects within the target area, and assign a content identifier to each spatial object. The knowledge base includes explanatory content and a 3D model associated with the content identifier. A point cloud map of the target area is constructed, and the point cloud map is divided into multiple local point cloud maps. The position and extent of each spatial object are marked in each local point cloud map, and a spatial identifier is assigned to each spatial object. The spatial identifier contains the position coordinates and boundary information of the spatial object. Establish the binding relationship between the content identifier and the spatial identifier; The smart wearable device acquires location information and real-time images, and determines the corresponding local point cloud map based on the location information; the real-time image is matched with the local point cloud map to obtain the pose of the smart wearable device in the local point cloud map. Based on the pose, the target object is determined; the target object is a spatial object that needs to be identified and explained. The corresponding content identifier is queried based on the spatial identifier of the target object, the explanatory content and 3D model associated with the content identifier are retrieved from the knowledge base, and output through the smart wearable device.

2. The method as described in claim 1, characterized in that, Determining the target object based on the pose includes: Based on the pose, the detection ray emitted from the smart wearable device is determined; Determine whether the detected ray intersects with the spatial identifier of a spatial object within the local point cloud map; If they intersect and the duration of the intersection reaches a preset threshold, then the spatial object is identified as the target object.

3. The method as described in claim 2, characterized in that, The step of determining whether the detected ray intersects with the spatial identifier of a spatial object within the local point cloud map also includes a priority determination step: If the detection ray intersects with the spatial identifiers of at least two spatial objects, then one of the at least two spatial objects is determined as the target object according to a predetermined priority rule; The priority rule is configured as follows: a priority score is calculated for each intersecting spatial object, and the spatial object with the highest score is selected as the target object; the priority score is obtained by quantifying and weighting at least one of the following factors: The spatial object occupies the projected area in the current field of view of the smart wearable device. The larger the projected area, the higher the score. The closer the spatial object is to the smart wearable device, the higher the score. The smaller the offset of the center position of the spatial object relative to the current field of view of the smart wearable device, the higher the score.

4. The method as described in claim 1, characterized in that, The criteria for dividing the point cloud map into multiple local point cloud maps include: Divide according to geofencing, building boundaries, or grids of predetermined size.

5. The method as described in claim 1, characterized in that, The step of performing feature matching between the real-time image and the local point cloud map to obtain the pose of the smart wearable device in the local point cloud map specifically includes: Extract image feature points from the real-time image; The image feature points are matched with map feature points in the local point cloud map; Based on the successfully matched feature point pairs, the pose of the smart wearable device on the local point cloud map is calculated using a pose calculation algorithm. The pose includes three-dimensional position coordinates and three-dimensional rotation angles.

6. The method as described in claim 1, characterized in that, The output of narration content and 3D model through the smart wearable device specifically includes: Play the explanatory audio corresponding to the target object, and process the explanatory audio using spatial audio technology to associate its sound image location with the actual spatial location of the target object; The three-dimensional model is superimposed on the display interface of the smart wearable device, and its display position corresponds to the actual spatial position of the target object.

7. The method as described in claim 1, characterized in that, The construction of the point cloud map of the target area includes: scanning the target area using synchronous positioning and mapping technology, extracting spatial feature points, and generating the point cloud map based on the spatial feature points.

8. The method as described in claim 1, characterized in that, The acquisition of location information through smart wearable devices specifically includes: The smart wearable device obtains location information through at least one of GPS, wireless network or Bluetooth.

9. A smart wearable device for implementing the method according to any one of claims 1 to 7, characterized in that, include: Image sensors are used to acquire real-time images; A positioning module is used to obtain positioning information via at least one of GPS, wireless network or Bluetooth; The communication module is used to receive positioning information and local point cloud maps, and communicate with the knowledge base service to obtain explanatory content and 3D models; The processor is configured to perform the following operations: The corresponding local point cloud map is determined based on the positioning information; the real-time image is matched with the local point cloud map to obtain the pose of the smart wearable device in the local point cloud map. Based on the pose, the target object is determined; the target object is a spatial object that needs to be identified and explained. Based on the spatial identifier of the target object, query the corresponding content identifier, and retrieve the explanatory content and 3D model associated with the content identifier from the knowledge base; The display module is used to output the narration content and 3D models in an augmented reality format.

10. A map-based object recognition and explanation system, characterized in that, include: A knowledge base server is used to build a knowledge base for all spatial objects within the target area, and assign a content identifier to each spatial object. The knowledge base includes explanatory content and a 3D model associated with the content identifier. A map building and partitioning server is used to build a point cloud map of the target area, and divide the point cloud map into multiple local point cloud maps. In each local point cloud map, the position and extent of each spatial object are marked, and a spatial identifier is assigned to each spatial object. The spatial identifier contains the position coordinates and boundary information of the spatial object. The knowledge base server is communicatively connected to the map construction and partitioning server, and is used to establish a binding relationship between the content identifier and the spatial identifier; The smart wearable device according to claim 9; in, The map building and segmentation server is also used to respond to the request from the smart wearable device by providing the local point cloud map corresponding to the positioning information; the knowledge base server is used to respond to the request from the smart wearable device by providing explanatory content and a 3D model corresponding to the target object.

Images