XR Content Providing System Using SLAM-Based Spatial Computing

The XR content delivery system uses SLAM-based spatial computing to integrate 3D spatial data and environment recognition for precise virtual object placement, addressing the limitations of conventional AR by ensuring alignment with real space and maintaining performance and immersion.

KR103045472B1Active Publication Date: 2026-11-10GLAM & 507 CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
KR1020250127980
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2025-09-09
Publication Date
2026-11-10
Estimated Expiration
2045-09-09

AI Technical Summary

Technical Problem

Conventional augmented reality technologies lack the capability to precisely recognize the shape and arrangement of objects in real space and calculate user location, resulting in poor consistency with real space and a diminished sense of immersion due to unnatural virtual object placement.

Method used

An XR content delivery system using SLAM-based spatial computing integrates 3D spatial data, environment recognition information, and user location information to accurately place virtual objects at specific points in real space, adjusting brightness, shadows, and detail levels to reflect real-world conditions and user movement.

Benefits of technology

The system provides immersive augmented reality content by aligning virtual objects with reality, maintaining performance and quality under dynamic conditions, ensuring precise placement and natural integration of virtual objects with real space.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112025103398825-PAT00022_ABST
    Figure 112025103398825-PAT00022_ABST
Patent Text Reader

Abstract

The XR content provision system using SLAM-based spatial computing according to the present invention is characterized by comprising: a spatial recognition module that recognizes an environment including the shape and arrangement of objects in a space where a user is located using simultaneous localization and mapping (SLAM) technology, generates environment recognition information, and calculates user location information based on the environment recognition information; a spatial data acquisition module that acquires three-dimensional spatial data of the space through a smartphone held by the user; a data management module including an integrated data management unit that generates and stores integrated data including the environment recognition information and user location information in the three-dimensional spatial data, and a virtual object data storage unit that stores a plurality of virtual object data including three-dimensional shape and attribute information; and a content provision module that generates extended reality (XR) content by placing virtual object data selected by the user in correspondence with the user's environment and location based on the integrated data, and outputs it at a specific point selected by the user in the space displayed on the screen of the user's smartphone. According to the XR content provision system using SLAM-based spatial computing according to the present invention, integrated data is generated by reflecting the structure of the real space and user location information, and based on this, virtual objects can be stably placed at a point designated by the user, thereby having the effect of providing XR content that is aligned with reality.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to an XR content delivery system using SLAM-based spatial computing, and more specifically, to an XR content delivery system that integrates 3D spatial data, environment recognition information, and user location information based on SLAM technology to accurately place a virtual object at a specific point in real space and naturally output it to a smartphone screen, thereby stably providing augmented reality content in which reality and virtuality are aligned. Background Technology

[0002] With the recent advancement of smartphone and camera technology, a variety of augmented reality services are being provided that capture real-world spaces and overlay simple virtual objects onto the user's screen.

[0003] For example, when a user activates the camera at a specific location, a method is commonly used where a pre-defined marker is recognized or a pre-prepared virtual object based on simple coordinates is displayed on the screen.

[0004] While such conventional technology is relatively simple to implement and can provide basic visual effects, it has limitations in that it cannot precisely recognize the shape or arrangement of objects in real space and cannot calculate the user's current location based on environmental information.

[0005] Furthermore, in this method, virtual objects are displayed only at arbitrary coordinates or simple marker locations on the screen, resulting in poor consistency with real space, which lowers the user's sense of immersion, and may cause problems where object placement appears unnatural compared to the actual environment.

[0006] In other words, conventional technology lacks the capability to reliably place virtual objects at user-specified points by integrating data based on the structure of real space and user location information.

[0007] Therefore, a novel and advanced technical means is required to generate integrated data by combining real-world spatial recognition and user location calculation, and to naturally place virtual objects based on this data. Prior art literature

[0008] Korean Registered Patent No. 10-2533209 The problem to be solved

[0009] The present invention was devised to overcome the problems of the above technology, and its main purpose is to provide an XR content providing system that uses SLAM technology to recognize an environment including the shape and arrangement of objects in a space where a user is located, calculates user location information based on the environment recognition information, integrates such information with 3D spatial data obtained through a smartphone to place a virtual object at a specific point designated by the user, and outputs it.

[0010] Another objective of the present invention is to provide natural and stable XR content by automatically determining the position and orientation of virtual objects in generated XR content, applying brightness, shadows, and colors that reflect the lighting conditions of real space, and adjusting the level of detail according to the number and degree of depiction of virtual objects.

[0011] Another objective of the present invention is to maintain a balanced quality and performance of XR content even under dynamic conditions, such as user movement or hand tremors, by recalibrating the level of detail to reflect the movement speed and camera stability indicators of the smartphone. means of solving the problem

[0012] To achieve the above objective, the XR content provision system using SLAM-based spatial computing according to the present invention is characterized by comprising: a spatial recognition module that recognizes an environment including the shape and arrangement of objects in a space where a user is located using simultaneous localization and mapping (SLAM) technology, generates environment recognition information, and calculates user location information based on the environment recognition information; a spatial data acquisition module that acquires three-dimensional spatial data of the space through a smartphone held by the user; a data management module including an integrated data management unit that generates and stores integrated data including the environment recognition information and user location information in the three-dimensional spatial data, and a virtual object data storage unit that stores a plurality of virtual object data including three-dimensional shape and attribute information; and a content provision module that generates extended reality (XR) content by placing virtual object data selected by the user in correspondence with the user's environment and location based on the integrated data, and outputs it at a specific point selected by the user in the space displayed on the screen of the user's smartphone.

[0013] In addition, the integrated data management unit is characterized by transmitting the integrated data to a cloud server for storage and providing an application programming interface to enable external applications to utilize the integrated data stored on the cloud server.

[0014] In addition, the content providing module is characterized by including a spatial placement unit that automatically determines the position and orientation of the virtual object, a visual effects unit that controls the brightness, shadow, and color of the virtual object, and a level of detail management unit that adjusts the level of detail (LOD) of the virtual object according to the number and degree of detail of the virtual object. Effects of the invention

[0015] According to the XR content provision system using SLAM-based spatial computing according to the present invention,

[0016] 1) It has the advantage of providing XR content aligned with reality by generating integrated data that reflects the structure of real space and user location information, and stably placing virtual objects at points designated by the user based on this data, and

[0017] 2) By reflecting the number of virtual objects, the degree of detail, processing performance, and the gaze fixation time index, the level of detail can be quantitatively calculated, thereby ensuring both performance and quality simultaneously, and

[0018] 3) By comprehensively considering the user's movement speed and camera stabilization index, the level of detail can be recalibrated, which has the effect of providing immersive XR content while saving performance even in dynamic environments. Brief explanation of the drawing

[0019] FIG. 1 is a block diagram illustrating the configuration of the system of the present invention. FIG. 2 is a conceptual diagram illustrating the process of a spatial recognition module generating environment recognition information and calculating user location information using SLAM technology. FIG. 3 is a conceptual diagram illustrating how a content provision module places and outputs virtual object data on a smartphone screen. Figure 4 is a conceptual diagram of an integrated data management unit uploading integrated data to a cloud server for management and providing it to external applications. FIG. 5 is a conceptual diagram illustrating a docent providing unit calling virtual docent data to provide guidance information to a user. Specific details for implementing the invention

[0020] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. The attached drawings are not drawn to scale, and the same reference numerals in each drawing refer to the same components.

[0021] The system of the present invention can be implemented through a central control server, and the central control server is composed of a complex structure of hardware and software for stably generating augmented reality content and providing it to users.

[0022] In terms of hardware, the central control server may include a central processing unit (CPU) to efficiently receive and process three-dimensional spatial data, environmental recognition information, and user location information transmitted from a user's smartphone, a graphics processing unit (GPU) or an artificial intelligence-dedicated computational processor (NPU, TPU, etc.) for executing real-time spatial mapping and object placement, memory (RAM) for high-speed data processing, and a high-speed storage means (SSD or HDD, etc.) for long-term storage and management of collected spatial data and virtual object data.

[0023] In addition, it is possible to include wired and wireless network interfaces and input / output interfaces for stable data communication and interaction with smartphones, administrator terminals, and various user devices.

[0024] Software-wise, the central control server may include a spatial recognition module (110) that processes environment recognition information and user location information obtained through SLAM technology in the space where the user is located, a spatial data acquisition module (120) that acquires 3D spatial data from a smartphone and transmits it to the server, a data management module (130) that generates integrated data including the environment recognition information, user location information, and 3D spatial data and manages a plurality of virtual object data, and a content provision module (140) that generates augmented reality content by placing a virtual object at a specific point selected by the user based on the integrated data and outputs it to the user's smartphone screen.

[0025] These software modules are not limited to logical components implemented solely in software, but may be partially or entirely implemented in hardware forms such as FPGAs (Field Programmable Gate Arrays) and ASICs (Application-Specific Integrated Circuits), and may be configured in various forms including processes, functions, procedures, subroutines, program code segments, drivers, firmware, circuits, databases, data structures, and variables.

[0026] The central control server of the present invention can operate by organically linking with various user devices used by an administrator, such as smartphones, tablet PCs, desktops, and laptops, through wired or wireless networks, and can provide various services such as spatial data collection, user location determination, verification of virtual object placement results, and experience and management of augmented reality content by linking with a client application installed on a smartphone (hereinafter also referred to as 'app' or 'application').

[0027] In addition, as an embodiment, the smartphone itself possessed by the user can perform the function of a central control server. In this case, the central processing unit, graphics processing unit, memory, and storage means of the smartphone are utilized as hardware resources of the central control server, and an application installed on the smartphone operates as a software module of the central control server to implement the same service.

[0028] Hereinafter, the configuration and operation of the system of the present invention will be described in detail with reference to the attached drawings.

[0029] FIG. 1 is a block diagram illustrating the configuration of the system of the present invention.

[0030] As can be seen from FIG. 1, the system of the present invention is based on including a spatial recognition module (110), a spatial data acquisition module (120), a data management module (130), and a content provision module (140).

[0031] Figure 2 is a conceptual diagram illustrating the process in which a spatial recognition module generates environment recognition information and calculates user location information using SLAM technology.

[0032] The spatial recognition module (110) uses simultaneous localization and mapping (SLAM) technology to recognize an environment including the shape and arrangement of objects in the space where the user is located, generates environment recognition information, and provides a function to calculate user location information based on the environment recognition information.

[0033] Simultaneous Localization and Mapping (SLAM) technology utilizes various sensors, such as cameras, gyroscopes, accelerometers, and GPS, to recognize the space where a user is located, simultaneously estimate the user's position, and create a spatial map in real time.

[0034] SLAM technology calculates the coordinates of where a user is located in space through a localization function and constructs a map by recording the structure of the surrounding environment in feature point units through a mapping function. By performing these two functions simultaneously, the spatial map is progressively expanded and corrected according to the path the user moves along.

[0035] In the system of the present invention, this SLAM technology can be applied to a spatial recognition module (110) to produce environment recognition information and user location information.

[0036] That is, the spatial recognition module (110) may, in terms of hardware, operate based on a central processing unit (CPU), a graphics processing unit (GPU), and inertial sensors such as a camera sensor, a magnetic field sensor, a gyroscope sensor, and an accelerometer sensor embedded in a central control server or a smartphone, preferably a smartphone having the sensor described later, and may additionally include a LiDAR sensor, and in terms of software, SLAM technology may be implemented by including an algorithm that extracts feature points from video frames and calculates a movement trajectory by matching feature points between frames, a filtering program that corrects position coordinates by fusing inertial sensor data, and map building software that creates environment map data.

[0037] Specifically, the spatial recognition module (110) performs SLAM technology to generate environment recognition information. First, it acquires raw data from a plurality of sensors including a camera sensor, a magnetic field sensor, a gyroscope sensor, and an accelerometer sensor of a smartphone, and uses this to recognize an environment including the shape and arrangement of objects.

[0038] In this process, video frames are continuously acquired through the smartphone camera, and feature elements such as table corners, door frames, window boundaries, and junctions between floors and walls are detected in each frame.

[0039] The magnetic field sensor detects changes in the direction of the magnetic field within space and combines them with feature elements extracted from camera images to enable directional correction, while the gyroscope and accelerometer sensors correct for tilt or shaking occurring between frames to support stable data acquisition.

[0040] The spatial recognition module (110) analyzes the matching relationship of identical feature elements between consecutive frames based on data acquired from multiple sensors in this way, and generates environment recognition information that reflects the shape and positional relationship of objects such as walls, pillars, and furniture in a specific space.

[0041] The generated environment perception information includes not only simple image information but also the outlines, relative distances, and arrangement structures of major objects within the space, enabling precise user location calculation in subsequent steps.

[0042] In addition, the spatial recognition module (110) calculates user location information based on the generated environment recognition information.

[0043] First, feature elements extracted from newly input video frames are matched with feature elements included in existing environment recognition information, and the user's relative movement distance and direction are calculated using the result of this match.

[0044] Next, the error in the distance traveled and direction calculated by fusing inertial data collected from the gyroscope sensor and the accelerometer sensor is corrected, and the user location coordinates are calculated through a process of aligning the coordinate system of the smartphone with the coordinate system of the environment recognition information.

[0045] Through this procedure, the spatial recognition module (110) can go beyond simply estimating the approximate location of the user and provide the user's location as coordinate data in a form that matches the actual structure of the environment.

[0046] For example, when a user moves inside an exhibition hall holding a smartphone, the spatial recognition module (110) detects feature elements such as corners of exhibits, wall boundaries, and floor patterns through camera images, corrects the east, west, north, and south directions through a magnetic field sensor, and corrects shaking and rotational changes that occur during movement through a gyroscope and accelerometer sensor.

[0047] Through this processing, environment recognition information reflecting the shape and arrangement of exhibits within the exhibition hall is created, and subsequently, the spatial recognition module (110) determines that the user has reached the front of the exhibit by comparing the environment recognition information with the feature elements of the newly input camera image and correcting the detailed coordinates using inertial sensor data to produce specific user location information such as "the user is at a point 1.5 meters away from the east wall of the exhibition hall, in front of a specific exhibit."

[0048] This spatial recognition module (110) can be equipped with a foundation to include environment recognition information and user location information in the integrated data and store and utilize it in the data management module (130) to be described later.

[0049] The spatial data acquisition module (120) provides the function of acquiring three-dimensional spatial data of the space through a smartphone held by the user.

[0050] 3D spatial data is data that expresses the structure of real space and the positional relationship of objects in the form of three-dimensional coordinates based on image and depth information acquired through a smartphone camera and sensors, where each point or mesh has a coordinate value in space and reflects the geometric shape and arrangement of space.

[0051] In the system of the present invention, since the 3D spatial data includes depth, distance, and location information of the space itself, unlike photos or videos, it is combined with environment recognition information and user location information in the data management module (130) described later and utilized as core data to form integrated data.

[0052] In other words, 3D spatial data has fundamental significance in ensuring that virtual objects can be accurately placed at specific points in real space when creating augmented reality content.

[0053] Specifically, the spatial data acquisition module (120) acquires continuous video frames using the smartphone camera, analyzes the difference between frames to extract depth information of the space displayed on the smartphone screen, calculates the camera's movement trajectory by combining gyroscope sensor and accelerometer sensor data, and fuses this video information and inertial information to calculate the coordinate values ​​of each point in the space displayed on the smartphone screen.

[0054] Next, the spatial data acquisition module (120) organizes the calculated coordinate values ​​into a point cloud to generate three-dimensional spatial data that reflects the outline and relative position of objects in space. The finally generated three-dimensional spatial data is provided as a data set that reproduces the physical environment around the user as a digital environment.

[0055] For example, when a user holds a smartphone and films the living room, the spatial data acquisition module (120) detects feature regions such as the table surface, the outline of the sofa, and the wall boundary from the camera image, and analyzes the parallax due to movement between frames to calculate depth information that the table is 1.2 meters away from the camera and the sofa is 2.5 meters away.

[0056] In addition, the degree to which the smartphone moves or is tilted is corrected by applying gyroscope and accelerometer data together, and the point cloud data produced through this is expressed as 3D spatial data that is reproduced similarly to the actual structure of the living room.

[0057] The 3D spatial data generated in this way is managed as integrated data along with environment recognition information and user location information in the data management module (130) to be described later, enabling the content provision module (140) to create augmented reality content, such as placing a virtual object at a specific point on the central table of the living room.

[0058] The data management module (130) includes an integrated data management unit (131) and a virtual object data storage unit (132).

[0059] The integrated data management unit (131) provides the function of creating and storing integrated data by including the environment recognition information and user location information in the three-dimensional spatial data.

[0060] Integrated data is information formed into a single data set by combining three-dimensional spatial data generated by the spatial data acquisition module (120) with environment recognition information and user location information calculated by the spatial recognition module (110), and is data in which not only the geometric structure of the actual space but also the shape and arrangement of objects within the space and the user location are simultaneously expressed.

[0061] In the present invention, the integrated data has significance in that it is used as reference data necessary to place a virtual object at a specific location in real space, going beyond simple three-dimensional space reproduction. That is, the integrated data serves as a foundation that enables the content provision module (140), which will be described later, to ensure consistency between the real world and the virtual object during the process of generating augmented reality content.

[0062] The integrated data management unit (131) first receives three-dimensional spatial data transmitted from the spatial data acquisition module (120) and records the point cloud information and mesh structure included in the data in the initial database. Then, it combines the environment recognition information generated by the spatial recognition module (110) to reflect the shape and arrangement of objects in the three-dimensional spatial data. Afterward, user location information calculated by the same module is added to the coordinate system so that the location of the user within the space is included.

[0063] Through this procedure, the 3D spatial data is expanded into integrated data that simultaneously reflects environmental recognition information and user location information, and the integrated data management unit (131) records this data on a storage device of a server or smartphone so that it can be utilized in subsequent steps.

[0064] For example, when a user holds a smartphone and photographs the living room, the spatial data acquisition module (120) generates three-dimensional spatial data reflecting the table, sofa, and wall and transmits it to the integrated data management unit (131).

[0065] The integrated data management unit (131) combines the environment recognition information calculated by the spatial recognition module (110) with this data to reinforce structural information such as the relative arrangement of the table and sofa and the boundary of the wall. Then, when user location information is included, integrated data is finally generated that reflects the spatial structure and user coordinates together, such as "the user is located in the center of the living room, the table is placed 1.2 meters in front of the user, and the sofa is placed 2.5 meters to the left."

[0066] The virtual object data storage unit (132) provides the function of storing multiple virtual object data including three-dimensional shape and attribute information.

[0067] A virtual object is a digital object designed on the premise of direct interaction with real space, referring to a virtual visual element that can be placed in a combined form with actual objects when a user observes real space through a smartphone screen.

[0068] These virtual objects are not simple two-dimensional images or icons, but are implemented as three-dimensional data that can be displayed in an appropriate size and position according to location and spatial information of the real world, and thus have significance as a core element constituting augmented reality content in the present invention.

[0069] That is, in the system of the present invention, a virtual object acts as a basic unit placed at a specific point in real space that can be visually or functionally perceived by the user, and is defined in the form of virtual object data to manage it as data.

[0070] Virtual object data is a dataset that includes 3D shape information and attribute information necessary for the implementation of virtual objects. The 3D shape information is data that defines the outline, surface, size, and shape of virtual objects based on a coordinate system, and the attribute information is data that defines visual and physical characteristics such as color, material, transparency, reflectivity, and animation behavior.

[0071] This virtual object data provides a foundation that, when placed at specific coordinates in real space, not only displays the shape but also expresses color and texture, and enables the execution of actions or animations if necessary.

[0072] The virtual object data storage unit (132) stores and manages a plurality of virtual object data as described above in a database structure, thereby supporting the content provision module (140), which will be described later, to call and place virtual objects in conjunction with integrated data.

[0073] That is, the virtual object data storage unit (132) systematically accumulates predefined virtual object data, thereby providing a foundation to retrieve the data as needed at any time and place it on the real space.

[0074] Examples of virtual object data may include various cases such as guide characters explaining artworks in exhibition halls, 3D vases placed on tables, monsters or items appearing in game environments, human models or mathematical shapes provided in educational content, and virtual signs or directional arrows shown in city guidance services.

[0075] These virtual object data are each stored with 3D shape information and attribute information, and are called when needed and placed at a specific point in real space to be implemented as augmented reality content that users can intuitively perceive.

[0076] Figure 3 is a conceptual diagram illustrating how a content provision module places and outputs virtual object data on the screen of a smartphone.

[0077] The content provision module (140) provides a function to generate extended reality (XR) content in which virtual object data selected by the user is placed in response to the user's environment and location based on integrated data, and to output it at a specific point selected by the user within the aforementioned space displayed on the screen of the user's smartphone.

[0078] Extended Reality (XR) content is a concept that encompasses technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). It refers to digital content that combines virtual objects with the real-world space and the user's location, enabling the user to perceive and interact with virtual objects within a real-world environment.

[0079] In the present invention, the augmented reality content is not merely about displaying virtual images, but rather has significance in that virtual objects are placed in a form aligned with the structure and coordinate system of the actual space where the user is located, thereby providing a fusion experience of reality and virtuality.

[0080] As examples of augmented reality content, in educational environments, virtual human anatomical models can be placed on classroom desks for anatomy learning; in museum exhibitions, virtual guide characters can be positioned next to exhibits to provide explanations; and in navigation services, directional arrows or signs can be placed at specific locations on the road where the user is situated to provide intuitive route guidance. All of these examples involve virtual objects fusing into real-world space, allowing users to experience integrated content that blends reality and the virtual world through their smartphones.

[0081] Specifically, the content provision module (140) first retrieves integrated data provided by the integrated data management unit (131), analyzes environment recognition information and user location information, and calls virtual object data selected by the user. Then, it converts the 3D shape information and attribute information of the called virtual object data to match the coordinate system of the integrated data, and calculates candidate points that can be placed considering the user location and the environmental structure. Afterward, it identifies a specific point designated by the user, places the virtual object data at that point, and creates augmented reality content within a coordinate system that matches the real space structure.

[0082] For example, if a user designates a specific point on the table in the center of the living room through a smartphone screen, the content providing module (140) checks the environment recognition information, including the shape and location of the living room table, in the integrated data, and calls the vase virtual object data selected by the user to adjust the size and texture of the object to the coordinate system of the integrated data. Subsequently, by placing the vase virtual object data at the center point of the table designated by the user, an augmented reality content is created that is “the appearance of a vase placed on a living room table.”

[0083] Afterwards, the content provision module (140) provides the function of displaying the extended reality (XR) content generated through the above-described procedure on the user's smartphone screen.

[0084] Specifically, a rendering process is performed so that the generated content matches the coordinate system of the integrated data, and a transformation process is performed so that the virtual object is not distorted according to the user's viewpoint and screen size. In addition, the content provision module (140) corrects the output coordinates so that the virtual object is accurately visually placed at a specific point designated by the user, and renders the attribute information of the placed virtual object, such as size, color, transparency, and texture, so that it naturally blends with the real object on the smartphone screen.

[0085] For example, if there is augmented reality content created by a user placing a vase virtual object on a central living room table, the content providing module (140) corrects the rendering coordinates of the smartphone screen so that the vase is displayed at the exact coordinates on the top of the table, and continuously displays the vase on the screen so that it appears to be fixed on the table even when the user's viewpoint moves.

[0086] Through this output process, users can be provided with an immersive augmented reality experience, such as having a real vase placed on a table in the real world.

[0087] The system of the present invention has the characteristic of stably providing augmented reality content in which reality and virtuality are aligned by integrating 3D spatial data, environment recognition information, and user location information based on SLAM technology to accurately place a virtual object at a specific point in real space and naturally display it on a smartphone screen.

[0088] The aforementioned integrated data management unit (131) can provide a foundation for stably sharing and long-term management of integrated data for multiple users or various service environments by not limiting the generated integrated data to within a terminal such as a smartphone but by transmitting and storing it on a cloud server. In addition, by providing an application programming interface (API) to link the integrated data stored on the cloud server with external applications, it is possible to perform the function of enabling augmented reality content to be utilized on various external services and platforms, rather than being implemented only within the smartphone.

[0089] To this end, the integrated data management unit (131) can transmit the integrated data to a cloud server for storage and provide an application program interface so that the integrated data stored on the cloud server can be utilized by an external application.

[0090] Figure 4 is a conceptual diagram of an integrated data management unit uploading integrated data to a cloud server for management and providing it to external applications.

[0091] First, the integrated data management unit (131) divides the integrated data generated from the smartphone into packets to set a transmission path, safely uploads it to a cloud server through a network interface, and applies an error detection code or encryption algorithm to verify data integrity during the upload process.

[0092] Next, the cloud server records the transmitted integrated data in a database and assigns metadata to manage it so that it can be efficiently indexed, and the integrated data management unit (131) provides the data stored in the cloud server in a form that allows an application programming interface (API) to be called so that it can be utilized by an external application.

[0093] This API includes the ability to query specific parts of integrated data or return information related to the placement of necessary virtual objects when requested by external applications, thereby supporting the integrated data to be used as a foundation for various service integrations rather than merely for storage.

[0094] For example, when providing augmented reality content based on a classroom environment in an educational application, the integrated data management unit (131) transmits and stores integrated data containing the arrangement of desks, blackboards, and windows of the classroom to a cloud server, and the educational application can retrieve the integrated data of the classroom via an API and place virtual teaching materials or learning shapes on a specific desk. As another example, in a museum guidance service, integrated data containing the structure of the exhibition hall and user location information is stored on a cloud server, and the guidance application checks the location in front of a specific exhibit via an API call, and then places a virtual docent object at that point so that it can be displayed on the user's smartphone screen.

[0095] This integrated data management unit (131) has the advantage of being able to efficiently utilize the integrated data in various service environments by simultaneously ensuring the sharing and scalability of the integrated data through the application programming interface (API) and by stably storing the integrated data on a cloud server and interacting with external applications.

[0096] If the augmented reality content described earlier remains at the level of merely placing virtual objects, it may lead to the problem of being unable to provide users with sufficient immersion and understanding.

[0097] To solve this, the virtual object data storage unit (132) includes virtual docent data configured to output guidance voice, explanatory text, and motion animation, and in response, the content providing module (140) may include a docent providing unit (141) that calls the virtual docent data and provides guidance information to the user through guidance voice, explanatory text, or motion.

[0098] Figure 5 is a conceptual diagram illustrating how a docent providing unit calls virtual docent data to provide guidance information to a user.

[0099] Virtual docent data refers to virtual object data configured to be placed at a specific location in a real-world space to provide explanations or guidance to users. It is not merely a simple visual model, but rather data configured to include multiple means of expression such as voice, text, and animation, enabling users to experience something similar to interacting with a real guide.

[0100] When called by the content provision module (140), the virtual docent data outputs guidance voice so that the user can obtain information audiovisually, displays explanatory text so that the user can accurately verify necessary information, and performs gaze guidance or emphasis actions through specific motion animations to help the user understand. In other words, the virtual docent data is utilized as an element that performs interactive guidance and information delivery beyond simple object placement in an augmented reality content environment.

[0101] For example, in a museum exhibition environment, a virtual docent can be placed in front of a specific exhibit to explain the background of its creation via voice, while supplementary explanatory text is simultaneously displayed on a smartphone screen.

[0102] In addition, in educational content, a virtual docent can take on the role of a teacher, appearing before students to explain concepts verbally and highlight specific parts of the textbook through hand gesture animations.

[0103] In addition, the navigation service can provide intuitive guidance by having a virtual docent appear at an intersection near the user's location to provide directions verbally and point to the route with hand gestures.

[0104] Specifically, the docent providing unit (141) calls virtual docent data included in the virtual object data storage unit (132), and converts the 3D shape information and attribute information of the called virtual docent data to match the coordinate system of the integrated data to prepare it in a form suitable for the user environment.

[0105] Next, the location where the virtual docent will be placed is calculated by referring to user location information and environment recognition information, and once the placement is confirmed, guidance information is provided by sequentially executing voice, text, and animation actions included in the virtual docent data.

[0106] In this process, the docent providing unit (141) may have the function of providing guidance information in multiple channels, such as displaying explanatory text together when outputting voice or guiding the user's gaze to a specific point through animation movements.

[0107] For example, when a user is positioned in front of a specific exhibit in a museum exhibition hall, the docent providing unit (141) calls virtual docent data designed to match the exhibition description from the virtual object data storage unit (132) and places the data at the coordinates in front of the exhibit.

[0108] Afterwards, the docent provision unit (141) guides the user to understand the exhibit more easily by having a virtual docent explain the background of the exhibit's production in voice, simultaneously displaying supplementary explanation text on the screen and pointing to a specific part of the exhibit with a hand gesture animation.

[0109] This docent providing unit (141) has the advantage of simultaneously improving information delivery and user experience by calling virtual docent data and providing multi-channel guidance information that combines voice, text, and animation actions, thereby enabling the user to receive intuitive and immersive explanations in an augmented reality environment.

[0110] The above-described content provision module (140) performs the function of creating augmented reality content by combining integrated data and virtual object data, but there is a possibility that the placement of virtual objects may be displayed unnaturally due to a mismatch in coordinate systems between the real space and virtual objects or changes in the user's viewpoint.

[0111] In particular, if virtual objects fail to maintain the correct position and orientation within space, or if brightness, shadows, and colors are expressed in a manner incompatible with ambient light conditions, and simultaneously, if the number of virtual objects or the level of detail increases, there may be a problem where the content output quality deteriorates due to smartphone performance degradation.

[0112] To solve this, the content provision module (140) is specialized into a spatial arrangement unit (142), a visual effect unit (143), and a detail level management unit (144) to place virtual objects in the correct position and direction, provide visual effects suitable for light source conditions, and appropriately adjust the detail level according to the number and degree of depiction of objects.

[0113] The spatial arrangement unit (142) provides the function of automatically determining the location and orientation of virtual objects.

[0114] First, the spatial arrangement unit (142) derives candidate points where virtual objects can be placed by referring to environment recognition information and user location information included in the integrated data, and then, when a specific point selected by the user is input, it corrects the coordinates of the point to match the coordinate system of the integrated data.

[0115] Next, the 3D shape information and attribute information of the virtual object data are converted to fit the coordinate system, the position of the virtual object is specified based on the converted data, and the direction is automatically determined so that the virtual object maintains the same direction as the object in the real space by considering the user's viewpoint change. Finally, the spatial arrangement unit (142) transmits the virtual object data, with the position and direction simultaneously determined, to the rendering process inside the content provision module.

[0116] For example, if the user designates a specific point on the table in the center of the living room through the smartphone screen, the space arrangement unit (142) derives a candidate point where a virtual object can be placed by referring to the table coordinates included in the integrated data and the user location information, and calls the vase virtual object data based on the table center coordinates selected by the user.

[0117] Afterward, the space arrangement unit (142) determines the position so that the vase is naturally placed at the top coordinates of the table, and automatically determines the direction so that the vase always maintains the same direction as the table even when the user moves the smartphone.

[0118] Through this, the vase displayed on the screen appears aligned as if it were actually placed on a real table.

[0119] The visual effects unit (143) provides the function of controlling the brightness, shadows, and colors of the virtual object.

[0120] Specifically, the visual effects unit (143) first estimates the direction and intensity of the light source from the environment recognition information included in the integrated data and performs initial adjustments by comparing it with the surface attribute information of the virtual object data so that the virtual object has brightness and color similar to the actual environment.

[0121] Next, during the rendering process, the position and length of the shadow are calculated to generate a shadow that matches the light source conditions of the real space, and the color and brightness are corrected by reflecting the mutual reflection between the virtual object and surrounding objects.

[0122] Next, the visual effects unit (143) tracks the change in the relative position of the light source that occurs according to the user's viewpoint change in real time, and continuously updates the brightness, shadow, and color of the virtual object in response to this to minimize the visual difference with the real space.

[0123] For example, when a user checks a virtual object of a vase placed on a living room table through a smartphone screen, the visual effects unit (143) extracts the lighting position and intensity of the living room ceiling from the environment recognition information and compares it with the surface attribute information of the vase to simulate the brightness and color under actual lighting.

[0124] Next, when the vase is placed on the table, it automatically generates a shadow cast on the floor and corrects the reflection between the table surface and the vase to achieve natural colors. When the user moves the smartphone to change the viewpoint, the visual effects unit (143) recalculates the relative positions of the light and the vase in real time and immediately updates the brightness and shadow, and as a result, the vase displayed on the screen appears visually aligned like an object placed under actual living room lighting.

[0125] The level of detail management unit (144) provides a function to adjust the level of detail (LOD) of virtual objects according to the number of virtual objects and the degree of detail description.

[0126] Level of Detail (LOD) refers to the degree of precision with which visual elements, such as the geometric shape, surface details, and texture resolution of a virtual object are displayed when representing it; this characteristic can vary depending on the complexity of the virtual object and the processing performance of the user device.

[0127] The level of detail is not merely an abstract concept but can be expressed as numerical indicators such as the number of polygons constituting the virtual object, texture resolution, and animation frames, or as a single value calculated by progressively reflecting the complexity, visual importance, and dynamic stability of the virtual object.

[0128] For example, the higher the LOD value, the richer the details of virtual objects are displayed, while the lower the value, the more simplified they are displayed.

[0129] Specifically, the detail level management unit (144) first calculates the number of virtual objects and the degree of detail description from the integrated data, and defines the complexity of the objects based on this.

[0130] In this context, the number of virtual objects refers to the total number of virtual objects simultaneously placed and rendered within the space displayed on the smartphone screen, and the level of detail refers to an indicator that comprehensively reflects factors such as the number of polygons constituting each virtual object, the resolution of textures applied to the surface, and the number of frames required to implement animation movements.

[0131] Here, the number of virtual objects can be determined by counting the total number of virtual objects registered in the rendering queue during the content delivery process, the number of polygons can be calculated by reading the polygon definitions included in the mesh information of the virtual object data, the resolution of the texture can be measured by checking the resolution metadata of the texture file included in the virtual object data, and the number of animation frames can be determined by referring to the total number of frames of the animation sequence included in the virtual object data.

[0132] For example, when 5 virtual objects are displayed in the same space with a level of detail of approximately 1,000 polygons, 512×512 textures, and 10 frames of animation, the level of detail is adjusted to a low value, and when 20 virtual objects are displayed with a level of detail of approximately 10,000 polygons, 2048×2048 textures, and 60 frames of animation, the level of detail is adjusted to a high value.

[0133] Next, the maximum complexity that can be rendered is determined by referring to the processing performance value of the smartphone, and the level of detail of the virtual object selected by the user is adjusted accordingly.

[0134] Afterward, the detail level management unit (144) divides the representation stage of the object into multiple stages during the rendering process, for example, when the LOD stage is 1, only the minimum shape is displayed, and when it is 3, the entire texture and animation are included.

[0135] In addition, the detail level management unit (144) monitors the user's gaze direction, the projection size within the screen, and the complexity of the entire space displayed on the smartphone screen in real time, and maintains a balance of performance and quality by automatically adjusting the LOD value of the object upward or downward accordingly.

[0136] For example, in an exhibition hall environment where multiple virtual objects are placed simultaneously, the level of detail management unit (144) adjusts the display to a low level of detail that displays only simple outlines and colors for exhibits that are far away, depending on the processing performance of the smartphone, and displays the display that the user approaches and fixes their gaze on as a high level of detail that includes high-resolution textures and details.

[0137] In this way, the detail level management unit (144) numerically adjusts the detail level of each object according to the situation, thereby providing the effect of precisely representing the objects needed by the user without degrading the performance of the smartphone.

[0138] In summary, the content provision module (140), specialized in the three parts described above, has the advantage of providing a more stable and natural augmented reality content experience to the user by automatically determining the position and direction of virtual objects through the spatial arrangement part (142), correcting brightness, shadows, and colors to match the real environment through the visual effects part (143), and adjusting the precision of expression according to the number and degree of depiction of virtual objects through the detail level management part (144).

[0139] Since the detailed level management unit (144) described above adjusts the detailed level according to the number of virtual objects and the degree of detailed description, there is a need to manage the detailed level in a more quantitative and stepwise manner. In particular, rather than adjusting the detailed level uniformly based on a single standard, it is desirable to calculate the number of virtual objects and the degree of detailed description separately and to synthesize them to reflect them stepwise.

[0140] To this end, the detailed level management unit (144) is specified into a complexity calculation part (144a), a first analysis part (144b), and an adjustment part (144e) so that the detailed level can be systematically calculated and adjusted.

[0141] The complexity calculation part (144a) provides a function to calculate complexity based on the number of virtual objects and the degree of detail.

[0142] In the present invention, complexity refers to a value calculated by combining the number of virtual objects placed in a specific space and the degree of detail of each virtual object, and is an indicator that quantitatively represents the overall rendering burden of virtual objects.

[0143] Complexity is not merely an abstract concept, but is calculated based on the number of virtual objects (N) and the level of detail (D) of each virtual object, where the level of detail reflects factors such as the number of polygons constituting the virtual object, the resolution of the textures applied to the surface, and the number of frames for animation implementation.

[0144] Specifically, the complexity can be defined as a simplified multiplication operation as follows, where N is the total number of virtual objects in the space displayed on the smartphone screen and D is the level of detail for each virtual object.

[0145]

[0146] Here, C represents complexity, N represents the total number of virtual objects within the space, and D is an indicator (detail of detail indicator) representing the degree of detail by quantifying and combining the number of polygons constituting the virtual objects, texture resolution, and the number of animation frames.

[0147] The degree of description index (D) can be defined as the average of normalized indices of the elements constituting the virtual object.

[0148] More specifically, the degree of description index (D) can be calculated through the following formula.

[0149]

[0150] Here, U is the value normalized by dividing the number of virtual object polygons by the reference number of polygons, H is the value normalized by dividing the texture resolution by the reference resolution, and J is the value normalized by dividing the number of animation frames by the reference number of frames.

[0151] Each value is defined as a dimensionless index and has a range between 0 and 1, and 1 is fixed as an upper limit when greater than or equal to the threshold value.

[0152] This formula calculates the average of the normalized values ​​for polygon count, texture resolution, and animation frame count, offering the advantage of representing the level of detail of virtual objects simply yet in a balanced manner as a single numerical value. Furthermore, since each element is weighted equally, it avoids bias toward specific items and provides the effect of fairly comparing and evaluating the characteristics of various virtual objects.

[0153] For example, assume that the reference polygon count is set to 10,000, the reference resolution to 2048, and the reference frame count to 60.

[0154] For Example 1 (simple virtual object), when a single virtual object is composed of 1,000 polygons and includes a 512×512 level texture and a 10-frame animation, the normalized values ​​are calculated as U=0.1, H=0.25, and J=approx. 0.17, respectively, and accordingly, the detail index (D) is calculated as (0.1+0.25+0.17) / 3 = approx. 0.17. In this case, it can be seen that the virtual object has a low detail index because it has a simple structure.

[0155] On the other hand, in Example 2 (sophisticated virtual object), when a single virtual object consists of 10,000 polygons, 2048×2048 level textures, and 60 frames animation, each normalized value is calculated as U=1.0, H=1.0, and J=1.0, and the detail index (D) is calculated as (1+1+1) / 3 = 1.0. In other words, since it is a virtual object with a sophisticated structure, it has the highest level of detail index.

[0156] In the present invention, complexity is an important reference value used as an input variable when the level of detail management unit (144) determines the Level of Detail (LOD), so it has significance in that it can simultaneously control system performance and visual quality in a balanced way.

[0157] For example, when the reference values ​​are set to 10,000 polygons, 2048 texture resolutions, and 60 animation frames, and the total number of virtual objects in space N is 5 and the detail index (D) of each virtual object is calculated to be 0.17, the complexity is calculated as C = 5 × 0.17 = 0.85.

[0158] On the other hand, under the same conditions, if the total number of virtual objects in space N is 20 and the descriptiveness index (D) of each virtual object is calculated as 1.0, the complexity is calculated as C = 20 × 1.0 = 20.

[0159] As such, the complexity (value) increases significantly as the number of virtual objects and the degree of detail increase, and in subsequent steps, the detail level management unit (144) uses it as a standard for calculating LOD according to mathematical formula 1.

[0160] The first analysis part (144b) provides a function to determine a processing performance value obtained by dividing the processing performance of the smartphone by a reference value and a fixation time index obtained by dividing the fixation time of the user's gaze on a virtual object by a reference time.

[0161] That is, the first analysis part (144b) provides a function that increases the reliability of the Level of Detail (LOD) calculation by quantitatively reflecting the processing performance of the smartphone and the user's gaze behavior along with the complexity value calculated by the level of detail management unit (144).

[0162] Specifically, the processing performance value calculated by dividing the smartphone throughput by a reference value serves as an indicator of the computational limit that the device can render simultaneously, and is meaningful for setting the range of virtual object representation suitable for the device's performance.

[0163] In addition, the fixation time index, calculated by dividing the user's fixation time by the reference time, is a metric that quantifies how carefully a user looks at a specific virtual object, and it is significant as a criterion for judging whether to raise or lower the level of detail of the object depending on the user's level of interest.

[0164] The first analysis part (144b) calculates a processing performance value by comparing the processing throughput per second measured in the CPU and GPU of the smartphone (or central control server) with a reference value, and calculates a gaze fixation time index by analyzing the gaze tracking data included in the camera image and comparing the time the user keeps their gaze on a specific virtual object with a reference time.

[0165] In other words, the processing performance value is used as a criterion to prevent frame drops by reflecting the hardware resource status of the terminal, and the gaze fixation time index is used as a criterion to relatively increase the level of detail of high-attention virtual objects by considering user experience.

[0166] For example, if the standard throughput of a smartphone is set to 100 GFLOPS and the actual measured throughput is 80 GFLOPS, the throughput performance value is calculated as 80 / 100 = 0.8.

[0167] Also, if the reference time is set to 2 seconds and the user fixes their gaze on a specific virtual object for 3 seconds, the gaze fixation time index is calculated as 3 / 2 = 1.5.

[0168] The processing performance value 0.8 and the gaze fixation time index 1.5 calculated in this way are input into Equation 1 of the subsequent adjustment part (144e) and used as variables to determine the detailed level calculation value (s1).

[0169] The adjustment part (144e) provides a function to calculate the Level of Detail (LOD) of the virtual object according to the following mathematical formula 1 based on complexity, the processing performance value of the smartphone, and the gaze fixation time.

[0170] Mathematical formula 1.

[0171]

[0172] Here, s1 is the calculated level of detail, C is the complexity based on the number of virtual objects and the degree of detail, P is the processing performance value of the smartphone, and T is the user's fixation time.

[0173] This mathematical formula non-linearly compresses the ratio of complexity to processing performance through the tanh(C / P) term, and has the characteristic that the output converges stably between -1 and 1 even if the value becomes excessively large.

[0174] In addition, the (1+ln(1+T)) term reflects the gradual increase of the logarithmic function as the gaze fixation time increases, ensuring that the level of detail is appropriately reinforced for virtual objects that the user focuses on for extended periods.

[0175] As a result, Equation 1 has the advantage of providing a balanced LOD output that suppresses complexity according to device performance while reflecting user attention and representing important objects more precisely.

[0176] For example, assume that the complexity reflecting the total number of virtual objects in space and the level of detail is calculated as C = 10, the processing performance value obtained by dividing the smartphone's throughput by the reference value is P = 0.8, and the user's fixation time index is T = 1.5.

[0177] If we substitute these into mathematical equation 1,

[0178]

[0179] The calculated level of detail value of 1.915 means that it converges stably without exceeding computational limits even in environments of high complexity, and at the same time, the level of detail for virtual objects that the user has focused on for a long time is increased so that they can be represented more precisely.

[0180] This level of detail management unit (144) has the advantage of simultaneously securing visual quality and processing efficiency by quantitatively calculating the level of detail through mathematical formula 1 by combining complexity, processing performance value and gaze fixation time index, thereby suppressing excessive computation in accordance with system performance while precisely representing the virtual object that the user is paying attention to.

[0181] The detailed level management unit (144) described above has the advantage of being able to simultaneously secure performance stability and visual quality by quantitatively calculating the detailed level by combining complexity, processing performance value and gaze fixation time index, but even if the detailed level calculation value is the same, there is a possibility that the actual perceived quality may differ depending on the user's gaze direction or the size of objects on the screen.

[0182] In particular, when virtual objects are projected very small on the screen or form a large angle with the user's line of sight, an unnecessarily high level of detail is maintained, which can lead to a problem of wasted computational resources.

[0183] To resolve this, the detail level management unit (144) can be configured to identify the angle difference between the projection area ratio of a virtual object within the screen and the user's gaze vector through the second analysis part (144c), and for the adjustment part (144e) to correct the detail level calculated by applying Equation 2.

[0184] Specifically, the detail level management unit (144) may include a second analysis part (144c) that identifies the angle difference between the projection area ratio and the user's gaze vector in the screen space of the virtual object.

[0185] The projection area ratio (A) of a virtual object in the screen space refers to the ratio of the area occupied by a specific virtual object relative to the total screen area when the smartphone screen is considered as a two-dimensional plane. That is, it is expressed as the value obtained by dividing the number of pixels where the virtual object is displayed by the total number of pixels when the screen size is considered to be 100%.

[0186] If this projection area ratio is included as an analysis target, objects displayed very small on the screen can be automatically corrected so that there is almost no loss of visual quality even when the level of detail is reduced, while objects displayed large on the screen maintain a high level of detail. Therefore, the system of the present invention can simultaneously achieve performance optimization and quality maintenance by reducing unnecessary computational resources and precisely representing important objects.

[0187] The second analysis part (144c) estimates the attitude (rotation matrix / view matrix) of the smartphone using values ​​obtained from the smartphone’s gyroscope sensor, accelerometer sensor, and magnetic field sensor, constructs a screen projection transformation with the camera intrinsic parameters (focal length, principal point), and then applies the virtual object anchor coordinates of the integrated data to this projection transformation to convert it into the screen coordinate system of the current frame.

[0188] Afterwards, the transformed outer polygon is divided into pixel units in the screen coordinate system to calculate the actual number of occupied pixels, and this is divided by the total number of pixels on the screen to calculate the projection area ratio (A) of the virtual object in the screen space, and the calculated A is limited to a value between 0 and 1, and a time filter such as an exponential moving average can be applied to reduce fluctuation between frames.

[0189] Alternatively, if the virtual object is small or the mesh subdivision is excessive and the amount of computation is large, the second analysis part (144c) can calculate A by using the screen area of ​​the circumscribed rectangle (bounding box) as an approximation or by utilizing the actual occupied area based on an alpha mask.

[0190] For example, assuming the smartphone screen resolution is 1080×2400 pixels, the total number of pixels is 2,592,000. Additionally, as a result of the second analysis part (144c) converting the virtual object into a screen coordinate system, if the outer edge of the virtual object occupies an area of ​​270 pixels wide × 960 pixels high, the number of occupied pixels is calculated as 270×960 = 259,200. Therefore, the transparent area ratio

[0191] It is calculated as.

[0192] The angle difference (θ) with respect to the user's gaze vector represents the geometric angle between the direction vector of the user's eyes or smartphone camera's gaze and the direction vector where the virtual object is located. A value closer to 0° indicates that the user is looking directly at the object, while a value closer to 90° indicates that the object is positioned perpendicular to the user's gaze, indicating a low level of interest.

[0193] If these angle differences are included in the analysis, objects the user focuses on from the front can be rendered with a higher level of detail for precision, while objects located to the side or rear out of the line of sight can have their detail level lowered to reduce the rendering burden. This enables efficient resource allocation that reflects the user's focus and enhances immersion.

[0194] The second analysis part (144c) defines the camera forward vector obtained from the above sensor-based pose estimation as the gaze vector, and calculates the object direction vector using the user location information of the integrated data as the camera location and the virtual object anchor coordinates as the target point. Then, the angle difference is calculated using the dot product of the two unit vectors as θ = arccos(·).

[0195] θ is interpreted as looking forward as it is close to 0 and looking sideways as it is close to π / 2, and inertial sensor-based attitude correction and time filters can be applied to suppress noise.

[0196] Alternatively, if the smartphone has an eye tracking function, the actual gaze vector can be used instead of the camera forward vector, and for large virtual objects, the weighted center point within the screen can be used instead of an anchor point to calculate θ.

[0197] For example, assume that the second analysis part (144c) defines the gaze vector of the smartphone estimated through the inertial sensor as v=[0,0,1]. This means that when the smartphone is placed facing the front of the screen, the user's gaze direction coincides with the z-axis direction.

[0198] Meanwhile, if the user's location coordinates in the integrated data are the origin [0,0,0] and the virtual object's anchor coordinates are [1,0,1], the direction vector from the user to the virtual object is

[0199] w = [1-0,0-0,1-0] = [1.0.1], and if we normalize this,

[0200] It becomes. Thus, the inner product of two unit vectors is,

[0201]

[0202] And finally, the angle difference is,

[0203] It can be calculated as.

[0204] In response to this, the adjustment part (144e) can correct the above level of detail according to the following mathematical formula 2.

[0205] Mathematical formula 2.

[0206]

[0207] Here, s2 is the calculated value of the corrected level of detail, A is the ratio of the in-screen projected area of ​​the virtual object, θ is the angle difference between the user's gaze vector and the virtual object display direction, and k is a weighting factor set as a natural number in the range of 10 to 15.

[0208] Mathematical Equation 2 has the characteristic that the value increases as the virtual object is displayed larger on the screen, and the value increases as the angle difference with the user's gaze direction becomes smaller, and the correction effect gradually converges as the input increases through the exponential function term.

[0209] This mathematical formula 2 has the advantage of achieving continuous correction based on visual importance without hard clipping (without discontinuity) because, for small objects (small A) and non-frontal views (large θ), the tanh function returns a small value to smoothly attenuate the effect of s1, and for large objects and frontal views, the tanh function approaches 1 so that s2 naturally approaches s1.

[0210] For example, when the detail level calculated by Equation 1 is s1=1.915, the projection area ratio of the virtual object in the screen space is A=0.10, and the angle difference between the user's gaze vector and the virtual object's direction vector is θ=45 degrees (cosθ=approx. 0.707), if the weighting factor is set to k=15, the corrected detail level calculated according to Equation 2 is calculated as follows.

[0211] = It is calculated to be approximately 1.50.

[0212] This result shows that the detail level s1=1.915, which was calculated as a high value in Equation 1, decreases continuously and gradually through the correction process of Equation 2 and stabilizes at s2= approximately 1.50.

[0213] In other words, even when objects are displayed small on the screen and the user observes them from a 45° angle rather than directly, the level of detail of the virtual objects does not drop drastically but is corrected to a reasonable level, thereby achieving both the effects of performance reduction and maintaining perceived quality simultaneously.

[0214] These second analysis part (144c) and adjustment part (144e) can provide a basis for controlling virtual objects that are of high importance to maintain high detail and virtual objects of relatively low importance to be simplified by comprehensively reflecting the projection area ratio of virtual objects in screen space and the angle difference with the user's gaze vector to correct the calculated detail level.

[0215] Meanwhile, the second analysis part (144c) and the adjustment part (144e) according to mathematical formula 2 perform the function of correcting the level of detail by reflecting the projection area ratio within the screen and the user's viewing angle. However, if the user moves the smartphone or hand shake occurs, the position of the virtual object within the screen may shake unstably or become distorted, and thus the fluctuation of the calculated level of detail may become excessive, so it is necessary to stabilize it.

[0216] In other words, corrections based solely on visual importance cannot adequately reflect dynamic conditions such as user movement speed or device shaking, resulting in limitations in the balance between the quality and performance of the virtual objects actually displayed.

[0217] To solve this, the detailed level management unit (144) may be configured to include a third analysis part (144d) to determine the movement speed index and camera stabilization index of the smartphone, and to allow the adjustment part (144e) to recalculate the detailed level calculation value by applying mathematical formula 4.

[0218] Specifically, the third analysis part (144d) provides a function to determine the movement speed index calculated by dividing the movement speed of the smartphone by the reference speed, and the camera stabilization index calculated according to the following mathematical formula 3.

[0219] Mathematical formula 3.

[0220]

[0221] Here, Q is the camera stabilization index, ω is the change in angular velocity between frames measured by the gyroscope sensor, and a is the change in acceleration between frames measured by the accelerometer sensor.

[0222] The motion speed index (V) is defined as a dimensionless value obtained by dividing the distance actually traveled by the smartphone per unit time (i.e., the smartphone's movement speed) by the reference speed.

[0223] The reason for including this in the analysis is that when a user moves the smartphone quickly, the relative position of virtual objects displayed on the screen fluctuates rapidly, which can make it difficult to represent a stable level of detail.

[0224] In other words, by additionally incorporating the movement speed index as an analysis target, it is possible to secure performance by appropriately lowering the level of detail when the smartphone is moving quickly, and to guarantee visual quality by maintaining the level of detail when there is almost no movement.

[0225] The third analysis part (144d) estimates the position change between frames through the accelerometer and gyroscope sensors built into the smartphone and calculates the distance traveled per unit time by integrating it.

[0226] The calculated movement speed is divided by a predefined reference speed to convert it into a dimensionless value, and this result is defined as the movement speed index (V). The reference speed can be set, for example, to the average speed at which a smartphone is typically held and moved (e.g., 1 m / s), and this allows for comparison with a consistent index value even in various user operation situations.

[0227] For example, if the distance traveled per unit time (1 second) estimated from a sensor built into a smartphone is 0.5 m and the reference speed is set to 1.0 m / s, the motion speed index is,

[0228] It is calculated as.

[0229] On the other hand, if the distance traveled in 2 seconds under the same conditions is 3m, the average speed becomes 1.5 m / s, and the speed of movement index is calculated as 1.5.

[0230] The camera stabilization index (Q) is an indicator that quantitatively represents the degree of minute shaking or hand shake that occurs when shooting with a smartphone, and is defined as the reciprocal of the sum of the change in angular velocity between frames (ω) measured by the gyroscope sensor and the change in acceleration between frames (a) measured by the accelerometer sensor, divided by 1 as shown in Equation 3.

[0231] The Q value falls within the range between 0 and 1, and a value closer to 1 means it is maintained stably, while a value closer to 0 means there is significant fluctuation.

[0232] The reason for including this camera stabilization index in the analysis is to ensure performance by maintaining a level of detail to enhance visual quality when the user holds the smartphone securely, while reducing the level of detail when there is significant hand tremor or sudden movement.

[0233] At this time, Equation 3 is an equation that defines an indicator for quantifying the camera stability of a smartphone, and uses the change in angular velocity between frames (ω) measured through the gyroscope sensor and the change in acceleration between frames (a) measured through the accelerometer sensor as variables.

[0234] This equation has the characteristic that as the absolute values ​​of ω and a increase—that is, as the shaking or sudden movement of the smartphone becomes more severe—the Q value decreases, and conversely, as ω and a decrease—that is, as the smartphone remains stable—the Q value approaches 1. Therefore, Q is defined as a dimensionless stabilization index with a range between 0 and 1, and has the characteristic of being able to intuitively express stability even in a simple form.

[0235] This mathematical formula 3 can calculate the degree of camera stabilization using only the basic output values ​​of the gyroscope and accelerometer sensors without complex algorithms, making it simple to implement and suitable for real-time processing. Additionally, since it allows for an intuitive interpretation that the closer the Q value is to 1, the more stable it is, it is easy to use for recalibration in the detailed level management unit (144).

[0236] In particular, it can be used as a standard to maintain a level of detail when stability is high and automatically reduce it as instability increases, offering the advantage of enhancing user-perceived quality and efficiently controlling system performance.

[0237] The third analysis part (144d) measures changes in angular velocity between frames through the gyroscope sensor of the smartphone and changes in acceleration between frames through the accelerometer sensor. Then, the absolute values ​​obtained from each sensor are summed to estimate the factors causing stability degradation, and the stability index (Q) is calculated by reflecting this in the denominator.

[0238] This method is designed so that the Q value remains high in the case of small vibrations, having no significant impact on the level of detail, while the Q value decreases in the case of large vibrations, thereby suppressing the level of detail.

[0239] For example, if the change in angular velocity between frames measured by the smartphone's gyroscope sensor is ω=0.05 and the change in acceleration between frames measured by the accelerometer sensor is a=0.02,

[0240] It is calculated as.

[0241] This value means that the smartphone is being maintained very stably.

[0242] The third analysis part (144d) uses these Q values ​​to enable the visual quality to be maintained during the recalibration stage of the detail level management part (144).

[0243] In response to this, the adjustment part (144e) can readjust the level of detail according to the following mathematical formula 4.

[0244] Mathematical formula 4.

[0245]

[0246] Here, s3 is the corrected level of detail output, and V is the motion speed index.

[0247] Mathematical Equation 4 defines a relationship for deriving a final recalculated level of detail (s3) by reflecting the smartphone's motion speed index (V) and camera stabilization index (Q) to the level of detail calculated value (s2) corrected in the 6th term.

[0248] This mathematical formula 4, as the movement speed increases It is designed so that the level of detail is gradually reduced as the term decreases, and as the camera stabilization index is lower, the correction effect of the tanh(Q) term becomes smaller, thereby further reducing the level of detail.

[0249] In other words, it has a continuous correction characteristic in which the level of detail is reasonably reduced when the user moves quickly or the smartphone is shaking, and is maintained as the situation becomes more stable.

[0250] This mathematical formula 4 simultaneously reflects two dynamic factors (movement speed and camera stabilization degree), thereby quantifying changes in user behavior that are difficult to reflect solely through corrections based on screen size or viewing angle, and using this as a recalibration standard. Therefore, it has the advantage of ensuring performance by reducing unnecessary calculations caused by rapid movement or hand shake, while maintaining quality in stable conditions to guarantee the user's sense of immersion.

[0251] As an example thereof, assuming that the correction detail level is s2 = 1.50, the motion velocity index is V=0.30, and the camera stabilization index is Q=0.935 (tanh(Q)= approx. 0.733),

[0252]

[0253] thus, It is calculated as.

[0254] This result means that the level of detail s2 = 1.50 corrected in Equation 2 is gradually reduced to s3 = approximately 1.39 through Equation 4.

[0255] In other words, it can be seen that even under conditions where there is some user movement and camera stability is slightly shaken, the level of detail does not drop sharply to less than half, but rather decreases gradually within a reasonable range.

[0256] The fact that the detail level output value is corrected to 1.39 means that the virtual object is not rendered with full maximum detail (e.g., level 2.0), but is not reduced to a minimum simplification level (e.g., 0.5 or less), and maintains a state of intermediate or higher detail.

[0257] This is explained in more detail with Examples 1 to 3 as follows.

[0258] In Example 1 (Texture Quality), if the texture applied to a virtual object has an original resolution of 2048×2048, Detail Level 1.39 displays it downscaled to approximately 1500×1500. In other words, performance is reduced on small screens while retaining sufficient detail that the user can perceive.

[0259] In Example 2 (Polygon Simplification), if a virtual object originally consists of 10,000 polygons, it is rendered with approximately 7,000 to 8,000 polygons at Detail Level 1.39. This reduces the amount of rendering computation, thereby decreasing heat generation and battery consumption, while maintaining the object's outline and details.

[0260] In Example 3 (Animation Frame), if the animation is originally implemented at 60 fps, at a detail level of 1.39, it is output at approximately 45–50 fps. In other words, performance efficiency can be secured by reducing unnecessary frame calculations while maintaining smooth movement.

[0261] Therefore, when the level of detail is adjusted to 1.39, texture resolution, polygon count, and animation frames are all reduced by a certain ratio, reducing computational load while minimizing user perceived quality. In other words, this figure serves as a "practical reference value that reduces unnecessarily excessive precision and maintains a balance between performance and immersion."

[0262] Through this, the system of the present invention has the advantage of saving performance by suppressing unnecessary high-precision rendering while maintaining visual quality, allowing users to experience augmented reality content immersively.

[0263] In summary, the third analysis part (144d) and the adjustment part (144e) based on mathematical formula 4 comprehensively reflect the smartphone's movement speed index and camera stabilization index to recalibrate the level of detail, thereby reducing computational resources while maintaining stable quality when the user is moving or shaking is severe, and preserving the level of detail in a stable state to provide visual immersion.

[0264] As explained above, the configuration and operation of the XR content provision system using SLAM-based spatial computing according to the present invention have been described and drawn in the above description and drawings, but this is merely an example and the concept of the present invention is not limited to the above description and drawings, and it is obvious that various changes and modifications are possible within the scope of the technical concept of the present invention. Explanation of the symbols

[0265] 110: Spatial Recognition Module 120: Spatial Data Acquisition Module 130: Data Management Module 131: Integrated Data Management Department 132: Virtual object data storage unit 140: Content provision module 141: Docent Provision Section 142: Spatial Layout Section 143: Visual Effects Department 144: Detail Level Management Department 144a: Complexity Calculation Part 144b: Analysis Part 1 144c: Analysis Part 2 144d: Analysis Part 3 144e: Adjustment part

Claims

Claim 1 An XR content provision system using SLAM-based spatial computing, comprising: a spatial recognition module that uses Simultaneous Localization and Mapping (SLAM) technology to recognize an environment including the shape and arrangement of objects in a space where a user is located, generates environment recognition information, and calculates user location information based on said environment recognition information; a spatial data acquisition module that acquires three-dimensional spatial data of said space through a smartphone carried by a user; and a data management module including an integrated data management unit that generates and stores integrated data by including said environment recognition information and user location information in said three-dimensional spatial data, and a virtual object data storage unit that stores a plurality of virtual object data including three-dimensional shape and attribute information. The XR content providing system comprises: a content providing module that generates extended reality (XR) content by placing virtual object data selected by a user in correspondence with the user's environment and location based on the integrated data, and outputs the content to a specific point selected by the user in the space displayed on the screen of the user's smartphone; wherein the content providing module includes a spatial placement unit that automatically determines the position and direction of the virtual object, a visual effects unit that controls the brightness, shadow, and color of the virtual object, and a level of detail management unit that adjusts the level of detail (LOD) of the virtual object according to the number and degree of detail of the virtual object, and wherein the level of detail management unit includes a complexity calculation part that calculates complexity according to the number and degree of detail of the virtual object, a first analysis part that identifies a processing performance value obtained by dividing the processing volume of the smartphone by a reference value and a gaze fixation time index obtained by dividing the user's gaze fixation time on the virtual object by a reference time, and an adjustment part that calculates the level of detail (LOD) of the virtual object according to the following mathematical formula 1 based on the complexity, the processing performance value of the smartphone, and the gaze fixation time.Mathematical formula 1. (Here, s1 is the output value for the level of detail, C is the complexity, P is the processing performance value of the smartphone, and T is the user's gaze fixation time) Claim 2 An XR content providing system according to claim 1, wherein the integrated data management unit transmits and stores the integrated data to a cloud server and provides an application programming interface so that the integrated data stored in the cloud server can be utilized by an external application. Claim 3 An XR content providing system according to claim 1, wherein the virtual object data storage unit includes virtual docent data configured to output guidance voice, explanatory text, and motion animation, and the content providing module includes a docent providing unit that calls the virtual docent data and provides guidance information to a user through guidance voice, explanatory text, or motion. Claim 4 delete Claim 5 delete Claim 6 In claim 1, the detail level management unit includes a second analysis part that identifies the angle difference between the projection area ratio and the user gaze vector in the screen space of the virtual object, and the adjustment part corrects the detail level according to the following mathematical formula 2, characterized in that it is an XR content providing system. Mathematical formula 2. (Here, s2 is the calculated value of the corrected level of detail, A is the ratio of the projected area in the virtual object's in-screen space, θ is the angular difference between the user's gaze vector and the virtual object's display direction, and k is a weighting factor set as a natural number in the range of 10 to 15) Claim 7 In Clause 6, the above-mentioned detailed level management unit includes a third analysis part that identifies a motion speed index obtained by dividing the movement speed of the smartphone by the reference speed, and a camera stabilization index calculated according to the following mathematical formula 3, and mathematical formula 3. (Here, Q is the camera stabilization index, ω is the change in angular velocity between frames measured through the gyroscope sensor, and a is the change in acceleration between frames measured through the accelerometer sensor) The above adjustment part is characterized by recalibrating the above level of detail according to the following mathematical formula 4, in an XR content delivery system. Mathematical formula 4. (Here, s3 is the corrected level of detail output, V is the motion velocity index)

Citation Information

Patent Citations

  • Portable device, virtual reality system and method

    KR1020140083015A

  • Systems, methods, and graphical user interfaces for interacting with augmented and virtual reality environments.

    KR1020200031666A

  • Image-based indoor AR navigation method and system

    KR1020250093960A

  • Low-latency adaptive streaming for augmented reality scenes

    US20210304706A1