system

Abstract

We provide the system. [Solution] Means of obtaining publicly available geographic information, A means of converting acquired geographic information into a format suitable for augmented reality display, A means of transmitting the converted geographic information to a mobile device, A means of determining the location of a mobile device using the Global Positioning System, A means of overlaying a virtual boundary onto real-world images in a form that can be displayed on a mobile device, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Boundary markers of land play an important role in the accurate management of land and real estate transactions. However, boundary markers may be buried in the ground or lost, resulting in difficulty in confirming the boundaries and problems in land transactions. This may also cause misunderstandings and disputes among the parties. Therefore, there is a need for a means to easily and quickly confirm the boundaries of land on-site.

Means for Solving the Problems

[0005] This invention provides a system that acquires publicly available geographic information and converts it into a format suitable for augmented reality display. This system uses geographic information and the Global Positioning System to determine the location of a mobile device, and then uses this information to overlay virtual boundaries onto real-world images in a format displayable on the mobile device, thereby enabling easy confirmation of land boundaries. This method allows users to visually confirm accurate land boundaries based on their current location, facilitating land management and real estate transactions.

[0006] "Publicly available geographic information" refers to geographical information that is made publicly accessible through government agencies, public databases, and other means.

[0007] "Augmented reality display" refers to a technology that overlays virtual information onto images of the real world.

[0008] A "mobile device" refers to an electronic device that is portable and equipped with wireless communication capabilities, and includes smartphones and tablets.

[0009] The "Global Positioning System" refers to a system that uses artificial satellites to measure precise location information on Earth, with GPS being one example.

[0010] A "virtual boundary" refers to boundary information that is represented in digital format based on actual physical boundaries.

[0011] "Real-world images" refers to visual information acquired in real time through the camera of a mobile device. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3]It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0014] First, the language used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention provides a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device. This system is realized by integrating functions such as geographic information acquisition, data conversion, location determination, and augmented reality display.

[0034] First, the server retrieves publicly available geographical information online. This information includes land boundaries and map data. The server then uses a generative AI to analyze the retrieved information and convert it into a data format suitable for augmented reality display. The converted data is then stored in the format required by the mobile device.

[0035] Next, the server sends the converted geographic information data to the terminal. When the user starts up their mobile device with the dedicated application installed, the terminal receives this data. The user then uses this terminal to go to the field.

[0036] On-site, the device uses the Global Positioning System (GPS) to obtain accurate, real-time location information. Based on this location information, the device's AR rendering engine uses converted geographic data to overlay virtual boundaries onto the real-world camera footage.

[0037] By holding a mobile device and activating the camera to scan the terrain, a virtual boundary line based on publicly available geographical information is displayed on the screen. This allows users to visually confirm land boundaries in a concise and effective manner. Furthermore, users can manipulate the boundary information on the screen in detail and display additional information through the interface.

[0038] For example, in real estate transactions and land development, users can facilitate communication with landowners by visually confirming actual boundaries in advance. Furthermore, by obtaining visual and real-time feedback without relying on traditional maps or paper media, it is expected that the reliability and efficiency of map information will be improved.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The server accesses an online geographic information database to retrieve the latest geographic information. The server uses an API to download data, including land boundary information.

[0042] Step 2:

[0043] The server uses a generative AI to analyze the acquired geographic information and convert it into a data format suitable for augmented reality display. The server then saves the conversion results as data for AR display.

[0044] Step 3:

[0045] The server sends the converted data to the user's mobile device. The server uses encryption protocols to ensure secure data transmission.

[0046] Step 4:

[0047] The device receives the transmitted geographic information data and prepares to read the data through the application. The user launches the dedicated application and performs the necessary initial setup.

[0048] Step 5:

[0049] The device uses the Global Positioning System (GPS) to obtain the user's current location in real time. To ensure highly accurate location information, the device receives signals from multiple satellites.

[0050] Step 6:

[0051] The device uses an AR rendering engine to overlay and display geographical boundaries converted from images acquired through the camera. This allows users to simultaneously view the real-world scenery and virtual boundaries.

[0052] Step 7:

[0053] The user observes the site with a mobile device and checks the boundaries displayed on the screen. The user can operate the interface to change the display mode or obtain detailed information.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] Conventional geographic data display systems lack a simple and accurate way for users to visually confirm geographical boundaries in real space. Furthermore, the display on mobile devices is not highly customizable, making it difficult for users to intuitively manipulate information according to their needs.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes means for collecting publicly available geographic data, means for utilizing an artificial intelligence model to convert the geographic data into a format suitable for augmented reality display, and means for communicating the converted geographic data to a mobile device. This allows users to accurately and visually confirm virtual boundaries in real geospace and customize them according to their needs.

[0059] "Publicly available geographic data" refers to geographical information resources that can be obtained from government agencies or publicly available platforms, and includes map data, land boundary information, topographic information, etc.

[0060] An "artificial intelligence model" refers to an information processing device that processes large amounts of data and exhibits capabilities equivalent to or exceeding those of a human in specific tasks, and includes machine learning and deep learning.

[0061] "Mobile devices" refer to portable electronic devices such as mobile phones and tablets that have wireless communication capabilities and computing power.

[0062] A "satellite positioning system" refers to a system that uses artificial satellites to provide accurate location information for any point on Earth, and includes GPS and similar technologies.

[0063] "Augmented reality display" refers to a technology that overlays digital information onto images of the real world, presenting information directly and in real time to the user.

[0064] A "virtually defined boundary" refers to a boundary line or area in real geographical space that is defined in digital data and presented in a way that the user can visually recognize.

[0065] "User operation" refers to instructions or actions performed through a user interface via a mobile device, and includes means that allow for intuitive operation.

[0066] This invention provides a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device. This system integrates functions for geographic data acquisition, data transformation, location determination, and augmented reality display.

[0067] First, the server collects publicly available geographic data. To do this, it accesses publicly available databases and web resources on the internet to obtain the latest map information and land boundary information. The server analyzes the acquired geographic data and uses an artificial intelligence model to convert the data into a format suitable for augmented reality display. Examples of artificial intelligence models used in this process include "large-scale language models." The server stores the converted geographic data on its own so that mobile devices can receive it.

[0068] Next, the server communicates the converted geographic data to the mobile device. A common data transfer protocol is used for communication, and encryption technology is employed to ensure a secure connection.

[0069] On-site, the device uses a satellite positioning system to determine its current location. The device is equipped with positioning technologies such as GPS, enabling accurate real-time location tracking of the user.

[0070] Subsequently, the device's AR rendering engine overlays the converted geographic information, based on the acquired location data, onto the real-world camera footage. This process utilizes an "augmented reality-enabled framework." For example, when a user activates the camera and points the device towards the surrounding terrain, virtual boundaries based on public geographic data are displayed on the screen.

[0071] Users can operate the mobile device's interface to fine-tune the displayed virtual boundaries and view related information. For example, it can be used during pre-inspection work at real estate sites to provide effective visual effects.

[0072] An example of a prompt message might be, "Retrieve the latest land boundary information for this area and convert it to a format suitable for AR display." This system will make it easier for users to verify information in the real world and enable rapid visualization of geographic data.

[0073] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0074] Step 1:

[0075] The server collects publicly available geographic data from the internet. Inputs include pre-specified map URLs and API keys. Web crawling technology is used for data collection. The obtained geographic data includes land boundary information and road information, and is output in a format for storage in a database.

[0076] Step 2:

[0077] The server analyzes the collected geographic data and converts it into a format suitable for augmented reality display. The input is the geographic data acquired in Step 1. A generative AI model is used for data conversion, performing noise reduction and data formatting. Specifically, latitude and longitude information is reconstructed into a 3D model format and converted into the Geo-JSON format used for real-time AR rendering. This converted data is then output in a format that mobile devices can receive.

[0078] Step 3:

[0079] The server communicates the converted geographic data to the mobile device. The data converted in step 2 is used as input. Encryption protocols such as TLS are used for communication to ensure secure data transfer. The output is provided in a format that can be stored in the mobile device's memory.

[0080] Step 4:

[0081] The terminal receives geographic data stored on the mobile device and then uses the Global Positioning System (GPS) to obtain its current location in real time. The input consists of the geographic data received in step 3 and the current GPS data. Based on this data, the terminal performs a position calculation to map the user's current location to the geographic data and outputs it as input to the AR rendering engine.

[0082] Step 5:

[0083] The device's AR rendering engine uses the results of position calculations to overlay a virtual boundary onto the real-world camera image. The input is the position data calculated in step 4. Specifically, it renders the boundary that should be visible from the user's viewpoint within the camera image in real time. The output is the AR display, which is shown on the mobile device's screen.

[0084] Step 6:

[0085] Users interact with the AR display on their mobile device to examine the displayed virtual boundary in detail. Input is provided through the device's interface. Specifically, users can adjust the displayed content using gestures such as pinching and swiping, and obtain additional information through voice input. The output provides visual information tailored to the user's intentions.

[0086] (Application Example 1)

[0087] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0088] Without relying on existing map information or paper media, there is a need to provide consumers and users with a means to recognize property boundaries and store layouts accurately in real time, thereby improving their shopping experience. In particular, by clearly understanding the layout and layout within stores, it is necessary to create a more convenient shopping environment for consumers and to provide store operators with efficient methods for layout changes and sales promotions.

[0089] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0090] In this invention, the server includes means for acquiring publicly available spatial information, means for converting the spatial information into a format suitable for augmented realization display using data processing technology, and means for transmitting the converted spatial information to a communication device. This enables consumers to use the communication device to recognize information about the area inside and around the store in real time, thereby improving the shopping experience.

[0091] "Spatial information" refers to data related to geographical location and extent, including information such as real estate and the layout of a store.

[0092] "Data processing technology" refers to the technology of analyzing and transforming data according to a specific purpose, and in this invention, it refers to the transformation into a data format for extended realization and display.

[0093] "Augmented Reality Display" refers to a technology that overlays virtual information onto real-world images, and is a form of AR (Augmented Reality).

[0094] "Communication equipment" refers to a device capable of sending and receiving digital information, and in this invention, it mainly refers to a portable terminal.

[0095] A "location system" is a technology that accurately measures physical location, and the Global Positioning System (GPS) is one example.

[0096] A "virtual boundary" is a line that does not physically exist but is displayed as information, visually presenting the user with the boundaries of a specific area.

[0097] "Store layout information" refers to information about the division and arrangement of space within a store, and is used in designing product displays and customer flow.

[0098] The system implementing this invention primarily consists of three components: a server, communication equipment (mobile terminal), and a user. The server first acquires publicly available spatial information from the internet. A dedicated API can be used for this acquisition. Subsequently, the server converts the acquired spatial information into a format suitable for augmented reality display using data processing technology. This conversion process uses a generative AI model to format the data into an AR display data format in real time. The converted data is stored so that it can be quickly transmitted to the communication equipment.

[0099] Mobile devices receive converted data from a server via the network. During this process, the device precisely determines its location using the Global Positioning System (GPS), and based on this location information, overlays virtual boundaries onto real-world camera footage. This allows users to visually confirm spatial and real estate parcel information in real time.

[0100] Users can use this mobile device to activate the camera at specific locations inside and outside the store, displaying virtual area information along with the video feed. For example, when introducing a new product section, this system can be used to check the store layout and determine the optimal placement of spaces. Based on a prompt message such as, "Please convert the surrounding geographic data to AR format to suggest the optimal store layout," a generative AI model can be used to present information in a practical and intuitive way for the user.

[0101] One specific use case is when a store's design team is selecting a location for a new display. The user can use the terminal to quickly obtain the information needed to achieve the most efficient product placement while viewing the available space within the store.

[0102] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0103] Step 1:

[0104] The server first retrieves publicly available spatial information from the internet. An internet connection is required as input, and requests are made to endpoints such as geographic information APIs. The output is spatial data including land parcel and boundary information.

[0105] Step 2:

[0106] The server transforms the acquired spatial information using data processing technology. Using a generative AI model, it converts the spatial data obtained as input into a data format suitable for augmented reality display. The output is a data format for AR display. During this process, prompt statements are used to control the behavior of the AI ​​model.

[0107] Step 3:

[0108] The server transmits the converted data to the communication device. The converted data is used as input, and the data is sent to the mobile terminal as output. A network connection is required for communication.

[0109] Step 4:

[0110] The device uses the Global Positioning System (GPS) to determine its current location. It receives GPS signals as input and obtains precise latitude and longitude location information as output.

[0111] Step 5:

[0112] The device uses location information obtained from GPS and converted data received from a server to overlay a virtual boundary onto the real-world camera image. Using camera footage and AR data as input, the output is a visually superimposed image.

[0113] Step 6:

[0114] The user utilizes the camera function of their device to view virtual area information presented by the application. The input requires real-world video captured by the device, and the output provides a visual display including virtual information overlaid on the camera footage. This allows the user to perform spatial recognition and confirm the necessary information.

[0115] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0116] This invention realizes a next-generation AR system that combines a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device with an emotion engine that recognizes the user's emotions. This makes it possible to further personalize the user experience.

[0117] This system first uses a server to retrieve geographic information publicly available online and convert it into a format suitable for AR display. Next, the server sends the converted data to the user's mobile device. The user then uses a mobile device with a dedicated app installed to activate the system at the site.

[0118] The device uses the Global Positioning System (GPS) to determine the user's location in real time and overlays virtual boundaries onto real-world images captured by the camera. This allows the user to visually confirm accurate land boundaries through the camera. Furthermore, the device is equipped with an emotion engine that recognizes the user's emotions by analyzing their facial expressions. Emotional data is collected through the device's camera and sensors and analyzed in real time.

[0119] When a user uses this system, the emotion engine changes the AR display according to the user's emotional state. For example, if the user is surprised, additional information is highlighted, while if they are relaxed, a simpler display is maintained. This interactive adjustment allows users to receive information that is most relevant to their state, enabling them to effectively proceed with land inspections and transactions.

[0120] For example, in a land transaction, if the system detects that a user is experiencing stress, it can help the user understand by re-displaying the complex geographical information that is believed to be the cause of the stress in an easy-to-understand format. Furthermore, when the user is at ease, the system maintains the standard mode while providing easy access to more detailed information, thereby optimizing the user experience.

[0121] The following describes the processing flow.

[0122] Step 1:

[0123] The server accesses an online geographic information database and retrieves the latest geographic information according to the user's request. The server uses an API to download the necessary data, including land boundary information.

[0124] Step 2:

[0125] The server analyzes the acquired geographic information using a generation AI and converts it into a format suitable for augmented reality display. This conversion process formats the geographic information into a format usable by the AR rendering engine.

[0126] Step 3:

[0127] The server transmits the converted geographical information to the user's mobile device. This transmission is carried out through a secure communication protocol, ensuring that accurate data is delivered to the user's device.

[0128] Step 4:

[0129] The device uses the Global Positioning System (GPS) to obtain its current location in real time. The device utilizes multiple satellite signals to ensure centimeter-level positional accuracy.

[0130] Step 5:

[0131] The device activates its camera and captures real-world images. The AR rendering engine overlays virtual boundaries onto the camera images, providing the user with visual information.

[0132] Step 6:

[0133] The device activates an emotion engine and collects emotional data from the user's facial expressions and voice through the camera and other sensors. The emotion engine analyzes this data to determine the user's emotional state.

[0134] Step 7:

[0135] The device dynamically adjusts the AR display content based on the analysis results of the emotion engine. For example, if the system determines that the user is confused, it simplifies the displayed information and provides navigation to aid understanding. Furthermore, the color scheme and emphasis of information change according to the user's emotions.

[0136] Step 8:

[0137] Users observe the site using a mobile device and check the boundaries and information displayed on the screen. Through the interface, users can access detailed information and change the display mode. This allows users to easily grasp the necessary information regarding land boundaries.

[0138] (Example 2)

[0139] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0140] The present invention aims to provide an augmented reality system that allows users to easily visualize geographical boundaries in the real world. In particular, it aims to solve the problem of improving the user experience by dynamically responding to the user's emotional state and displaying information according to the situation.

[0141] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0142] In this invention, the server includes means for acquiring publicly available spatial information, means for converting the acquired spatial information into a format suitable for augmented reality display, and means for transmitting the converted spatial information to a mobile terminal. This allows the user to easily perceive virtual boundaries on real-world visual information and to adjust the information display based on the user's emotional state.

[0143] "Publicly available spatial information" refers to information about physical spaces that is available in open databases such as the internet.

[0144] "A format suitable for augmented reality display" refers to the data structure necessary to display a combination of the user's real-world visual information and digital information.

[0145] A "mobile device" refers to an electronic device that is portable by a user and has communication and computing capabilities.

[0146] "Global positioning technology" refers to satellite systems and signal receiving technologies used to determine any location on Earth.

[0147] A "virtual boundary" refers to lines or edges that are superimposed on a real-world landscape to visually indicate a specific area within a space.

[0148] "Real-world visual information" refers to video and image information acquired from the physical landscape of the real world using cameras and various sensors.

[0149] "User emotional state" refers to the emotions and psychological state a user is experiencing at a particular moment, and is analyzed through methods such as facial recognition.

[0150] "Adjusting information display" means dynamically changing the type of information displayed and its visual presentation according to the user's situation and requests.

[0151] This invention provides an augmented reality system that allows users to visually perceive virtual boundaries in real space using a mobile device, and combines this with technology for recognizing the user's emotions. This makes it possible to further personalize the user experience.

[0152] The server first retrieves spatial information publicly available on the internet from a database. This information includes coordinate data and details about geographical areas. Specifically, it uses a "Geographic Information Service API" to retrieve the data. Since the retrieved data is difficult to use directly, it is converted into a format suitable for augmented reality display using a "3D engine platform" or similar. This conversion process includes polygonizing the data and generating meshes. The converted data is then quickly transmitted to the user's mobile device via a wireless network.

[0153] The device uses the Global Positioning System to determine the user's location in real time. Furthermore, the device uses a camera to acquire real-world visual information and overlays virtual boundaries onto that image. An "AR platform library" may be used for this display. The device also incorporates emotion recognition technology, analyzing the user's facial expression data acquired through the camera and sensors using "emotion analysis software." This determines the user's current emotions. Based on this emotion data, the device adjusts the displayed information to provide the most appropriate interface for the user's situation.

[0154] When a user uses this system, emotion recognition technology changes the AR display according to the user's emotional state. For example, if a user is surprised when trying to confirm a specific physical boundary, the system will take measures to attract the user's attention, such as enlarging summary information. Conversely, if the user is relaxed, the information display will be simplified, and the interface will be designed so that additional details can be easily viewed later.

[0155] Examples of prompts for a generative AI model include the following:

[0156] "Please demonstrate how to utilize an emotion engine to adjust information presentation based on the user's emotions."

[0157] "Please tell me about methods for visualizing geographical boundaries in augmented reality systems and how to optimize them."

[0158] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0159] Step 1:

[0160] The server retrieves spatial information publicly available on the internet. The input is a request for coordinate data concerning a specific geographic area. Based on this request, the server uses a geographic information service API to obtain detailed spatial data about the target area. The retrieved data is unconverted data, including coordinate and topographic information. The output of this step is this unconverted data.

[0161] Step 2:

[0162] The server converts the acquired unconverted data into a format that can be displayed in augmented reality. The input is the unconverted data acquired in step 1. Using the "3D engine platform," the data is polygonized and mesh generated, converting it into an advanced data format. The converted data is in a format suitable for a graphical user interface, and this is the output of step 2. Specifically, the data structure is rearranged within the server to create a format that can be computed efficiently.

[0163] Step 3:

[0164] The server transmits the converted data to the mobile device. The input is the advanced data format generated in step 2. The server transmits this data to the user's mobile device using a wireless network. The output is a data stream in a format usable by the mobile device. The server optimizes data transfer during this process to maintain transmission speed and signal quality.

[0165] Step 4:

[0166] The device uses the Global Positioning System to determine the user's location in real time. The input is location data from the device's GPS sensor. The device analyzes this location data and performs calculations to accurately determine the user's current location. The output is location information bound to the user interface. Specifically, the device periodically updates its location information to keep it always up-to-date.

[0167] Step 5:

[0168] The device uses a camera to acquire real-world visual information and overlays it with virtual boundaries. Inputs include the user's location information and a data stream received from a server. Utilizing an AR platform library, it performs a virtual overlay on the real-world image. The output is visual information in an augmented reality environment. Specifically, boundaries and terrain information are displayed in real time on the camera feed, helping the user intuitively understand the space.

[0169] Step 6:

[0170] The device uses a camera and sensors to acquire user facial expression data and analyzes it with emotion analysis software. The input is facial expression data from the camera. Based on this data, an emotion recognition algorithm analyzes the user's emotional state and generates information about it. The output is evaluation data regarding the user's emotional state. Specifically, the device dynamically adjusts the AR display according to the user's emotions.

[0171] Step 7:

[0172] Users take action based on information refined by the system. Inputs are feedback from the system and emotionally-driven emphasis. Users interact directly through the user interface and acquire additional information as needed. Outputs are the receipt of information optimized for the user, and concrete actions include making decisions based on the information provided.

[0173] (Application Example 2)

[0174] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0175] Conventional augmented reality systems have the challenge of displaying personalized information tailored to individual emotional states, as they display uniform information to all users. Furthermore, they are insufficient in optimizing the user experience based on the user's emotional state, and there is a particular need to address the sense of security and tension inherent in real estate transactions.

[0176] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0177] In this invention, the server includes means for acquiring publicly available geographic information, means for converting the acquired geographic information into a format suitable for augmented reality display, and means for transmitting the converted geographic information to a mobile terminal. This enables personalized augmented reality display on the mobile terminal that responds to the user's emotional state, thereby optimizing the user experience in specific fields such as real estate transactions.

[0178] "Publicly available geographic information" refers to geographical data that is accessible to the general public through the internet or public resources.

[0179] A "format suitable for augmented reality display" is data that has been converted into a format necessary for displaying it overlaid on real-world images.

[0180] A "mobile device" is a portable information processing device such as a smartphone or tablet.

[0181] The Global Positioning System (GSO) is a satellite system used to determine one's current location on Earth.

[0182] A "virtual boundary" is a digital geographical boundary that is superimposed onto real-world images.

[0183] An "emotion recognition engine" is an algorithm or software that analyzes emotions in real time from a customer's facial expressions and other data.

[0184] "Means for adjusting augmented reality displays according to the user's emotions" refers to a mechanism for dynamically changing the content and format of displayed information based on the user's emotional information.

[0185] The server retrieves geographic information publicly available via the internet and converts it into a format suitable for augmented reality display. The converted data is transmitted to mobile devices in real time. As the central hub for data processing, the server requires high-performance processing equipment and a stable network connection to enable efficient and rapid information delivery.

[0186] The device uses the Global Positioning System (GPS) to determine the user's location in real time, captures real-world images through its camera, and overlays virtual boundaries onto them. In addition, the device is equipped with an emotion recognition engine that analyzes the user's facial expressions captured by the camera in real time. Based on the emotion data obtained from this analysis, the displayed information is personalized.

[0187] The hardware used will be mobile devices such as smartphones and tablets, and the software will utilize augmented reality frameworks such as Google's ARCore and Apple's ARKit. For sentiment analysis, high-precision analysis will be achieved by using Google Cloud's Vision API and Microsoft Azure's Face API.

[0188] As a concrete example, during a real estate site visit, when a user is viewing a plot of land, the terminal automatically displays the site boundaries and provides detailed information if the user appears nervous. An example of a prompt message would be, "When the system detects that the user is nervous, please prioritize displaying detailed real estate information." This system allows users to receive the most appropriate information in real time, tailored to their situation.

[0189] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0190] Step 1:

[0191] The server retrieves geographic information published via the internet. The input is online geographic data, and the output is raw data stored in an internal database. This data is used for subsequent processing and is updated regularly using a stable communication line.

[0192] Step 2:

[0193] The server converts acquired geographic information into a format suitable for augmented reality display. The input is raw geographic data, and the output is converted AR format data. This step involves formatting and encoding the data to make it usable in AR frameworks.

[0194] Step 3:

[0195] The server transmits the converted geographic information to the mobile device. The input is data in AR format, and the output is the data that reaches the device's inbox. This process involves data compression and efficient transfer using a communication protocol.

[0196] Step 4:

[0197] The device uses the Global Positioning System (GPS) to determine the user's location. The input is a GPS signal, and the output is the coordinate data of the current location. Here, a GPS module is used to acquire real-time location information, which is then reflected in the subsequent video.

[0198] Step 5:

[0199] The device acquires real-world images through its camera and overlays them with virtual boundaries for display. The input consists of camera footage and coordinate data, while the output is an AR-displayed image. The AR framework precisely positions and overlays the virtual boundaries onto the camera footage.

[0200] Step 6:

[0201] The device's built-in emotion recognition engine analyzes the user's facial expressions. The input is a facial image of the user captured by the camera, and the output is emotion data. The emotion analysis algorithm determines the user's emotions in real time and expresses them numerically.

[0202] Step 7:

[0203] The device dynamically adjusts the augmented reality display according to the user's emotions. Input consists of emotion data and AR display data, and output is the adjusted AR display. If the emotional state is recognized as "tension," detailed information is displayed; when the user is at ease, it reverts to a concise information display. This automatic adjustment optimizes the user experience.

[0204] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0205] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0206] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0207] [Second Embodiment]

[0208] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0209] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0210] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0211] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0212] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0213] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0214] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0215] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0216] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0217] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0218] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0219] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0220] This invention provides a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device. This system is realized by integrating functions such as geographic information acquisition, data conversion, location determination, and augmented reality display.

[0221] First, the server retrieves publicly available geographical information online. This information includes land boundaries and map data. The server then uses a generative AI to analyze the retrieved information and convert it into a data format suitable for augmented reality display. The converted data is then stored in the format required by the mobile device.

[0222] Next, the server sends the converted geographic information data to the terminal. When the user starts up their mobile device with the dedicated application installed, the terminal receives this data. The user then uses this terminal to go to the field.

[0223] On-site, the device uses the Global Positioning System (GPS) to obtain accurate, real-time location information. Based on this location information, the device's AR rendering engine uses converted geographic data to overlay virtual boundaries onto the real-world camera footage.

[0224] By holding a mobile device and activating the camera to scan the terrain, a virtual boundary line based on publicly available geographical information is displayed on the screen. This allows users to visually confirm land boundaries in a concise and effective manner. Furthermore, users can manipulate the boundary information on the screen in detail and display additional information through the interface.

[0225] For example, in real estate transactions and land development, users can facilitate communication with landowners by visually confirming actual boundaries in advance. Furthermore, by obtaining visual and real-time feedback without relying on traditional maps or paper media, it is expected that the reliability and efficiency of map information will be improved.

[0226] The following describes the processing flow.

[0227] Step 1:

[0228] The server accesses an online geographic information database to retrieve the latest geographic information. The server uses an API to download data, including land boundary information.

[0229] Step 2:

[0230] The server uses a generative AI to analyze the acquired geographic information and convert it into a data format suitable for augmented reality display. The server then saves the conversion results as data for AR display.

[0231] Step 3:

[0232] The server sends the converted data to the user's mobile device. The server uses encryption protocols to ensure secure data transmission.

[0233] Step 4:

[0234] The device receives the transmitted geographic information data and prepares to read the data through the application. The user launches the dedicated application and performs the necessary initial setup.

[0235] Step 5:

[0236] The device uses the Global Positioning System (GPS) to obtain the user's current location in real time. To ensure highly accurate location information, the device receives signals from multiple satellites.

[0237] Step 6:

[0238] The device uses an AR rendering engine to overlay and display geographical boundaries converted from images acquired through the camera. This allows users to simultaneously view the real-world scenery and virtual boundaries.

[0239] Step 7:

[0240] The user observes the site with a mobile device and checks the boundaries displayed on the screen. The user can operate the interface to change the display mode or obtain detailed information.

[0241] (Example 1)

[0242] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0243] Conventional geographic data display systems lack a simple and accurate way for users to visually confirm geographical boundaries in real space. Furthermore, the display on mobile devices is not highly customizable, making it difficult for users to intuitively manipulate information according to their needs.

[0244] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0245] In this invention, the server includes means for collecting publicly available geographic data, means for utilizing an artificial intelligence model to convert the geographic data into a format suitable for augmented reality display, and means for communicating the converted geographic data to a mobile device. This allows users to accurately and visually confirm virtual boundaries in real geospace and customize them according to their needs.

[0246] "Publicly available geographic data" refers to geographical information resources that can be obtained from government agencies or publicly available platforms, and includes map data, land boundary information, topographic information, etc.

[0247] An "artificial intelligence model" refers to an information processing device that processes large amounts of data and exhibits capabilities equivalent to or exceeding those of a human in specific tasks, and includes machine learning and deep learning.

[0248] "Mobile devices" refer to portable electronic devices such as mobile phones and tablets that have wireless communication capabilities and computing power.

[0249] A "satellite positioning system" refers to a system that uses artificial satellites to provide accurate location information for any point on Earth, and includes GPS and similar technologies.

[0250] "Augmented reality display" refers to a technology that overlays digital information onto images of the real world, presenting information directly and in real time to the user.

[0251] A "virtually defined boundary" refers to a boundary line or area in real geographical space that is defined in digital data and presented in a way that the user can visually recognize.

[0252] "User operation" refers to instructions or actions performed through a user interface via a mobile device, and includes means that allow for intuitive operation.

[0253] This invention provides a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device. This system integrates functions for geographic data acquisition, data transformation, location determination, and augmented reality display.

[0254] First, the server collects publicly available geographic data. To do this, it accesses publicly available databases and web resources on the internet to obtain the latest map information and land boundary information. The server analyzes the acquired geographic data and uses an artificial intelligence model to convert the data into a format suitable for augmented reality display. Examples of artificial intelligence models used in this process include "large-scale language models." The server stores the converted geographic data on its own so that mobile devices can receive it.

[0255] Next, the server communicates the converted geographic data to the mobile device. A common data transfer protocol is used for communication, and encryption technology is employed to ensure a secure connection.

[0256] On-site, the device uses a satellite positioning system to determine its current location. The device is equipped with positioning technologies such as GPS, enabling accurate real-time location tracking of the user.

[0257] Subsequently, the device's AR rendering engine overlays the converted geographic information, based on the acquired location data, onto the real-world camera footage. This process utilizes an "augmented reality-enabled framework." For example, when a user activates the camera and points the device towards the surrounding terrain, virtual boundaries based on public geographic data are displayed on the screen.

[0258] Users can operate the mobile device's interface to fine-tune the displayed virtual boundaries and view related information. For example, it can be used during pre-inspection work at real estate sites to provide effective visual effects.

[0259] An example of a prompt message might be, "Retrieve the latest land boundary information for this area and convert it to a format suitable for AR display." This system will make it easier for users to verify information in the real world and enable rapid visualization of geographic data.

[0260] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0261] Step 1:

[0262] The server collects publicly available geographic data from the internet. Inputs include pre-specified map URLs and API keys. Web crawling technology is used for data collection. The obtained geographic data includes land boundary information and road information, and is output in a format for storage in a database.

[0263] Step 2:

[0264] The server analyzes the collected geographic data and converts it into a format suitable for augmented reality display. The input is the geographic data acquired in Step 1. A generative AI model is used for data conversion, performing noise reduction and data formatting. Specifically, latitude and longitude information is reconstructed into a 3D model format and converted into the Geo-JSON format used for real-time AR rendering. This converted data is then output in a format that mobile devices can receive.

[0265] Step 3:

[0266] The server communicates the converted geographic data to the mobile device. The data converted in step 2 is used as input. Encryption protocols such as TLS are used for communication to ensure secure data transfer. The output is provided in a format that can be stored in the mobile device's memory.

[0267] Step 4:

[0268] The terminal receives geographic data stored on the mobile device and then uses the Global Positioning System (GPS) to obtain its current location in real time. The input consists of the geographic data received in step 3 and the current GPS data. Based on this data, the terminal performs a position calculation to map the user's current location to the geographic data and outputs it as input to the AR rendering engine.

[0269] Step 5:

[0270] The device's AR rendering engine uses the results of position calculations to overlay a virtual boundary onto the real-world camera image. The input is the position data calculated in step 4. Specifically, it renders the boundary that should be visible from the user's viewpoint within the camera image in real time. The output is the AR display, which is shown on the mobile device's screen.

[0271] Step 6:

[0272] Users interact with the AR display on their mobile device to examine the displayed virtual boundary in detail. Input is provided through the device's interface. Specifically, users can adjust the displayed content using gestures such as pinching and swiping, and obtain additional information through voice input. The output provides visual information tailored to the user's intentions.

[0273] (Application Example 1)

[0274] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0275] Without relying on existing map information or paper media, there is a need to provide consumers and users with a means to recognize property boundaries and store layouts accurately in real time, thereby improving their shopping experience. In particular, by clearly understanding the layout and layout within stores, it is necessary to create a more convenient shopping environment for consumers and to provide store operators with efficient methods for layout changes and sales promotions.

[0276] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0277] In this invention, the server includes means for acquiring publicly available spatial information, means for converting the spatial information into a format suitable for augmented realization display using data processing technology, and means for transmitting the converted spatial information to a communication device. This enables consumers to use the communication device to recognize information about the area inside and around the store in real time, thereby improving the shopping experience.

[0278] "Spatial information" refers to data related to geographical location and extent, including information such as real estate and the layout of a store.

[0279] "Data processing technology" refers to the technology of analyzing and transforming data according to a specific purpose, and in this invention, it refers to the transformation into a data format for extended realization and display.

[0280] "Extended Reality Display" refers to the technology of superimposing virtual information on real images, which is a form of AR (Augmented Reality).

[0281] "Communication device" refers to a device capable of transmitting and receiving digital information, and in this invention, it mainly refers to a mobile terminal.

[0282] "Positioning system" refers to the technology of accurately measuring physical positions, and the Global Positioning System (GPS) is an example of it.

[0283] "Virtual boundary line" refers to a line that does not physically exist but is displayed as information, and it visually presents the boundary of a specific area to the user.

[0284] "Store partition information" refers to information related to the division and arrangement of spaces within a store, and it is utilized in the design of product displays and customer flow.

[0285] The system for implementing this invention mainly consists of three entities: a server, a communication device (mobile terminal), and a user. First, the server obtains publicly available spatial information from the Internet. For this acquisition, a dedicated API or the like can be used. Then, the server converts the acquired spatial information into a format suitable for extended reality display using data processing technology. In this conversion process, a generative AI model is used to format the data into a data format for AR display in real time. The converted data is saved so that it can be quickly transmitted to the communication device.

[0286] The mobile terminal receives the converted data from the server through the network. At that time, the terminal accurately identifies its own position using the Global Positioning System (GPS), and based on the obtained position information, it superimposes a virtual boundary line on the real camera image. As a result, the user can visually confirm spatial and real estate partition information in real time.

[0287] Users can use this mobile device to activate the camera at specific locations inside and outside the store, displaying virtual area information along with the video feed. For example, when introducing a new product section, this system can be used to check the store layout and determine the optimal placement of spaces. Based on a prompt message such as, "Please convert the surrounding geographic data to AR format to suggest the optimal store layout," a generative AI model can be used to present information in a practical and intuitive way for the user.

[0288] One specific use case is when a store's design team is selecting a location for a new display. The user can use the terminal to quickly obtain the information needed to achieve the most efficient product placement while viewing the available space within the store.

[0289] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0290] Step 1:

[0291] The server first retrieves publicly available spatial information from the internet. An internet connection is required as input, and requests are made to endpoints such as geographic information APIs. The output is spatial data including land parcel and boundary information.

[0292] Step 2:

[0293] The server transforms the acquired spatial information using data processing technology. Using a generative AI model, it converts the spatial data obtained as input into a data format suitable for augmented reality display. The output is a data format for AR display. During this process, prompt statements are used to control the behavior of the AI ​​model.

[0294] Step 3:

[0295] The server transmits the converted data to the communication device. The converted data is used as input, and the data is sent to the mobile terminal as output. A network connection is required for communication.

[0296] Step 4:

[0297] The device uses the Global Positioning System (GPS) to determine its current location. It receives GPS signals as input and obtains precise latitude and longitude location information as output.

[0298] Step 5:

[0299] The device uses location information obtained from GPS and converted data received from a server to overlay a virtual boundary onto the real-world camera image. Using camera footage and AR data as input, the output is a visually superimposed image.

[0300] Step 6:

[0301] The user utilizes the camera function of their device to view virtual area information presented by the application. The input requires real-world video captured by the device, and the output provides a visual display including virtual information overlaid on the camera footage. This allows the user to perform spatial recognition and confirm the necessary information.

[0302] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0303] This invention realizes a next-generation AR system that combines a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device with an emotion engine that recognizes the user's emotions. This makes it possible to further personalize the user experience.

[0304] First, the system has the server acquire the geographical information that is publicly available online and convert it into a format suitable for AR display. Next, the server transmits the converted data to the user's mobile terminal. The user has a mobile terminal installed with a dedicated app and starts it at the site.

[0305] The terminal uses the Global Positioning System (GPS) to identify the user's position in real time and overlays virtual boundary lines on the real-world video captured by the camera for display. This allows the user to visually confirm the exact land boundaries through the camera. Additionally, the terminal is equipped with an emotion engine that recognizes the user's emotions by analyzing the user's facial expressions. The emotion data is collected through the terminal's camera and sensors and analyzed in real time.

[0306] When the user uses this system, the AR display changes according to the user's emotional state by the emotion engine. For example, when the user is surprised, additional information is highlighted, or when the user is relaxed, adjustments are made to maintain a simple display. Through this interactive adjustment, the user can receive the information most suitable for their state and can effectively proceed with land confirmation and transactions.

[0307] As a specific example, at the site of a land transaction, if the system recognizes that the user is feeling stressed, it can help the user's understanding by redisplaying the complex geographical information that is considered the cause in an easy-to-understand manner. Also, when the user is at ease, while maintaining the standard mode, the user experience can be optimized by making it easy to access further detailed information.

[0308] The processing flow will be described below.

[0309] Step 1:

[0310] The server accesses an online geographic information database and retrieves the latest geographic information according to the user's request. The server uses an API to download the necessary data, including land boundary information.

[0311] Step 2:

[0312] The server analyzes the acquired geographic information using a generation AI and converts it into a format suitable for augmented reality display. This conversion process formats the geographic information into a format usable by the AR rendering engine.

[0313] Step 3:

[0314] The server transmits the converted geographical information to the user's mobile device. This transmission is carried out through a secure communication protocol, ensuring that accurate data is delivered to the user's device.

[0315] Step 4:

[0316] The device uses the Global Positioning System (GPS) to obtain its current location in real time. The device utilizes multiple satellite signals to ensure centimeter-level positional accuracy.

[0317] Step 5:

[0318] The device activates its camera and captures real-world images. The AR rendering engine overlays virtual boundaries onto the camera images, providing the user with visual information.

[0319] Step 6:

[0320] The device activates an emotion engine and collects emotional data from the user's facial expressions and voice through the camera and other sensors. The emotion engine analyzes this data to determine the user's emotional state.

[0321] Step 7:

[0322] The device dynamically adjusts the AR display content based on the analysis results of the emotion engine. For example, if the system determines that the user is confused, it simplifies the displayed information and provides navigation to aid understanding. Furthermore, the color scheme and emphasis of information change according to the user's emotions.

[0323] Step 8:

[0324] Users observe the site using a mobile device and check the boundaries and information displayed on the screen. Through the interface, users can access detailed information and change the display mode. This allows users to easily grasp the necessary information regarding land boundaries.

[0325] (Example 2)

[0326] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0327] The present invention aims to provide an augmented reality system that allows users to easily visualize geographical boundaries in the real world. In particular, it aims to solve the problem of improving the user experience by dynamically responding to the user's emotional state and displaying information according to the situation.

[0328] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0329] In this invention, the server includes means for acquiring publicly available spatial information, means for converting the acquired spatial information into a format suitable for augmented reality display, and means for transmitting the converted spatial information to a mobile terminal. This allows the user to easily perceive virtual boundaries on real-world visual information and to adjust the information display based on the user's emotional state.

[0330] "Publicly available spatial information" refers to information about physical spaces that is available in open databases such as the internet.

[0331] "A format suitable for augmented reality display" refers to the data structure necessary to display a combination of the user's real-world visual information and digital information.

[0332] A "mobile device" refers to an electronic device that is portable by a user and has communication and computing capabilities.

[0333] "Global positioning technology" refers to satellite systems and signal receiving technologies used to determine any location on Earth.

[0334] A "virtual boundary" refers to lines or edges that are superimposed on a real-world landscape to visually indicate a specific area within a space.

[0335] "Real-world visual information" refers to video and image information acquired from the physical landscape of the real world using cameras and various sensors.

[0336] "User emotional state" refers to the emotions and psychological state a user is experiencing at a particular moment, and is analyzed through methods such as facial recognition.

[0337] "Adjusting information display" means dynamically changing the type of information displayed and its visual presentation according to the user's situation and requests.

[0338] This invention provides an augmented reality system that allows users to visually perceive virtual boundaries in real space using a mobile device, and combines this with technology for recognizing the user's emotions. This makes it possible to further personalize the user experience.

[0339] The server first retrieves spatial information publicly available on the internet from a database. This information includes coordinate data and details about geographical areas. Specifically, it uses a "Geographic Information Service API" to retrieve the data. Since the retrieved data is difficult to use directly, it is converted into a format suitable for augmented reality display using a "3D engine platform" or similar. This conversion process includes polygonizing the data and generating meshes. The converted data is then quickly transmitted to the user's mobile device via a wireless network.

[0340] The device uses the Global Positioning System to determine the user's location in real time. Furthermore, the device uses a camera to acquire real-world visual information and overlays virtual boundaries onto that image. An "AR platform library" may be used for this display. The device also incorporates emotion recognition technology, analyzing the user's facial expression data acquired through the camera and sensors using "emotion analysis software." This determines the user's current emotions. Based on this emotion data, the device adjusts the displayed information to provide the most appropriate interface for the user's situation.

[0341] When a user uses this system, emotion recognition technology changes the AR display according to the user's emotional state. For example, if a user is surprised when trying to confirm a specific physical boundary, the system will take measures to attract the user's attention, such as enlarging summary information. Conversely, if the user is relaxed, the information display will be simplified, and the interface will be designed so that additional details can be easily viewed later.

[0342] Examples of prompts for a generative AI model include the following:

[0343] "Please demonstrate how to utilize an emotion engine to adjust information presentation based on the user's emotions."

[0344] "Please tell me about methods for visualizing geographical boundaries in augmented reality systems and how to optimize them."

[0345] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0346] Step 1:

[0347] The server retrieves spatial information publicly available on the internet. The input is a request for coordinate data concerning a specific geographic area. Based on this request, the server uses a geographic information service API to obtain detailed spatial data about the target area. The retrieved data is unconverted data, including coordinate and topographic information. The output of this step is this unconverted data.

[0348] Step 2:

[0349] The server converts the acquired unconverted data into a format that can be displayed in augmented reality. The input is the unconverted data acquired in step 1. Using the "3D engine platform," the data is polygonized and mesh generated, converting it into an advanced data format. The converted data is in a format suitable for a graphical user interface, and this is the output of step 2. Specifically, the data structure is rearranged within the server to create a format that can be computed efficiently.

[0350] Step 3:

[0351] The server transmits the converted data to the mobile device. The input is the advanced data format generated in step 2. The server transmits this data to the user's mobile device using a wireless network. The output is a data stream in a format usable by the mobile device. The server optimizes data transfer during this process to maintain transmission speed and signal quality.

[0352] Step 4:

[0353] The device uses the Global Positioning System to determine the user's location in real time. The input is location data from the device's GPS sensor. The device analyzes this location data and performs calculations to accurately determine the user's current location. The output is location information bound to the user interface. Specifically, the device periodically updates its location information to keep it always up-to-date.

[0354] Step 5:

[0355] The device uses a camera to acquire real-world visual information and overlays it with virtual boundaries. Inputs include the user's location information and a data stream received from a server. Utilizing an AR platform library, it performs a virtual overlay on the real-world image. The output is visual information in an augmented reality environment. Specifically, boundaries and terrain information are displayed in real time on the camera feed, helping the user intuitively understand the space.

[0356] Step 6:

[0357] The device uses a camera and sensors to acquire user facial expression data and analyzes it with emotion analysis software. The input is facial expression data from the camera. Based on this data, an emotion recognition algorithm analyzes the user's emotional state and generates information about it. The output is evaluation data regarding the user's emotional state. Specifically, the device dynamically adjusts the AR display according to the user's emotions.

[0358] Step 7:

[0359] Users take action based on information refined by the system. Inputs are feedback from the system and emotionally-driven emphasis. Users interact directly through the user interface and acquire additional information as needed. Outputs are the receipt of information optimized for the user, and concrete actions include making decisions based on the information provided.

[0360] (Application Example 2)

[0361] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0362] Conventional augmented reality systems have the challenge of displaying personalized information tailored to individual emotional states, as they display uniform information to all users. Furthermore, they are insufficient in optimizing the user experience based on the user's emotional state, and there is a particular need to address the sense of security and tension inherent in real estate transactions.

[0363] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0364] In this invention, the server includes means for acquiring publicly available geographic information, means for converting the acquired geographic information into a format suitable for augmented reality display, and means for transmitting the converted geographic information to a mobile terminal. This enables personalized augmented reality display on the mobile terminal that responds to the user's emotional state, thereby optimizing the user experience in specific fields such as real estate transactions.

[0365] "Publicly available geographic information" refers to geographical data that is accessible to the general public through the internet or public resources.

[0366] A "format suitable for augmented reality display" is data that has been converted into a format necessary for displaying it overlaid on real-world images.

[0367] A "mobile device" is a portable information processing device such as a smartphone or tablet.

[0368] The Global Positioning System (GSO) is a satellite system used to determine one's current location on Earth.

[0369] A "virtual boundary" is a digital geographical boundary that is superimposed onto real-world images.

[0370] An "emotion recognition engine" is an algorithm or software that analyzes emotions in real time from a customer's facial expressions and other data.

[0371] "Means for adjusting augmented reality displays according to the user's emotions" refers to a mechanism for dynamically changing the content and format of displayed information based on the user's emotional information.

[0372] The server retrieves geographic information publicly available via the internet and converts it into a format suitable for augmented reality display. The converted data is transmitted to mobile devices in real time. As the central hub for data processing, the server requires high-performance processing equipment and a stable network connection to enable efficient and rapid information delivery.

[0373] The device uses the Global Positioning System (GPS) to determine the user's location in real time, captures real-world images through its camera, and overlays virtual boundaries onto them. In addition, the device is equipped with an emotion recognition engine that analyzes the user's facial expressions captured by the camera in real time. Based on the emotion data obtained from this analysis, the displayed information is personalized.

[0374] The hardware used will be mobile devices such as smartphones and tablets, and the software will utilize augmented reality frameworks such as Google's ARCore and Apple's ARKit. For sentiment analysis, high-precision analysis will be achieved by using Google Cloud's Vision API and Microsoft Azure's Face API.

[0375] As a concrete example, during a real estate site visit, when a user is viewing a plot of land, the terminal automatically displays the site boundaries and provides detailed information if the user appears nervous. An example of a prompt message would be, "When the system detects that the user is nervous, please prioritize displaying detailed real estate information." This system allows users to receive the most appropriate information in real time, tailored to their situation.

[0376] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0377] Step 1:

[0378] The server retrieves geographic information published via the internet. The input is online geographic data, and the output is raw data stored in an internal database. This data is used for subsequent processing and is updated regularly using a stable communication line.

[0379] Step 2:

[0380] The server converts acquired geographic information into a format suitable for augmented reality display. The input is raw geographic data, and the output is converted AR format data. This step involves formatting and encoding the data to make it usable in AR frameworks.

[0381] Step 3:

[0382] The server transmits the converted geographic information to the mobile device. The input is data in AR format, and the output is the data that reaches the device's inbox. This process involves data compression and efficient transfer using a communication protocol.

[0383] Step 4:

[0384] The device uses the Global Positioning System (GPS) to determine the user's location. The input is a GPS signal, and the output is the coordinate data of the current location. Here, a GPS module is used to acquire real-time location information, which is then reflected in the subsequent video.

[0385] Step 5:

[0386] The device acquires real-world images through its camera and overlays them with virtual boundaries for display. The input consists of camera footage and coordinate data, while the output is an AR-displayed image. The AR framework precisely positions and overlays the virtual boundaries onto the camera footage.

[0387] Step 6:

[0388] The device's built-in emotion recognition engine analyzes the user's facial expressions. The input is a facial image of the user captured by the camera, and the output is emotion data. The emotion analysis algorithm determines the user's emotions in real time and expresses them numerically.

[0389] Step 7:

[0390] The device dynamically adjusts the augmented reality display according to the user's emotions. Input consists of emotion data and AR display data, and output is the adjusted AR display. If the emotional state is recognized as "tension," detailed information is displayed; when the user is at ease, it reverts to a concise information display. This automatic adjustment optimizes the user experience.

[0391] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0392] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0393] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0394] [Third Embodiment]

[0395] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0396] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0397] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0398] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0399] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0400] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0401] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0402] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0403] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0404] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0405] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0406] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0407] This invention provides a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device. This system is realized by integrating functions such as geographic information acquisition, data conversion, location determination, and augmented reality display.

[0408] First, the server retrieves publicly available geographical information online. This information includes land boundaries and map data. The server then uses a generative AI to analyze the retrieved information and convert it into a data format suitable for augmented reality display. The converted data is then stored in the format required by the mobile device.

[0409] Next, the server sends the converted geographic information data to the terminal. When the user starts up their mobile device with the dedicated application installed, the terminal receives this data. The user then uses this terminal to go to the field.

[0410] On-site, the device uses the Global Positioning System (GPS) to obtain accurate, real-time location information. Based on this location information, the device's AR rendering engine uses converted geographic data to overlay virtual boundaries onto the real-world camera footage.

[0411] By holding a mobile device and activating the camera to scan the terrain, a virtual boundary line based on publicly available geographical information is displayed on the screen. This allows users to visually confirm land boundaries in a concise and effective manner. Furthermore, users can manipulate the boundary information on the screen in detail and display additional information through the interface.

[0412] For example, in real estate transactions and land development, users can facilitate communication with landowners by visually confirming actual boundaries in advance. Furthermore, by obtaining visual and real-time feedback without relying on traditional maps or paper media, it is expected that the reliability and efficiency of map information will be improved.

[0413] The following describes the processing flow.

[0414] Step 1:

[0415] The server accesses an online geographic information database to retrieve the latest geographic information. The server uses an API to download data, including land boundary information.

[0416] Step 2:

[0417] The server uses a generative AI to analyze the acquired geographic information and convert it into a data format suitable for augmented reality display. The server then saves the conversion results as data for AR display.

[0418] Step 3:

[0419] The server sends the converted data to the user's mobile device. The server uses encryption protocols to ensure secure data transmission.

[0420] Step 4:

[0421] The device receives the transmitted geographic information data and prepares to read the data through the application. The user launches the dedicated application and performs the necessary initial setup.

[0422] Step 5:

[0423] The device uses the Global Positioning System (GPS) to obtain the user's current location in real time. To ensure highly accurate location information, the device receives signals from multiple satellites.

[0424] Step 6:

[0425] The device uses an AR rendering engine to overlay and display geographical boundaries converted from images acquired through the camera. This allows users to simultaneously view the real-world scenery and virtual boundaries.

[0426] Step 7:

[0427] The user observes the site with a mobile device and checks the boundaries displayed on the screen. The user can operate the interface to change the display mode or obtain detailed information.

[0428] (Example 1)

[0429] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0430] Conventional geographic data display systems lack a simple and accurate way for users to visually confirm geographical boundaries in real space. Furthermore, the display on mobile devices is not highly customizable, making it difficult for users to intuitively manipulate information according to their needs.

[0431] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0432] In this invention, the server includes means for collecting publicly available geographic data, means for utilizing an artificial intelligence model to convert the geographic data into a format suitable for augmented reality display, and means for communicating the converted geographic data to a mobile device. This allows users to accurately and visually confirm virtual boundaries in real geospace and customize them according to their needs.

[0433] "Publicly available geographic data" refers to geographical information resources that can be obtained from government agencies or publicly available platforms, and includes map data, land boundary information, topographic information, etc.

[0434] An "artificial intelligence model" refers to an information processing device that processes large amounts of data and exhibits capabilities equivalent to or exceeding those of a human in specific tasks, and includes machine learning and deep learning.

[0435] "Mobile devices" refer to portable electronic devices such as mobile phones and tablets that have wireless communication capabilities and computing power.

[0436] A "satellite positioning system" refers to a system that uses artificial satellites to provide accurate location information for any point on Earth, and includes GPS and similar technologies.

[0437] "Augmented reality display" refers to a technology that overlays digital information onto images of the real world, presenting information directly and in real time to the user.

[0438] A "virtually defined boundary" refers to a boundary line or area in real geographical space that is defined in digital data and presented in a way that the user can visually recognize.

[0439] "User operation" refers to instructions or actions performed through a user interface via a mobile device, and includes means that allow for intuitive operation.

[0440] This invention provides a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device. This system integrates functions for geographic data acquisition, data transformation, location determination, and augmented reality display.

[0441] First, the server collects publicly available geographic data. To do this, it accesses publicly available databases and web resources on the internet to obtain the latest map information and land boundary information. The server analyzes the acquired geographic data and uses an artificial intelligence model to convert the data into a format suitable for augmented reality display. Examples of artificial intelligence models used in this process include "large-scale language models." The server stores the converted geographic data on its own so that mobile devices can receive it.

[0442] Next, the server communicates the converted geographic data to the mobile device. A common data transfer protocol is used for communication, and encryption technology is employed to ensure a secure connection.

[0443] On-site, the device uses a satellite positioning system to determine its current location. The device is equipped with positioning technologies such as GPS, enabling accurate real-time location tracking of the user.

[0444] Subsequently, the device's AR rendering engine overlays the converted geographic information, based on the acquired location data, onto the real-world camera footage. This process utilizes an "augmented reality-enabled framework." For example, when a user activates the camera and points the device towards the surrounding terrain, virtual boundaries based on public geographic data are displayed on the screen.

[0445] Users can operate the mobile device's interface to fine-tune the displayed virtual boundaries and view related information. For example, it can be used during pre-inspection work at real estate sites to provide effective visual effects.

[0446] An example of a prompt message might be, "Retrieve the latest land boundary information for this area and convert it to a format suitable for AR display." This system will make it easier for users to verify information in the real world and enable rapid visualization of geographic data.

[0447] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0448] Step 1:

[0449] The server collects publicly available geographic data from the internet. Inputs include pre-specified map URLs and API keys. Web crawling technology is used for data collection. The obtained geographic data includes land boundary information and road information, and is output in a format for storage in a database.

[0450] Step 2:

[0451] The server analyzes the collected geographic data and converts it into a format suitable for augmented reality display. The input is the geographic data acquired in Step 1. A generative AI model is used for data conversion, performing noise reduction and data formatting. Specifically, latitude and longitude information is reconstructed into a 3D model format and converted into the Geo-JSON format used for real-time AR rendering. This converted data is then output in a format that mobile devices can receive.

[0452] Step 3:

[0453] The server communicates the converted geographic data to the mobile device. The data converted in step 2 is used as input. Encryption protocols such as TLS are used for communication to ensure secure data transfer. The output is provided in a format that can be stored in the mobile device's memory.

[0454] Step 4:

[0455] The terminal receives geographic data stored on the mobile device and then uses the Global Positioning System (GPS) to obtain its current location in real time. The input consists of the geographic data received in step 3 and the current GPS data. Based on this data, the terminal performs a position calculation to map the user's current location to the geographic data and outputs it as input to the AR rendering engine.

[0456] Step 5:

[0457] The device's AR rendering engine uses the results of position calculations to overlay a virtual boundary onto the real-world camera image. The input is the position data calculated in step 4. Specifically, it renders the boundary that should be visible from the user's viewpoint within the camera image in real time. The output is the AR display, which is shown on the mobile device's screen.

[0458] Step 6:

[0459] Users interact with the AR display on their mobile device to examine the displayed virtual boundary in detail. Input is provided through the device's interface. Specifically, users can adjust the displayed content using gestures such as pinching and swiping, and obtain additional information through voice input. The output provides visual information tailored to the user's intentions.

[0460] (Application Example 1)

[0461] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0462] Without relying on existing map information or paper media, there is a need to provide consumers and users with a means to recognize property boundaries and store layouts accurately in real time, thereby improving their shopping experience. In particular, by clearly understanding the layout and layout within stores, it is necessary to create a more convenient shopping environment for consumers and to provide store operators with efficient methods for layout changes and sales promotions.

[0463] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0464] In this invention, the server includes means for acquiring publicly available spatial information, means for converting the spatial information into a format suitable for augmented realization display using data processing technology, and means for transmitting the converted spatial information to a communication device. This enables consumers to use the communication device to recognize information about the area inside and around the store in real time, thereby improving the shopping experience.

[0465] "Spatial information" refers to data related to geographical location and extent, including information such as real estate and the layout of a store.

[0466] "Data processing technology" refers to the technology of analyzing and transforming data according to a specific purpose, and in this invention, it refers to the transformation into a data format for extended realization and display.

[0467] "Augmented Reality Display" refers to a technology that overlays virtual information onto real-world images, and is a form of AR (Augmented Reality).

[0468] "Communication equipment" refers to a device capable of sending and receiving digital information, and in this invention, it mainly refers to a portable terminal.

[0469] A "location system" is a technology that accurately measures physical location, and the Global Positioning System (GPS) is one example.

[0470] A "virtual boundary" is a line that does not physically exist but is displayed as information, visually presenting the user with the boundaries of a specific area.

[0471] "Store layout information" refers to information about the division and arrangement of space within a store, and is used in designing product displays and customer flow.

[0472] The system implementing this invention primarily consists of three components: a server, communication equipment (mobile terminal), and a user. The server first acquires publicly available spatial information from the internet. A dedicated API can be used for this acquisition. Subsequently, the server converts the acquired spatial information into a format suitable for augmented reality display using data processing technology. This conversion process uses a generative AI model to format the data into an AR display data format in real time. The converted data is stored so that it can be quickly transmitted to the communication equipment.

[0473] Mobile devices receive converted data from a server via the network. During this process, the device precisely determines its location using the Global Positioning System (GPS), and based on this location information, overlays virtual boundaries onto real-world camera footage. This allows users to visually confirm spatial and real estate parcel information in real time.

[0474] Users can use this mobile device to activate the camera at specific locations inside and outside the store, displaying virtual area information along with the video feed. For example, when introducing a new product section, this system can be used to check the store layout and determine the optimal placement of spaces. Based on a prompt message such as, "Please convert the surrounding geographic data to AR format to suggest the optimal store layout," a generative AI model can be used to present information in a practical and intuitive way for the user.

[0475] One specific use case is when a store's design team is selecting a location for a new display. The user can use the terminal to quickly obtain the information needed to achieve the most efficient product placement while viewing the available space within the store.

[0476] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0477] Step 1:

[0478] The server first retrieves publicly available spatial information from the internet. An internet connection is required as input, and requests are made to endpoints such as geographic information APIs. The output is spatial data including land parcel and boundary information.

[0479] Step 2:

[0480] The server transforms the acquired spatial information using data processing technology. Using a generative AI model, it converts the spatial data obtained as input into a data format suitable for augmented reality display. The output is a data format for AR display. During this process, prompt statements are used to control the behavior of the AI ​​model.

[0481] Step 3:

[0482] The server transmits the converted data to the communication device. The converted data is used as input, and the data is sent to the mobile terminal as output. A network connection is required for communication.

[0483] Step 4:

[0484] The device uses the Global Positioning System (GPS) to determine its current location. It receives GPS signals as input and obtains precise latitude and longitude location information as output.

[0485] Step 5:

[0486] The device uses location information obtained from GPS and converted data received from a server to overlay a virtual boundary onto the real-world camera image. Using camera footage and AR data as input, the output is a visually superimposed image.

[0487] Step 6:

[0488] The user utilizes the camera function of their device to view virtual area information presented by the application. The input requires real-world video captured by the device, and the output provides a visual display including virtual information overlaid on the camera footage. This allows the user to perform spatial recognition and confirm the necessary information.

[0489] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0490] This invention realizes a next-generation AR system that combines a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device with an emotion engine that recognizes the user's emotions. This makes it possible to further personalize the user experience.

[0491] This system first uses a server to retrieve geographic information publicly available online and convert it into a format suitable for AR display. Next, the server sends the converted data to the user's mobile device. The user then uses a mobile device with a dedicated app installed to activate the system at the site.

[0492] The device uses the Global Positioning System (GPS) to determine the user's location in real time and overlays virtual boundaries onto real-world images captured by the camera. This allows the user to visually confirm accurate land boundaries through the camera. Furthermore, the device is equipped with an emotion engine that recognizes the user's emotions by analyzing their facial expressions. Emotional data is collected through the device's camera and sensors and analyzed in real time.

[0493] When a user uses this system, the emotion engine changes the AR display according to the user's emotional state. For example, if the user is surprised, additional information is highlighted, while if they are relaxed, a simpler display is maintained. This interactive adjustment allows users to receive information that is most relevant to their state, enabling them to effectively proceed with land inspections and transactions.

[0494] For example, in a land transaction, if the system detects that a user is experiencing stress, it can help the user understand by re-displaying the complex geographical information that is believed to be the cause of the stress in an easy-to-understand format. Furthermore, when the user is at ease, the system maintains the standard mode while providing easy access to more detailed information, thereby optimizing the user experience.

[0495] The following describes the processing flow.

[0496] Step 1:

[0497] The server accesses an online geographic information database and retrieves the latest geographic information according to the user's request. The server uses an API to download the necessary data, including land boundary information.

[0498] Step 2:

[0499] The server analyzes the acquired geographic information using a generation AI and converts it into a format suitable for augmented reality display. This conversion process formats the geographic information into a format usable by the AR rendering engine.

[0500] Step 3:

[0501] The server transmits the converted geographical information to the user's mobile device. This transmission is carried out through a secure communication protocol, ensuring that accurate data is delivered to the user's device.

[0502] Step 4:

[0503] The device uses the Global Positioning System (GPS) to obtain its current location in real time. The device utilizes multiple satellite signals to ensure centimeter-level positional accuracy.

[0504] Step 5:

[0505] The device activates its camera and captures real-world images. The AR rendering engine overlays virtual boundaries onto the camera images, providing the user with visual information.

[0506] Step 6:

[0507] The device activates an emotion engine and collects emotional data from the user's facial expressions and voice through the camera and other sensors. The emotion engine analyzes this data to determine the user's emotional state.

[0508] Step 7:

[0509] The device dynamically adjusts the AR display content based on the analysis results of the emotion engine. For example, if the system determines that the user is confused, it simplifies the displayed information and provides navigation to aid understanding. Furthermore, the color scheme and emphasis of information change according to the user's emotions.

[0510] Step 8:

[0511] Users observe the site using a mobile device and check the boundaries and information displayed on the screen. Through the interface, users can access detailed information and change the display mode. This allows users to easily grasp the necessary information regarding land boundaries.

[0512] (Example 2)

[0513] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0514] The present invention aims to provide an augmented reality system that allows users to easily visualize geographical boundaries in the real world. In particular, it aims to solve the problem of improving the user experience by dynamically responding to the user's emotional state and displaying information according to the situation.

[0515] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0516] In this invention, the server includes means for acquiring publicly available spatial information, means for converting the acquired spatial information into a format suitable for augmented reality display, and means for transmitting the converted spatial information to a mobile terminal. This allows the user to easily perceive virtual boundaries on real-world visual information and to adjust the information display based on the user's emotional state.

[0517] "Publicly available spatial information" refers to information about physical spaces that is available in open databases such as the internet.

[0518] "A format suitable for augmented reality display" refers to the data structure necessary to display a combination of the user's real-world visual information and digital information.

[0519] A "mobile device" refers to an electronic device that is portable by a user and has communication and computing capabilities.

[0520] "Global positioning technology" refers to satellite systems and signal receiving technologies used to determine any location on Earth.

[0521] A "virtual boundary" refers to lines or edges that are superimposed on a real-world landscape to visually indicate a specific area within a space.

[0522] "Real-world visual information" refers to video and image information acquired from the physical landscape of the real world using cameras and various sensors.

[0523] "User emotional state" refers to the emotions and psychological state a user is experiencing at a particular moment, and is analyzed through methods such as facial recognition.

[0524] "Adjusting information display" means dynamically changing the type of information displayed and its visual presentation according to the user's situation and requests.

[0525] This invention provides an augmented reality system that allows users to visually perceive virtual boundaries in real space using a mobile device, and combines this with technology for recognizing the user's emotions. This makes it possible to further personalize the user experience.

[0526] The server first retrieves spatial information publicly available on the internet from a database. This information includes coordinate data and details about geographical areas. Specifically, it uses a "Geographic Information Service API" to retrieve the data. Since the retrieved data is difficult to use directly, it is converted into a format suitable for augmented reality display using a "3D engine platform" or similar. This conversion process includes polygonizing the data and generating meshes. The converted data is then quickly transmitted to the user's mobile device via a wireless network.

[0527] The device uses the Global Positioning System to determine the user's location in real time. Furthermore, the device uses a camera to acquire real-world visual information and overlays virtual boundaries onto that image. An "AR platform library" may be used for this display. The device also incorporates emotion recognition technology, analyzing the user's facial expression data acquired through the camera and sensors using "emotion analysis software." This determines the user's current emotions. Based on this emotion data, the device adjusts the displayed information to provide the most appropriate interface for the user's situation.

[0528] When a user uses this system, emotion recognition technology changes the AR display according to the user's emotional state. For example, if a user is surprised when trying to confirm a specific physical boundary, the system will take measures to attract the user's attention, such as enlarging summary information. Conversely, if the user is relaxed, the information display will be simplified, and the interface will be designed so that additional details can be easily viewed later.

[0529] Examples of prompts for a generative AI model include the following:

[0530] "Please demonstrate how to utilize an emotion engine to adjust information presentation based on the user's emotions."

[0531] "Please tell me about methods for visualizing geographical boundaries in augmented reality systems and how to optimize them."

[0532] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0533] Step 1:

[0534] The server retrieves spatial information publicly available on the internet. The input is a request for coordinate data concerning a specific geographic area. Based on this request, the server uses a geographic information service API to obtain detailed spatial data about the target area. The retrieved data is unconverted data, including coordinate and topographic information. The output of this step is this unconverted data.

[0535] Step 2:

[0536] The server converts the acquired unconverted data into a format that can be displayed in augmented reality. The input is the unconverted data acquired in step 1. Using the "3D engine platform," the data is polygonized and mesh generated, converting it into an advanced data format. The converted data is in a format suitable for a graphical user interface, and this is the output of step 2. Specifically, the data structure is rearranged within the server to create a format that can be computed efficiently.

[0537] Step 3:

[0538] The server transmits the converted data to the mobile device. The input is the advanced data format generated in step 2. The server transmits this data to the user's mobile device using a wireless network. The output is a data stream in a format usable by the mobile device. The server optimizes data transfer during this process to maintain transmission speed and signal quality.

[0539] Step 4:

[0540] The device uses the Global Positioning System to determine the user's location in real time. The input is location data from the device's GPS sensor. The device analyzes this location data and performs calculations to accurately determine the user's current location. The output is location information bound to the user interface. Specifically, the device periodically updates its location information to keep it always up-to-date.

[0541] Step 5:

[0542] The device uses a camera to acquire real-world visual information and overlays it with virtual boundaries. Inputs include the user's location information and a data stream received from a server. Utilizing an AR platform library, it performs a virtual overlay on the real-world image. The output is visual information in an augmented reality environment. Specifically, boundaries and terrain information are displayed in real time on the camera feed, helping the user intuitively understand the space.

[0543] Step 6:

[0544] The device uses a camera and sensors to acquire user facial expression data and analyzes it with emotion analysis software. The input is facial expression data from the camera. Based on this data, an emotion recognition algorithm analyzes the user's emotional state and generates information about it. The output is evaluation data regarding the user's emotional state. Specifically, the device dynamically adjusts the AR display according to the user's emotions.

[0545] Step 7:

[0546] Users take action based on information refined by the system. Inputs are feedback from the system and emotionally-driven emphasis. Users interact directly through the user interface and acquire additional information as needed. Outputs are the receipt of information optimized for the user, and concrete actions include making decisions based on the information provided.

[0547] (Application Example 2)

[0548] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0549] Conventional augmented reality systems have the challenge of displaying personalized information tailored to individual emotional states, as they display uniform information to all users. Furthermore, they are insufficient in optimizing the user experience based on the user's emotional state, and there is a particular need to address the sense of security and tension inherent in real estate transactions.

[0550] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0551] In this invention, the server includes means for acquiring publicly available geographic information, means for converting the acquired geographic information into a format suitable for augmented reality display, and means for transmitting the converted geographic information to a mobile terminal. This enables personalized augmented reality display on the mobile terminal that responds to the user's emotional state, thereby optimizing the user experience in specific fields such as real estate transactions.

[0552] "Publicly available geographic information" refers to geographical data that is accessible to the general public through the internet or public resources.

[0553] A "format suitable for augmented reality display" is data that has been converted into a format necessary for displaying it overlaid on real-world images.

[0554] A "mobile device" is a portable information processing device such as a smartphone or tablet.

[0555] The Global Positioning System (GSO) is a satellite system used to determine one's current location on Earth.

[0556] A "virtual boundary" is a digital geographical boundary that is superimposed onto real-world images.

[0557] An "emotion recognition engine" is an algorithm or software that analyzes emotions in real time from a customer's facial expressions and other data.

[0558] "Means for adjusting augmented reality displays according to the user's emotions" refers to a mechanism for dynamically changing the content and format of displayed information based on the user's emotional information.

[0559] The server retrieves geographic information publicly available via the internet and converts it into a format suitable for augmented reality display. The converted data is transmitted to mobile devices in real time. As the central hub for data processing, the server requires high-performance processing equipment and a stable network connection to enable efficient and rapid information delivery.

[0560] The device uses the Global Positioning System (GPS) to determine the user's location in real time, captures real-world images through its camera, and overlays virtual boundaries onto them. In addition, the device is equipped with an emotion recognition engine that analyzes the user's facial expressions captured by the camera in real time. Based on the emotion data obtained from this analysis, the displayed information is personalized.

[0561] The hardware used will be mobile devices such as smartphones and tablets, and the software will utilize augmented reality frameworks such as Google's ARCore and Apple's ARKit. For sentiment analysis, high-precision analysis will be achieved by using Google Cloud's Vision API and Microsoft Azure's Face API.

[0562] As a concrete example, during a real estate site visit, when a user is viewing a plot of land, the terminal automatically displays the site boundaries and provides detailed information if the user appears nervous. An example of a prompt message would be, "When the system detects that the user is nervous, please prioritize displaying detailed real estate information." This system allows users to receive the most appropriate information in real time, tailored to their situation.

[0563] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0564] Step 1:

[0565] The server retrieves geographic information published via the internet. The input is online geographic data, and the output is raw data stored in an internal database. This data is used for subsequent processing and is updated regularly using a stable communication line.

[0566] Step 2:

[0567] The server converts acquired geographic information into a format suitable for augmented reality display. The input is raw geographic data, and the output is converted AR format data. This step involves formatting and encoding the data to make it usable in AR frameworks.

[0568] Step 3:

[0569] The server transmits the converted geographic information to the mobile device. The input is data in AR format, and the output is the data that reaches the device's inbox. This process involves data compression and efficient transfer using a communication protocol.

[0570] Step 4:

[0571] The device uses the Global Positioning System (GPS) to determine the user's location. The input is a GPS signal, and the output is the coordinate data of the current location. Here, a GPS module is used to acquire real-time location information, which is then reflected in the subsequent video.

[0572] Step 5:

[0573] The device acquires real-world images through its camera and overlays them with virtual boundaries for display. The input consists of camera footage and coordinate data, while the output is an AR-displayed image. The AR framework precisely positions and overlays the virtual boundaries onto the camera footage.

[0574] Step 6:

[0575] The device's built-in emotion recognition engine analyzes the user's facial expressions. The input is a facial image of the user captured by the camera, and the output is emotion data. The emotion analysis algorithm determines the user's emotions in real time and expresses them numerically.

[0576] Step 7:

[0577] The device dynamically adjusts the augmented reality display according to the user's emotions. Input consists of emotion data and AR display data, and output is the adjusted AR display. If the emotional state is recognized as "tension," detailed information is displayed; when the user is at ease, it reverts to a concise information display. This automatic adjustment optimizes the user experience.

[0578] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0579] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0580] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0581] [Fourth Embodiment]

[0582] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0583] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0584] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0585] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0586] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0587] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0588] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0589] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0590] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0591] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0592] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0593] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0594] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0595] This invention provides a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device. This system is realized by integrating functions such as geographic information acquisition, data conversion, location determination, and augmented reality display.

[0596] First, the server retrieves publicly available geographical information online. This information includes land boundaries and map data. The server then uses a generative AI to analyze the retrieved information and convert it into a data format suitable for augmented reality display. The converted data is then stored in the format required by the mobile device.

[0597] Next, the server sends the converted geographic information data to the terminal. When the user starts up their mobile device with the dedicated application installed, the terminal receives this data. The user then uses this terminal to go to the field.

[0598] On-site, the device uses the Global Positioning System (GPS) to obtain accurate, real-time location information. Based on this location information, the device's AR rendering engine uses converted geographic data to overlay virtual boundaries onto the real-world camera footage.

[0599] By holding a mobile device and activating the camera to scan the terrain, a virtual boundary line based on publicly available geographical information is displayed on the screen. This allows users to visually confirm land boundaries in a concise and effective manner. Furthermore, users can manipulate the boundary information on the screen in detail and display additional information through the interface.

[0600] For example, in real estate transactions and land development, users can facilitate communication with landowners by visually confirming actual boundaries in advance. Furthermore, by obtaining visual and real-time feedback without relying on traditional maps or paper media, it is expected that the reliability and efficiency of map information will be improved.

[0601] The following describes the processing flow.

[0602] Step 1:

[0603] The server accesses an online geographic information database to retrieve the latest geographic information. The server uses an API to download data, including land boundary information.

[0604] Step 2:

[0605] The server uses a generative AI to analyze the acquired geographic information and convert it into a data format suitable for augmented reality display. The server then saves the conversion results as data for AR display.

[0606] Step 3:

[0607] The server sends the converted data to the user's mobile device. The server uses encryption protocols to ensure secure data transmission.

[0608] Step 4:

[0609] The device receives the transmitted geographic information data and prepares to read the data through the application. The user launches the dedicated application and performs the necessary initial setup.

[0610] Step 5:

[0611] The device uses the Global Positioning System (GPS) to obtain the user's current location in real time. To ensure highly accurate location information, the device receives signals from multiple satellites.

[0612] Step 6:

[0613] The device uses an AR rendering engine to overlay and display geographical boundaries converted from images acquired through the camera. This allows users to simultaneously view the real-world scenery and virtual boundaries.

[0614] Step 7:

[0615] The user observes the site with a mobile device and checks the boundaries displayed on the screen. The user can operate the interface to change the display mode or obtain detailed information.

[0616] (Example 1)

[0617] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0618] Conventional geographic data display systems lack a simple and accurate way for users to visually confirm geographical boundaries in real space. Furthermore, the display on mobile devices is not highly customizable, making it difficult for users to intuitively manipulate information according to their needs.

[0619] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0620] In this invention, the server includes means for collecting publicly available geographic data, means for utilizing an artificial intelligence model to convert the geographic data into a format suitable for augmented reality display, and means for communicating the converted geographic data to a mobile device. This allows users to accurately and visually confirm virtual boundaries in real geospace and customize them according to their needs.

[0621] "Publicly available geographic data" refers to geographical information resources that can be obtained from government agencies or publicly available platforms, and includes map data, land boundary information, topographic information, etc.

[0622] An "artificial intelligence model" refers to an information processing device that processes large amounts of data and exhibits capabilities equivalent to or exceeding those of a human in specific tasks, and includes machine learning and deep learning.

[0623] "Mobile devices" refer to portable electronic devices such as mobile phones and tablets that have wireless communication capabilities and computing power.

[0624] A "satellite positioning system" refers to a system that uses artificial satellites to provide accurate location information for any point on Earth, and includes GPS and similar technologies.

[0625] "Augmented reality display" refers to a technology that overlays digital information onto images of the real world, presenting information directly and in real time to the user.

[0626] A "virtually defined boundary" refers to a boundary line or area in real geographical space that is defined in digital data and presented in a way that the user can visually recognize.

[0627] "User operation" refers to instructions or actions performed through a user interface via a mobile device, and includes means that allow for intuitive operation.

[0628] This invention provides a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device. This system integrates functions for geographic data acquisition, data transformation, location determination, and augmented reality display.

[0629] First, the server collects publicly available geographic data. To do this, it accesses publicly available databases and web resources on the internet to obtain the latest map information and land boundary information. The server analyzes the acquired geographic data and uses an artificial intelligence model to convert the data into a format suitable for augmented reality display. Examples of artificial intelligence models used in this process include "large-scale language models." The server stores the converted geographic data on its own so that mobile devices can receive it.

[0630] Next, the server communicates the converted geographic data to the mobile device. A common data transfer protocol is used for communication, and encryption technology is employed to ensure a secure connection.

[0631] On-site, the device uses a satellite positioning system to determine its current location. The device is equipped with positioning technologies such as GPS, enabling accurate real-time location tracking of the user.

[0632] Subsequently, the device's AR rendering engine overlays the converted geographic information, based on the acquired location data, onto the real-world camera footage. This process utilizes an "augmented reality-enabled framework." For example, when a user activates the camera and points the device towards the surrounding terrain, virtual boundaries based on public geographic data are displayed on the screen.

[0633] Users can operate the mobile device's interface to fine-tune the displayed virtual boundaries and view related information. For example, it can be used during pre-inspection work at real estate sites to provide effective visual effects.

[0634] An example of a prompt message might be, "Retrieve the latest land boundary information for this area and convert it to a format suitable for AR display." This system will make it easier for users to verify information in the real world and enable rapid visualization of geographic data.

[0635] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0636] Step 1:

[0637] The server collects publicly available geographic data from the internet. Inputs include pre-specified map URLs and API keys. Web crawling technology is used for data collection. The obtained geographic data includes land boundary information and road information, and is output in a format for storage in a database.

[0638] Step 2:

[0639] The server analyzes the collected geographic data and converts it into a format suitable for augmented reality display. The input is the geographic data acquired in Step 1. A generative AI model is used for data conversion, performing noise reduction and data formatting. Specifically, latitude and longitude information is reconstructed into a 3D model format and converted into the Geo-JSON format used for real-time AR rendering. This converted data is then output in a format that mobile devices can receive.

[0640] Step 3:

[0641] The server communicates the converted geographic data to the mobile device. The data converted in step 2 is used as input. Encryption protocols such as TLS are used for communication to ensure secure data transfer. The output is provided in a format that can be stored in the mobile device's memory.

[0642] Step 4:

[0643] The terminal receives geographic data stored on the mobile device and then uses the Global Positioning System (GPS) to obtain its current location in real time. The input consists of the geographic data received in step 3 and the current GPS data. Based on this data, the terminal performs a position calculation to map the user's current location to the geographic data and outputs it as input to the AR rendering engine.

[0644] Step 5:

[0645] The device's AR rendering engine uses the results of position calculations to overlay a virtual boundary onto the real-world camera image. The input is the position data calculated in step 4. Specifically, it renders the boundary that should be visible from the user's viewpoint within the camera image in real time. The output is the AR display, which is shown on the mobile device's screen.

[0646] Step 6:

[0647] Users interact with the AR display on their mobile device to examine the displayed virtual boundary in detail. Input is provided through the device's interface. Specifically, users can adjust the displayed content using gestures such as pinching and swiping, and obtain additional information through voice input. The output provides visual information tailored to the user's intentions.

[0648] (Application Example 1)

[0649] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0650] Without relying on existing map information or paper media, there is a need to provide consumers and users with a means to recognize property boundaries and store layouts accurately in real time, thereby improving their shopping experience. In particular, by clearly understanding the layout and layout within stores, it is necessary to create a more convenient shopping environment for consumers and to provide store operators with efficient methods for layout changes and sales promotions.

[0651] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0652] In this invention, the server includes means for acquiring publicly available spatial information, means for converting the spatial information into a format suitable for augmented realization display using data processing technology, and means for transmitting the converted spatial information to a communication device. This enables consumers to use the communication device to recognize information about the area inside and around the store in real time, thereby improving the shopping experience.

[0653] "Spatial information" refers to data related to geographical location and extent, including information such as real estate and the layout of a store.

[0654] "Data processing technology" refers to the technology of analyzing and transforming data according to a specific purpose, and in this invention, it refers to the transformation into a data format for extended realization and display.

[0655] "Augmented Reality Display" refers to a technology that overlays virtual information onto real-world images, and is a form of AR (Augmented Reality).

[0656] "Communication equipment" refers to a device capable of sending and receiving digital information, and in this invention, it mainly refers to a portable terminal.

[0657] A "location system" is a technology that accurately measures physical location, and the Global Positioning System (GPS) is one example.

[0658] A "virtual boundary" is a line that does not physically exist but is displayed as information, visually presenting the user with the boundaries of a specific area.

[0659] "Store layout information" refers to information about the division and arrangement of space within a store, and is used in designing product displays and customer flow.

[0660] The system implementing this invention primarily consists of three components: a server, communication equipment (mobile terminal), and a user. The server first acquires publicly available spatial information from the internet. A dedicated API can be used for this acquisition. Subsequently, the server converts the acquired spatial information into a format suitable for augmented reality display using data processing technology. This conversion process uses a generative AI model to format the data into an AR display data format in real time. The converted data is stored so that it can be quickly transmitted to the communication equipment.

[0661] Mobile devices receive converted data from a server via the network. During this process, the device precisely determines its location using the Global Positioning System (GPS), and based on this location information, overlays virtual boundaries onto real-world camera footage. This allows users to visually confirm spatial and real estate parcel information in real time.

[0662] Users can use this mobile device to activate the camera at specific locations inside and outside the store, displaying virtual area information along with the video feed. For example, when introducing a new product section, this system can be used to check the store layout and determine the optimal placement of spaces. Based on a prompt message such as, "Please convert the surrounding geographic data to AR format to suggest the optimal store layout," a generative AI model can be used to present information in a practical and intuitive way for the user.

[0663] One specific use case is when a store's design team is selecting a location for a new display. The user can use the terminal to quickly obtain the information needed to achieve the most efficient product placement while viewing the available space within the store.

[0664] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0665] Step 1:

[0666] The server first retrieves publicly available spatial information from the internet. An internet connection is required as input, and requests are made to endpoints such as geographic information APIs. The output is spatial data including land parcel and boundary information.

[0667] Step 2:

[0668] The server transforms the acquired spatial information using data processing technology. Using a generative AI model, it converts the spatial data obtained as input into a data format suitable for augmented reality display. The output is a data format for AR display. During this process, prompt statements are used to control the behavior of the AI ​​model.

[0669] Step 3:

[0670] The server transmits the converted data to the communication device. The converted data is used as input, and the data is sent to the mobile terminal as output. A network connection is required for communication.

[0671] Step 4:

[0672] The device uses the Global Positioning System (GPS) to determine its current location. It receives GPS signals as input and obtains precise latitude and longitude location information as output.

[0673] Step 5:

[0674] The device uses location information obtained from GPS and converted data received from a server to overlay a virtual boundary onto the real-world camera image. Using camera footage and AR data as input, the output is a visually superimposed image.

[0675] Step 6:

[0676] The user utilizes the camera function of their device to view virtual area information presented by the application. The input requires real-world video captured by the device, and the output provides a visual display including virtual information overlaid on the camera footage. This allows the user to perform spatial recognition and confirm the necessary information.

[0677] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0678] This invention realizes a next-generation AR system that combines a system that allows users to visually confirm virtual boundaries in real geographic space using a mobile device with an emotion engine that recognizes the user's emotions. This makes it possible to further personalize the user experience.

[0679] This system first uses a server to retrieve geographic information publicly available online and convert it into a format suitable for AR display. Next, the server sends the converted data to the user's mobile device. The user then uses a mobile device with a dedicated app installed to activate the system at the site.

[0680] The device uses the Global Positioning System (GPS) to determine the user's location in real time and overlays virtual boundaries onto real-world images captured by the camera. This allows the user to visually confirm accurate land boundaries through the camera. Furthermore, the device is equipped with an emotion engine that recognizes the user's emotions by analyzing their facial expressions. Emotional data is collected through the device's camera and sensors and analyzed in real time.

[0681] When a user uses this system, the emotion engine changes the AR display according to the user's emotional state. For example, if the user is surprised, additional information is highlighted, while if they are relaxed, a simpler display is maintained. This interactive adjustment allows users to receive information that is most relevant to their state, enabling them to effectively proceed with land inspections and transactions.

[0682] For example, in a land transaction, if the system detects that a user is experiencing stress, it can help the user understand by re-displaying the complex geographical information that is believed to be the cause of the stress in an easy-to-understand format. Furthermore, when the user is at ease, the system maintains the standard mode while providing easy access to more detailed information, thereby optimizing the user experience.

[0683] The following describes the processing flow.

[0684] Step 1:

[0685] The server accesses an online geographic information database and retrieves the latest geographic information according to the user's request. The server uses an API to download the necessary data, including land boundary information.

[0686] Step 2:

[0687] The server analyzes the acquired geographic information using a generation AI and converts it into a format suitable for augmented reality display. This conversion process formats the geographic information into a format usable by the AR rendering engine.

[0688] Step 3:

[0689] The server transmits the converted geographical information to the user's mobile device. This transmission is carried out through a secure communication protocol, ensuring that accurate data is delivered to the user's device.

[0690] Step 4:

[0691] The device uses the Global Positioning System (GPS) to obtain its current location in real time. The device utilizes multiple satellite signals to ensure centimeter-level positional accuracy.

[0692] Step 5:

[0693] The device activates its camera and captures real-world images. The AR rendering engine overlays virtual boundaries onto the camera images, providing the user with visual information.

[0694] Step 6:

[0695] The device activates an emotion engine and collects emotional data from the user's facial expressions and voice through the camera and other sensors. The emotion engine analyzes this data to determine the user's emotional state.

[0696] Step 7:

[0697] The device dynamically adjusts the AR display content based on the analysis results of the emotion engine. For example, if the system determines that the user is confused, it simplifies the displayed information and provides navigation to aid understanding. Furthermore, the color scheme and emphasis of information change according to the user's emotions.

[0698] Step 8:

[0699] Users observe the site using a mobile device and check the boundaries and information displayed on the screen. Through the interface, users can access detailed information and change the display mode. This allows users to easily grasp the necessary information regarding land boundaries.

[0700] (Example 2)

[0701] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0702] The present invention aims to provide an augmented reality system that allows users to easily visualize geographical boundaries in the real world. In particular, it aims to solve the problem of improving the user experience by dynamically responding to the user's emotional state and displaying information according to the situation.

[0703] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0704] In this invention, the server includes means for acquiring publicly available spatial information, means for converting the acquired spatial information into a format suitable for augmented reality display, and means for transmitting the converted spatial information to a mobile terminal. This allows the user to easily perceive virtual boundaries on real-world visual information and to adjust the information display based on the user's emotional state.

[0705] "Publicly available spatial information" refers to information about physical spaces that is available in open databases such as the internet.

[0706] "A format suitable for augmented reality display" refers to the data structure necessary to display a combination of the user's real-world visual information and digital information.

[0707] A "mobile device" refers to an electronic device that is portable by a user and has communication and computing capabilities.

[0708] "Global positioning technology" refers to satellite systems and signal receiving technologies used to determine any location on Earth.

[0709] A "virtual boundary" refers to lines or edges that are superimposed on a real-world landscape to visually indicate a specific area within a space.

[0710] "Real-world visual information" refers to video and image information acquired from the physical landscape of the real world using cameras and various sensors.

[0711] "User emotional state" refers to the emotions and psychological state a user is experiencing at a particular moment, and is analyzed through methods such as facial recognition.

[0712] "Adjusting information display" means dynamically changing the type of information displayed and its visual presentation according to the user's situation and requests.

[0713] This invention provides an augmented reality system that allows users to visually perceive virtual boundaries in real space using a mobile device, and combines this with technology for recognizing the user's emotions. This makes it possible to further personalize the user experience.

[0714] The server first retrieves spatial information publicly available on the internet from a database. This information includes coordinate data and details about geographical areas. Specifically, it uses a "Geographic Information Service API" to retrieve the data. Since the retrieved data is difficult to use directly, it is converted into a format suitable for augmented reality display using a "3D engine platform" or similar. This conversion process includes polygonizing the data and generating meshes. The converted data is then quickly transmitted to the user's mobile device via a wireless network.

[0715] The device uses the Global Positioning System to determine the user's location in real time. Furthermore, the device uses a camera to acquire real-world visual information and overlays virtual boundaries onto that image. An "AR platform library" may be used for this display. The device also incorporates emotion recognition technology, analyzing the user's facial expression data acquired through the camera and sensors using "emotion analysis software." This determines the user's current emotions. Based on this emotion data, the device adjusts the displayed information to provide the most appropriate interface for the user's situation.

[0716] When a user uses this system, emotion recognition technology changes the AR display according to the user's emotional state. For example, if a user is surprised when trying to confirm a specific physical boundary, the system will take measures to attract the user's attention, such as enlarging summary information. Conversely, if the user is relaxed, the information display will be simplified, and the interface will be designed so that additional details can be easily viewed later.

[0717] Examples of prompts for a generative AI model include the following:

[0718] "Please demonstrate how to utilize an emotion engine to adjust information presentation based on the user's emotions."

[0719] "Please tell me about methods for visualizing geographical boundaries in augmented reality systems and how to optimize them."

[0720] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0721] Step 1:

[0722] The server retrieves spatial information publicly available on the internet. The input is a request for coordinate data concerning a specific geographic area. Based on this request, the server uses a geographic information service API to obtain detailed spatial data about the target area. The retrieved data is unconverted data, including coordinate and topographic information. The output of this step is this unconverted data.

[0723] Step 2:

[0724] The server converts the acquired unconverted data into a format that can be displayed in augmented reality. The input is the unconverted data acquired in step 1. Using the "3D engine platform," the data is polygonized and mesh generated, converting it into an advanced data format. The converted data is in a format suitable for a graphical user interface, and this is the output of step 2. Specifically, the data structure is rearranged within the server to create a format that can be computed efficiently.

[0725] Step 3:

[0726] The server transmits the converted data to the mobile device. The input is the advanced data format generated in step 2. The server transmits this data to the user's mobile device using a wireless network. The output is a data stream in a format usable by the mobile device. The server optimizes data transfer during this process to maintain transmission speed and signal quality.

[0727] Step 4:

[0728] The device uses the Global Positioning System to determine the user's location in real time. The input is location data from the device's GPS sensor. The device analyzes this location data and performs calculations to accurately determine the user's current location. The output is location information bound to the user interface. Specifically, the device periodically updates its location information to keep it always up-to-date.

[0729] Step 5:

[0730] The device uses a camera to acquire real-world visual information and overlays it with virtual boundaries. Inputs include the user's location information and a data stream received from a server. Utilizing an AR platform library, it performs a virtual overlay on the real-world image. The output is visual information in an augmented reality environment. Specifically, boundaries and terrain information are displayed in real time on the camera feed, helping the user intuitively understand the space.

[0731] Step 6:

[0732] The device uses a camera and sensors to acquire user facial expression data and analyzes it with emotion analysis software. The input is facial expression data from the camera. Based on this data, an emotion recognition algorithm analyzes the user's emotional state and generates information about it. The output is evaluation data regarding the user's emotional state. Specifically, the device dynamically adjusts the AR display according to the user's emotions.

[0733] Step 7:

[0734] Users take action based on information refined by the system. Inputs are feedback from the system and emotionally-driven emphasis. Users interact directly through the user interface and acquire additional information as needed. Outputs are the receipt of information optimized for the user, and concrete actions include making decisions based on the information provided.

[0735] (Application Example 2)

[0736] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0737] Conventional augmented reality systems have the challenge of displaying personalized information tailored to individual emotional states, as they display uniform information to all users. Furthermore, they are insufficient in optimizing the user experience based on the user's emotional state, and there is a particular need to address the sense of security and tension inherent in real estate transactions.

[0738] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0739] In this invention, the server includes means for acquiring publicly available geographic information, means for converting the acquired geographic information into a format suitable for augmented reality display, and means for transmitting the converted geographic information to a mobile terminal. This enables personalized augmented reality display on the mobile terminal that responds to the user's emotional state, thereby optimizing the user experience in specific fields such as real estate transactions.

[0740] "Publicly available geographic information" refers to geographical data that is accessible to the general public through the internet or public resources.

[0741] A "format suitable for augmented reality display" is data that has been converted into a format necessary for displaying it overlaid on real-world images.

[0742] A "mobile device" is a portable information processing device such as a smartphone or tablet.

[0743] The Global Positioning System (GSO) is a satellite system used to determine one's current location on Earth.

[0744] A "virtual boundary" is a digital geographical boundary that is superimposed onto real-world images.

[0745] An "emotion recognition engine" is an algorithm or software that analyzes emotions in real time from a customer's facial expressions and other data.

[0746] "Means for adjusting augmented reality displays according to the user's emotions" refers to a mechanism for dynamically changing the content and format of displayed information based on the user's emotional information.

[0747] The server retrieves geographic information publicly available via the internet and converts it into a format suitable for augmented reality display. The converted data is transmitted to mobile devices in real time. As the central hub for data processing, the server requires high-performance processing equipment and a stable network connection to enable efficient and rapid information delivery.

[0748] The device uses the Global Positioning System (GPS) to determine the user's location in real time, captures real-world images through its camera, and overlays virtual boundaries onto them. In addition, the device is equipped with an emotion recognition engine that analyzes the user's facial expressions captured by the camera in real time. Based on the emotion data obtained from this analysis, the displayed information is personalized.

[0749] The hardware used will be mobile devices such as smartphones and tablets, and the software will utilize augmented reality frameworks such as Google's ARCore and Apple's ARKit. For sentiment analysis, high-precision analysis will be achieved by using Google Cloud's Vision API and Microsoft Azure's Face API.

[0750] As a concrete example, during a real estate site visit, when a user is viewing a plot of land, the terminal automatically displays the site boundaries and provides detailed information if the user appears nervous. An example of a prompt message would be, "When the system detects that the user is nervous, please prioritize displaying detailed real estate information." This system allows users to receive the most appropriate information in real time, tailored to their situation.

[0751] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0752] Step 1:

[0753] The server retrieves geographic information published via the internet. The input is online geographic data, and the output is raw data stored in an internal database. This data is used for subsequent processing and is updated regularly using a stable communication line.

[0754] Step 2:

[0755] The server converts acquired geographic information into a format suitable for augmented reality display. The input is raw geographic data, and the output is converted AR format data. This step involves formatting and encoding the data to make it usable in AR frameworks.

[0756] Step 3:

[0757] The server transmits the converted geographic information to the mobile device. The input is data in AR format, and the output is the data that reaches the device's inbox. This process involves data compression and efficient transfer using a communication protocol.

[0758] Step 4:

[0759] The device uses the Global Positioning System (GPS) to determine the user's location. The input is a GPS signal, and the output is the coordinate data of the current location. Here, a GPS module is used to acquire real-time location information, which is then reflected in the subsequent video.

[0760] Step 5:

[0761] The device acquires real-world images through its camera and overlays them with virtual boundaries for display. The input consists of camera footage and coordinate data, while the output is an AR-displayed image. The AR framework precisely positions and overlays the virtual boundaries onto the camera footage.

[0762] Step 6:

[0763] The device's built-in emotion recognition engine analyzes the user's facial expressions. The input is a facial image of the user captured by the camera, and the output is emotion data. The emotion analysis algorithm determines the user's emotions in real time and expresses them numerically.

[0764] Step 7:

[0765] The device dynamically adjusts the augmented reality display according to the user's emotions. Input consists of emotion data and AR display data, and output is the adjusted AR display. If the emotional state is recognized as "tension," detailed information is displayed; when the user is at ease, it reverts to a concise information display. This automatic adjustment optimizes the user experience.

[0766] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0767] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0768] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0769] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0770] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0771] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0772] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0773] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0774] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0775] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0776] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0777] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0778] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0779] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0780] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0781] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0782] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0783] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0784] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0785] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0786] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0787] The following is further disclosed regarding the embodiments described above.

[0788] (Claim 1)

[0789] Means of obtaining publicly available geographic information,

[0790] A means of converting acquired geographic information into a format suitable for augmented reality display,

[0791] A means of transmitting the converted geographic information to a mobile device,

[0792] A means of determining the location of a mobile device using the Global Positioning System,

[0793] A means of overlaying a virtual boundary onto real-world images in a form that can be displayed on a mobile device,

[0794] A system that includes this.

[0795] (Claim 2)

[0796] The system according to claim 1, wherein the geographic information includes land boundary information.

[0797] (Claim 3)

[0798] The system according to claim 1, wherein a mobile device can operate an augmented reality display through user input.

[0799] "Example 1"

[0800] (Claim 1)

[0801] Means for collecting publicly available geographic data,

[0802] A means of using artificial intelligence models to convert geographic data into a format suitable for augmented reality display,

[0803] A means for communicating the converted geographic data to a mobile device,

[0804] A means of determining the current location of a mobile device via a satellite positioning system,

[0805] A means for overlaying a virtually defined boundary, in a visually synthesizable form, onto an actual image on a mobile device,

[0806] A system that includes this.

[0807] (Claim 2)

[0808] The system according to claim 1, wherein geographic data includes real estate boundary information.

[0809] (Claim 3)

[0810] The system according to claim 1, wherein the mobile device is capable of customizing the augmented reality display through user operation.

[0811] "Application Example 1"

[0812] (Claim 1)

[0813] Means for acquiring publicly available spatial information,

[0814] A means for converting acquired spatial information into a format suitable for augmented and realized display using data processing technology,

[0815] Means for transmitting the converted spatial information to a communication device,

[0816] A means of determining the location of communication devices using a location tracking system,

[0817] A means of overlaying a virtual boundary onto real-world images in a form that can be displayed on communication devices,

[0818] A means of visually presenting information about the store layout and supporting the consumer's purchasing experience,

[0819] A system that includes this.

[0820] (Claim 2)

[0821] The system according to claim 1, wherein the spatial information includes real estate boundary information.

[0822] (Claim 3)

[0823] The system according to claim 1, wherein the communication device allows the user to operate the extended display.

[0824] "Example 2 of combining an emotion engine"

[0825] (Claim 1)

[0826] Means for acquiring publicly available spatial information,

[0827] A means for converting acquired spatial information into a format suitable for augmented reality display,

[0828] A means for transmitting the converted spatial information to a mobile device,

[0829] A means of determining the location of a mobile device using global positioning technology,

[0830] A means of overlaying a virtual boundary onto real-world visual information in a form that can be displayed on a mobile device,

[0831] A means of analyzing user facial expression data to recognize the user's emotional state,

[0832] Means for adjusting the augmented reality display based on recognized emotional states,

[0833] A system that includes this.

[0834] (Claim 2)

[0835] The system according to claim 1, wherein the spatial information includes domain boundary information.

[0836] (Claim 3)

[0837] The system according to claim 1, wherein a mobile device can be operated by the user to control augmented reality display and highlighting of additional information.

[0838] "Application example 2 when combining with an emotional engine"

[0839] (Claim 1)

[0840] Means of obtaining publicly available geographic information,

[0841] A means of converting acquired geographic information into a format suitable for augmented reality display,

[0842] A means of transmitting the converted geographic information to a mobile device,

[0843] A means of determining the location of a mobile device using the Global Positioning System,

[0844] A means of overlaying a virtual boundary onto real-world images in a form that can be displayed on a mobile device,

[0845] A means by which an emotion recognition engine installed in a mobile device analyzes the user's emotions,

[0846] A means of adjusting the augmented reality display according to the user's emotions,

[0847] A system that includes this.

[0848] (Claim 2)

[0849] The system according to claim 1, wherein geographic information includes area boundary information, and an emotion recognition engine determines whether the user is in a state of tension or relaxation, and changes the displayed content accordingly.

[0850] (Claim 3)

[0851] The system according to claim 1, wherein the mobile device allows the user to operate the augmented reality display, and also includes an automatic adjustment function based on emotion recognition. [Explanation of Symbols]

[0852] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means of obtaining publicly available geographic information, A means of converting acquired geographic information into a format suitable for augmented reality display, A means of transmitting the converted geographic information to a mobile device, A means of determining the location of a mobile device using the Global Positioning System, A means of overlaying a virtual boundary onto real-world images in a form that can be displayed on a mobile device, A system that includes this.

2. The system according to claim 1, wherein the geographic information includes land boundary information.

3. The system according to claim 1, wherein a mobile device can operate an augmented reality display through user input.

Images