system

Abstract

We provide the system. [Solution] A means for obtaining geolocation information to obtain the user's location information, A posture detection means for detecting the orientation of the user's device, A data acquisition means for acquiring celestial body data based on the above-mentioned positional information and direction, A display means for displaying acquired celestial data in augmented reality format on the user's device, Information provision means for providing information about celestial bodies in response to user input, A system that includes this.

Description

Technical Field

[0001] The technology of this disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a problem that it is difficult for ordinary users to easily observe and understand celestial bodies and constellations in the night sky even without specialized equipment and knowledge. In conventional celestial observation methods, astronomical knowledge is required, which is one of the factors hindering participation in celestial observation. Also, it is difficult to obtain accurate information about celestial bodies at specific positions and times. Furthermore, since an environment where educational information about astronomy can be immediately obtained is not common, it inhibits users' learning motivation

Means for Solving the Problems

[0005] To address this challenge, we propose a system that utilizes smartphone sensor technology. This system includes a geolocation information acquisition means to accurately determine the user's current location and an attitude detection means to detect the device's orientation. It also provides a means to acquire corresponding celestial body data based on the current location and orientation and display that data on the device in augmented reality format. In addition, it includes an information provision means that instantly provides information about celestial bodies in response to user input. Specifically, when a user requests detailed information about a particular celestial body, the system provides the corresponding information and notifies the user when a specific astronomical event occurs. In this way, it provides an environment where users can easily enjoy and learn about astronomical observation even without specialized knowledge.

[0006] "Means for acquiring geographic location information" refers to a device or function for acquiring information such as latitude, longitude, and altitude that indicates the user's current location.

[0007] "Attitude detection means" refers to sensors and functions that process the orientation and tilt of the user's device.

[0008] "Astronomical data" refers to a collection of information about constellations, planets, and stars that can be observed based on a specific time and location.

[0009] "Data acquisition means" refers to the means of obtaining necessary astronomical data from external databases or servers.

[0010] "Display means" refers to a device or function for visually displaying information on a user's device.

[0011] Augmented reality is a technological technique that overlays virtual information onto the real world environment.

[0012] "Information provision means" refers to functions or systems that provide specific information in response to user requests.

[0013] A "notification method" is a means of informing users about events or important information. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The present invention is a system that enables users to perform astronomical observations and easily obtain information about constellations and celestial bodies using a smartphone or similar portable device without special equipment. The main components of this system include means for acquiring geographic location information, means for detecting attitude, means for acquiring celestial body data, means for augmented reality display, means for providing information, and means for notification.

[0036] First, the device uses its built-in GPS function to obtain the user's current location. This information is used as the viewpoint when the user observes celestial bodies. Next, the device uses its gyroscope and accelerometer to detect its orientation and determine which direction it is facing.

[0037] Based on this location and orientation information, the device queries the server to request celestial data visible at its current location and time. The server retrieves data about specific constellations and celestial objects from its database and sends it to the device.

[0038] Subsequently, the device uses augmented reality (AR) technology to display constellations and celestial bodies on the screen, corresponding to the direction the user is pointing the device, based on the received astronomical data. This allows the user to visually recognize constellations and individual stars superimposed on the real night sky.

[0039] Furthermore, when a user selects a specific constellation or celestial body, the device utilizes AI technology to provide the user with explanations and related information about that celestial body. This information includes the mythology of the constellation, its astronomical characteristics, and observational considerations.

[0040] Furthermore, the device sends notifications to the user when specific astronomical events, such as the peak of a meteor shower or a lunar eclipse, are approaching. This notification feature allows users to enjoy important astronomical phenomena without missing them.

[0041] In this way, the present invention provides a user-friendly and educational astronomical observation experience for a wide range of users. Furthermore, it is designed to be easy for beginners to use through an intuitive user interface.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The user launches the app. This causes the app to begin preparing to retrieve and process data.

[0045] Step 2:

[0046] The device activates its built-in GPS sensor to obtain the user's current location. This location data includes latitude and longitude and is used as the basis for calculations within the device.

[0047] Step 3:

[0048] The device activates its gyroscope and accelerometer to detect its current orientation and tilt. This information is used to calculate the precise direction the device is facing.

[0049] Step 4:

[0050] The device sends a request to the server based on its current location and orientation information, requesting celestial object data visible at a specified location and time.

[0051] Step 5:

[0052] The server processes the received request, retrieves appropriate celestial data from the database based on the user's location and time, and sends it to the terminal.

[0053] Step 6:

[0054] The device analyzes astronomical data received from the server and prepares it for display on the device's screen. The data is overlaid on the screen in an augmented reality format.

[0055] Step 7:

[0056] Users can view information about constellations and celestial bodies displayed on their device screen. They can tap on a star or constellation that interests them to request more detailed information.

[0057] Step 8:

[0058] The device responds to user input and uses AI technology to provide detailed information about selected celestial objects. This information is displayed on the screen in text and audio formats.

[0059] Step 9:

[0060] When the device detects an astronomical event (e.g., a meteor shower or a lunar eclipse), it sends a notification to the user. This allows the user to enjoy important events without missing them.

[0061] Step 10:

[0062] When the user closes the app, the device stops the sensors and saves the app's state. This allows for quick resumption the next time the app is launched.

[0063] (Example 1)

[0064] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0065] Traditional astronomical observation systems require users to have specialized equipment, making it difficult to obtain detailed information about celestial objects. Furthermore, users may miss important astronomical events. This limits opportunities for beginners and the general public to become familiar with astronomical observation.

[0066] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0067] In this invention, the server includes means for acquiring location information, means for detecting attitude, and means for acquiring celestial body information. This makes it possible to easily perform astronomical observations using a portable device and obtain detailed celestial body information without requiring special equipment. In addition, a notification function for astronomical events allows users to enjoy important astronomical phenomena without missing them.

[0068] "Location information acquisition means" refers to a device or system that has the function of determining the geographical location of a user.

[0069] "Attitude detection means" refers to sensors or technologies that measure the orientation and tilt of the device used by the user.

[0070] "Data acquisition means" refers to a system for collecting information about celestial bodies based on the user's position and orientation.

[0071] "Display means" refers to technologies and devices for visually displaying acquired celestial information on a user's terminal.

[0072] An "information provision method" refers to a system that has the function of providing users with information about celestial bodies they are looking for.

[0073] "Explanatory means" refers to a system that uses a generative AI model to generate detailed explanations about celestial bodies selected by the user and presents them to the user.

[0074] A "notification device" refers to a device or system that has the function of informing users of the occurrence of astronomical phenomena or events.

[0075] This invention is a system that allows users to perform astronomical observations using a portable information terminal and easily obtain information about constellations and other celestial bodies. The system mainly comprises means for acquiring location information, means for detecting attitude, means for acquiring data, means for display, means for providing information, means for explaining, and means for notifying.

[0076] The device uses its built-in GPS function to obtain the user's current geographical location, thereby establishing a reference point for observation. This location information is combined with the device's orientation, detected by its gyroscope and accelerometer. This allows the device to determine the direction and angle the user is actually facing.

[0077] Using this acquired information, the device sends a request to a server via the internet to obtain information about celestial objects visible from the current time and location. In response to this request, the server extracts the relevant information from a celestial object database and sends it back to the device. Based on this, the device uses AR technology to overlay constellations and celestial objects that match the direction the user is facing onto the screen.

[0078] Users can tap to select constellations and celestial bodies that interest them from those displayed on the screen. Detailed information about the selected celestial body is generated by a generative AI model based on prompts. This information includes myths, physical characteristics, and observational considerations. For example, by inputting the prompt, "Please tell me the myth about the Big Dipper," into the AI ​​model, the relevant myths will be displayed.

[0079] Furthermore, the device will send push notifications to users when important astronomical events such as meteor showers and lunar eclipses occur, ensuring they don't miss out on these events.

[0080] This system is an educational tool that allows anyone to easily deepen their knowledge of celestial bodies without requiring any special equipment, and it is designed to be easy to use even for beginners in astronomy.

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] The device acquires location information.

[0084] Input: Latitude and longitude obtained from the device's GPS sensor.

[0085] Processing: The system processes location data obtained from GPS to determine the user's current geographical location.

[0086] Output: Latitude and longitude data indicating the user's current location.

[0087] Specific operation: The message "Locating current location" will be displayed on the device screen, and upon success, the current location will be displayed on the screen.

[0088] Step 2:

[0089] The device detects its posture.

[0090] Input: Data from the gyroscope and accelerometer.

[0091] Processing: Based on information obtained from sensors, the device's orientation and tilt are analyzed to determine which direction the device is facing.

[0092] Output: The direction and angle the device is facing.

[0093] Specific operation: When the device moves, the compass display updates and the icon indicating direction moves.

[0094] Step 3:

[0095] The device sends a request to the server to obtain astronomical information.

[0096] Input: User's current location information (latitude, longitude) and device orientation information (direction, angle).

[0097] Processing: Send location and orientation information to the server and create a request for celestial object data observable at the given time.

[0098] Output: Sends a request for astronomical data to the server.

[0099] Specific operation: A progress bar is displayed while data is being transmitted to inform the user of the progress.

[0100] Step 4:

[0101] The server searches for celestial data based on the request.

[0102] Input: User's location and orientation information sent from the device.

[0103] Processing: Query the database to collect information on constellations and celestial objects observable at the corresponding location and time.

[0104] Output: A set of information on the corresponding constellation or celestial body.

[0105] Specific operation: After obtaining accurate data, the server sends the data to the terminal.

[0106] Step 5:

[0107] Displays astronomical data acquired by the device.

[0108] Input: Celestial information received from the server.

[0109] Processing: Based on the received information, constellations and celestial bodies are drawn on the device screen using AR technology.

[0110] Output: Real-time visuals of constellations and celestial bodies displayed on the screen.

[0111] Specific operation: Constellations and celestial bodies are displayed on the screen, with the sky the user is pointing the screen at as the background.

[0112] Step 6:

[0113] The user selects a specific celestial body and retrieves detailed information.

[0114] Input: The ID or name of the selected celestial body.

[0115] Processing: Create prompts for the generating AI model and generate explanations about the selected celestial body.

[0116] Output: Detailed information text about the selected celestial object.

[0117] Specific operation: When the user taps on a celestial body, information pops up on the screen, and an audio guide plays.

[0118] Step 7:

[0119] The device notifies the user of astronomical events.

[0120] Input: Data from a pre-configured astronomical event.

[0121] Processing: Check the current date and time and trigger a notification when a relevant astronomical event is approaching.

[0122] Output: Notification message provided to the user.

[0123] Specific operation: When a specific event approaches, a pop-up notification will appear along with vibration.

[0124] (Application Example 1)

[0125] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0126] Traditional astronomical observation requires specialized equipment and knowledge, making it inaccessible to the average user. Furthermore, it's difficult to observe celestial events without missing them. Additionally, non-expert users lack the means to gain a deep understanding of the celestial objects they observe. As a result, interest in astronomical phenomena and their educational potential are not being fully utilized.

[0127] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0128] In this invention, the server includes location information acquisition means for obtaining the user's current location information, attitude detection means for detecting the attitude of the mobile device, and data acquisition means for obtaining celestial body information based on the location information and attitude information. This enables the user to perform astronomical observations in real time without special equipment, visually recognize celestial bodies using augmented reality technology, and further enjoy detailed explanations provided by artificial intelligence.

[0129] "Location information acquisition means" refers to a device or system that identifies the user's current location and collects that location information.

[0130] "Attitude detection means" refers to a device or system that uses a gyroscope or accelerometer to detect the direction in which the user's mobile device is facing.

[0131] "Data acquisition means" refers to a device or system that acquires information about celestial bodies visible from the user's viewpoint from a server, based on location information and orientation information.

[0132] "Display means" refers to a device or system that visually displays acquired celestial body information on a user's mobile device using augmented reality technology.

[0133] "Information provision means" refers to a device or system that provides information about a celestial body selected by the user in response to their actions.

[0134] A "notification means" is a device or system that notifies users of information during important celestial events.

[0135] "Guidance means" refers to a function or system for providing live guidance on astronomical events on a mobile device.

[0136] "Artificial intelligence" is a technology that allows computer programs to learn and respond independently in order to generate detailed explanations of celestial objects selected by the user.

[0137] This invention is a system for users to perform astronomical observations using a mobile device without special equipment. The system includes means for acquiring location information, means for detecting attitude, means for acquiring data, means for displaying, means for providing information, means for notifying, means for guiding, and means for providing information using artificial intelligence.

[0138] The server retrieves relevant celestial object information from a celestial object database based on the current location information received from the mobile device and sends it to the device. The device receives the celestial object information based on the user's current location and orientation obtained using its built-in GPS and gyroscope sensor, and displays it using augmented reality technology. Open-source AR libraries such as ARKit / iOS and ARCore / ANDROID (registered trademarks) can be used.

[0139] When providing information, cloud-based AI (e.g., Google® Cloud AI) is used to generate detailed explanations about celestial objects selected by the user. Furthermore, as astronomical events approach, the device sends push notifications to the user and provides detailed guidance through a live guide function.

[0140] As a concrete example, if a user points their mobile device towards the night sky and selects "Orion," the device will display information about the mythology of Orion and related astronomical information. Furthermore, during the peak of the Orionian meteor shower, it will notify the user of this information and guide them to the best time and direction to view the meteor shower.

[0141] As an example of a prompt using a generative AI model, in response to the input "Tell me about the myth of Orion," the AI ​​can provide relevant stories and information.

[0142] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0143] Step 1:

[0144] The device uses its built-in GPS sensor to obtain its current location. This location information is collected as latitude and longitude. The input is the signal from the GPS sensor, and the output is the user's location data.

[0145] Step 2:

[0146] The device uses a gyroscope and accelerometer to determine its orientation, that is, the direction it is facing. This orientation information is obtained as elevation angle and azimuth angle. The input is the signal from the sensors, and the output is the orientation data.

[0147] Step 3:

[0148] The device connects to the server and transmits location and orientation information. Based on this data, the server retrieves information about celestial objects visible at that location and direction from its database. The input is location and orientation information, and the output is the corresponding celestial object data.

[0149] Step 4:

[0150] The device uses AR libraries (e.g., ARKit / iOS or ARCore / Android) based on celestial data received from the server to display celestial objects on the user's screen in augmented reality. The input is celestial data, and the output is an augmented reality display.

[0151] Step 5:

[0152] When the user selects a celestial object from the displayed list, the terminal runs a generation AI model on the server to generate a detailed explanation related to that object. The input is the information of the selected celestial object, and the output is a detailed explanatory text.

[0153] Step 6:

[0154] The device checks the system calendar and astronomical database when important astronomical events are approaching and pushes event notifications to the user. The input is event information from the astronomical calendar, and the output is the notification to the user.

[0155] Step 7:

[0156] When a user receives a notification, the device activates the live guide function, providing live coverage of the event and optimal viewing conditions. The input is event information and the current time, and the output is the content of the live guide.

[0157] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0158] This invention relates to a system that uses a user's smartphone or similar portable device to allow them to enjoy astronomical observation while recognizing the user's emotions and providing information and notifications accordingly. This system is particularly composed of a combination of means for acquiring geographic location information, attitude detection, astronomical data acquisition, augmented reality display, information provision, emotion recognition, emotion response, and notification.

[0159] First, the device obtains the user's location information using its built-in GPS function through the application. This location information forms the basis for the celestial information obtained by the celestial data acquisition method described above. Next, the device determines its orientation using information from the gyroscope and accelerometer, and calculates the precise position of the celestial bodies and constellations to be displayed.

[0160] Next, the device communicates with a server to acquire astronomical data for a specific time and location, and overlays it onto the device screen in augmented reality format. This allows the user to intuitively understand what stars and constellations are in the sky in front of them.

[0161] Furthermore, this system uses emotion recognition to determine the user's emotions from their facial expressions and voice. Based on this information, the terminal activates emotion response mechanisms to provide appropriate information according to the user's emotional state. For example, if the user shows interest or surprise, it can present more detailed and interesting information.

[0162] Furthermore, when astronomical events occur, the device notifies the user through a notification system and provides personalized suggestions that take into account the user's emotions and timing. In this way, users can enjoy an even more enriching and interactive astronomical observation experience.

[0163] For example, if a user is observing the night sky with a big smile on their face, the emotion engine could recognize that emotion and the device could then provide information about particularly memorable constellation myths or unique astronomical events. Furthermore, if the user appears tired, the system could advise them to refrain from observing late at night.

[0164] Thus, the system of the present invention can empathize with the user's emotions through astronomical observation and provide a more personalized experience.

[0165] The following describes the processing flow.

[0166] Step 1:

[0167] When a user launches an application and authorizes the use of location information, the device uses its built-in GPS sensor to obtain its current location. This location information includes latitude, longitude, and altitude, and is used to determine the spatial position of celestial bodies.

[0168] Step 2:

[0169] The device activates its gyroscope and accelerometer to obtain its current orientation. This orientation information is used to determine which direction the device is facing and to decide which area of ​​the sky to observe.

[0170] Step 3:

[0171] The device sends a request to the server based on its current location and orientation information, requesting the corresponding celestial object data.

[0172] Step 4:

[0173] The server searches a celestial database based on location and time, and sends information about matching constellations and celestial objects to the terminal.

[0174] Step 5:

[0175] The device analyzes the astronomical data it receives and displays it on the device's screen in augmented reality format. For example, it can overlay the actual night sky, connect constellations with lines, and display each star with a name.

[0176] Step 6:

[0177] The device analyzes the user's facial expressions and voice through its built-in camera and microphone, and uses emotion recognition technology to determine the user's emotions.

[0178] Step 7:

[0179] The device uses emotion response mechanisms to customize the content of the information displayed based on the user's emotions as perceived by the device. For example, if it detects surprise, it will add and display previously unpublished interesting astronomical facts.

[0180] Step 8:

[0181] When a specific celestial event (e.g., the peak of a meteor shower) approaches, the server sends information, and the device sends a push notification to the user. This notification is personalized based on the user's emotions and past observation history.

[0182] Step 9:

[0183] When a user receives a notification, they can initiate observation or change settings based on that notification. The device continuously monitors the user's responses and updates information and suggestions as needed.

[0184] (Example 2)

[0185] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0186] Traditional astronomical observation systems have struggled to provide an engaging experience for users because they merely offer astronomical information without considering the user's emotions or interests. Furthermore, they have been insufficient in providing effective notifications and information timely to astronomical events.

[0187] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0188] In this invention, the server includes a location acquisition means for determining the spatial location of the user, an orientation detection means for determining the orientation of the device used by the user, and a data generation means for generating celestial body-related data based on the location and orientation. This enables not only the generation and display of accurate celestial body data based on the user's location and orientation, but also the provision of personalized information that responds to the user's emotions and appropriate notifications of celestial events.

[0189] "Location acquisition means" refers to a function for accurately determining the user's spatial location.

[0190] "Directional detection means" refers to a function for determining the direction of the device being used by the user.

[0191] "Data generation means" refers to a function that generates celestial body-related data based on the user's location and orientation information.

[0192] "Visualization means" refers to a function that visualizes generated data related to celestial bodies as augmented reality on the user's device.

[0193] An "emotional response mechanism" is a function that determines the user's emotional state and provides information appropriate to that state.

[0194] The "notification function" is a feature that provides users with appropriate notifications when celestial events begin.

[0195] The "information provision function" is a function that provides detailed information about a specific celestial body when the user requests it.

[0196] In an embodiment of the present invention, a system is constructed that enables personalized information provision linked to emotion recognition when a user performs astronomical observations using a smartphone or similar portable device. This system is implemented using a geographic information system (GPS), a gyroscope, a camera, a microphone, augmented reality (AR) software, and software equipped with an emotion recognition algorithm.

[0197] The device uses its built-in GPS function to obtain the user's location information. This allows it to accurately determine the user's current location and provides a foundation for displaying celestial information for that location. The gyroscope and accelerometer are used to determine the device's orientation and display celestial data related to the specific part of the sky the user is observing.

[0198] The server receives this location and orientation information and generates the corresponding celestial data. This process involves linking with astronomical databases to extract accurate data based on the user's current location and time.

[0199] After the data is generated, the device overlays the celestial data onto the user's device screen in augmented reality. This allows the user to intuitively visualize constellations and celestial bodies associated with the current sky.

[0200] The emotion response function uses the camera and microphone to analyze the user's facial expressions and voice, and an emotion recognition algorithm to understand the user's emotional state. For example, if the user is surprised, the system can provide interesting information such as "the myth of the birth of Orion."

[0201] A concrete example of a prompt might be something like, "Suggest astronomical information to provide if the user smiles while stargazing on their smartphone." Based on this prompt, the generative AI model can provide the user with relevant and detailed information.

[0202] In this way, the system of the present invention provides a complementary astronomical observation experience according to the user's location, orientation, and emotional state, resulting in a more engaging and interactive experience for the user.

[0203] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0204] Step 1:

[0205] The device uses the smartphone's built-in GPS function to obtain the user's current location information. In this step, the input is location data from the GPS sensor, and the output is the user's current location information. This data is used as the basis for subsequent astronomical data generation.

[0206] Step 2:

[0207] The device uses data from its gyroscope and accelerometer to determine its orientation. The input is orientation information from each sensor, and the output is the direction the user is observing. Based on this information, the position of the stars to be displayed is calculated.

[0208] Step 3:

[0209] The server receives location and orientation information and queries an astronomical database to generate relevant celestial object data. The input is the user's location and orientation information, and the output is a list of currently observable celestial objects and constellations. Database processing extracts accurate celestial object information.

[0210] Step 4:

[0211] The device receives astronomical data from the server and overlays it onto the screen in augmented reality format. The input here is astronomical data, and the output is a visualized AR display. Users can overlay the astronomical data onto the real sky.

[0212] Step 5:

[0213] The device uses its camera and microphone to capture the user's facial expressions and voice, and then executes an emotion recognition algorithm. The input is image and audio data, and the output is the user's emotional state. The algorithm determines the user's emotion (e.g., joy, surprise).

[0214] Step 6:

[0215] Based on the emotional state detected by the device, a generative AI model is used to generate appropriate prompt messages and provide information to the user. The input for this step is the user's emotional state and astronomical data, and the output is a customized information presentation. Specifically, when the user shows interest, detailed information about a particular constellation or astronomical phenomenon is displayed.

[0216] (Application Example 2)

[0217] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0218] When enjoying astronomical observation, there is a need for a method that provides appropriate information in real time, tailored to the user's emotions when they show excitement or interest, thereby creating a more fulfilling observation experience. However, conventional astronomical observation support systems only provide simple information without considering the user's emotional state, making it difficult to improve the individual user experience. To solve this problem, it is necessary to develop a system that can provide information in accordance with the user's emotions.

[0219] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0220] In this invention, the server includes emotion recognition means for analyzing the user's emotional state, emotion response means for presenting information based on the analysis results, and notification means for notifying the user when an astronomical event occurs. This enables the provision of appropriate information according to the user's emotions, thereby improving the individual user experience.

[0221] "Geographic location information acquisition means" refers to a device or system that has the function of identifying the user's current location and acquiring that location information via a digital terminal.

[0222] "Attitude detection means" refers to a device or method for detecting the orientation of an electronic device using technology that senses the physical orientation and tilt of the device.

[0223] "Data acquisition means" refers to a device or program that has the function of collecting necessary information in digital format and incorporating it into a system.

[0224] "Display means" refers to technologies and devices for providing information to users visually through a device screen.

[0225] "Information provision means" refers to a system or function that provides appropriate data or content in response to user requests or actions.

[0226] "Emotion recognition means" refers to technologies and processes that analyze data such as a user's facial expressions and voice to identify their emotional state.

[0227] An "emotional response system" is a mechanism that provides appropriate information and services based on the user's emotions, as determined by an emotion recognition system.

[0228] A "notification device" is a device or system that has the function of sending warnings or messages to inform users of specific events or information.

[0229] The system realizing this invention utilizes devices such as smartphones and smart glasses to provide users with astronomical observations in an augmented reality format. The server obtains the user's location from the device's GPS module using geolocation information acquisition means. Furthermore, the terminal uses a gyroscope and accelerometer to detect the device's orientation. This allows for the precise determination of the device's orientation and, combined with astronomical data, to display appropriate constellation information relative to the real sky.

[0230] The server collects celestial information based on the user's location and device orientation using methods such as the Google Sky Map API. This information is displayed on the device in augmented reality format. This allows users to see constellations and celestial objects superimposed onto the actual night sky through their device screen.

[0231] The device incorporates a function that analyzes the user's emotional state using emotion recognition APIs such as Microsoft® Azure® Face API. The emotions the user exhibits during astronomical observation (e.g., excitement, surprise) are analyzed, and the server activates an emotion response mechanism based on the results. This makes it possible to provide information about interesting astronomical events and constellations in real time.

[0232] In addition, when astronomical events occur, users will be directly notified using notification methods. For example, when a meteor shower is approaching, warnings and advice such as "A special meteor shower can be observed in the southern sky today" will be provided.

[0233] Examples of prompts could include, "Show the constellations the user can see when they point the app at the sky," or "Search for information about the constellation Orion for a smiling user."

[0234] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0235] Step 1:

[0236] The server uses geolocation information acquisition methods to obtain the user's location information from the GPS module. The input is location data from the device, and the output is the user's latitude and longitude information. Based on this location information, the observation location is determined and used as the basis for acquiring astronomical data.

[0237] Step 2:

[0238] The device acquires orientation data from a gyroscope and accelerometer through attitude detection. The sensor data obtained as input is analyzed to determine the device's orientation (azimuth, tilt angle) as output. This provides information to identify the celestial object in the direction the user is facing.

[0239] Step 3:

[0240] The server utilizes data acquisition methods to collect celestial data using the Google Sky Map API and other tools, based on the user's location and device orientation. It takes location information and device orientation as input and outputs information about specific celestial bodies or constellations.

[0241] Step 4:

[0242] The device displays acquired astronomical data on its screen in AR format via an augmented reality display system. This involves combining astronomical data with the device's camera image in real time, providing an intuitive astronomical display in the user's field of view as output.

[0243] Step 5:

[0244] The user observes through the device and sends facial expressions and voice as input to the terminal using emotion recognition means. The terminal analyzes this input data using an API and identifies the user's emotional state as output.

[0245] Step 6:

[0246] The server activates an emotion response mechanism and, based on the analyzed emotional state, provides relevant astronomical information and myths. In this step, the input is the user's emotional state, and the output is customized content corresponding to that emotion.

[0247] Step 7:

[0248] When an astronomical event occurs, the server sends an alert to the user through a notification system. The server monitors a database of astronomical events and sends an alert message to the user as output, using information that meets specific conditions as input.

[0249] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0250] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0251] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0252] [Second Embodiment]

[0253] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0254] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0255] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0256] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0257] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0258] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0259] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0260] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0261] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0262] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0263] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0264] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0265] The present invention is a system that enables users to perform astronomical observations and easily obtain information about constellations and celestial bodies using a smartphone or similar portable device without special equipment. The main components of this system include means for acquiring geographic location information, means for detecting attitude, means for acquiring celestial body data, means for augmented reality display, means for providing information, and means for notification.

[0266] First, the device uses its built-in GPS function to obtain the user's current location. This information is used as the viewpoint when the user observes celestial bodies. Next, the device uses its gyroscope and accelerometer to detect its orientation and determine which direction it is facing.

[0267] Based on this location and orientation information, the device queries the server to request celestial data visible at its current location and time. The server retrieves data about specific constellations and celestial objects from its database and sends it to the device.

[0268] Subsequently, the device uses augmented reality (AR) technology to display constellations and celestial bodies on the screen, corresponding to the direction the user is pointing the device, based on the received astronomical data. This allows the user to visually recognize constellations and individual stars superimposed on the real night sky.

[0269] Furthermore, when a user selects a specific constellation or celestial body, the device utilizes AI technology to provide the user with explanations and related information about that celestial body. This information includes the mythology of the constellation, its astronomical characteristics, and observational considerations.

[0270] Furthermore, the device sends notifications to the user when specific astronomical events, such as the peak of a meteor shower or a lunar eclipse, are approaching. This notification feature allows users to enjoy important astronomical phenomena without missing them.

[0271] In this way, the present invention provides a user-friendly and educational astronomical observation experience for a wide range of users. Furthermore, it is designed to be easy for beginners to use through an intuitive user interface.

[0272] The following describes the processing flow.

[0273] Step 1:

[0274] The user launches the app. This causes the app to begin preparing to retrieve and process data.

[0275] Step 2:

[0276] The terminal activates the built-in GPS sensor to obtain the user's current location. The location data includes latitude and longitude and is used as a basis for calculations within the device.

[0277] Step 3:

[0278] The terminal activates the gyroscope sensor and the acceleration sensor to detect the current orientation and tilt of the device. This information is used to calculate the exact direction the device is facing.

[0279] Step 4:

[0280] Based on the current location information and attitude information, the terminal sends a request to the server to request celestial data that will be visible at the specified location and time.

[0281] Step 5:

[0282] The server processes the received request, retrieves appropriate celestial data based on the user's location and time from the database, and sends it to the terminal.

[0283] Step 6:

[0284] The terminal analyzes the celestial data received from the server and prepares it for display on the device's screen. The data is overlaid on the screen in augmented reality format.

[0285] Step 7:

[0286] The user looks at the device's screen and checks the information about the displayed constellations and celestial bodies. Tapping on the stars or constellations of interest allows the user to request detailed information.

[0287] Step 8:

[0288] The device responds to user input and uses AI technology to provide detailed information about selected celestial objects. This information is displayed on the screen in text and audio formats.

[0289] Step 9:

[0290] When the device detects an astronomical event (e.g., a meteor shower or a lunar eclipse), it sends a notification to the user. This allows the user to enjoy important events without missing them.

[0291] Step 10:

[0292] When the user closes the app, the device stops the sensors and saves the app's state. This allows for quick resumption the next time the app is launched.

[0293] (Example 1)

[0294] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0295] Traditional astronomical observation systems require users to have specialized equipment, making it difficult to obtain detailed information about celestial objects. Furthermore, users may miss important astronomical events. This limits opportunities for beginners and the general public to become familiar with astronomical observation.

[0296] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0297] In this invention, the server includes means for acquiring location information, means for detecting attitude, and means for acquiring celestial body information. This makes it possible to easily perform astronomical observations using a portable device and obtain detailed celestial body information without requiring special equipment. In addition, a notification function for astronomical events allows users to enjoy important astronomical phenomena without missing them.

[0298] "Location information acquisition means" refers to a device or system that has the function of determining the geographical location of a user.

[0299] "Attitude detection means" refers to sensors or technologies that measure the orientation and tilt of the device used by the user.

[0300] "Data acquisition means" refers to a system for collecting information about celestial bodies based on the user's position and orientation.

[0301] "Display means" refers to technologies and devices for visually displaying acquired celestial information on a user's terminal.

[0302] An "information provision method" refers to a system that has the function of providing users with information about celestial bodies they are looking for.

[0303] "Explanatory means" refers to a system that uses a generative AI model to generate detailed explanations about celestial bodies selected by the user and presents them to the user.

[0304] A "notification device" refers to a device or system that has the function of informing users of the occurrence of astronomical phenomena or events.

[0305] This invention is a system that allows users to perform astronomical observations using a portable information terminal and easily obtain information about constellations and other celestial bodies. The system mainly comprises means for acquiring location information, means for detecting attitude, means for acquiring data, means for display, means for providing information, means for explaining, and means for notifying.

[0306] The device uses its built-in GPS function to obtain the user's current geographical location, thereby establishing a reference point for observation. This location information is combined with the device's orientation, detected by its gyroscope and accelerometer. This allows the device to determine the direction and angle the user is actually facing.

[0307] Utilizing these acquired information, the terminal sends a request to the server via the Internet to obtain information about celestial bodies visible from the current time and location. The server extracts the corresponding information from the celestial body database in response to this request and returns it to the terminal. Based on this, the terminal uses AR technology to superimpose and display constellations and celestial bodies that match the direction the user is facing on the screen.

[0308] The user can tap and select the celestial bodies or constellations displayed on the screen that they are interested in. Detailed information about the selected celestial body is generated based on a prompt by the generative AI model. This information includes mythology, physical characteristics, observational precautions, etc. As a specific example, by inputting a prompt such as "Please tell me the mythology about the Big Dipper" into the AI model, the related mythology is displayed.

[0309] Furthermore, when important astronomical phenomena such as meteor showers and lunar eclipses occur, the terminal sends push notifications to the user so that they can experience these events without missing them.

[0310] This system is an educational tool that allows anyone to easily deepen their knowledge about celestial bodies without the need for special equipment, and is designed to be user-friendly for beginners in astronomy.

[0311] The flow of the specific process in Example 1 will be described using FIG. 11.

[0312] Step 1:

[0313] The terminal acquires location information.

[0314] Input: Latitude and longitude acquired by the GPS sensor of the terminal.

[0315] Process: Process the position data obtained from the GPS to identify the user's current geographical location.

[0316] Output: Data of latitude and longitude indicating the user's current location.

[0317] Specific operation: The message "Locating current location" will be displayed on the device screen, and upon success, the current location will be displayed on the screen.

[0318] Step 2:

[0319] The device detects its posture.

[0320] Input: Data from the gyroscope and accelerometer.

[0321] Processing: Based on information obtained from sensors, the device's orientation and tilt are analyzed to determine which direction the device is facing.

[0322] Output: The direction and angle the device is facing.

[0323] Specific operation: When the device moves, the compass display updates and the icon indicating direction moves.

[0324] Step 3:

[0325] The device sends a request to the server to obtain astronomical information.

[0326] Input: User's current location information (latitude, longitude) and device orientation information (direction, angle).

[0327] Processing: Send location and orientation information to the server and create a request for celestial object data observable at the given time.

[0328] Output: Sends a request for astronomical data to the server.

[0329] Specific operation: A progress bar is displayed while data is being transmitted to inform the user of the progress.

[0330] Step 4:

[0331] The server searches for celestial data based on the request.

[0332] Input: User's location and orientation information sent from the device.

[0333] Processing: Query the database to collect information on constellations and celestial objects observable at the corresponding location and time.

[0334] Output: A set of information on the corresponding constellation or celestial body.

[0335] Specific operation: After obtaining accurate data, the server sends the data to the terminal.

[0336] Step 5:

[0337] Displays astronomical data acquired by the device.

[0338] Input: Celestial information received from the server.

[0339] Processing: Based on the received information, constellations and celestial bodies are drawn on the device screen using AR technology.

[0340] Output: Real-time visuals of constellations and celestial bodies displayed on the screen.

[0341] Specific operation: Constellations and celestial bodies are displayed on the screen, with the sky the user is pointing the screen at as the background.

[0342] Step 6:

[0343] The user selects a specific celestial body and retrieves detailed information.

[0344] Input: The ID or name of the selected celestial body.

[0345] Processing: Create prompts for the generating AI model and generate explanations about the selected celestial body.

[0346] Output: Detailed information text about the selected celestial object.

[0347] Specific operation: When the user taps on a celestial body, information pops up on the screen, and an audio guide plays.

[0348] Step 7:

[0349] The device notifies the user of astronomical events.

[0350] Input: Data from a pre-configured astronomical event.

[0351] Processing: Check the current date and time and trigger a notification when a relevant astronomical event is approaching.

[0352] Output: Notification message provided to the user.

[0353] Specific operation: When a specific event approaches, a pop-up notification will appear along with vibration.

[0354] (Application Example 1)

[0355] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0356] Traditional astronomical observation requires specialized equipment and knowledge, making it inaccessible to the average user. Furthermore, it's difficult to observe celestial events without missing them. Additionally, non-expert users lack the means to gain a deep understanding of the celestial objects they observe. As a result, interest in astronomical phenomena and their educational potential are not being fully utilized.

[0357] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0358] In this invention, the server includes location information acquisition means for obtaining the user's current location information, attitude detection means for detecting the attitude of the mobile device, and data acquisition means for obtaining celestial body information based on the location information and attitude information. This enables the user to perform astronomical observations in real time without special equipment, visually recognize celestial bodies using augmented reality technology, and further enjoy detailed explanations provided by artificial intelligence.

[0359] "Location information acquisition means" refers to a device or system that identifies the user's current location and collects that location information.

[0360] "Attitude detection means" refers to a device or system that uses a gyroscope or accelerometer to detect the direction in which the user's mobile device is facing.

[0361] "Data acquisition means" refers to a device or system that acquires information about celestial bodies visible from the user's viewpoint from a server, based on location information and orientation information.

[0362] "Display means" refers to a device or system that visually displays acquired celestial body information on a user's mobile device using augmented reality technology.

[0363] "Information provision means" refers to a device or system that provides information about a celestial body selected by the user in response to their actions.

[0364] A "notification means" is a device or system that notifies users of information during important celestial events.

[0365] "Guidance means" refers to a function or system for providing live guidance on astronomical events on a mobile device.

[0366] "Artificial intelligence" is a technology that allows computer programs to learn and respond independently in order to generate detailed explanations of celestial objects selected by the user.

[0367] This invention is a system for users to perform astronomical observations using a mobile device without special equipment. The system includes means for acquiring location information, means for detecting attitude, means for acquiring data, means for displaying, means for providing information, means for notifying, means for guiding, and means for providing information using artificial intelligence.

[0368] The server retrieves relevant celestial object information from a celestial database based on the current location information received from the mobile device and sends it to the device. The device receives the celestial object information based on the user's current location and orientation obtained using its built-in GPS and gyroscope sensor, and displays it using augmented reality technology. Open-source AR libraries such as ARKit / iOS and ARCore / Android can be used.

[0369] When providing information, cloud-based AI (e.g., Google Cloud AI) is used to generate detailed explanations about celestial objects selected by the user. Furthermore, as astronomical events approach, the device sends push notifications to the user and provides detailed guidance through a live guide function.

[0370] As a concrete example, if a user points their mobile device towards the night sky and selects "Orion," the device will display information about the mythology of Orion and related astronomical information. Furthermore, during the peak of the Orionian meteor shower, it will notify the user of this information and guide them to the best time and direction to view the meteor shower.

[0371] As an example of a prompt using a generative AI model, in response to the input "Tell me about the myth of Orion," the AI ​​can provide relevant stories and information.

[0372] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0373] Step 1:

[0374] The device uses its built-in GPS sensor to obtain its current location. This location information is collected as latitude and longitude. The input is the signal from the GPS sensor, and the output is the user's location data.

[0375] Step 2:

[0376] The device uses a gyroscope and accelerometer to determine its orientation, that is, the direction it is facing. This orientation information is obtained as elevation angle and azimuth angle. The input is the signal from the sensors, and the output is the orientation data.

[0377] Step 3:

[0378] The device connects to the server and transmits location and orientation information. Based on this data, the server retrieves information about celestial objects visible at that location and direction from its database. The input is location and orientation information, and the output is the corresponding celestial object data.

[0379] Step 4:

[0380] The device uses AR libraries (e.g., ARKit / iOS or ARCore / Android) based on celestial data received from the server to display celestial objects on the user's screen in augmented reality. The input is celestial data, and the output is an augmented reality display.

[0381] Step 5:

[0382] When the user selects a celestial object from the displayed list, the terminal runs a generation AI model on the server to generate a detailed explanation related to that object. The input is the information of the selected celestial object, and the output is a detailed explanatory text.

[0383] Step 6:

[0384] The device checks the system calendar and astronomical database when important astronomical events are approaching and pushes event notifications to the user. The input is event information from the astronomical calendar, and the output is the notification to the user.

[0385] Step 7:

[0386] When a user receives a notification, the device activates the live guide function, providing live coverage of the event and optimal viewing conditions. The input is event information and the current time, and the output is the content of the live guide.

[0387] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0388] This invention relates to a system that uses a user's smartphone or similar portable device to allow them to enjoy astronomical observation while recognizing the user's emotions and providing information and notifications accordingly. This system is particularly composed of a combination of means for acquiring geographic location information, attitude detection, astronomical data acquisition, augmented reality display, information provision, emotion recognition, emotion response, and notification.

[0389] First, the device obtains the user's location information using its built-in GPS function through the application. This location information forms the basis for the celestial information obtained by the celestial data acquisition method described above. Next, the device determines its orientation using information from the gyroscope and accelerometer, and calculates the precise position of the celestial bodies and constellations to be displayed.

[0390] Next, the device communicates with a server to acquire astronomical data for a specific time and location, and overlays it onto the device screen in augmented reality format. This allows the user to intuitively understand what stars and constellations are in the sky in front of them.

[0391] Furthermore, this system uses emotion recognition to determine the user's emotions from their facial expressions and voice. Based on this information, the terminal activates emotion response mechanisms to provide appropriate information according to the user's emotional state. For example, if the user shows interest or surprise, it can present more detailed and interesting information.

[0392] Furthermore, when astronomical events occur, the device notifies the user through a notification system and provides personalized suggestions that take into account the user's emotions and timing. In this way, users can enjoy an even more enriching and interactive astronomical observation experience.

[0393] For example, if a user is observing the night sky with a big smile on their face, the emotion engine could recognize that emotion and the device could then provide information about particularly memorable constellation myths or unique astronomical events. Furthermore, if the user appears tired, the system could advise them to refrain from observing late at night.

[0394] Thus, the system of the present invention can empathize with the user's emotions through astronomical observation and provide a more personalized experience.

[0395] The following describes the processing flow.

[0396] Step 1:

[0397] When a user launches an application and authorizes the use of location information, the device uses its built-in GPS sensor to obtain its current location. This location information includes latitude, longitude, and altitude, and is used to determine the spatial position of celestial bodies.

[0398] Step 2:

[0399] The device activates its gyroscope and accelerometer to obtain its current orientation. This orientation information is used to determine which direction the device is facing and to decide which area of ​​the sky to observe.

[0400] Step 3:

[0401] The device sends a request to the server based on its current location and orientation information, requesting the corresponding celestial object data.

[0402] Step 4:

[0403] The server searches a celestial database based on location and time, and sends information about matching constellations and celestial objects to the terminal.

[0404] Step 5:

[0405] The device analyzes the astronomical data it receives and displays it on the device's screen in augmented reality format. For example, it can overlay the actual night sky, connect constellations with lines, and display each star with a name.

[0406] Step 6:

[0407] The device analyzes the user's facial expressions and voice through its built-in camera and microphone, and uses emotion recognition technology to determine the user's emotions.

[0408] Step 7:

[0409] The device uses emotion response mechanisms to customize the content of the information displayed based on the user's emotions as perceived by the device. For example, if it detects surprise, it will add and display previously unpublished interesting astronomical facts.

[0410] Step 8:

[0411] When a specific celestial event (e.g., the peak of a meteor shower) approaches, the server sends information, and the device sends a push notification to the user. This notification is personalized based on the user's emotions and past observation history.

[0412] Step 9:

[0413] When a user receives a notification, they can initiate observation or change settings based on that notification. The device continuously monitors the user's responses and updates information and suggestions as needed.

[0414] (Example 2)

[0415] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0416] Traditional astronomical observation systems have struggled to provide an engaging experience for users because they merely offer astronomical information without considering the user's emotions or interests. Furthermore, they have been insufficient in providing effective notifications and information timely to astronomical events.

[0417] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0418] In this invention, the server includes a location acquisition means for determining the spatial location of the user, an orientation detection means for determining the orientation of the device used by the user, and a data generation means for generating celestial body-related data based on the location and orientation. This enables not only the generation and display of accurate celestial body data based on the user's location and orientation, but also the provision of personalized information that responds to the user's emotions and appropriate notifications of celestial events.

[0419] "Location acquisition means" refers to a function for accurately determining the user's spatial location.

[0420] "Directional detection means" refers to a function for determining the direction of the device being used by the user.

[0421] "Data generation means" refers to a function that generates celestial body-related data based on the user's location and orientation information.

[0422] "Visualization means" refers to a function that visualizes generated data related to celestial bodies as augmented reality on the user's device.

[0423] An "emotional response mechanism" is a function that determines the user's emotional state and provides information appropriate to that state.

[0424] The "notification function" is a feature that provides users with appropriate notifications when celestial events begin.

[0425] The "information provision function" is a function that provides detailed information about a specific celestial body when the user requests it.

[0426] In an embodiment of the present invention, a system is constructed that enables personalized information provision linked to emotion recognition when a user performs astronomical observations using a smartphone or similar portable device. This system is implemented using a geographic information system (GPS), a gyroscope, a camera, a microphone, augmented reality (AR) software, and software equipped with an emotion recognition algorithm.

[0427] The device uses its built-in GPS function to obtain the user's location information. This allows it to accurately determine the user's current location and provides a foundation for displaying celestial information for that location. The gyroscope and accelerometer are used to determine the device's orientation and display celestial data related to the specific part of the sky the user is observing.

[0428] The server receives this location and orientation information and generates the corresponding celestial data. This process involves linking with astronomical databases to extract accurate data based on the user's current location and time.

[0429] After the data is generated, the device overlays the celestial data onto the user's device screen in augmented reality. This allows the user to intuitively visualize constellations and celestial bodies associated with the current sky.

[0430] The emotion response function uses the camera and microphone to analyze the user's facial expressions and voice, and an emotion recognition algorithm to understand the user's emotional state. For example, if the user is surprised, the system can provide interesting information such as "the myth of the birth of Orion."

[0431] A concrete example of a prompt might be something like, "Suggest astronomical information to provide if the user smiles while stargazing on their smartphone." Based on this prompt, the generative AI model can provide the user with relevant and detailed information.

[0432] In this way, the system of the present invention provides a complementary astronomical observation experience according to the user's location, orientation, and emotional state, resulting in a more engaging and interactive experience for the user.

[0433] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0434] Step 1:

[0435] The device uses the smartphone's built-in GPS function to obtain the user's current location information. In this step, the input is location data from the GPS sensor, and the output is the user's current location information. This data is used as the basis for subsequent astronomical data generation.

[0436] Step 2:

[0437] The device uses data from its gyroscope and accelerometer to determine its orientation. The input is orientation information from each sensor, and the output is the direction the user is observing. Based on this information, the position of the stars to be displayed is calculated.

[0438] Step 3:

[0439] The server receives location and orientation information and queries an astronomical database to generate relevant celestial object data. The input is the user's location and orientation information, and the output is a list of currently observable celestial objects and constellations. Database processing extracts accurate celestial object information.

[0440] Step 4:

[0441] The device receives astronomical data from the server and overlays it onto the screen in augmented reality format. The input here is astronomical data, and the output is a visualized AR display. Users can overlay the astronomical data onto the real sky.

[0442] Step 5:

[0443] The device uses its camera and microphone to capture the user's facial expressions and voice, and then executes an emotion recognition algorithm. The input is image and audio data, and the output is the user's emotional state. The algorithm determines the user's emotion (e.g., joy, surprise).

[0444] Step 6:

[0445] Based on the emotional state detected by the device, a generative AI model is used to generate appropriate prompt messages and provide information to the user. The input for this step is the user's emotional state and astronomical data, and the output is a customized information presentation. Specifically, when the user shows interest, detailed information about a particular constellation or astronomical phenomenon is displayed.

[0446] (Application Example 2)

[0447] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0448] When enjoying astronomical observation, there is a need for a method that provides appropriate information in real time, tailored to the user's emotions when they show excitement or interest, thereby creating a more fulfilling observation experience. However, conventional astronomical observation support systems only provide simple information without considering the user's emotional state, making it difficult to improve the individual user experience. To solve this problem, it is necessary to develop a system that can provide information in accordance with the user's emotions.

[0449] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0450] In this invention, the server includes emotion recognition means for analyzing the user's emotional state, emotion response means for presenting information based on the analysis results, and notification means for notifying the user when an astronomical event occurs. This enables the provision of appropriate information according to the user's emotions, thereby improving the individual user experience.

[0451] "Geographic location information acquisition means" refers to a device or system that has the function of identifying the user's current location and acquiring that location information via a digital terminal.

[0452] "Attitude detection means" refers to a device or method for detecting the orientation of an electronic device using technology that senses the physical orientation and tilt of the device.

[0453] "Data acquisition means" refers to a device or program that has the function of collecting necessary information in digital format and incorporating it into a system.

[0454] "Display means" refers to technologies and devices for providing information to users visually through a device screen.

[0455] "Information provision means" refers to a system or function that provides appropriate data or content in response to user requests or actions.

[0456] "Emotion recognition means" refers to technologies and processes that analyze data such as a user's facial expressions and voice to identify their emotional state.

[0457] An "emotional response system" is a mechanism that provides appropriate information and services based on the user's emotions, as determined by an emotion recognition system.

[0458] A "notification device" is a device or system that has the function of sending warnings or messages to inform users of specific events or information.

[0459] The system realizing this invention utilizes devices such as smartphones and smart glasses to provide users with astronomical observations in an augmented reality format. The server obtains the user's location from the device's GPS module using geolocation information acquisition means. Furthermore, the terminal uses a gyroscope and accelerometer to detect the device's orientation. This allows for the precise determination of the device's orientation and, combined with astronomical data, to display appropriate constellation information relative to the real sky.

[0460] The server collects celestial information based on the user's location and device orientation using methods such as the Google Sky Map API. This information is displayed on the device in augmented reality format. This allows users to see constellations and celestial objects superimposed onto the actual night sky through their device screen.

[0461] The device incorporates a function that analyzes the user's emotional state using emotion recognition APIs such as Microsoft Azure Face API. The emotions the user exhibits during astronomical observation (e.g., excitement, surprise) are analyzed, and the server activates an emotion response mechanism based on the results. This makes it possible to provide information about interesting astronomical events and constellations in real time.

[0462] In addition, when astronomical events occur, users will be directly notified using notification methods. For example, when a meteor shower is approaching, warnings and advice such as "A special meteor shower can be observed in the southern sky today" will be provided.

[0463] Examples of prompts could include, "Show the constellations the user can see when they point the app at the sky," or "Search for information about the constellation Orion for a smiling user."

[0464] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0465] Step 1:

[0466] The server uses geolocation information acquisition methods to obtain the user's location information from the GPS module. The input is location data from the device, and the output is the user's latitude and longitude information. Based on this location information, the observation location is determined and used as the basis for acquiring astronomical data.

[0467] Step 2:

[0468] The device acquires orientation data from a gyroscope and accelerometer through attitude detection. The sensor data obtained as input is analyzed to determine the device's orientation (azimuth, tilt angle) as output. This provides information to identify the celestial object in the direction the user is facing.

[0469] Step 3:

[0470] The server utilizes data acquisition methods to collect celestial data using the Google Sky Map API and other tools, based on the user's location and device orientation. It takes location information and device orientation as input and outputs information about specific celestial bodies or constellations.

[0471] Step 4:

[0472] The device displays acquired astronomical data on its screen in AR format via an augmented reality display system. This involves combining astronomical data with the device's camera image in real time, providing an intuitive astronomical display in the user's field of view as output.

[0473] Step 5:

[0474] The user observes through the device and sends facial expressions and voice as input to the terminal using emotion recognition means. The terminal analyzes this input data using an API and identifies the user's emotional state as output.

[0475] Step 6:

[0476] The server activates an emotion response mechanism and, based on the analyzed emotional state, provides relevant astronomical information and myths. In this step, the input is the user's emotional state, and the output is customized content corresponding to that emotion.

[0477] Step 7:

[0478] When an astronomical event occurs, the server sends an alert to the user through a notification system. The server monitors a database of astronomical events and sends an alert message to the user as output, using information that meets specific conditions as input.

[0479] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0480] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0481] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0482] [Third Embodiment]

[0483] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0484] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0485] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0486] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0487] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0488] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0489] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0490] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0491] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0492] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0493] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0494] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0495] The present invention is a system that enables users to perform astronomical observations and easily obtain information about constellations and celestial bodies using a smartphone or similar portable device without special equipment. The main components of this system include means for acquiring geographic location information, means for detecting attitude, means for acquiring celestial body data, means for augmented reality display, means for providing information, and means for notification.

[0496] First, the device uses its built-in GPS function to obtain the user's current location. This information is used as the viewpoint when the user observes celestial bodies. Next, the device uses its gyroscope and accelerometer to detect its orientation and determine which direction it is facing.

[0497] Based on this location and orientation information, the device queries the server to request celestial data visible at its current location and time. The server retrieves data about specific constellations and celestial objects from its database and sends it to the device.

[0498] Subsequently, the device uses augmented reality (AR) technology to display constellations and celestial bodies on the screen, corresponding to the direction the user is pointing the device, based on the received astronomical data. This allows the user to visually recognize constellations and individual stars superimposed on the real night sky.

[0499] Furthermore, when a user selects a specific constellation or celestial body, the device utilizes AI technology to provide the user with explanations and related information about that celestial body. This information includes the mythology of the constellation, its astronomical characteristics, and observational considerations.

[0500] Furthermore, the device sends notifications to the user when specific astronomical events, such as the peak of a meteor shower or a lunar eclipse, are approaching. This notification feature allows users to enjoy important astronomical phenomena without missing them.

[0501] In this way, the present invention provides a user-friendly and educational astronomical observation experience for a wide range of users. Furthermore, it is designed to be easy for beginners to use through an intuitive user interface.

[0502] The following describes the processing flow.

[0503] Step 1:

[0504] The user launches the app. This causes the app to begin preparing to retrieve and process data.

[0505] Step 2:

[0506] The device activates its built-in GPS sensor to obtain the user's current location. This location data includes latitude and longitude and is used as the basis for calculations within the device.

[0507] Step 3:

[0508] The device activates its gyroscope and accelerometer to detect its current orientation and tilt. This information is used to calculate the precise direction the device is facing.

[0509] Step 4:

[0510] The device sends a request to the server based on its current location and orientation information, requesting celestial object data visible at a specified location and time.

[0511] Step 5:

[0512] The server processes the received request, retrieves appropriate celestial data from the database based on the user's location and time, and sends it to the terminal.

[0513] Step 6:

[0514] The device analyzes astronomical data received from the server and prepares it for display on the device's screen. The data is overlaid on the screen in an augmented reality format.

[0515] Step 7:

[0516] Users can view information about constellations and celestial bodies displayed on their device screen. They can tap on a star or constellation that interests them to request more detailed information.

[0517] Step 8:

[0518] The device responds to user input and uses AI technology to provide detailed information about selected celestial objects. This information is displayed on the screen in text and audio formats.

[0519] Step 9:

[0520] When the device detects an astronomical event (e.g., a meteor shower or a lunar eclipse), it sends a notification to the user. This allows the user to enjoy important events without missing them.

[0521] Step 10:

[0522] When the user closes the app, the device stops the sensors and saves the app's state. This allows for quick resumption the next time the app is launched.

[0523] (Example 1)

[0524] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0525] Traditional astronomical observation systems require users to have specialized equipment, making it difficult to obtain detailed information about celestial objects. Furthermore, users may miss important astronomical events. This limits opportunities for beginners and the general public to become familiar with astronomical observation.

[0526] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0527] In this invention, the server includes means for acquiring location information, means for detecting attitude, and means for acquiring celestial body information. This makes it possible to easily perform astronomical observations using a portable device and obtain detailed celestial body information without requiring special equipment. In addition, a notification function for astronomical events allows users to enjoy important astronomical phenomena without missing them.

[0528] "Location information acquisition means" refers to a device or system that has the function of determining the geographical location of a user.

[0529] "Attitude detection means" refers to sensors or technologies that measure the orientation and tilt of the device used by the user.

[0530] "Data acquisition means" refers to a system for collecting information about celestial bodies based on the user's position and orientation.

[0531] "Display means" refers to technologies and devices for visually displaying acquired celestial information on a user's terminal.

[0532] An "information provision method" refers to a system that has the function of providing users with information about celestial bodies they are looking for.

[0533] "Explanatory means" refers to a system that uses a generative AI model to generate detailed explanations about celestial bodies selected by the user and presents them to the user.

[0534] A "notification device" refers to a device or system that has the function of informing users of the occurrence of astronomical phenomena or events.

[0535] This invention is a system that allows users to perform astronomical observations using a portable information terminal and easily obtain information about constellations and other celestial bodies. The system mainly comprises means for acquiring location information, means for detecting attitude, means for acquiring data, means for display, means for providing information, means for explaining, and means for notifying.

[0536] The device uses its built-in GPS function to obtain the user's current geographical location, thereby establishing a reference point for observation. This location information is combined with the device's orientation, detected by its gyroscope and accelerometer. This allows the device to determine the direction and angle the user is actually facing.

[0537] Using this acquired information, the device sends a request to a server via the internet to obtain information about celestial objects visible from the current time and location. In response to this request, the server extracts the relevant information from a celestial object database and sends it back to the device. Based on this, the device uses AR technology to overlay constellations and celestial objects that match the direction the user is facing onto the screen.

[0538] Users can tap to select constellations and celestial bodies that interest them from those displayed on the screen. Detailed information about the selected celestial body is generated by a generative AI model based on prompts. This information includes myths, physical characteristics, and observational considerations. For example, by inputting the prompt, "Please tell me the myth about the Big Dipper," into the AI ​​model, the relevant myths will be displayed.

[0539] Furthermore, the device will send push notifications to users when important astronomical events such as meteor showers and lunar eclipses occur, ensuring they don't miss out on these events.

[0540] This system is an educational tool that allows anyone to easily deepen their knowledge of celestial bodies without requiring any special equipment, and it is designed to be easy to use even for beginners in astronomy.

[0541] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0542] Step 1:

[0543] The device acquires location information.

[0544] Input: Latitude and longitude obtained from the device's GPS sensor.

[0545] Processing: The system processes location data obtained from GPS to determine the user's current geographical location.

[0546] Output: Latitude and longitude data indicating the user's current location.

[0547] Specific operation: The message "Locating current location" will be displayed on the device screen, and upon success, the current location will be displayed on the screen.

[0548] Step 2:

[0549] The device detects its posture.

[0550] Input: Data from the gyroscope and accelerometer.

[0551] Processing: Based on information obtained from sensors, the device's orientation and tilt are analyzed to determine which direction the device is facing.

[0552] Output: The direction and angle the device is facing.

[0553] Specific operation: When the device moves, the compass display updates and the icon indicating direction moves.

[0554] Step 3:

[0555] The device sends a request to the server to obtain astronomical information.

[0556] Input: User's current location information (latitude, longitude) and device orientation information (direction, angle).

[0557] Processing: Send location and orientation information to the server and create a request for celestial object data observable at the given time.

[0558] Output: Sends a request for astronomical data to the server.

[0559] Specific operation: A progress bar is displayed while data is being transmitted to inform the user of the progress.

[0560] Step 4:

[0561] The server searches for celestial data based on the request.

[0562] Input: User's location and orientation information sent from the device.

[0563] Processing: Query the database to collect information on constellations and celestial objects observable at the corresponding location and time.

[0564] Output: A set of information on the corresponding constellation or celestial body.

[0565] Specific operation: After obtaining accurate data, the server sends the data to the terminal.

[0566] Step 5:

[0567] Displays astronomical data acquired by the device.

[0568] Input: Celestial information received from the server.

[0569] Processing: Based on the received information, constellations and celestial bodies are drawn on the device screen using AR technology.

[0570] Output: Real-time visuals of constellations and celestial bodies displayed on the screen.

[0571] Specific operation: Constellations and celestial bodies are displayed on the screen, with the sky the user is pointing the screen at as the background.

[0572] Step 6:

[0573] The user selects a specific celestial body and retrieves detailed information.

[0574] Input: The ID or name of the selected celestial body.

[0575] Processing: Create prompts for the generating AI model and generate explanations about the selected celestial body.

[0576] Output: Detailed information text about the selected celestial object.

[0577] Specific operation: When the user taps on a celestial body, information pops up on the screen, and an audio guide plays.

[0578] Step 7:

[0579] The device notifies the user of astronomical events.

[0580] Input: Data from a pre-configured astronomical event.

[0581] Processing: Check the current date and time and trigger a notification when a relevant astronomical event is approaching.

[0582] Output: Notification message provided to the user.

[0583] Specific operation: When a specific event approaches, a pop-up notification will appear along with vibration.

[0584] (Application Example 1)

[0585] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0586] Traditional astronomical observation requires specialized equipment and knowledge, making it inaccessible to the average user. Furthermore, it's difficult to observe celestial events without missing them. Additionally, non-expert users lack the means to gain a deep understanding of the celestial objects they observe. As a result, interest in astronomical phenomena and their educational potential are not being fully utilized.

[0587] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0588] In this invention, the server includes location information acquisition means for obtaining the user's current location information, attitude detection means for detecting the attitude of the mobile device, and data acquisition means for obtaining celestial body information based on the location information and attitude information. This enables the user to perform astronomical observations in real time without special equipment, visually recognize celestial bodies using augmented reality technology, and further enjoy detailed explanations provided by artificial intelligence.

[0589] "Location information acquisition means" refers to a device or system that identifies the user's current location and collects that location information.

[0590] "Attitude detection means" refers to a device or system that uses a gyroscope or accelerometer to detect the direction in which the user's mobile device is facing.

[0591] "Data acquisition means" refers to a device or system that acquires information about celestial bodies visible from the user's viewpoint from a server, based on location information and orientation information.

[0592] "Display means" refers to a device or system that visually displays acquired celestial body information on a user's mobile device using augmented reality technology.

[0593] "Information provision means" refers to a device or system that provides information about a celestial body selected by the user in response to their actions.

[0594] A "notification means" is a device or system that notifies users of information during important celestial events.

[0595] "Guidance means" refers to a function or system for providing live guidance on astronomical events on a mobile device.

[0596] "Artificial intelligence" is a technology that allows computer programs to learn and respond independently in order to generate detailed explanations of celestial objects selected by the user.

[0597] This invention is a system for users to perform astronomical observations using a mobile device without special equipment. The system includes means for acquiring location information, means for detecting attitude, means for acquiring data, means for displaying, means for providing information, means for notifying, means for guiding, and means for providing information using artificial intelligence.

[0598] The server retrieves relevant celestial object information from a celestial database based on the current location information received from the mobile device and sends it to the device. The device receives the celestial object information based on the user's current location and orientation obtained using its built-in GPS and gyroscope sensor, and displays it using augmented reality technology. Open-source AR libraries such as ARKit / iOS and ARCore / Android can be used.

[0599] When providing information, cloud-based AI (e.g., Google Cloud AI) is used to generate detailed explanations about celestial objects selected by the user. Furthermore, as astronomical events approach, the device sends push notifications to the user and provides detailed guidance through a live guide function.

[0600] As a concrete example, if a user points their mobile device towards the night sky and selects "Orion," the device will display information about the mythology of Orion and related astronomical information. Furthermore, during the peak of the Orionian meteor shower, it will notify the user of this information and guide them to the best time and direction to view the meteor shower.

[0601] As an example of a prompt using a generative AI model, in response to the input "Tell me about the myth of Orion," the AI ​​can provide relevant stories and information.

[0602] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0603] Step 1:

[0604] The device uses its built-in GPS sensor to obtain its current location. This location information is collected as latitude and longitude. The input is the signal from the GPS sensor, and the output is the user's location data.

[0605] Step 2:

[0606] The device uses a gyroscope and accelerometer to determine its orientation, that is, the direction it is facing. This orientation information is obtained as elevation angle and azimuth angle. The input is the signal from the sensors, and the output is the orientation data.

[0607] Step 3:

[0608] The device connects to the server and transmits location and orientation information. Based on this data, the server retrieves information about celestial objects visible at that location and direction from its database. The input is location and orientation information, and the output is the corresponding celestial object data.

[0609] Step 4:

[0610] The device uses AR libraries (e.g., ARKit / iOS or ARCore / Android) based on celestial data received from the server to display celestial objects on the user's screen in augmented reality. The input is celestial data, and the output is an augmented reality display.

[0611] Step 5:

[0612] When the user selects a celestial object from the displayed list, the terminal runs a generation AI model on the server to generate a detailed explanation related to that object. The input is the information of the selected celestial object, and the output is a detailed explanatory text.

[0613] Step 6:

[0614] The device checks the system calendar and astronomical database when important astronomical events are approaching and pushes event notifications to the user. The input is event information from the astronomical calendar, and the output is the notification to the user.

[0615] Step 7:

[0616] When a user receives a notification, the device activates the live guide function, providing live coverage of the event and optimal viewing conditions. The input is event information and the current time, and the output is the content of the live guide.

[0617] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0618] This invention relates to a system that uses a user's smartphone or similar portable device to allow them to enjoy astronomical observation while recognizing the user's emotions and providing information and notifications accordingly. This system is particularly composed of a combination of means for acquiring geographic location information, attitude detection, astronomical data acquisition, augmented reality display, information provision, emotion recognition, emotion response, and notification.

[0619] First, the device obtains the user's location information using its built-in GPS function through the application. This location information forms the basis for the celestial information obtained by the celestial data acquisition method described above. Next, the device determines its orientation using information from the gyroscope and accelerometer, and calculates the precise position of the celestial bodies and constellations to be displayed.

[0620] Next, the device communicates with a server to acquire astronomical data for a specific time and location, and overlays it onto the device screen in augmented reality format. This allows the user to intuitively understand what stars and constellations are in the sky in front of them.

[0621] Furthermore, this system uses emotion recognition to determine the user's emotions from their facial expressions and voice. Based on this information, the terminal activates emotion response mechanisms to provide appropriate information according to the user's emotional state. For example, if the user shows interest or surprise, it can present more detailed and interesting information.

[0622] Furthermore, when astronomical events occur, the device notifies the user through a notification system and provides personalized suggestions that take into account the user's emotions and timing. In this way, users can enjoy an even more enriching and interactive astronomical observation experience.

[0623] For example, if a user is observing the night sky with a big smile on their face, the emotion engine could recognize that emotion and the device could then provide information about particularly memorable constellation myths or unique astronomical events. Furthermore, if the user appears tired, the system could advise them to refrain from observing late at night.

[0624] Thus, the system of the present invention can empathize with the user's emotions through astronomical observation and provide a more personalized experience.

[0625] The following describes the processing flow.

[0626] Step 1:

[0627] When a user launches an application and authorizes the use of location information, the device uses its built-in GPS sensor to obtain its current location. This location information includes latitude, longitude, and altitude, and is used to determine the spatial position of celestial bodies.

[0628] Step 2:

[0629] The device activates its gyroscope and accelerometer to obtain its current orientation. This orientation information is used to determine which direction the device is facing and to decide which area of ​​the sky to observe.

[0630] Step 3:

[0631] The device sends a request to the server based on its current location and orientation information, requesting the corresponding celestial object data.

[0632] Step 4:

[0633] The server searches a celestial database based on location and time, and sends information about matching constellations and celestial objects to the terminal.

[0634] Step 5:

[0635] The device analyzes the astronomical data it receives and displays it on the device's screen in augmented reality format. For example, it can overlay the actual night sky, connect constellations with lines, and display each star with a name.

[0636] Step 6:

[0637] The device analyzes the user's facial expressions and voice through its built-in camera and microphone, and uses emotion recognition technology to determine the user's emotions.

[0638] Step 7:

[0639] The device uses emotion response mechanisms to customize the content of the information displayed based on the user's emotions as perceived by the device. For example, if it detects surprise, it will add and display previously unpublished interesting astronomical facts.

[0640] Step 8:

[0641] When a specific celestial event (e.g., the peak of a meteor shower) approaches, the server sends information, and the device sends a push notification to the user. This notification is personalized based on the user's emotions and past observation history.

[0642] Step 9:

[0643] When a user receives a notification, they can initiate observation or change settings based on that notification. The device continuously monitors the user's responses and updates information and suggestions as needed.

[0644] (Example 2)

[0645] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0646] Traditional astronomical observation systems have struggled to provide an engaging experience for users because they merely offer astronomical information without considering the user's emotions or interests. Furthermore, they have been insufficient in providing effective notifications and information timely to astronomical events.

[0647] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0648] In this invention, the server includes a location acquisition means for determining the spatial location of the user, an orientation detection means for determining the orientation of the device used by the user, and a data generation means for generating celestial body-related data based on the location and orientation. This enables not only the generation and display of accurate celestial body data based on the user's location and orientation, but also the provision of personalized information that responds to the user's emotions and appropriate notifications of celestial events.

[0649] "Location acquisition means" refers to a function for accurately determining the user's spatial location.

[0650] "Directional detection means" refers to a function for determining the direction of the device being used by the user.

[0651] "Data generation means" refers to a function that generates celestial body-related data based on the user's location and orientation information.

[0652] "Visualization means" refers to a function that visualizes generated data related to celestial bodies as augmented reality on the user's device.

[0653] An "emotional response mechanism" is a function that determines the user's emotional state and provides information appropriate to that state.

[0654] The "notification function" is a feature that provides users with appropriate notifications when celestial events begin.

[0655] The "information provision function" is a function that provides detailed information about a specific celestial body when the user requests it.

[0656] In an embodiment of the present invention, a system is constructed that enables personalized information provision linked to emotion recognition when a user performs astronomical observations using a smartphone or similar portable device. This system is implemented using a geographic information system (GPS), a gyroscope, a camera, a microphone, augmented reality (AR) software, and software equipped with an emotion recognition algorithm.

[0657] The device uses its built-in GPS function to obtain the user's location information. This allows it to accurately determine the user's current location and provides a foundation for displaying celestial information for that location. The gyroscope and accelerometer are used to determine the device's orientation and display celestial data related to the specific part of the sky the user is observing.

[0658] The server receives this location and orientation information and generates the corresponding celestial data. This process involves linking with astronomical databases to extract accurate data based on the user's current location and time.

[0659] After the data is generated, the device overlays the celestial data onto the user's device screen in augmented reality. This allows the user to intuitively visualize constellations and celestial bodies associated with the current sky.

[0660] The emotion response function uses the camera and microphone to analyze the user's facial expressions and voice, and an emotion recognition algorithm to understand the user's emotional state. For example, if the user is surprised, the system can provide interesting information such as "the myth of the birth of Orion."

[0661] A concrete example of a prompt might be something like, "Suggest astronomical information to provide if the user smiles while stargazing on their smartphone." Based on this prompt, the generative AI model can provide the user with relevant and detailed information.

[0662] In this way, the system of the present invention provides a complementary astronomical observation experience according to the user's location, orientation, and emotional state, resulting in a more engaging and interactive experience for the user.

[0663] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0664] Step 1:

[0665] The device uses the smartphone's built-in GPS function to obtain the user's current location information. In this step, the input is location data from the GPS sensor, and the output is the user's current location information. This data is used as the basis for subsequent astronomical data generation.

[0666] Step 2:

[0667] The device uses data from its gyroscope and accelerometer to determine its orientation. The input is orientation information from each sensor, and the output is the direction the user is observing. Based on this information, the position of the stars to be displayed is calculated.

[0668] Step 3:

[0669] The server receives location and orientation information and queries an astronomical database to generate relevant celestial object data. The input is the user's location and orientation information, and the output is a list of currently observable celestial objects and constellations. Database processing extracts accurate celestial object information.

[0670] Step 4:

[0671] The device receives astronomical data from the server and overlays it onto the screen in augmented reality format. The input here is astronomical data, and the output is a visualized AR display. Users can overlay the astronomical data onto the real sky.

[0672] Step 5:

[0673] The device uses its camera and microphone to capture the user's facial expressions and voice, and then executes an emotion recognition algorithm. The input is image and audio data, and the output is the user's emotional state. The algorithm determines the user's emotion (e.g., joy, surprise).

[0674] Step 6:

[0675] Based on the emotional state detected by the device, a generative AI model is used to generate appropriate prompt messages and provide information to the user. The input for this step is the user's emotional state and astronomical data, and the output is a customized information presentation. Specifically, when the user shows interest, detailed information about a particular constellation or astronomical phenomenon is displayed.

[0676] (Application Example 2)

[0677] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0678] When enjoying astronomical observation, there is a need for a method that provides appropriate information in real time, tailored to the user's emotions when they show excitement or interest, thereby creating a more fulfilling observation experience. However, conventional astronomical observation support systems only provide simple information without considering the user's emotional state, making it difficult to improve the individual user experience. To solve this problem, it is necessary to develop a system that can provide information in accordance with the user's emotions.

[0679] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0680] In this invention, the server includes emotion recognition means for analyzing the user's emotional state, emotion response means for presenting information based on the analysis results, and notification means for notifying the user when an astronomical event occurs. This enables the provision of appropriate information according to the user's emotions, thereby improving the individual user experience.

[0681] "Geographic location information acquisition means" refers to a device or system that has the function of identifying the user's current location and acquiring that location information via a digital terminal.

[0682] "Attitude detection means" refers to a device or method for detecting the orientation of an electronic device using technology that senses the physical orientation and tilt of the device.

[0683] "Data acquisition means" refers to a device or program that has the function of collecting necessary information in digital format and incorporating it into a system.

[0684] "Display means" refers to technologies and devices for providing information to users visually through a device screen.

[0685] "Information provision means" refers to a system or function that provides appropriate data or content in response to user requests or actions.

[0686] "Emotion recognition means" refers to technologies and processes that analyze data such as a user's facial expressions and voice to identify their emotional state.

[0687] An "emotional response system" is a mechanism that provides appropriate information and services based on the user's emotions, as determined by an emotion recognition system.

[0688] A "notification device" is a device or system that has the function of sending warnings or messages to inform users of specific events or information.

[0689] The system realizing this invention utilizes devices such as smartphones and smart glasses to provide users with astronomical observations in an augmented reality format. The server obtains the user's location from the device's GPS module using geolocation information acquisition means. Furthermore, the terminal uses a gyroscope and accelerometer to detect the device's orientation. This allows for the precise determination of the device's orientation and, combined with astronomical data, to display appropriate constellation information relative to the real sky.

[0690] The server collects celestial information based on the user's location and device orientation using methods such as the Google Sky Map API. This information is displayed on the device in augmented reality format. This allows users to see constellations and celestial objects superimposed onto the actual night sky through their device screen.

[0691] The device incorporates a function that analyzes the user's emotional state using emotion recognition APIs such as Microsoft Azure Face API. The emotions the user exhibits during astronomical observation (e.g., excitement, surprise) are analyzed, and the server activates an emotion response mechanism based on the results. This makes it possible to provide information about interesting astronomical events and constellations in real time.

[0692] In addition, when astronomical events occur, users will be directly notified using notification methods. For example, when a meteor shower is approaching, warnings and advice such as "A special meteor shower can be observed in the southern sky today" will be provided.

[0693] Examples of prompts could include, "Show the constellations the user can see when they point the app at the sky," or "Search for information about the constellation Orion for a smiling user."

[0694] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0695] Step 1:

[0696] The server uses geolocation information acquisition methods to obtain the user's location information from the GPS module. The input is location data from the device, and the output is the user's latitude and longitude information. Based on this location information, the observation location is determined and used as the basis for acquiring astronomical data.

[0697] Step 2:

[0698] The device acquires orientation data from a gyroscope and accelerometer through attitude detection. The sensor data obtained as input is analyzed to determine the device's orientation (azimuth, tilt angle) as output. This provides information to identify the celestial object in the direction the user is facing.

[0699] Step 3:

[0700] The server utilizes data acquisition methods to collect celestial data using the Google Sky Map API and other tools, based on the user's location and device orientation. It takes location information and device orientation as input and outputs information about specific celestial bodies or constellations.

[0701] Step 4:

[0702] The device displays acquired astronomical data on its screen in AR format via an augmented reality display system. This involves combining astronomical data with the device's camera image in real time, providing an intuitive astronomical display in the user's field of view as output.

[0703] Step 5:

[0704] The user observes through the device and sends facial expressions and voice as input to the terminal using emotion recognition means. The terminal analyzes this input data using an API and identifies the user's emotional state as output.

[0705] Step 6:

[0706] The server activates an emotion response mechanism and, based on the analyzed emotional state, provides relevant astronomical information and myths. In this step, the input is the user's emotional state, and the output is customized content corresponding to that emotion.

[0707] Step 7:

[0708] When an astronomical event occurs, the server sends an alert to the user through a notification system. The server monitors a database of astronomical events and sends an alert message to the user as output, using information that meets specific conditions as input.

[0709] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0710] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0711] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0712] [Fourth Embodiment]

[0713] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0714] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0715] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0716] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0717] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0718] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0719] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0720] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0721] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0722] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0723] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0724] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0725] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0726] The present invention is a system that enables users to perform astronomical observations and easily obtain information about constellations and celestial bodies using a smartphone or similar portable device without special equipment. The main components of this system include means for acquiring geographic location information, means for detecting attitude, means for acquiring celestial body data, means for augmented reality display, means for providing information, and means for notification.

[0727] First, the device uses its built-in GPS function to obtain the user's current location. This information is used as the viewpoint when the user observes celestial bodies. Next, the device uses its gyroscope and accelerometer to detect its orientation and determine which direction it is facing.

[0728] Based on this location and orientation information, the device queries the server to request celestial data visible at its current location and time. The server retrieves data about specific constellations and celestial objects from its database and sends it to the device.

[0729] Subsequently, the device uses augmented reality (AR) technology to display constellations and celestial bodies on the screen, corresponding to the direction the user is pointing the device, based on the received astronomical data. This allows the user to visually recognize constellations and individual stars superimposed on the real night sky.

[0730] Furthermore, when a user selects a specific constellation or celestial body, the device utilizes AI technology to provide the user with explanations and related information about that celestial body. This information includes the mythology of the constellation, its astronomical characteristics, and observational considerations.

[0731] Furthermore, the device sends notifications to the user when specific astronomical events, such as the peak of a meteor shower or a lunar eclipse, are approaching. This notification feature allows users to enjoy important astronomical phenomena without missing them.

[0732] In this way, the present invention provides a user-friendly and educational astronomical observation experience for a wide range of users. Furthermore, it is designed to be easy for beginners to use through an intuitive user interface.

[0733] The following describes the processing flow.

[0734] Step 1:

[0735] The user launches the app. This causes the app to begin preparing to retrieve and process data.

[0736] Step 2:

[0737] The device activates its built-in GPS sensor to obtain the user's current location. This location data includes latitude and longitude and is used as the basis for calculations within the device.

[0738] Step 3:

[0739] The device activates its gyroscope and accelerometer to detect its current orientation and tilt. This information is used to calculate the precise direction the device is facing.

[0740] Step 4:

[0741] The device sends a request to the server based on its current location and orientation information, requesting celestial object data visible at a specified location and time.

[0742] Step 5:

[0743] The server processes the received request, retrieves appropriate celestial data from the database based on the user's location and time, and sends it to the terminal.

[0744] Step 6:

[0745] The device analyzes astronomical data received from the server and prepares it for display on the device's screen. The data is overlaid on the screen in an augmented reality format.

[0746] Step 7:

[0747] Users can view information about constellations and celestial bodies displayed on their device screen. They can tap on a star or constellation that interests them to request more detailed information.

[0748] Step 8:

[0749] The device responds to user input and uses AI technology to provide detailed information about selected celestial objects. This information is displayed on the screen in text and audio formats.

[0750] Step 9:

[0751] When the device detects an astronomical event (e.g., a meteor shower or a lunar eclipse), it sends a notification to the user. This allows the user to enjoy important events without missing them.

[0752] Step 10:

[0753] When the user closes the app, the device stops the sensors and saves the app's state. This allows for quick resumption the next time the app is launched.

[0754] (Example 1)

[0755] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0756] Traditional astronomical observation systems require users to have specialized equipment, making it difficult to obtain detailed information about celestial objects. Furthermore, users may miss important astronomical events. This limits opportunities for beginners and the general public to become familiar with astronomical observation.

[0757] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0758] In this invention, the server includes means for acquiring location information, means for detecting attitude, and means for acquiring celestial body information. This makes it possible to easily perform astronomical observations using a portable device and obtain detailed celestial body information without requiring special equipment. In addition, a notification function for astronomical events allows users to enjoy important astronomical phenomena without missing them.

[0759] "Location information acquisition means" refers to a device or system that has the function of determining the geographical location of a user.

[0760] "Attitude detection means" refers to sensors or technologies that measure the orientation and tilt of the device used by the user.

[0761] "Data acquisition means" refers to a system for collecting information about celestial bodies based on the user's position and orientation.

[0762] "Display means" refers to technologies and devices for visually displaying acquired celestial information on a user's terminal.

[0763] An "information provision method" refers to a system that has the function of providing users with information about celestial bodies they are looking for.

[0764] "Explanatory means" refers to a system that uses a generative AI model to generate detailed explanations about celestial bodies selected by the user and presents them to the user.

[0765] A "notification device" refers to a device or system that has the function of informing users of the occurrence of astronomical phenomena or events.

[0766] This invention is a system that allows users to perform astronomical observations using a portable information terminal and easily obtain information about constellations and other celestial bodies. The system mainly comprises means for acquiring location information, means for detecting attitude, means for acquiring data, means for display, means for providing information, means for explaining, and means for notifying.

[0767] The device uses its built-in GPS function to obtain the user's current geographical location, thereby establishing a reference point for observation. This location information is combined with the device's orientation, detected by its gyroscope and accelerometer. This allows the device to determine the direction and angle the user is actually facing.

[0768] Using this acquired information, the device sends a request to a server via the internet to obtain information about celestial objects visible from the current time and location. In response to this request, the server extracts the relevant information from a celestial object database and sends it back to the device. Based on this, the device uses AR technology to overlay constellations and celestial objects that match the direction the user is facing onto the screen.

[0769] Users can tap to select constellations and celestial bodies that interest them from those displayed on the screen. Detailed information about the selected celestial body is generated by a generative AI model based on prompts. This information includes myths, physical characteristics, and observational considerations. For example, by inputting the prompt, "Please tell me the myth about the Big Dipper," into the AI ​​model, the relevant myths will be displayed.

[0770] Furthermore, the device will send push notifications to users when important astronomical events such as meteor showers and lunar eclipses occur, ensuring they don't miss out on these events.

[0771] This system is an educational tool that allows anyone to easily deepen their knowledge of celestial bodies without requiring any special equipment, and it is designed to be easy to use even for beginners in astronomy.

[0772] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0773] Step 1:

[0774] The device acquires location information.

[0775] Input: Latitude and longitude obtained from the device's GPS sensor.

[0776] Processing: The system processes location data obtained from GPS to determine the user's current geographical location.

[0777] Output: Latitude and longitude data indicating the user's current location.

[0778] Specific operation: The message "Locating current location" will be displayed on the device screen, and upon success, the current location will be displayed on the screen.

[0779] Step 2:

[0780] The device detects its posture.

[0781] Input: Data from the gyroscope and accelerometer.

[0782] Processing: Based on information obtained from sensors, the device's orientation and tilt are analyzed to determine which direction the device is facing.

[0783] Output: The direction and angle the device is facing.

[0784] Specific operation: When the device moves, the compass display updates and the icon indicating direction moves.

[0785] Step 3:

[0786] The device sends a request to the server to obtain astronomical information.

[0787] Input: User's current location information (latitude, longitude) and device orientation information (direction, angle).

[0788] Processing: Send location and orientation information to the server and create a request for celestial object data observable at the given time.

[0789] Output: Sends a request for astronomical data to the server.

[0790] Specific operation: A progress bar is displayed while data is being transmitted to inform the user of the progress.

[0791] Step 4:

[0792] The server searches for celestial data based on the request.

[0793] Input: User's location and orientation information sent from the device.

[0794] Processing: Query the database to collect information on constellations and celestial objects observable at the corresponding location and time.

[0795] Output: A set of information on the corresponding constellation or celestial body.

[0796] Specific operation: After obtaining accurate data, the server sends the data to the terminal.

[0797] Step 5:

[0798] Displays astronomical data acquired by the device.

[0799] Input: Celestial information received from the server.

[0800] Processing: Based on the received information, constellations and celestial bodies are drawn on the device screen using AR technology.

[0801] Output: Real-time visuals of constellations and celestial bodies displayed on the screen.

[0802] Specific operation: Constellations and celestial bodies are displayed on the screen, with the sky the user is pointing the screen at as the background.

[0803] Step 6:

[0804] The user selects a specific celestial body and retrieves detailed information.

[0805] Input: The ID or name of the selected celestial body.

[0806] Processing: Create prompts for the generating AI model and generate explanations about the selected celestial body.

[0807] Output: Detailed information text about the selected celestial object.

[0808] Specific operation: When the user taps on a celestial body, information pops up on the screen, and an audio guide plays.

[0809] Step 7:

[0810] The device notifies the user of astronomical events.

[0811] Input: Data from a pre-configured astronomical event.

[0812] Processing: Check the current date and time and trigger a notification when a relevant astronomical event is approaching.

[0813] Output: Notification message provided to the user.

[0814] Specific operation: When a specific event approaches, a pop-up notification will appear along with vibration.

[0815] (Application Example 1)

[0816] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0817] Traditional astronomical observation requires specialized equipment and knowledge, making it inaccessible to the average user. Furthermore, it's difficult to observe celestial events without missing them. Additionally, non-expert users lack the means to gain a deep understanding of the celestial objects they observe. As a result, interest in astronomical phenomena and their educational potential are not being fully utilized.

[0818] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0819] In this invention, the server includes location information acquisition means for obtaining the user's current location information, attitude detection means for detecting the attitude of the mobile device, and data acquisition means for obtaining celestial body information based on the location information and attitude information. This enables the user to perform astronomical observations in real time without special equipment, visually recognize celestial bodies using augmented reality technology, and further enjoy detailed explanations provided by artificial intelligence.

[0820] "Location information acquisition means" refers to a device or system that identifies the user's current location and collects that location information.

[0821] "Attitude detection means" refers to a device or system that uses a gyroscope or accelerometer to detect the direction in which the user's mobile device is facing.

[0822] "Data acquisition means" refers to a device or system that acquires information about celestial bodies visible from the user's viewpoint from a server, based on location information and orientation information.

[0823] "Display means" refers to a device or system that visually displays acquired celestial body information on a user's mobile device using augmented reality technology.

[0824] "Information provision means" refers to a device or system that provides information about a celestial body selected by the user in response to their actions.

[0825] A "notification means" is a device or system that notifies users of information during important celestial events.

[0826] "Guidance means" refers to a function or system for providing live guidance on astronomical events on a mobile device.

[0827] "Artificial intelligence" is a technology that allows computer programs to learn and respond independently in order to generate detailed explanations of celestial objects selected by the user.

[0828] This invention is a system for users to perform astronomical observations using a mobile device without special equipment. The system includes means for acquiring location information, means for detecting attitude, means for acquiring data, means for displaying, means for providing information, means for notifying, means for guiding, and means for providing information using artificial intelligence.

[0829] The server retrieves relevant celestial object information from a celestial database based on the current location information received from the mobile device and sends it to the device. The device receives the celestial object information based on the user's current location and orientation obtained using its built-in GPS and gyroscope sensor, and displays it using augmented reality technology. Open-source AR libraries such as ARKit / iOS and ARCore / Android can be used.

[0830] When providing information, cloud-based AI (e.g., Google Cloud AI) is used to generate detailed explanations about celestial objects selected by the user. Furthermore, as astronomical events approach, the device sends push notifications to the user and provides detailed guidance through a live guide function.

[0831] As a concrete example, if a user points their mobile device towards the night sky and selects "Orion," the device will display information about the mythology of Orion and related astronomical information. Furthermore, during the peak of the Orionian meteor shower, it will notify the user of this information and guide them to the best time and direction to view the meteor shower.

[0832] As an example of a prompt using a generative AI model, in response to the input "Tell me about the myth of Orion," the AI ​​can provide relevant stories and information.

[0833] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0834] Step 1:

[0835] The device uses its built-in GPS sensor to obtain its current location. This location information is collected as latitude and longitude. The input is the signal from the GPS sensor, and the output is the user's location data.

[0836] Step 2:

[0837] The device uses a gyroscope and accelerometer to determine its orientation, that is, the direction it is facing. This orientation information is obtained as elevation angle and azimuth angle. The input is the signal from the sensors, and the output is the orientation data.

[0838] Step 3:

[0839] The device connects to the server and transmits location and orientation information. Based on this data, the server retrieves information about celestial objects visible at that location and direction from its database. The input is location and orientation information, and the output is the corresponding celestial object data.

[0840] Step 4:

[0841] The device uses AR libraries (e.g., ARKit / iOS or ARCore / Android) based on celestial data received from the server to display celestial objects on the user's screen in augmented reality. The input is celestial data, and the output is an augmented reality display.

[0842] Step 5:

[0843] When the user selects a celestial object from the displayed list, the terminal runs a generation AI model on the server to generate a detailed explanation related to that object. The input is the information of the selected celestial object, and the output is a detailed explanatory text.

[0844] Step 6:

[0845] The device checks the system calendar and astronomical database when important astronomical events are approaching and pushes event notifications to the user. The input is event information from the astronomical calendar, and the output is the notification to the user.

[0846] Step 7:

[0847] When a user receives a notification, the device activates the live guide function, providing live coverage of the event and optimal viewing conditions. The input is event information and the current time, and the output is the content of the live guide.

[0848] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0849] This invention relates to a system that uses a user's smartphone or similar portable device to allow them to enjoy astronomical observation while recognizing the user's emotions and providing information and notifications accordingly. This system is particularly composed of a combination of means for acquiring geographic location information, attitude detection, astronomical data acquisition, augmented reality display, information provision, emotion recognition, emotion response, and notification.

[0850] First, the device obtains the user's location information using its built-in GPS function through the application. This location information forms the basis for the celestial information obtained by the celestial data acquisition method described above. Next, the device determines its orientation using information from the gyroscope and accelerometer, and calculates the precise position of the celestial bodies and constellations to be displayed.

[0851] Next, the device communicates with a server to acquire astronomical data for a specific time and location, and overlays it onto the device screen in augmented reality format. This allows the user to intuitively understand what stars and constellations are in the sky in front of them.

[0852] Furthermore, this system uses emotion recognition to determine the user's emotions from their facial expressions and voice. Based on this information, the terminal activates emotion response mechanisms to provide appropriate information according to the user's emotional state. For example, if the user shows interest or surprise, it can present more detailed and interesting information.

[0853] Furthermore, when astronomical events occur, the device notifies the user through a notification system and provides personalized suggestions that take into account the user's emotions and timing. In this way, users can enjoy an even more enriching and interactive astronomical observation experience.

[0854] For example, if a user is observing the night sky with a big smile on their face, the emotion engine could recognize that emotion and the device could then provide information about particularly memorable constellation myths or unique astronomical events. Furthermore, if the user appears tired, the system could advise them to refrain from observing late at night.

[0855] Thus, the system of the present invention can empathize with the user's emotions through astronomical observation and provide a more personalized experience.

[0856] The following describes the processing flow.

[0857] Step 1:

[0858] When a user launches an application and authorizes the use of location information, the device uses its built-in GPS sensor to obtain its current location. This location information includes latitude, longitude, and altitude, and is used to determine the spatial position of celestial bodies.

[0859] Step 2:

[0860] The device activates its gyroscope and accelerometer to obtain its current orientation. This orientation information is used to determine which direction the device is facing and to decide which area of ​​the sky to observe.

[0861] Step 3:

[0862] The device sends a request to the server based on its current location and orientation information, requesting the corresponding celestial object data.

[0863] Step 4:

[0864] The server searches a celestial database based on location and time, and sends information about matching constellations and celestial objects to the terminal.

[0865] Step 5:

[0866] The device analyzes the astronomical data it receives and displays it on the device's screen in augmented reality format. For example, it can overlay the actual night sky, connect constellations with lines, and display each star with a name.

[0867] Step 6:

[0868] The device analyzes the user's facial expressions and voice through its built-in camera and microphone, and uses emotion recognition technology to determine the user's emotions.

[0869] Step 7:

[0870] The device uses emotion response mechanisms to customize the content of the information displayed based on the user's emotions as perceived by the device. For example, if it detects surprise, it will add and display previously unpublished interesting astronomical facts.

[0871] Step 8:

[0872] When a specific celestial event (e.g., the peak of a meteor shower) approaches, the server sends information, and the device sends a push notification to the user. This notification is personalized based on the user's emotions and past observation history.

[0873] Step 9:

[0874] When a user receives a notification, they can initiate observation or change settings based on that notification. The device continuously monitors the user's responses and updates information and suggestions as needed.

[0875] (Example 2)

[0876] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0877] Traditional astronomical observation systems have struggled to provide an engaging experience for users because they merely offer astronomical information without considering the user's emotions or interests. Furthermore, they have been insufficient in providing effective notifications and information timely to astronomical events.

[0878] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0879] In this invention, the server includes a location acquisition means for determining the spatial location of the user, an orientation detection means for determining the orientation of the device used by the user, and a data generation means for generating celestial body-related data based on the location and orientation. This enables not only the generation and display of accurate celestial body data based on the user's location and orientation, but also the provision of personalized information that responds to the user's emotions and appropriate notifications of celestial events.

[0880] "Location acquisition means" refers to a function for accurately determining the user's spatial location.

[0881] "Directional detection means" refers to a function for determining the direction of the device being used by the user.

[0882] "Data generation means" refers to a function that generates celestial body-related data based on the user's location and orientation information.

[0883] "Visualization means" refers to a function that visualizes generated data related to celestial bodies as augmented reality on the user's device.

[0884] An "emotional response mechanism" is a function that determines the user's emotional state and provides information appropriate to that state.

[0885] The "notification function" is a feature that provides users with appropriate notifications when celestial events begin.

[0886] The "information provision function" is a function that provides detailed information about a specific celestial body when the user requests it.

[0887] In an embodiment of the present invention, a system is constructed that enables personalized information provision linked to emotion recognition when a user performs astronomical observations using a smartphone or similar portable device. This system is implemented using a geographic information system (GPS), a gyroscope, a camera, a microphone, augmented reality (AR) software, and software equipped with an emotion recognition algorithm.

[0888] The device uses its built-in GPS function to obtain the user's location information. This allows it to accurately determine the user's current location and provides a foundation for displaying celestial information for that location. The gyroscope and accelerometer are used to determine the device's orientation and display celestial data related to the specific part of the sky the user is observing.

[0889] The server receives this location and orientation information and generates the corresponding celestial data. This process involves linking with astronomical databases to extract accurate data based on the user's current location and time.

[0890] After the data is generated, the device overlays the celestial data onto the user's device screen in augmented reality. This allows the user to intuitively visualize constellations and celestial bodies associated with the current sky.

[0891] The emotion response function uses the camera and microphone to analyze the user's facial expressions and voice, and an emotion recognition algorithm to understand the user's emotional state. For example, if the user is surprised, the system can provide interesting information such as "the myth of the birth of Orion."

[0892] A concrete example of a prompt might be something like, "Suggest astronomical information to provide if the user smiles while stargazing on their smartphone." Based on this prompt, the generative AI model can provide the user with relevant and detailed information.

[0893] In this way, the system of the present invention provides a complementary astronomical observation experience according to the user's location, orientation, and emotional state, resulting in a more engaging and interactive experience for the user.

[0894] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0895] Step 1:

[0896] The device uses the smartphone's built-in GPS function to obtain the user's current location information. In this step, the input is location data from the GPS sensor, and the output is the user's current location information. This data is used as the basis for subsequent astronomical data generation.

[0897] Step 2:

[0898] The device uses data from its gyroscope and accelerometer to determine its orientation. The input is orientation information from each sensor, and the output is the direction the user is observing. Based on this information, the position of the stars to be displayed is calculated.

[0899] Step 3:

[0900] The server receives location and orientation information and queries an astronomical database to generate relevant celestial object data. The input is the user's location and orientation information, and the output is a list of currently observable celestial objects and constellations. Database processing extracts accurate celestial object information.

[0901] Step 4:

[0902] The device receives astronomical data from the server and overlays it onto the screen in augmented reality format. The input here is astronomical data, and the output is a visualized AR display. Users can overlay the astronomical data onto the real sky.

[0903] Step 5:

[0904] The device uses its camera and microphone to capture the user's facial expressions and voice, and then executes an emotion recognition algorithm. The input is image and audio data, and the output is the user's emotional state. The algorithm determines the user's emotion (e.g., joy, surprise).

[0905] Step 6:

[0906] Based on the emotional state detected by the device, a generative AI model is used to generate appropriate prompt messages and provide information to the user. The input for this step is the user's emotional state and astronomical data, and the output is a customized information presentation. Specifically, when the user shows interest, detailed information about a particular constellation or astronomical phenomenon is displayed.

[0907] (Application Example 2)

[0908] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0909] When enjoying astronomical observation, there is a need for a method that provides appropriate information in real time, tailored to the user's emotions when they show excitement or interest, thereby creating a more fulfilling observation experience. However, conventional astronomical observation support systems only provide simple information without considering the user's emotional state, making it difficult to improve the individual user experience. To solve this problem, it is necessary to develop a system that can provide information in accordance with the user's emotions.

[0910] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0911] In this invention, the server includes emotion recognition means for analyzing the user's emotional state, emotion response means for presenting information based on the analysis results, and notification means for notifying the user when an astronomical event occurs. This enables the provision of appropriate information according to the user's emotions, thereby improving the individual user experience.

[0912] "Geographic location information acquisition means" refers to a device or system that has the function of identifying the user's current location and acquiring that location information via a digital terminal.

[0913] "Attitude detection means" refers to a device or method for detecting the orientation of an electronic device using technology that senses the physical orientation and tilt of the device.

[0914] "Data acquisition means" refers to a device or program that has the function of collecting necessary information in digital format and incorporating it into a system.

[0915] "Display means" refers to technologies and devices for providing information to users visually through a device screen.

[0916] "Information provision means" refers to a system or function that provides appropriate data or content in response to user requests or actions.

[0917] "Emotion recognition means" refers to technologies and processes that analyze data such as a user's facial expressions and voice to identify their emotional state.

[0918] An "emotional response system" is a mechanism that provides appropriate information and services based on the user's emotions, as determined by an emotion recognition system.

[0919] A "notification device" is a device or system that has the function of sending warnings or messages to inform users of specific events or information.

[0920] The system realizing this invention utilizes devices such as smartphones and smart glasses to provide users with astronomical observations in an augmented reality format. The server obtains the user's location from the device's GPS module using geolocation information acquisition means. Furthermore, the terminal uses a gyroscope and accelerometer to detect the device's orientation. This allows for the precise determination of the device's orientation and, combined with astronomical data, to display appropriate constellation information relative to the real sky.

[0921] The server collects celestial information based on the user's location and device orientation using methods such as the Google Sky Map API. This information is displayed on the device in augmented reality format. This allows users to see constellations and celestial objects superimposed onto the actual night sky through their device screen.

[0922] The device incorporates a function that analyzes the user's emotional state using emotion recognition APIs such as Microsoft Azure Face API. The emotions the user exhibits during astronomical observation (e.g., excitement, surprise) are analyzed, and the server activates an emotion response mechanism based on the results. This makes it possible to provide information about interesting astronomical events and constellations in real time.

[0923] In addition, when astronomical events occur, users will be directly notified using notification methods. For example, when a meteor shower is approaching, warnings and advice such as "A special meteor shower can be observed in the southern sky today" will be provided.

[0924] Examples of prompts could include, "Show the constellations the user can see when they point the app at the sky," or "Search for information about the constellation Orion for a smiling user."

[0925] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0926] Step 1:

[0927] The server uses geolocation information acquisition methods to obtain the user's location information from the GPS module. The input is location data from the device, and the output is the user's latitude and longitude information. Based on this location information, the observation location is determined and used as the basis for acquiring astronomical data.

[0928] Step 2:

[0929] The device acquires orientation data from a gyroscope and accelerometer through attitude detection. The sensor data obtained as input is analyzed to determine the device's orientation (azimuth, tilt angle) as output. This provides information to identify the celestial object in the direction the user is facing.

[0930] Step 3:

[0931] The server utilizes data acquisition methods to collect celestial data using the Google Sky Map API and other tools, based on the user's location and device orientation. It takes location information and device orientation as input and outputs information about specific celestial bodies or constellations.

[0932] Step 4:

[0933] The device displays acquired astronomical data on its screen in AR format via an augmented reality display system. This involves combining astronomical data with the device's camera image in real time, providing an intuitive astronomical display in the user's field of view as output.

[0934] Step 5:

[0935] The user observes through the device and sends facial expressions and voice as input to the terminal using emotion recognition means. The terminal analyzes this input data using an API and identifies the user's emotional state as output.

[0936] Step 6:

[0937] The server activates an emotion response mechanism and, based on the analyzed emotional state, provides relevant astronomical information and myths. In this step, the input is the user's emotional state, and the output is customized content corresponding to that emotion.

[0938] Step 7:

[0939] When an astronomical event occurs, the server sends an alert to the user through a notification system. The server monitors a database of astronomical events and sends an alert message to the user as output, using information that meets specific conditions as input.

[0940] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0941] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0942] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0943] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0944] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0945] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0946] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0947] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0948] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0949] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0950] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0951] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0952] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0953] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0954] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0955] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0956] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0957] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0958] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0959] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0960] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0961] The following is further disclosed regarding the embodiments described above.

[0962] (Claim 1)

[0963] A means for obtaining geolocation information to obtain the user's location information,

[0964] A posture detection means for detecting the orientation of the user's device,

[0965] A data acquisition means for acquiring celestial body data based on the above-mentioned positional information and direction,

[0966] A display means for displaying acquired celestial data in augmented reality format on the user's device,

[0967] Information provision means for providing information about celestial bodies in response to user input,

[0968] A system that includes this.

[0969] (Claim 2)

[0970] The system according to claim 1, comprising information response means for providing relevant information when a user requests detailed information about a specific celestial body.

[0971] (Claim 3)

[0972] The system according to claim 1, further comprising a notification means for notifying a user when an astronomical event occurs.

[0973] "Example 1"

[0974] (Claim 1)

[0975] A means for obtaining the user's location information,

[0976] A posture detection means for detecting the orientation of the user's device,

[0977] A data acquisition means for acquiring celestial body information based on the above positional and orientation information,

[0978] A display means for displaying acquired celestial information on the user's device in augmented reality format,

[0979] Information provision means for providing information about celestial bodies in response to user input,

[0980] An explanatory means that generates a detailed explanation of a celestial body selected by the user using a generative AI model,

[0981] A notification method for informing users when astronomical phenomena occur,

[0982] A system that includes this.

[0983] (Claim 2)

[0984] The system according to claim 1, comprising an information response means for generating and providing relevant information using a generative AI model when a user requests detailed information about a specific celestial body.

[0985] (Claim 3)

[0986] The system according to claim 1, further comprising notification means for providing a notification to a user, accompanied by sound or animation, in the event of an astronomical event.

[0987] "Application Example 1"

[0988] (Claim 1)

[0989] A means for obtaining the user's current location information,

[0990] A posture detection means for detecting the posture of the user's mobile device,

[0991] A data acquisition means for acquiring celestial body information based on the aforementioned positional and orientation information,

[0992] A display means for displaying acquired celestial information on a mobile device in augmented reality format,

[0993] Information provision means for providing information related to celestial bodies in accordance with user operations,

[0994] A notification method for informing users of important astronomical events,

[0995] A guiding method for providing live guides for astronomical events on mobile devices,

[0996] A system that includes this.

[0997] (Claim 2)

[0998] The system according to claim 1, comprising means for providing informational responses using artificial intelligence to generate a detailed explanation of a celestial body selected by the user.

[0999] (Claim 3)

[1000] The system according to claim 1, further comprising a means for providing users with explanations using a generative AI model related to astronomical events.

[1001] "Example 2 of combining an emotion engine"

[1002] (Claim 1)

[1003] A means for obtaining location information to determine the spatial location of the user,

[1004] Direction detection means for determining the orientation of the device used by the user,

[1005] A data generation means for generating data related to celestial bodies based on the aforementioned position and orientation,

[1006] A visualization means for visualizing generated celestial body-related data as augmented reality on the user's device,

[1007] An emotional response means for determining the user's emotional state and providing information according to that state,

[1008] A system that includes a notification function to inform users when celestial events begin.

[1009] (Claim 2)

[1010] The system according to claim 1, further comprising an information provision function for providing relevant information when a user requests detailed information about a particular celestial body.

[1011] (Claim 3)

[1012] The system according to claim 1, further comprising a processing function for recognizing emotions from the user's facial expressions and voice, and for supplying information optimized for the user's emotions.

[1013] "Application example 2 of combining emotional engines"

[1014] (Claim 1)

[1015] A means for obtaining geolocation information to obtain the user's location information,

[1016] A posture detection means for detecting the orientation of the user's device,

[1017] A data acquisition means for acquiring celestial body data based on the above-mentioned positional information and direction,

[1018] A display means for displaying acquired celestial data in augmented reality format on the user's device,

[1019] Information provision means for providing information about celestial bodies in response to user input,

[1020] An emotion recognition method for analyzing the user's emotional state,

[1021] An emotional response means that presents information based on the analysis results,

[1022] A system that includes this.

[1023] (Claim 2)

[1024] The system according to claim 1, comprising information response means for providing relevant information when a user requests detailed information about a specific celestial body.

[1025] (Claim 3)

[1026] The system according to claim 1, comprising notification means for notifying a user when an astronomical event occurs. [Explanation of Symbols]

[1027] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for obtaining geolocation information to obtain the user's location information, A posture detection means for detecting the orientation of the user's device, A data acquisition means for acquiring celestial body data based on the above-mentioned positional information and direction, A display means for displaying acquired celestial data in augmented reality format on the user's device, Information provision means for providing information about celestial bodies in response to user input, A system that includes this.

2. The system according to claim 1, further comprising information response means for providing relevant information when a user requests detailed information about a specific celestial body.

3. The system according to claim 1, further comprising a notification means for notifying a user when an astronomical event occurs.

Images