system

Abstract

To provide navigation for a user to accurately grasp his / her own position and reach a destination in an indoor environment where a GPS cannot be used. In addition, store information and a congestion situation are provided in real time to improve convenience of the user.SOLUTION: A system comprising: means for obtaining a video image; means for determining a location from the video image; means for generating map information in real time; and means for interactively providing information based on the location information and a query.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by this invention is to provide navigation that enables users to accurately determine their location and reach their destination in indoor environments where GPS is unavailable. Specifically, the object is to support users in smoothly reaching their desired store or facility in complex structures such as large commercial facilities or underground shopping malls. Another object is to improve user convenience by providing real-time store information and congestion status. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means. That is, a system including a means for acquiring video images, a means for identifying a location from the video images, a means for generating map information in real time, and a means for interactively providing information based on location information and questions. This system analyzes camera images to acquire information about the surrounding area, and can provide an optimal route based on the user's current location, as well as provide the user with real-time store and facility information. This allows users to easily use navigation even in indoor environments and smoothly reach their desired store or facility.

[0006] The "means for acquiring moving images" refers to a means for a user to capture the indoor scene using a device such as a camera and acquire the video data.

[0007] The "means for identifying a position from a moving image" is a means for analyzing the acquired moving image data and accurately identifying the current position of the user.

[0008] The "means for generating map information in real time" is a means for generating map information of the current environment in real time based on position information identified from the video image.

[0009] The "means for interactively providing information based on location information and questions" refers to a means for generating appropriate information in accordance with the user's location information and questions from the user, and interactively providing the information to the user.

[0010] "Means for analyzing camera footage and acquiring surrounding information" refers to means for analyzing acquired camera footage and extracting and acquiring information such as objects, text, and signs within the footage.

[0011] The "means for providing an optimal route based on the user's current location" is a means for calculating the optimal route to the destination, starting from the user's current location, and providing navigation to the user. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] In an embodiment of the present invention, a system is constructed that identifies a user's position, particularly in an indoor environment where GPS cannot be used, and provides navigation and store information. A specific embodiment of the system is described below.

[0034] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server. The server is equipped with a high-performance image analysis engine that can analyze the video image data and identify the user's current location.

[0035] By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to pinpoint the user's exact location. Based on this information, the server generates map information in real time and sends it to the device as feedback to the user.

[0036] The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation. In addition, the server provides dynamic information such as store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage.

[0037] For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays that information on the screen and provides visual and audio feedback to the user.

[0038] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[0039] According to the above-described embodiment, the system of the present invention can accurately identify the user's position and provide real-time navigation and useful information even in indoor environments where GPS cannot be used.

[0040] The processing flow will be explained below.

[0041] Step 1:

[0042] The user launches the smartphone camera app, which selects a dedicated navigation mode and the camera captures real-time images of the interior of the room.

[0043] Step 2:

[0044] The device processes the acquired video data in real time and sends it to the server. The data is divided at a fixed frame rate and sent as network packets.

[0045] Step 3:

[0046] The server analyzes the received video data frame by frame, and uses image analysis algorithms to extract feature points (e.g., walls, doors, signs) from the video.

[0047] Step 4:

[0048] The server identifies the user's current location based on feature points in the video, analyzes the relative positions of the feature points, and compares them with existing map data to determine the user's exact location.

[0049] Step 5:

[0050] The server transmits real-time generated map information and the user's current location information to the terminal, and the server also transmits surrounding map data related to the user's location.

[0051] Step 6:

[0052] The device displays the received map information and the user's current location. The user selects a destination on the app's interface, and the current location and route to the destination are displayed on the screen.

[0053] Step 7:

[0054] The device calculates and displays the optimal route based on the user's input, and the app provides users with multiple route options, such as the shortest route or a route that avoids congestion.

[0055] Step 8:

[0056] The user enters store information or other questions by text or voice, such as "Where is the nearest cafe?" or "Can you give me a review of this cafe?"

[0057] Step 9:

[0058] The server receives a question from the user, analyzes the question using AI, and uses natural language processing (NLP) to retrieve relevant information from a database.

[0059] Step 10:

[0060] The server sends the acquired information to the device, including details such as store location, opening hours, and reviews.

[0061] Step 11:

[0062] The device displays the received information to the user, and the app provides information to the user through text, images, and in some cases audio.

[0063] Step 12:

[0064] The user follows the navigation display in the app and begins moving towards their destination. The device updates their current location and provides real-time directions.

[0065] Step 13:

[0066] The device periodically sends video images to the server, which then analyzes them, providing accurate navigation from the user's current location until they reach their destination.

[0067] Through this series of processes, the system can accurately identify the user's location even in indoor environments, providing real-time navigation and interactive information.

[0068] Example 1

[0069] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0070] Conventional technologies have difficulty accurately determining a user's location in indoor environments where GPS is unavailable. Furthermore, there are problems with navigation and local information provision after location determination in real time. Furthermore, there is a lack of systems that can efficiently provide users with the optimal route to their destination.

[0071] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0072] In this invention, the server includes means for analyzing video data and extracting feature points, means for collating the feature points with map information to identify a location, and means for generating map information in real time, thereby enabling accurate identification of a user's location in an indoor environment and providing navigation information and dynamic store information in real time.

[0073] A "moving image" is an image that expresses movement by displaying multiple consecutive still images (usually 24 or more) per second.

[0074] "Motion image data" refers to data for recording or transferring motion images in digital format.

[0075] A "feature point" is an identifiable point in an image that serves as a reference for location specification and image recognition.

[0076] "Map information" refers to information that shows the geographical features and facility locations of a specific area.

[0077] "Real-time" is a term that refers to near-instant processing or reaction.

[0078] "Interactive" means that it allows for two-way interaction with the user.

[0079] "Location information" refers to information that indicates the current location of an object or person.

[0080] "Analysis" is the process of examining data or information in detail to uncover meanings and relationships.

[0081] "Matching" is the process of comparing different data or information to identify matches and connections.

[0082] A "route" is a route or path to a particular destination.

[0083] An "optimal route" is a route or path that is evaluated as being the most efficient under certain conditions.

[0084] "Dynamic store information" refers to information about a store that changes depending on the time or situation (e.g., occupancy status, latest reviews).

[0085] The present invention relates to a system for locating a user's position and providing navigation and store information, particularly in an indoor environment where GPS is unavailable. Specific embodiments of the system are described below.

[0086] Hardware and Software Configuration

[0087] Hardware

[0088] Smartphone / tablet (device): A device with a camera function and internet connection.

[0089] Server: A server capable of high-performance processing (e.g., a server provided by a cloud service provider).

[0090] software

[0091] Terminal application: A dedicated application that captures camera footage and sends it to a server in real time.

[0092] Image analysis engine: Software that analyzes video data using libraries such as OpenCV.

[0093] Database: A database for managing location information and store information (e.g., a relational database system).

[0094] System Overview

[0095] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server.

[0096] The server is equipped with a high-performance image analysis engine that can analyze video data to identify the user's current location. By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to identify the user's exact location.

[0097] The server generates map information in real time based on this information and sends it to the device as feedback to the user. The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation.

[0098] Furthermore, the server provides real-time dynamic information such as store information and congestion status based on the user's location information and surrounding information obtained from camera footage. For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server obtains the latest review information for that cafe and sends it to the device. The device displays that information on the screen and provides the user with visual and audio feedback.

[0099] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[0100] Specific examples

[0101] For example, if a user is in a large shopping mall, they can use their smartphone camera to capture video of the store. The application sends this video to a server, which analyzes the video data to determine the user's location. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays this information on the screen and provides the user with visual and audio feedback.

[0102] Prompt Sentence Examples

[0103] An example of a prompt sentence might be:

[0104] "Tell me about cafes in large shopping malls. I'd like to know recommended cafes and the latest reviews."

[0105] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0106] Step 1:

[0107] Users launch a dedicated application on their smartphone or tablet and use the camera to capture indoor video. The input is video from the device's camera, and the output is a real-time video preview. Users can accurately capture video by pointing the camera in the appropriate direction and checking the video preview on the device screen.

[0108] Step 2:

[0109] The terminal transmits the captured video data to the server in real time. The input is the video data captured by the terminal, and the output is the data transmitted to the server via the Internet. The terminal confirms the success of the transmission and displays the transmission status to the user.

[0110] Step 3:

[0111] The server analyzes the received video data and uses an image analysis engine (e.g., OpenCV) to extract indoor feature points (walls, doors, signs, etc.). The input is the video data sent to the server, and the output is the extracted feature point data. The server uses edge detection and feature point matching algorithms to extract the feature points.

[0112] Step 4:

[0113] The server compares the extracted feature points with pre-registered indoor map information to identify the user's current location. The input is the extracted feature points and map data, and the output is the coordinate information of the user's current location. The server uses algorithms such as KD tree and Hough transform for the comparison.

[0114] Step 5:

[0115] The server generates the necessary data based on the user's current location, the latest map information, and dynamic store information. The input is the user's current location, map data, and store data, and the output is map information and store information packaged in JSON format. The server sends this to the device in real time.

[0116] Step 6:

[0117] The device displays the received map information, the user's current location, and the destination. The input is JSON format data received from the server, and the output is a visual map display on the user interface. The device uses a map drawing library such as Google Maps API or Mapbox SDK to display the map.

[0118] Step 7:

[0119] The user selects a destination on the screen and sets the navigation route. The input is the user's destination selection operation, and the output is the navigation route information. The device displays the route and guides the user visually and audibly.

[0120] Step 8:

[0121] The server obtains dynamic information from the connected store database and sensor network and provides the necessary information to the user. The input is a data acquisition request from the store database, and the output is the latest store information and congestion status. The server sends this information to the terminal in real time, and the terminal displays it on the screen. Specifically, by providing the user with the latest store reviews and congestion status, the user can make better decisions.

[0122] These steps enable the system to accurately locate users in indoor environments where GPS is unavailable, providing real-time navigation and dynamic information provision.

[0123] (Application example 1)

[0124] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] Existing indoor navigation systems have difficulty pinpointing a user's exact location in environments where GPS is unavailable, and their ability to provide real-time information is limited. Another problem is that users cannot instantly obtain information by voice input. This makes it difficult to move around efficiently and use the system, especially in shopping malls and large facilities.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0127] In this invention, the server includes a means for acquiring a moving image, a means for identifying a position from the moving image, a means for generating map information in real time, and a means for providing information interactively based on the position information and a voice input, thereby enabling the server to accurately identify a user's position even in an indoor environment where GPS is not available, and to provide information interactively at high speed via a voice input.

[0128] "Means for acquiring moving images" refers to devices or software that capture images of the user's surroundings in real time.

[0129] "Means for determining location from video images" refers to an algorithm or system for analyzing and determining a user's current location from captured video data.

[0130] "Means for generating map information in real time" refers to devices or programs that instantly create map information corresponding to the current environment based on the user's location information.

[0131] "Means for providing information interactively based on location information and voice input" refers to systems and technologies that respond to a user's current location information and voice questions, and provide relevant information on the spot.

[0132] "Means for analyzing camera footage and obtaining surrounding information and store information" refers to technology for analyzing footage captured by a camera and extracting detailed information about facilities and stores around the user.

[0133] "Audio and visual means" refers to interfaces and technologies for providing information to users through audio and screen displays.

[0134] To implement this invention, a smartphone, tablet, or other mobile device is used. When the user is in a shopping mall or large facility, the following system is used to provide indoor navigation and store information.

[0135] First, the user launches a dedicated app installed on their device and captures video of their surroundings through the camera. This video data is sent to the server in real time. The server then performs image analysis based on the received video data and determines the user's current location. The main technologies used for this are image analysis engines such as OpenCV and TensorFlow.

[0136] Specifically, the server extracts indoor features (e.g., walls, doors, signs, etc.) from camera footage and locates the location by comparing them with existing map information. Based on the identified location information, map information is generated in real time and fed back to the device. In addition, the system accepts voice input from the user and provides information interactively based on the location information and the question. Google Cloud Speech-to-Text and Text-to-Speech APIs are used for voice processing.

[0137] For example, if a user says, "Tell me a review of the cafe up ahead," the server will retrieve the location and review information of the nearest cafe and provide real-time audio and visual feedback, allowing the user to easily understand the congestion situation within the facility and detailed store information.

[0138] With the above system, the present invention can accurately locate a user's location even in indoor environments where GPS is not available, and provide real-time information and navigation. This convenience is particularly noticeable in large shopping malls and complex indoor facilities, providing users with comfortable and efficient mobility.

[0139] An example of a prompt sentence might be:

[0140] "When a user is searching for a cafe in a shopping mall, please explain the program that will identify their location using camera footage and provide information on the nearest cafe along with reviews."

[0141] The system of the present invention can greatly improve convenience at specific locations and dramatically improve the user experience.

[0142] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0143] Step 1:

[0144] The user launches a dedicated app on their smartphone and captures images of their surroundings through the camera. The input is real-time camera images, and the output is the captured image data. This allows the user's surroundings to be displayed in real time.

[0145] Step 2:

[0146] The device sends the captured video data to the server. The input is the video data from step 1, and the output is the data sent to the server. The device sends the video data using the HTTP or HTTPS protocol.

[0147] Step 3:

[0148] The server analyzes the received video data using an image analysis engine (e.g., OpenCV, TensorFlow) and extracts indoor feature points. The input is the transmitted video data, and the output is feature point data. The server identifies feature points such as walls, doors, and signs in the video.

[0149] Step 4:

[0150] The server matches the feature point data with existing map information to determine the user's current location. The input is feature point data and map information, and the output is the user's current location. The server determines the location using a database (e.g., MySQL, PostgreSQL).

[0151] Step 5:

[0152] The server generates map information in real time and sends it to the terminal. The input is the user's current location and map information, and the output is the generated map information. The server generates map information instantly and provides it to the terminal.

[0153] Step 6:

[0154] The terminal displays the received map information and location information on the screen. The input is the received map information and current location information, and the output is the map and current location displayed to the user. The terminal provides map information visually to the user.

[0155] Step 7:

[0156] The user asks a question by voice input. The input is a voice command such as "Please tell me a review of the cafe up ahead," and the output is the results of voice analysis. The user's voice information is analyzed by the device.

[0157] Step 8:

[0158] The device sends the analyzed voice input to the server and requests information based on the question. The input is the voice analysis result, and the output is a request to send to the server. The device converts the voice data into text and sends it to the server.

[0159] Step 9:

[0160] The server searches for relevant information based on the voice input and location information and sends it to the device. The input is the voice analysis results and location information, and the output is relevant information (e.g., cafe reviews). The server searches a database and provides the appropriate information.

[0161] Step 10:

[0162] The device provides the received information to the user via voice and visual display. The input is relevant information from the server, and the output is a voice response and visual display to the user. The device generates the voice response using Google Cloud Text-to-Speech.

[0163] Step 11:

[0164] The user receives audio and visual feedback from the device and acts based on the navigation and information. The input is audio and visual feedback, and the output is the user's actions. The user decides on their next action based on the displayed information.

[0165] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0166] In an embodiment of the present invention, a system is constructed to identify a user's location, particularly in an indoor environment where GPS is not available, and provide navigation and store information. Furthermore, the present invention combines an emotion engine that recognizes the user's emotions and optimizes the information provided based on the emotions.

[0167] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server. The server is equipped with a high-performance image analysis engine that can analyze the video image data and identify the user's current location.

[0168] By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to pinpoint the user's exact location. Based on this information, the server generates map information in real time and sends it to the device as feedback to the user.

[0169] The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation. In addition, the server provides dynamic information such as store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage.

[0170] For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays that information on the screen and provides visual and audio feedback to the user.

[0171] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[0172] Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions, and by analyzing the user's facial expressions and tone of voice, it grasps their emotions in real time. For example, if the user looks impatient, the system will determine that they are in a hurry and will prioritize displaying the shortest route. Also, if the user is relaxing, it will provide information on rest spots and cafes where they can relax.

[0173] The navigation display and audio guidance provided can also be adjusted based on the user's emotions. For example, if the user is stressed, the audio guidance can be reassuring and avoid complex instructions. Conversely, if the user is excited, the audio guidance can be more entertaining, providing detailed information and more options.

[0174] By combining these functions, the system of the present invention can accurately locate the user's location even in indoor environments where GPS is unavailable, providing real-time navigation and interactive information, and can also provide personalized services based on the user's emotions, providing a more convenient and user-friendly experience.

[0175] The processing flow will be explained below.

[0176] Step 1:

[0177] The user launches the smartphone camera app, which selects a dedicated navigation mode and the camera captures real-time images of the interior of the room.

[0178] Step 2:

[0179] The device processes the acquired video data in real time and sends it to the server. The data is divided at a fixed frame rate and sent as network packets.

[0180] Step 3:

[0181] The server analyzes the received video data frame by frame, and uses image analysis algorithms to extract feature points (e.g., walls, doors, signs) from the video.

[0182] Step 4:

[0183] The server identifies the user's current location based on feature points in the video, analyzes the relative positions of the feature points, and compares them with existing map data to determine the user's exact location.

[0184] Step 5:

[0185] The server transmits real-time generated map information and the user's current location information to the terminal, and the server also transmits surrounding map data related to the user's location.

[0186] Step 6:

[0187] The device displays the received map information and the user's current location. The user selects a destination on the app's interface, and the current location and route to the destination are displayed on the screen.

[0188] Step 7:

[0189] The device calculates and displays the optimal route based on the user's input, and the app provides users with multiple route options, such as the shortest route or a route that avoids congestion.

[0190] Step 8:

[0191] The user enters store information or other questions by text or voice, such as "Where is the nearest cafe?" or "Can you give me a review of this cafe?"

[0192] Step 9:

[0193] The server receives a question from the user, analyzes the question using AI, and uses natural language processing (NLP) to retrieve relevant information from a database.

[0194] Step 10:

[0195] The server sends the acquired information to the device, including details such as store location, opening hours, and reviews.

[0196] Step 11:

[0197] The device displays the received information to the user, and the app provides information to the user through text, images, and in some cases audio.

[0198] Step 12:

[0199] The user follows the navigation display in the app and begins moving towards their destination. The device updates their current location and provides real-time directions.

[0200] Step 13:

[0201] The device periodically sends video images to the server, which then analyzes them, providing accurate navigation from the user's current location until they reach their destination.

[0202] Step 14:

[0203] The device analyzes the camera footage and the user's voice and uses an emotion engine to identify the user's emotional state. For example, it recognizes the user's current emotion by analyzing the user's facial expressions and tone of voice.

[0204] Step 15:

[0205] The server adjusts the information provided and navigation methods based on the emotion data sent from the emotion engine. For example, if it determines that the user is in a hurry, it will prioritize suggesting the shortest route.

[0206] Step 16:

[0207] The device will provide users with information and navigation based on their emotions. For example, if the user is feeling relaxed, it will provide information on rest spots and cafes.

[0208] Step 17:

[0209] When the user again inputs a question about store information or navigation, the server takes the emotional data into consideration and provides the most appropriate information.

[0210] Step 18:

[0211] The server and device work together to continuously monitor the user's current location and emotional state, thereby constantly providing the user with the most appropriate information and navigation.

[0212] Example 2

[0213] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0214] In indoor environments, GPS is unavailable, making it difficult to pinpoint the user's exact location, and providing navigation and store information is also difficult.In addition, the inability to provide information that takes into account the user's emotions results in an inadequate user experience.

[0215] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring a moving image, a means for transmitting the moving image to the server, a means for identifying a location from the moving image, a means for generating map information in real time, a means for interactively providing information based on the location information and a question, and a means for recognizing a user's emotion and optimizing information. This makes it possible to identify the user's exact location even in an indoor environment, and to provide real-time navigation and personalized information.

[0216] The "means for acquiring moving images" is a function that allows a user to capture images of the interior of a room using the camera on a smartphone or tablet.

[0217] The "means for transmitting moving images to a server" is a communication function for transferring moving image data acquired by a terminal to a server in real time.

[0218] The "means for identifying a location from a video image" is a function that analyzes video image data using an image analysis engine installed in the server and identifies the user's current location.

[0219] "Means for generating map information in real time" refers to a function in which the server instantly creates the latest map information based on the user's location information and provides it to the user.

[0220] The "means for interactively providing information based on location information and questions" is a function that provides appropriate information in real time based on the user's current location and questions.

[0221] "Means for recognizing the user's emotions and optimizing information" refers to a function in which the server analyzes the user's facial expressions and tone of voice to understand their emotions and provide optimal information based on those emotions.

[0222] The "means for analyzing camera images and acquiring surrounding information" is a function for analyzing images acquired through a camera and identifying information about the environment surrounding the user.

[0223] The "means for providing the optimal route based on the user's current location" is a function that calculates and displays the most efficient route to the destination set by the user.

[0224] The present invention builds a system that identifies a user's location and provides navigation and store information, especially in indoor environments where GPS is not available. Furthermore, the present invention combines an emotion engine that recognizes the user's emotions and optimizes the information provided based on those emotions.

[0225] Hardware and software used

[0226] The main hardware of this system is the user's smartphone or tablet, which must be equipped with a high-performance camera. As for software, a dedicated application is installed on the device, which is responsible for capturing and transmitting video images.

[0227] The server is equipped with a high-performance image analysis engine (e.g., OpenCV, TensorFlow) that analyzes video data in real time. The server also has an emotion engine that recognizes the user's emotions and has the ability to analyze the user's facial expressions and tone of voice.

[0228] Specific processing and operations

[0229] The system starts with the user using a smartphone or tablet to take pictures of the indoor environment with a camera. The user then launches a dedicated application and starts capturing video images in real time. At this time, the device sends the video image data to a server via Wi-Fi or a mobile network.

[0230] The server analyzes the received video data and extracts indoor features (e.g., walls, doors, signs). It then compares this with existing map information to determine the user's current location. The server generates map information in real time based on this information and sends it to the device as feedback.

[0231] The device displays the received map information and the user's current location, as well as calculates and displays the optimal route to the user's selected destination. When the user selects "I want to go to the Apple Store," the device displays the route and guides the user along the route.

[0232] Furthermore, the server provides dynamic store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage. For example, when a user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays the information on its screen and provides visual and audio feedback to the user.

[0233] The emotion engine allows the server to analyze the user's facial expressions and tone of voice to understand their emotions in real time. For example, if the user looks anxious, the system will determine that they are in a hurry and prioritize the shortest route. Conversely, if the user is relaxed, it will provide information on rest spots and cafes where they can relax.

[0234] Examples of specific examples and prompts

[0235] When a user is in a large shopping mall, the information of each store is analyzed through camera footage, and the server sends that information to the device. For example, if a user selects "I want to go to the Apple Store," the device will display the shortest route. Similarly, if the user asks "What are the reviews of this cafe?", the device will display the latest reviews.

[0236] An example prompt is:

[0237] "Please give us a review of the cafe."

[0238] "What's the best route to get to the nearest restroom?"

[0239] "How do I get to the cinema?"

[0240] This allows users to easily locate and navigate indoors, as well as receive personalized information based on their emotions.

[0241] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0242] Step 1:

[0243] A user launches a dedicated application on a smartphone or tablet and activates the camera. This application is designed to allow users to take pictures of indoor scenes. When the user presses the capture button, the device begins capturing video images in real time.

[0244] Input: User initiates camera and takes a photo

[0245] Output: Real-time video data

[0246] Step 2:

[0247] The device transmits the acquired video data to a server via Wi-Fi or mobile networks in real time, minimizing time lag.

[0248] Input: Real-time video data

[0249] Output: Send video data to the server

[0250] Step 3:

[0251] The server uses a high-performance image analysis engine (e.g., OpenCV, TensorFlow) to analyze the received video data. The server detects indoor features (e.g., walls, doors, signs) and compares them with existing map information.

[0252] Input: Video data

[0253] Output: Current location data of the user

[0254] Step 4:

[0255] The server generates real-time map information based on the extracted feature points. This map information reflects the user's current location. The server then sends the generated map information to the device.

[0256] Input: current location data of the user

[0257] Output: Generate real-time map information and send it to the device

[0258] Step 5:

[0259] The device displays the received map information to the user. The map shows the user's current location and information about nearby stores. When the user selects a destination (e.g., an Apple Store), the device sends that information to the server.

[0260] Input: Real-time map information

[0261] Output: Map display and destination selection information sent to server

[0262] Step 6:

[0263] The server calculates the optimal route based on the user's current location and selected destination, and can provide multiple route options (e.g., shortest route, congestion avoidance route, etc.).

[0264] Input: User's current location data and destination selection information

[0265] Output: Calculate optimal route information and send it to the device

[0266] Step 7:

[0267] The terminal displays the optimal route information received from the server to the user, providing interactive navigation. The user follows the presented route and is guided to the destination.

[0268] Input: Optimal route information

[0269] Output: Display route information and provide navigation

[0270] Step 8:

[0271] The server provides real-time store information and congestion status based on the user's current location and surrounding information obtained from camera footage. For example, when a user asks, "What are the reviews for this cafe?", the server retrieves the latest review information and sends it to the device.

[0272] Input: Local information, user questions

[0273] Output: Dynamic store information and review information sent to the device

[0274] Step 9:

[0275] The server uses an emotion engine to analyze the user's facial expressions and tone of voice, and based on the analysis results, it provides the shortest route if the user is in a hurry, or information on rest spots if the user is relaxing.

[0276] Input: Emotion recognition data (facial expressions, voice tone)

[0277] Output: Optimized information delivery

[0278] In this way, users can receive accurate location and real-time navigation even in indoor environments, as well as personalized information based on their emotions.

[0279] (Application example 2)

[0280] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] The present invention aims to provide a system that accurately identifies a user's location and provides navigation and information, especially in indoor environments where GPS is unavailable, and also aims to solve the problem of improving the user experience by recognizing the user's emotions and optimizing the information provided based on those emotions.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0283] In this invention, the server includes a means for acquiring video images, a means for identifying a location from the video images, a means for generating map information in real time, a means for interactively providing information based on the location information and questions, and a means for recognizing a user's emotions and optimizing the information. This enables highly accurate location identification and navigation even in indoor environments that do not support GPS, and further enables personalized information provision according to the user's emotions.

[0284] "Means for acquiring video images" refers to the function that allows users to capture video in real time using the camera on their smartphone or tablet.

[0285] "Means for identifying a position from a video image" refers to a function that analyzes the acquired video image and performs processing to identify the user's current position from characteristic points (walls, doors, signs, etc.).

[0286] "Means for generating map information in real time" refers to the function of the server generating and providing the latest map that reflects the user's location information based on the analysis results.

[0287] "Means for interactively providing information based on location information and questions" refers to a function that provides relevant information (store information, route information, etc.) in real time based on the user's current location and inquiries from the user.

[0288] "Means for recognizing the user's emotions and optimizing information" refers to the function of analyzing the user's emotions from their facial expressions and voice, and adjusting the information and navigation content provided based on those emotions.

[0289] This invention is a system that accurately identifies a user's location and provides navigation and information, especially in indoor environments where GPS is not available. It is also capable of recognizing the user's emotions and optimizing the information provided based on those emotions.

[0290] The main components of the system include a means for acquiring video images, a means for identifying a location from the video images, a means for generating map information in real time, a means for providing information interactively based on location information and questions, and a means for recognizing the user's emotions and optimizing the information.

[0291] Hardware and Software Configuration

[0292] 1. Acquiring video images

[0293] Users can capture indoor video images using mobile devices such as smartphones and tablets, using the device's built-in camera and image processing libraries such as OpenCV.

[0294] 2. Video data sent to the server

[0295] The acquired video data is sent from the device to the server. This sending process uses Requests, an HTTP request library.

[0296] 3. Location Identification and Map Generation

[0297] The server analyzes the video images sent and extracts indoor features (walls, doors, signs, etc.). Based on this, it locates the user's location by comparing it with existing map information. This analysis is performed using an image analysis engine.

[0298] The server generates the latest map information in real time and sends it to the terminal.

[0299] 4. Navigation and Information

[0300] The device displays the received map information and the user's current location, and the user can select a route to their destination. The route information is visualized, and the device guides the user to their destination. This guidance can provide multiple options, such as the shortest route or a route that avoids congestion.

[0301] 5. Emotion Recognition and Information Optimization

[0302] When a user provides video via a camera, the server analyzes the user's facial expressions and voice in real time using an emotion recognition engine.

[0303] Based on the recognized emotion, the information provided and navigation content are optimized. For example, if the user is in a hurry, the shortest route will be prioritized, and if the user is relaxed, information about rest spots will be provided.

[0304] Examples of specific examples and prompts

[0305] This system is extremely effective in large shopping malls and department stores, for example. When a user uses the system while walking around the mall, they can obtain real-time information on congestion and store reviews, along with navigation to their destination. Furthermore, the information provided is optimized according to the user's emotions, providing a more comfortable shopping experience.

[0306] Example prompt sentence:

[0307] "Your goal is to develop an application that identifies the user's location in a large shopping mall, analyzes camera footage to recognize emotions, and provides optimal store navigation based on those emotions."

[0308] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0309] Step 1:

[0310] A user captures indoor video using the camera on their smartphone or tablet. When the user launches the app and the camera acquires video, the device captures video in real time. The input of this step is the video captured by the camera, and the output is the captured video data.

[0311] Step 2:

[0312] The device sends the captured video data to the server. The device encodes the video data into a format such as JPEG and sends it to the server using an HTTP request library (e.g., Requests). The input to this step is the video data captured in step 1, and the output is the data sent to the server.

[0313] Step 3:

[0314] The server analyzes the received video data using its image analysis engine. The server decodes the transmitted data and extracts feature points (e.g., walls, doors, signs, etc.). The input of this step is the video data transmitted in step 2, and the output is the extracted feature point data.

[0315] Step 4:

[0316] The server performs location identification. The server compares the extracted feature points with existing map information to identify the user's current location. The input to this step is the feature point data and map information obtained in step 3, and the output is the identified user's location information.

[0317] Step 5:

[0318] The server generates map information in real time and sends it to the device. The server generates the latest map information based on the user's location information and sends it to the device. The input of this step is the identified location information, and the output is the generated map information.

[0319] Step 6:

[0320] The device displays the received map information and the user's current location. The device visually displays the received map information to the user, allowing the user to select a route to their destination. The input of this step is the map information received in step 5, and the output is the displayed map information and the user's selected route information.

[0321] Step 7:

[0322] The device navigates the user to the destination. The device provides multiple route options, such as the shortest route or a route that avoids congestion, and navigates along the route selected by the user. The input of this step is the route information selected by the user, and the output is the navigation information provided.

[0323] Step 8:

[0324] The server recognizes the user's emotions and optimizes the information. When the user provides camera footage, the server analyzes the user's facial expressions and voice in real time to recognize emotions. An emotion recognition engine is used for this analysis. The input for this step is camera footage and voice data, and the output is recognized emotional information.

[0325] Step 9:

[0326] The server optimizes information based on the recognized emotion. The navigation content and information provided are adjusted according to the recognized emotion. For example, if the user is in a hurry, the shortest route is prioritized, and if the user is relaxed, information about rest spots is provided. The input of this step is the recognized emotion information, and the output is optimized information.

[0327] Step 10:

[0328] The terminal displays the optimized information and provides it to the user. The terminal visually and audibly provides the optimized information received from the server to the user. The input of this step is the optimized information, and the output is the provided visual and audio feedback.

[0329] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0330] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0331] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0332] [Second embodiment]

[0333] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0334] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0335] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0336] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0337] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0338] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0339] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0340] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0341] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0342] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0343] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0344] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0345] In an embodiment of the present invention, a system is constructed that identifies a user's position, particularly in an indoor environment where GPS cannot be used, and provides navigation and store information. A specific embodiment of the system is described below.

[0346] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server. The server is equipped with a high-performance image analysis engine that can analyze the video image data and identify the user's current location.

[0347] By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to pinpoint the user's exact location. Based on this information, the server generates map information in real time and sends it to the device as feedback to the user.

[0348] The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation. In addition, the server provides dynamic information such as store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage.

[0349] For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays that information on the screen and provides visual and audio feedback to the user.

[0350] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[0351] According to the above-described embodiment, the system of the present invention can accurately identify the user's position and provide real-time navigation and useful information even in indoor environments where GPS cannot be used.

[0352] The processing flow will be explained below.

[0353] Step 1:

[0354] The user launches the smartphone camera app, which selects a dedicated navigation mode and the camera captures real-time images of the interior of the room.

[0355] Step 2:

[0356] The device processes the acquired video data in real time and sends it to the server. The data is divided at a fixed frame rate and sent as network packets.

[0357] Step 3:

[0358] The server analyzes the received video data frame by frame, and uses image analysis algorithms to extract feature points (e.g., walls, doors, signs) from the video.

[0359] Step 4:

[0360] The server identifies the user's current location based on feature points in the video, analyzes the relative positions of the feature points, and compares them with existing map data to determine the user's exact location.

[0361] Step 5:

[0362] The server transmits real-time generated map information and the user's current location information to the terminal, and the server also transmits surrounding map data related to the user's location.

[0363] Step 6:

[0364] The device displays the received map information and the user's current location. The user selects a destination on the app's interface, and the current location and route to the destination are displayed on the screen.

[0365] Step 7:

[0366] The device calculates and displays the optimal route based on the user's input, and the app provides users with multiple route options, such as the shortest route or a route that avoids congestion.

[0367] Step 8:

[0368] The user enters store information or other questions by text or voice, such as "Where is the nearest cafe?" or "Can you give me a review of this cafe?"

[0369] Step 9:

[0370] The server receives a question from the user, analyzes the question using AI, and uses natural language processing (NLP) to retrieve relevant information from a database.

[0371] Step 10:

[0372] The server sends the acquired information to the device, including details such as store location, opening hours, and reviews.

[0373] Step 11:

[0374] The device displays the received information to the user, and the app provides information to the user through text, images, and in some cases audio.

[0375] Step 12:

[0376] The user follows the navigation display in the app and begins moving towards their destination. The device updates their current location and provides real-time directions.

[0377] Step 13:

[0378] The device periodically sends video images to the server, which then analyzes them, providing accurate navigation from the user's current location until they reach their destination.

[0379] Through this series of processes, the system can accurately identify the user's location even in indoor environments, providing real-time navigation and interactive information.

[0380] Example 1

[0381] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0382] Conventional technologies have difficulty accurately determining a user's location in indoor environments where GPS is unavailable. Furthermore, there are problems with navigation and local information provision after location determination in real time. Furthermore, there is a lack of systems that can efficiently provide users with the optimal route to their destination.

[0383] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0384] In this invention, the server includes means for analyzing video data and extracting feature points, means for collating the feature points with map information to identify a location, and means for generating map information in real time, thereby enabling accurate identification of a user's location in an indoor environment and providing navigation information and dynamic store information in real time.

[0385] A "moving image" is an image that expresses movement by displaying multiple consecutive still images (usually 24 or more) per second.

[0386] "Motion image data" refers to data for recording or transferring motion images in digital format.

[0387] A "feature point" is an identifiable point in an image that serves as a reference for location specification and image recognition.

[0388] "Map information" refers to information that shows the geographical features and facility locations of a specific area.

[0389] "Real-time" is a term that refers to near-instant processing or reaction.

[0390] "Interactive" means that it allows for two-way interaction with the user.

[0391] "Location information" refers to information that indicates the current location of an object or person.

[0392] "Analysis" is the process of examining data or information in detail to uncover meanings and relationships.

[0393] "Matching" is the process of comparing different data or information to identify matches and connections.

[0394] A "route" is a route or path to a particular destination.

[0395] An "optimal route" is a route or path that is evaluated as being the most efficient under certain conditions.

[0396] "Dynamic store information" refers to information about a store that changes depending on the time or situation (e.g., occupancy status, latest reviews).

[0397] The present invention relates to a system for locating a user's position and providing navigation and store information, particularly in an indoor environment where GPS is unavailable. Specific embodiments of the system are described below.

[0398] Hardware and Software Configuration

[0399] Hardware

[0400] Smartphone / tablet (device): A device with a camera function and internet connection.

[0401] Server: A server capable of high-performance processing (e.g., a server provided by a cloud service provider).

[0402] software

[0403] Terminal application: A dedicated application that captures camera footage and sends it to a server in real time.

[0404] Image analysis engine: Software that analyzes video data using libraries such as OpenCV.

[0405] Database: A database for managing location information and store information (e.g., a relational database system).

[0406] System Overview

[0407] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server.

[0408] The server is equipped with a high-performance image analysis engine that can analyze video data to identify the user's current location. By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to identify the user's exact location.

[0409] The server generates map information in real time based on this information and sends it to the device as feedback to the user. The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation.

[0410] Furthermore, the server provides real-time dynamic information such as store information and congestion status based on the user's location information and surrounding information obtained from camera footage. For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server obtains the latest review information for that cafe and sends it to the device. The device displays that information on the screen and provides the user with visual and audio feedback.

[0411] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[0412] Specific examples

[0413] For example, if a user is in a large shopping mall, they can use their smartphone camera to capture video of the store. The application sends this video to a server, which analyzes the video data to determine the user's location. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays this information on the screen and provides the user with visual and audio feedback.

[0414] Prompt Sentence Examples

[0415] An example of a prompt sentence might be:

[0416] "Tell me about cafes in large shopping malls. I'd like to know recommended cafes and the latest reviews."

[0417] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0418] Step 1:

[0419] Users launch a dedicated application on their smartphone or tablet and use the camera to capture indoor video. The input is video from the device's camera, and the output is a real-time video preview. Users can accurately capture video by pointing the camera in the appropriate direction and checking the video preview on the device screen.

[0420] Step 2:

[0421] The terminal transmits the captured video data to the server in real time. The input is the video data captured by the terminal, and the output is the data transmitted to the server via the Internet. The terminal confirms the success of the transmission and displays the transmission status to the user.

[0422] Step 3:

[0423] The server analyzes the received video data and uses an image analysis engine (e.g., OpenCV) to extract indoor feature points (walls, doors, signs, etc.). The input is the video data sent to the server, and the output is the extracted feature point data. The server uses edge detection and feature point matching algorithms to extract the feature points.

[0424] Step 4:

[0425] The server compares the extracted feature points with pre-registered indoor map information to identify the user's current location. The input is the extracted feature points and map data, and the output is the coordinate information of the user's current location. The server uses algorithms such as KD tree and Hough transform for the comparison.

[0426] Step 5:

[0427] The server generates the necessary data based on the user's current location, the latest map information, and dynamic store information. The input is the user's current location, map data, and store data, and the output is map information and store information packaged in JSON format. The server sends this to the device in real time.

[0428] Step 6:

[0429] The device displays the received map information, the user's current location, and the destination. The input is JSON format data received from the server, and the output is a visual map display on the user interface. The device uses a map drawing library such as Google Maps API or Mapbox SDK to display the map.

[0430] Step 7:

[0431] The user selects a destination on the screen and sets the navigation route. The input is the user's destination selection operation, and the output is the navigation route information. The device displays the route and guides the user visually and audibly.

[0432] Step 8:

[0433] The server obtains dynamic information from the connected store database and sensor network and provides the necessary information to the user. The input is a data acquisition request from the store database, and the output is the latest store information and congestion status. The server sends this information to the terminal in real time, and the terminal displays it on the screen. Specifically, by providing the user with the latest store reviews and congestion status, the user can make better decisions.

[0434] These steps enable the system to accurately locate users in indoor environments where GPS is unavailable, providing real-time navigation and dynamic information provision.

[0435] (Application example 1)

[0436] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0437] Existing indoor navigation systems have difficulty pinpointing a user's exact location in environments where GPS is unavailable, and their ability to provide real-time information is limited. Another problem is that users cannot instantly obtain information by voice input. This makes it difficult to move around efficiently and use the system, especially in shopping malls and large facilities.

[0438] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0439] In this invention, the server includes a means for acquiring a moving image, a means for identifying a position from the moving image, a means for generating map information in real time, and a means for providing information interactively based on the position information and a voice input, thereby enabling the server to accurately identify a user's position even in an indoor environment where GPS is not available, and to provide information interactively at high speed via a voice input.

[0440] "Means for acquiring moving images" refers to devices or software that capture images of the user's surroundings in real time.

[0441] "Means for determining location from video images" refers to an algorithm or system for analyzing and determining a user's current location from captured video data.

[0442] "Means for generating map information in real time" refers to devices or programs that instantly create map information corresponding to the current environment based on the user's location information.

[0443] "Means for providing information interactively based on location information and voice input" refers to systems and technologies that respond to a user's current location information and voice questions, and provide relevant information on the spot.

[0444] "Means for analyzing camera footage and obtaining surrounding information and store information" refers to technology for analyzing footage captured by a camera and extracting detailed information about facilities and stores around the user.

[0445] "Audio and visual means" refers to interfaces and technologies for providing information to users through audio and screen displays.

[0446] To implement this invention, a smartphone, tablet, or other mobile device is used. When the user is in a shopping mall or large facility, the following system is used to provide indoor navigation and store information.

[0447] First, the user launches a dedicated app installed on their device and captures video of their surroundings through the camera. This video data is sent to the server in real time. The server then performs image analysis based on the received video data and determines the user's current location. The main technologies used for this are image analysis engines such as OpenCV and TensorFlow.

[0448] Specifically, the server extracts indoor features (e.g., walls, doors, signs, etc.) from camera footage and locates the location by comparing them with existing map information. Based on the identified location information, map information is generated in real time and fed back to the device. In addition, the system accepts voice input from the user and provides information interactively based on the location information and the question. Google Cloud Speech-to-Text and Text-to-Speech APIs are used for voice processing.

[0449] For example, if a user says, "Tell me a review of the cafe up ahead," the server will retrieve the location and review information of the nearest cafe and provide real-time audio and visual feedback, allowing the user to easily understand the congestion situation within the facility and detailed store information.

[0450] With the above system, the present invention can accurately locate a user's location even in indoor environments where GPS is not available, and provide real-time information and navigation. This convenience is particularly noticeable in large shopping malls and complex indoor facilities, providing users with comfortable and efficient mobility.

[0451] An example of a prompt sentence might be:

[0452] "When a user is searching for a cafe in a shopping mall, please explain the program that will identify their location using camera footage and provide information on the nearest cafe along with reviews."

[0453] The system of the present invention can greatly improve convenience at specific locations and dramatically improve the user experience.

[0454] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0455] Step 1:

[0456] The user launches a dedicated app on their smartphone and captures images of their surroundings through the camera. The input is real-time camera images, and the output is the captured image data. This allows the user's surroundings to be displayed in real time.

[0457] Step 2:

[0458] The device sends the captured video data to the server. The input is the video data from step 1, and the output is the data sent to the server. The device sends the video data using the HTTP or HTTPS protocol.

[0459] Step 3:

[0460] The server analyzes the received video data using an image analysis engine (e.g., OpenCV, TensorFlow) and extracts indoor feature points. The input is the transmitted video data, and the output is feature point data. The server identifies feature points such as walls, doors, and signs in the video.

[0461] Step 4:

[0462] The server matches the feature point data with existing map information to determine the user's current location. The input is feature point data and map information, and the output is the user's current location. The server determines the location using a database (e.g., MySQL, PostgreSQL).

[0463] Step 5:

[0464] The server generates map information in real time and sends it to the terminal. The input is the user's current location and map information, and the output is the generated map information. The server generates map information instantly and provides it to the terminal.

[0465] Step 6:

[0466] The terminal displays the received map information and location information on the screen. The input is the received map information and current location information, and the output is the map and current location displayed to the user. The terminal provides map information visually to the user.

[0467] Step 7:

[0468] The user asks a question by voice input. The input is a voice command such as "Please tell me a review of the cafe up ahead," and the output is the results of voice analysis. The user's voice information is analyzed by the device.

[0469] Step 8:

[0470] The device sends the analyzed voice input to the server and requests information based on the question. The input is the voice analysis result, and the output is a request to send to the server. The device converts the voice data into text and sends it to the server.

[0471] Step 9:

[0472] The server searches for relevant information based on the voice input and location information and sends it to the device. The input is the voice analysis results and location information, and the output is relevant information (e.g., cafe reviews). The server searches a database and provides the appropriate information.

[0473] Step 10:

[0474] The device provides the received information to the user via voice and visual display. The input is relevant information from the server, and the output is a voice response and visual display to the user. The device generates the voice response using Google Cloud Text-to-Speech.

[0475] Step 11:

[0476] The user receives audio and visual feedback from the device and acts based on the navigation and information. The input is audio and visual feedback, and the output is the user's actions. The user decides on their next action based on the displayed information.

[0477] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0478] In an embodiment of the present invention, a system is constructed to identify a user's location, particularly in an indoor environment where GPS is not available, and provide navigation and store information. Furthermore, the present invention combines an emotion engine that recognizes the user's emotions and optimizes the information provided based on the emotions.

[0479] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server. The server is equipped with a high-performance image analysis engine that can analyze the video image data and identify the user's current location.

[0480] By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to pinpoint the user's exact location. Based on this information, the server generates map information in real time and sends it to the device as feedback to the user.

[0481] The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation. In addition, the server provides dynamic information such as store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage.

[0482] For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays that information on the screen and provides visual and audio feedback to the user.

[0483] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[0484] Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions, and by analyzing the user's facial expressions and tone of voice, it grasps their emotions in real time. For example, if the user looks impatient, the system will determine that they are in a hurry and will prioritize displaying the shortest route. Also, if the user is relaxing, it will provide information on rest spots and cafes where they can relax.

[0485] The navigation display and audio guidance provided can also be adjusted based on the user's emotions. For example, if the user is stressed, the audio guidance can be reassuring and avoid complex instructions. Conversely, if the user is excited, the audio guidance can be more entertaining, providing detailed information and more options.

[0486] By combining these functions, the system of the present invention can accurately locate the user's location even in indoor environments where GPS is unavailable, providing real-time navigation and interactive information, and can also provide personalized services based on the user's emotions, providing a more convenient and user-friendly experience.

[0487] The processing flow will be explained below.

[0488] Step 1:

[0489] The user launches the smartphone camera app, which selects a dedicated navigation mode and the camera captures real-time images of the interior of the room.

[0490] Step 2:

[0491] The device processes the acquired video data in real time and sends it to the server. The data is divided at a fixed frame rate and sent as network packets.

[0492] Step 3:

[0493] The server analyzes the received video data frame by frame, and uses image analysis algorithms to extract feature points (e.g., walls, doors, signs) from the video.

[0494] Step 4:

[0495] The server identifies the user's current location based on feature points in the video, analyzes the relative positions of the feature points, and compares them with existing map data to determine the user's exact location.

[0496] Step 5:

[0497] The server transmits real-time generated map information and the user's current location information to the terminal, and the server also transmits surrounding map data related to the user's location.

[0498] Step 6:

[0499] The device displays the received map information and the user's current location. The user selects a destination on the app's interface, and the current location and route to the destination are displayed on the screen.

[0500] Step 7:

[0501] The device calculates and displays the optimal route based on the user's input, and the app provides users with multiple route options, such as the shortest route or a route that avoids congestion.

[0502] Step 8:

[0503] The user enters store information or other questions by text or voice, such as "Where is the nearest cafe?" or "Can you give me a review of this cafe?"

[0504] Step 9:

[0505] The server receives a question from the user, analyzes the question using AI, and uses natural language processing (NLP) to retrieve relevant information from a database.

[0506] Step 10:

[0507] The server sends the acquired information to the device, including details such as store location, opening hours, and reviews.

[0508] Step 11:

[0509] The device displays the received information to the user, and the app provides information to the user through text, images, and in some cases audio.

[0510] Step 12:

[0511] The user follows the navigation display in the app and begins moving towards their destination. The device updates their current location and provides real-time directions.

[0512] Step 13:

[0513] The device periodically sends video images to the server, which then analyzes them, providing accurate navigation from the user's current location until they reach their destination.

[0514] Step 14:

[0515] The device analyzes the camera footage and the user's voice and uses an emotion engine to identify the user's emotional state. For example, it recognizes the user's current emotion by analyzing the user's facial expressions and tone of voice.

[0516] Step 15:

[0517] The server adjusts the information provided and navigation methods based on the emotion data sent from the emotion engine. For example, if it determines that the user is in a hurry, it will prioritize suggesting the shortest route.

[0518] Step 16:

[0519] The device will provide users with information and navigation based on their emotions. For example, if the user is feeling relaxed, it will provide information on rest spots and cafes.

[0520] Step 17:

[0521] When the user again inputs a question about store information or navigation, the server takes the emotional data into consideration and provides the most appropriate information.

[0522] Step 18:

[0523] The server and device work together to continuously monitor the user's current location and emotional state, thereby constantly providing the user with the most appropriate information and navigation.

[0524] Example 2

[0525] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0526] In indoor environments, GPS is unavailable, making it difficult to pinpoint the user's exact location, and providing navigation and store information is also difficult.In addition, the inability to provide information that takes into account the user's emotions results in an inadequate user experience.

[0527] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring a moving image, a means for transmitting the moving image to the server, a means for identifying a location from the moving image, a means for generating map information in real time, a means for interactively providing information based on the location information and a question, and a means for recognizing a user's emotion and optimizing information. This makes it possible to identify the user's exact location even in an indoor environment, and to provide real-time navigation and personalized information.

[0528] The "means for acquiring moving images" is a function that allows a user to capture images of the interior of a room using the camera on a smartphone or tablet.

[0529] The "means for transmitting moving images to a server" is a communication function for transferring moving image data acquired by a terminal to a server in real time.

[0530] The "means for identifying a location from a video image" is a function that analyzes video image data using an image analysis engine installed in the server and identifies the user's current location.

[0531] "Means for generating map information in real time" refers to a function in which the server instantly creates the latest map information based on the user's location information and provides it to the user.

[0532] The "means for interactively providing information based on location information and questions" is a function that provides appropriate information in real time based on the user's current location and questions.

[0533] "Means for recognizing the user's emotions and optimizing information" refers to a function in which the server analyzes the user's facial expressions and tone of voice to understand their emotions and provide optimal information based on those emotions.

[0534] The "means for analyzing camera images and acquiring surrounding information" is a function for analyzing images acquired through a camera and identifying information about the environment surrounding the user.

[0535] The "means for providing the optimal route based on the user's current location" is a function that calculates and displays the most efficient route to the destination set by the user.

[0536] The present invention builds a system that identifies a user's location and provides navigation and store information, especially in indoor environments where GPS is not available. Furthermore, the present invention combines an emotion engine that recognizes the user's emotions and optimizes the information provided based on those emotions.

[0537] Hardware and software used

[0538] The main hardware of this system is the user's smartphone or tablet, which must be equipped with a high-performance camera. As for software, a dedicated application is installed on the device, which is responsible for capturing and transmitting video images.

[0539] The server is equipped with a high-performance image analysis engine (e.g., OpenCV, TensorFlow) that analyzes video data in real time. The server also has an emotion engine that recognizes the user's emotions and has the ability to analyze the user's facial expressions and tone of voice.

[0540] Specific processing and operations

[0541] The system starts with the user using a smartphone or tablet to take pictures of the indoor environment with a camera. The user then launches a dedicated application and starts capturing video images in real time. At this time, the device sends the video image data to a server via Wi-Fi or a mobile network.

[0542] The server analyzes the received video data and extracts indoor features (e.g., walls, doors, signs). It then compares this with existing map information to determine the user's current location. The server generates map information in real time based on this information and sends it to the device as feedback.

[0543] The device displays the received map information and the user's current location, as well as calculates and displays the optimal route to the user's selected destination. When the user selects "I want to go to the Apple Store," the device displays the route and guides the user along the route.

[0544] Furthermore, the server provides dynamic store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage. For example, when a user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays the information on its screen and provides visual and audio feedback to the user.

[0545] The emotion engine allows the server to analyze the user's facial expressions and tone of voice to understand their emotions in real time. For example, if the user looks anxious, the system will determine that they are in a hurry and prioritize the shortest route. Conversely, if the user is relaxed, it will provide information on rest spots and cafes where they can relax.

[0546] Examples of specific examples and prompts

[0547] When a user is in a large shopping mall, the information of each store is analyzed through camera footage, and the server sends that information to the device. For example, if a user selects "I want to go to the Apple Store," the device will display the shortest route. Similarly, if the user asks "What are the reviews of this cafe?", the device will display the latest reviews.

[0548] An example prompt is:

[0549] "Please give us a review of the cafe."

[0550] "What's the best route to get to the nearest restroom?"

[0551] "How do I get to the cinema?"

[0552] This allows users to easily locate and navigate indoors, as well as receive personalized information based on their emotions.

[0553] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0554] Step 1:

[0555] A user launches a dedicated application on a smartphone or tablet and activates the camera. This application is designed to allow users to take pictures of indoor scenes. When the user presses the capture button, the device begins capturing video images in real time.

[0556] Input: User initiates camera and takes a photo

[0557] Output: Real-time video data

[0558] Step 2:

[0559] The device transmits the acquired video data to a server via Wi-Fi or mobile networks in real time, minimizing time lag.

[0560] Input: Real-time video data

[0561] Output: Send video data to the server

[0562] Step 3:

[0563] The server uses a high-performance image analysis engine (e.g., OpenCV, TensorFlow) to analyze the received video data. The server detects indoor features (e.g., walls, doors, signs) and compares them with existing map information.

[0564] Input: Video data

[0565] Output: Current location data of the user

[0566] Step 4:

[0567] The server generates real-time map information based on the extracted feature points. This map information reflects the user's current location. The server then sends the generated map information to the device.

[0568] Input: current location data of the user

[0569] Output: Generate real-time map information and send it to the device

[0570] Step 5:

[0571] The device displays the received map information to the user. The map shows the user's current location and information about nearby stores. When the user selects a destination (e.g., an Apple Store), the device sends that information to the server.

[0572] Input: Real-time map information

[0573] Output: Map display and destination selection information sent to server

[0574] Step 6:

[0575] The server calculates the optimal route based on the user's current location and selected destination, and can provide multiple route options (e.g., shortest route, congestion avoidance route, etc.).

[0576] Input: User's current location data and destination selection information

[0577] Output: Calculate optimal route information and send it to the device

[0578] Step 7:

[0579] The terminal displays the optimal route information received from the server to the user, providing interactive navigation. The user follows the presented route and is guided to the destination.

[0580] Input: Optimal route information

[0581] Output: Display route information and provide navigation

[0582] Step 8:

[0583] The server provides real-time store information and congestion status based on the user's current location and surrounding information obtained from camera footage. For example, when a user asks, "What are the reviews for this cafe?", the server retrieves the latest review information and sends it to the device.

[0584] Input: Local information, user questions

[0585] Output: Dynamic store information and review information sent to the device

[0586] Step 9:

[0587] The server uses an emotion engine to analyze the user's facial expressions and tone of voice, and based on the analysis results, it provides the shortest route if the user is in a hurry, or information on rest spots if the user is relaxing.

[0588] Input: Emotion recognition data (facial expressions, voice tone)

[0589] Output: Optimized information delivery

[0590] In this way, users can receive accurate location and real-time navigation even in indoor environments, as well as personalized information based on their emotions.

[0591] (Application example 2)

[0592] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0593] The present invention aims to provide a system that accurately identifies a user's location and provides navigation and information, especially in indoor environments where GPS is unavailable, and also aims to solve the problem of improving the user experience by recognizing the user's emotions and optimizing the information provided based on those emotions.

[0594] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0595] In this invention, the server includes a means for acquiring video images, a means for identifying a location from the video images, a means for generating map information in real time, a means for interactively providing information based on the location information and questions, and a means for recognizing a user's emotions and optimizing the information. This enables highly accurate location identification and navigation even in indoor environments that do not support GPS, and further enables personalized information provision according to the user's emotions.

[0596] "Means for acquiring video images" refers to the function that allows users to capture video in real time using the camera on their smartphone or tablet.

[0597] "Means for identifying a position from a video image" refers to a function that analyzes the acquired video image and performs processing to identify the user's current position from characteristic points (walls, doors, signs, etc.).

[0598] "Means for generating map information in real time" refers to the function of the server generating and providing the latest map that reflects the user's location information based on the analysis results.

[0599] "Means for interactively providing information based on location information and questions" refers to a function that provides relevant information (store information, route information, etc.) in real time based on the user's current location and inquiries from the user.

[0600] "Means for recognizing the user's emotions and optimizing information" refers to the function of analyzing the user's emotions from their facial expressions and voice, and adjusting the information and navigation content provided based on those emotions.

[0601] This invention is a system that accurately identifies a user's location and provides navigation and information, especially in indoor environments where GPS is not available. It is also capable of recognizing the user's emotions and optimizing the information provided based on those emotions.

[0602] The main components of the system include a means for acquiring video images, a means for identifying a location from the video images, a means for generating map information in real time, a means for providing information interactively based on location information and questions, and a means for recognizing the user's emotions and optimizing the information.

[0603] Hardware and Software Configuration

[0604] 1. Acquiring video images

[0605] Users can capture indoor video images using mobile devices such as smartphones and tablets, using the device's built-in camera and image processing libraries such as OpenCV.

[0606] 2. Video data sent to the server

[0607] The acquired video data is sent from the device to the server. This sending process uses Requests, an HTTP request library.

[0608] 3. Location Identification and Map Generation

[0609] The server analyzes the video images sent and extracts indoor features (walls, doors, signs, etc.). Based on this, it locates the user's location by comparing it with existing map information. This analysis is performed using an image analysis engine.

[0610] The server generates the latest map information in real time and sends it to the terminal.

[0611] 4. Navigation and Information

[0612] The device displays the received map information and the user's current location, and the user can select a route to their destination. The route information is visualized, and the device guides the user to their destination. This guidance can provide multiple options, such as the shortest route or a route that avoids congestion.

[0613] 5. Emotion Recognition and Information Optimization

[0614] When a user provides video via a camera, the server analyzes the user's facial expressions and voice in real time using an emotion recognition engine.

[0615] Based on the recognized emotion, the information provided and navigation content are optimized. For example, if the user is in a hurry, the shortest route will be prioritized, and if the user is relaxed, information about rest spots will be provided.

[0616] Examples of specific examples and prompts

[0617] This system is extremely effective in large shopping malls and department stores, for example. When a user uses the system while walking around the mall, they can obtain real-time information on congestion and store reviews, along with navigation to their destination. Furthermore, the information provided is optimized according to the user's emotions, providing a more comfortable shopping experience.

[0618] Example prompt sentence:

[0619] "Your goal is to develop an application that identifies the user's location in a large shopping mall, analyzes camera footage to recognize emotions, and provides optimal store navigation based on those emotions."

[0620] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0621] Step 1:

[0622] A user captures indoor video using the camera on their smartphone or tablet. When the user launches the app and the camera acquires video, the device captures video in real time. The input of this step is the video captured by the camera, and the output is the captured video data.

[0623] Step 2:

[0624] The device sends the captured video data to the server. The device encodes the video data into a format such as JPEG and sends it to the server using an HTTP request library (e.g., Requests). The input to this step is the video data captured in step 1, and the output is the data sent to the server.

[0625] Step 3:

[0626] The server analyzes the received video data using its image analysis engine. The server decodes the transmitted data and extracts feature points (e.g., walls, doors, signs, etc.). The input of this step is the video data transmitted in step 2, and the output is the extracted feature point data.

[0627] Step 4:

[0628] The server performs location identification. The server compares the extracted feature points with existing map information to identify the user's current location. The input to this step is the feature point data and map information obtained in step 3, and the output is the identified user's location information.

[0629] Step 5:

[0630] The server generates map information in real time and sends it to the device. The server generates the latest map information based on the user's location information and sends it to the device. The input of this step is the identified location information, and the output is the generated map information.

[0631] Step 6:

[0632] The device displays the received map information and the user's current location. The device visually displays the received map information to the user, allowing the user to select a route to their destination. The input of this step is the map information received in step 5, and the output is the displayed map information and the user's selected route information.

[0633] Step 7:

[0634] The device navigates the user to the destination. The device provides multiple route options, such as the shortest route or a route that avoids congestion, and navigates along the route selected by the user. The input of this step is the route information selected by the user, and the output is the navigation information provided.

[0635] Step 8:

[0636] The server recognizes the user's emotions and optimizes the information. When the user provides camera footage, the server analyzes the user's facial expressions and voice in real time to recognize emotions. An emotion recognition engine is used for this analysis. The input for this step is camera footage and voice data, and the output is recognized emotional information.

[0637] Step 9:

[0638] The server optimizes information based on the recognized emotion. The navigation content and information provided are adjusted according to the recognized emotion. For example, if the user is in a hurry, the shortest route is prioritized, and if the user is relaxed, information about rest spots is provided. The input of this step is the recognized emotion information, and the output is optimized information.

[0639] Step 10:

[0640] The terminal displays the optimized information and provides it to the user. The terminal visually and audibly provides the optimized information received from the server to the user. The input of this step is the optimized information, and the output is the provided visual and audio feedback.

[0641] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0642] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0643] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0644] [Third embodiment]

[0645] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0646] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0647] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0648] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0649] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0650] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0651] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0652] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0653] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0654] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0655] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0656] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0657] In an embodiment of the present invention, a system is constructed that identifies a user's position, particularly in an indoor environment where GPS cannot be used, and provides navigation and store information. A specific embodiment of the system is described below.

[0658] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server. The server is equipped with a high-performance image analysis engine that can analyze the video image data and identify the user's current location.

[0659] By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to pinpoint the user's exact location. Based on this information, the server generates map information in real time and sends it to the device as feedback to the user.

[0660] The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation. In addition, the server provides dynamic information such as store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage.

[0661] For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays that information on the screen and provides visual and audio feedback to the user.

[0662] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[0663] According to the above-described embodiment, the system of the present invention can accurately identify the user's position and provide real-time navigation and useful information even in indoor environments where GPS cannot be used.

[0664] The processing flow will be explained below.

[0665] Step 1:

[0666] The user launches the smartphone camera app, which selects a dedicated navigation mode and the camera captures real-time images of the interior of the room.

[0667] Step 2:

[0668] The device processes the acquired video data in real time and sends it to the server. The data is divided at a fixed frame rate and sent as network packets.

[0669] Step 3:

[0670] The server analyzes the received video data frame by frame, and uses image analysis algorithms to extract feature points (e.g., walls, doors, signs) from the video.

[0671] Step 4:

[0672] The server identifies the user's current location based on feature points in the video, analyzes the relative positions of the feature points, and compares them with existing map data to determine the user's exact location.

[0673] Step 5:

[0674] The server transmits real-time generated map information and the user's current location information to the terminal, and the server also transmits surrounding map data related to the user's location.

[0675] Step 6:

[0676] The device displays the received map information and the user's current location. The user selects a destination on the app's interface, and the current location and route to the destination are displayed on the screen.

[0677] Step 7:

[0678] The device calculates and displays the optimal route based on the user's input, and the app provides users with multiple route options, such as the shortest route or a route that avoids congestion.

[0679] Step 8:

[0680] The user enters store information or other questions by text or voice, such as "Where is the nearest cafe?" or "Can you give me a review of this cafe?"

[0681] Step 9:

[0682] The server receives a question from the user, analyzes the question using AI, and uses natural language processing (NLP) to retrieve relevant information from a database.

[0683] Step 10:

[0684] The server sends the acquired information to the device, including details such as store location, opening hours, and reviews.

[0685] Step 11:

[0686] The device displays the received information to the user, and the app provides information to the user through text, images, and in some cases audio.

[0687] Step 12:

[0688] The user follows the navigation display in the app and begins moving towards their destination. The device updates their current location and provides real-time directions.

[0689] Step 13:

[0690] The device periodically sends video images to the server, which then analyzes them, providing accurate navigation from the user's current location until they reach their destination.

[0691] Through this series of processes, the system can accurately identify the user's location even in indoor environments, providing real-time navigation and interactive information.

[0692] Example 1

[0693] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0694] Conventional technologies have difficulty accurately determining a user's location in indoor environments where GPS is unavailable. Furthermore, there are problems with navigation and local information provision after location determination in real time. Furthermore, there is a lack of systems that can efficiently provide users with the optimal route to their destination.

[0695] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0696] In this invention, the server includes means for analyzing video data and extracting feature points, means for collating the feature points with map information to identify a location, and means for generating map information in real time, thereby enabling accurate identification of a user's location in an indoor environment and providing navigation information and dynamic store information in real time.

[0697] A "moving image" is an image that expresses movement by displaying multiple consecutive still images (usually 24 or more) per second.

[0698] "Motion image data" refers to data for recording or transferring motion images in digital format.

[0699] A "feature point" is an identifiable point in an image that serves as a reference for location specification and image recognition.

[0700] "Map information" refers to information that shows the geographical features and facility locations of a specific area.

[0701] "Real-time" is a term that refers to near-instant processing or reaction.

[0702] "Interactive" means that it allows for two-way interaction with the user.

[0703] "Location information" refers to information that indicates the current location of an object or person.

[0704] "Analysis" is the process of examining data or information in detail to uncover meanings and relationships.

[0705] "Matching" is the process of comparing different data or information to identify matches and connections.

[0706] A "route" is a route or path to a particular destination.

[0707] An "optimal route" is a route or path that is evaluated as being the most efficient under certain conditions.

[0708] "Dynamic store information" refers to information about a store that changes depending on the time or situation (e.g., occupancy status, latest reviews).

[0709] The present invention relates to a system for locating a user's position and providing navigation and store information, particularly in an indoor environment where GPS is unavailable. Specific embodiments of the system are described below.

[0710] Hardware and Software Configuration

[0711] Hardware

[0712] Smartphone / tablet (device): A device with a camera function and internet connection.

[0713] Server: A server capable of high-performance processing (e.g., a server provided by a cloud service provider).

[0714] software

[0715] Terminal application: A dedicated application that captures camera footage and sends it to a server in real time.

[0716] Image analysis engine: Software that analyzes video data using libraries such as OpenCV.

[0717] Database: A database for managing location information and store information (e.g., a relational database system).

[0718] System Overview

[0719] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server.

[0720] The server is equipped with a high-performance image analysis engine that can analyze video data to identify the user's current location. By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to identify the user's exact location.

[0721] The server generates map information in real time based on this information and sends it to the device as feedback to the user. The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation.

[0722] Furthermore, the server provides real-time dynamic information such as store information and congestion status based on the user's location information and surrounding information obtained from camera footage. For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server obtains the latest review information for that cafe and sends it to the device. The device displays that information on the screen and provides the user with visual and audio feedback.

[0723] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[0724] Specific examples

[0725] For example, if a user is in a large shopping mall, they can use their smartphone camera to capture video of the store. The application sends this video to a server, which analyzes the video data to determine the user's location. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays this information on the screen and provides the user with visual and audio feedback.

[0726] Prompt Sentence Examples

[0727] An example of a prompt sentence might be:

[0728] "Tell me about cafes in large shopping malls. I'd like to know recommended cafes and the latest reviews."

[0729] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0730] Step 1:

[0731] Users launch a dedicated application on their smartphone or tablet and use the camera to capture indoor video. The input is video from the device's camera, and the output is a real-time video preview. Users can accurately capture video by pointing the camera in the appropriate direction and checking the video preview on the device screen.

[0732] Step 2:

[0733] The terminal transmits the captured video data to the server in real time. The input is the video data captured by the terminal, and the output is the data transmitted to the server via the Internet. The terminal confirms the success of the transmission and displays the transmission status to the user.

[0734] Step 3:

[0735] The server analyzes the received video data and uses an image analysis engine (e.g., OpenCV) to extract indoor feature points (walls, doors, signs, etc.). The input is the video data sent to the server, and the output is the extracted feature point data. The server uses edge detection and feature point matching algorithms to extract the feature points.

[0736] Step 4:

[0737] The server compares the extracted feature points with pre-registered indoor map information to identify the user's current location. The input is the extracted feature points and map data, and the output is the coordinate information of the user's current location. The server uses algorithms such as KD tree and Hough transform for the comparison.

[0738] Step 5:

[0739] The server generates the necessary data based on the user's current location, the latest map information, and dynamic store information. The input is the user's current location, map data, and store data, and the output is map information and store information packaged in JSON format. The server sends this to the device in real time.

[0740] Step 6:

[0741] The device displays the received map information, the user's current location, and the destination. The input is JSON format data received from the server, and the output is a visual map display on the user interface. The device uses a map drawing library such as Google Maps API or Mapbox SDK to display the map.

[0742] Step 7:

[0743] The user selects a destination on the screen and sets the navigation route. The input is the user's destination selection operation, and the output is the navigation route information. The device displays the route and guides the user visually and audibly.

[0744] Step 8:

[0745] The server obtains dynamic information from the connected store database and sensor network and provides the necessary information to the user. The input is a data acquisition request from the store database, and the output is the latest store information and congestion status. The server sends this information to the terminal in real time, and the terminal displays it on the screen. Specifically, by providing the user with the latest store reviews and congestion status, the user can make better decisions.

[0746] These steps enable the system to accurately locate users in indoor environments where GPS is unavailable, providing real-time navigation and dynamic information provision.

[0747] (Application example 1)

[0748] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0749] Existing indoor navigation systems have difficulty pinpointing a user's exact location in environments where GPS is unavailable, and their ability to provide real-time information is limited. Another problem is that users cannot instantly obtain information by voice input. This makes it difficult to move around efficiently and use the system, especially in shopping malls and large facilities.

[0750] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0751] In this invention, the server includes a means for acquiring a moving image, a means for identifying a position from the moving image, a means for generating map information in real time, and a means for providing information interactively based on the position information and a voice input, thereby enabling the server to accurately identify a user's position even in an indoor environment where GPS is not available, and to provide information interactively at high speed via a voice input.

[0752] "Means for acquiring moving images" refers to devices or software that capture images of the user's surroundings in real time.

[0753] "Means for determining location from video images" refers to an algorithm or system for analyzing and determining a user's current location from captured video data.

[0754] "Means for generating map information in real time" refers to devices or programs that instantly create map information corresponding to the current environment based on the user's location information.

[0755] "Means for providing information interactively based on location information and voice input" refers to systems and technologies that respond to a user's current location information and voice questions, and provide relevant information on the spot.

[0756] "Means for analyzing camera footage and obtaining surrounding information and store information" refers to technology for analyzing footage captured by a camera and extracting detailed information about facilities and stores around the user.

[0757] "Audio and visual means" refers to interfaces and technologies for providing information to users through audio and screen displays.

[0758] To implement this invention, a smartphone, tablet, or other mobile device is used. When the user is in a shopping mall or large facility, the following system is used to provide indoor navigation and store information.

[0759] First, the user launches a dedicated app installed on their device and captures video of their surroundings through the camera. This video data is sent to the server in real time. The server then performs image analysis based on the received video data and determines the user's current location. The main technologies used for this are image analysis engines such as OpenCV and TensorFlow.

[0760] Specifically, the server extracts indoor features (e.g., walls, doors, signs, etc.) from camera footage and locates the location by comparing them with existing map information. Based on the identified location information, map information is generated in real time and fed back to the device. In addition, the system accepts voice input from the user and provides information interactively based on the location information and the question. Google Cloud Speech-to-Text and Text-to-Speech APIs are used for voice processing.

[0761] For example, if a user says, "Tell me a review of the cafe up ahead," the server will retrieve the location and review information of the nearest cafe and provide real-time audio and visual feedback, allowing the user to easily understand the congestion situation within the facility and detailed store information.

[0762] With the above system, the present invention can accurately locate a user's location even in indoor environments where GPS is not available, and provide real-time information and navigation. This convenience is particularly noticeable in large shopping malls and complex indoor facilities, providing users with comfortable and efficient mobility.

[0763] An example of a prompt sentence might be:

[0764] "When a user is searching for a cafe in a shopping mall, please explain the program that will identify their location using camera footage and provide information on the nearest cafe along with reviews."

[0765] The system of the present invention can greatly improve convenience at specific locations and dramatically improve the user experience.

[0766] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0767] Step 1:

[0768] The user launches a dedicated app on their smartphone and captures images of their surroundings through the camera. The input is real-time camera images, and the output is the captured image data. This allows the user's surroundings to be displayed in real time.

[0769] Step 2:

[0770] The device sends the captured video data to the server. The input is the video data from step 1, and the output is the data sent to the server. The device sends the video data using the HTTP or HTTPS protocol.

[0771] Step 3:

[0772] The server analyzes the received video data using an image analysis engine (e.g., OpenCV, TensorFlow) and extracts indoor feature points. The input is the transmitted video data, and the output is feature point data. The server identifies feature points such as walls, doors, and signs in the video.

[0773] Step 4:

[0774] The server matches the feature point data with existing map information to determine the user's current location. The input is feature point data and map information, and the output is the user's current location. The server determines the location using a database (e.g., MySQL, PostgreSQL).

[0775] Step 5:

[0776] The server generates map information in real time and sends it to the terminal. The input is the user's current location and map information, and the output is the generated map information. The server generates map information instantly and provides it to the terminal.

[0777] Step 6:

[0778] The terminal displays the received map information and location information on the screen. The input is the received map information and current location information, and the output is the map and current location displayed to the user. The terminal provides map information visually to the user.

[0779] Step 7:

[0780] The user asks a question by voice input. The input is a voice command such as "Please tell me a review of the cafe up ahead," and the output is the results of voice analysis. The user's voice information is analyzed by the device.

[0781] Step 8:

[0782] The device sends the analyzed voice input to the server and requests information based on the question. The input is the voice analysis result, and the output is a request to send to the server. The device converts the voice data into text and sends it to the server.

[0783] Step 9:

[0784] The server searches for relevant information based on the voice input and location information and sends it to the device. The input is the voice analysis results and location information, and the output is relevant information (e.g., cafe reviews). The server searches a database and provides the appropriate information.

[0785] Step 10:

[0786] The device provides the received information to the user via voice and visual display. The input is relevant information from the server, and the output is a voice response and visual display to the user. The device generates the voice response using Google Cloud Text-to-Speech.

[0787] Step 11:

[0788] The user receives audio and visual feedback from the device and acts based on the navigation and information. The input is audio and visual feedback, and the output is the user's actions. The user decides on their next action based on the displayed information.

[0789] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0790] In an embodiment of the present invention, a system is constructed to identify a user's location, particularly in an indoor environment where GPS is not available, and provide navigation and store information. Furthermore, the present invention combines an emotion engine that recognizes the user's emotions and optimizes the information provided based on the emotions.

[0791] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server. The server is equipped with a high-performance image analysis engine that can analyze the video image data and identify the user's current location.

[0792] By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to pinpoint the user's exact location. Based on this information, the server generates map information in real time and sends it to the device as feedback to the user.

[0793] The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation. In addition, the server provides dynamic information such as store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage.

[0794] For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays that information on the screen and provides visual and audio feedback to the user.

[0795] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[0796] Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions, and by analyzing the user's facial expressions and tone of voice, it grasps their emotions in real time. For example, if the user looks impatient, the system will determine that they are in a hurry and will prioritize displaying the shortest route. Also, if the user is relaxing, it will provide information on rest spots and cafes where they can relax.

[0797] The navigation display and audio guidance provided can also be adjusted based on the user's emotions. For example, if the user is stressed, the audio guidance can be reassuring and avoid complex instructions. Conversely, if the user is excited, the audio guidance can be more entertaining, providing detailed information and more options.

[0798] By combining these functions, the system of the present invention can accurately locate the user's location even in indoor environments where GPS is unavailable, providing real-time navigation and interactive information, and can also provide personalized services based on the user's emotions, providing a more convenient and user-friendly experience.

[0799] The processing flow will be explained below.

[0800] Step 1:

[0801] The user launches the smartphone camera app, which selects a dedicated navigation mode and the camera captures real-time images of the interior of the room.

[0802] Step 2:

[0803] The device processes the acquired video data in real time and sends it to the server. The data is divided at a fixed frame rate and sent as network packets.

[0804] Step 3:

[0805] The server analyzes the received video data frame by frame, and uses image analysis algorithms to extract feature points (e.g., walls, doors, signs) from the video.

[0806] Step 4:

[0807] The server identifies the user's current location based on feature points in the video, analyzes the relative positions of the feature points, and compares them with existing map data to determine the user's exact location.

[0808] Step 5:

[0809] The server transmits real-time generated map information and the user's current location information to the terminal, and the server also transmits surrounding map data related to the user's location.

[0810] Step 6:

[0811] The device displays the received map information and the user's current location. The user selects a destination on the app's interface, and the current location and route to the destination are displayed on the screen.

[0812] Step 7:

[0813] The device calculates and displays the optimal route based on the user's input, and the app provides users with multiple route options, such as the shortest route or a route that avoids congestion.

[0814] Step 8:

[0815] The user enters store information or other questions by text or voice, such as "Where is the nearest cafe?" or "Can you give me a review of this cafe?"

[0816] Step 9:

[0817] The server receives a question from the user, analyzes the question using AI, and uses natural language processing (NLP) to retrieve relevant information from a database.

[0818] Step 10:

[0819] The server sends the acquired information to the device, including details such as store location, opening hours, and reviews.

[0820] Step 11:

[0821] The device displays the received information to the user, and the app provides information to the user through text, images, and in some cases audio.

[0822] Step 12:

[0823] The user follows the navigation display in the app and begins moving towards their destination. The device updates their current location and provides real-time directions.

[0824] Step 13:

[0825] The device periodically sends video images to the server, which then analyzes them, providing accurate navigation from the user's current location until they reach their destination.

[0826] Step 14:

[0827] The device analyzes the camera footage and the user's voice and uses an emotion engine to identify the user's emotional state. For example, it recognizes the user's current emotion by analyzing the user's facial expressions and tone of voice.

[0828] Step 15:

[0829] The server adjusts the information provided and navigation methods based on the emotion data sent from the emotion engine. For example, if it determines that the user is in a hurry, it will prioritize suggesting the shortest route.

[0830] Step 16:

[0831] The device will provide users with information and navigation based on their emotions. For example, if the user is feeling relaxed, it will provide information on rest spots and cafes.

[0832] Step 17:

[0833] When the user again inputs a question about store information or navigation, the server takes the emotional data into consideration and provides the most appropriate information.

[0834] Step 18:

[0835] The server and device work together to continuously monitor the user's current location and emotional state, thereby constantly providing the user with the most appropriate information and navigation.

[0836] Example 2

[0837] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0838] In indoor environments, GPS is unavailable, making it difficult to pinpoint the user's exact location, and providing navigation and store information is also difficult.In addition, the inability to provide information that takes into account the user's emotions results in an inadequate user experience.

[0839] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring a moving image, a means for transmitting the moving image to the server, a means for identifying a location from the moving image, a means for generating map information in real time, a means for interactively providing information based on the location information and a question, and a means for recognizing a user's emotion and optimizing information. This makes it possible to identify the user's exact location even in an indoor environment, and to provide real-time navigation and personalized information.

[0840] The "means for acquiring moving images" is a function that allows a user to capture images of the interior of a room using the camera on a smartphone or tablet.

[0841] The "means for transmitting moving images to a server" is a communication function for transferring moving image data acquired by a terminal to a server in real time.

[0842] The "means for identifying a location from a video image" is a function that analyzes video image data using an image analysis engine installed in the server and identifies the user's current location.

[0843] "Means for generating map information in real time" refers to a function in which the server instantly creates the latest map information based on the user's location information and provides it to the user.

[0844] The "means for interactively providing information based on location information and questions" is a function that provides appropriate information in real time based on the user's current location and questions.

[0845] "Means for recognizing the user's emotions and optimizing information" refers to a function in which the server analyzes the user's facial expressions and tone of voice to understand their emotions and provide optimal information based on those emotions.

[0846] The "means for analyzing camera images and acquiring surrounding information" is a function for analyzing images acquired through a camera and identifying information about the environment surrounding the user.

[0847] The "means for providing the optimal route based on the user's current location" is a function that calculates and displays the most efficient route to the destination set by the user.

[0848] The present invention builds a system that identifies a user's location and provides navigation and store information, especially in indoor environments where GPS is not available. Furthermore, the present invention combines an emotion engine that recognizes the user's emotions and optimizes the information provided based on those emotions.

[0849] Hardware and software used

[0850] The main hardware of this system is the user's smartphone or tablet, which must be equipped with a high-performance camera. As for software, a dedicated application is installed on the device, which is responsible for capturing and transmitting video images.

[0851] The server is equipped with a high-performance image analysis engine (e.g., OpenCV, TensorFlow) that analyzes video data in real time. The server also has an emotion engine that recognizes the user's emotions and has the ability to analyze the user's facial expressions and tone of voice.

[0852] Specific processing and operations

[0853] The system starts with the user using a smartphone or tablet to take pictures of the indoor environment with a camera. The user then launches a dedicated application and starts capturing video images in real time. At this time, the device sends the video image data to a server via Wi-Fi or a mobile network.

[0854] The server analyzes the received video data and extracts indoor features (e.g., walls, doors, signs). It then compares this with existing map information to determine the user's current location. The server generates map information in real time based on this information and sends it to the device as feedback.

[0855] The device displays the received map information and the user's current location, as well as calculates and displays the optimal route to the user's selected destination. When the user selects "I want to go to the Apple Store," the device displays the route and guides the user along the route.

[0856] Furthermore, the server provides dynamic store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage. For example, when a user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays the information on its screen and provides visual and audio feedback to the user.

[0857] The emotion engine allows the server to analyze the user's facial expressions and tone of voice to understand their emotions in real time. For example, if the user looks anxious, the system will determine that they are in a hurry and prioritize the shortest route. Conversely, if the user is relaxed, it will provide information on rest spots and cafes where they can relax.

[0858] Examples of specific examples and prompts

[0859] When a user is in a large shopping mall, the information of each store is analyzed through camera footage, and the server sends that information to the device. For example, if a user selects "I want to go to the Apple Store," the device will display the shortest route. Similarly, if the user asks "What are the reviews of this cafe?", the device will display the latest reviews.

[0860] An example prompt is:

[0861] "Please give us a review of the cafe."

[0862] "What's the best route to get to the nearest restroom?"

[0863] "How do I get to the cinema?"

[0864] This allows users to easily locate and navigate indoors, as well as receive personalized information based on their emotions.

[0865] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0866] Step 1:

[0867] A user launches a dedicated application on a smartphone or tablet and activates the camera. This application is designed to allow users to take pictures of indoor scenes. When the user presses the capture button, the device begins capturing video images in real time.

[0868] Input: User initiates camera and takes a photo

[0869] Output: Real-time video data

[0870] Step 2:

[0871] The device transmits the acquired video data to a server via Wi-Fi or mobile networks in real time, minimizing time lag.

[0872] Input: Real-time video data

[0873] Output: Send video data to the server

[0874] Step 3:

[0875] The server uses a high-performance image analysis engine (e.g., OpenCV, TensorFlow) to analyze the received video data. The server detects indoor features (e.g., walls, doors, signs) and compares them with existing map information.

[0876] Input: Video data

[0877] Output: Current location data of the user

[0878] Step 4:

[0879] The server generates real-time map information based on the extracted feature points. This map information reflects the user's current location. The server then sends the generated map information to the device.

[0880] Input: current location data of the user

[0881] Output: Generate real-time map information and send it to the device

[0882] Step 5:

[0883] The device displays the received map information to the user. The map shows the user's current location and information about nearby stores. When the user selects a destination (e.g., an Apple Store), the device sends that information to the server.

[0884] Input: Real-time map information

[0885] Output: Map display and destination selection information sent to server

[0886] Step 6:

[0887] The server calculates the optimal route based on the user's current location and selected destination, and can provide multiple route options (e.g., shortest route, congestion avoidance route, etc.).

[0888] Input: User's current location data and destination selection information

[0889] Output: Calculate optimal route information and send it to the device

[0890] Step 7:

[0891] The terminal displays the optimal route information received from the server to the user, providing interactive navigation. The user follows the presented route and is guided to the destination.

[0892] Input: Optimal route information

[0893] Output: Display route information and provide navigation

[0894] Step 8:

[0895] The server provides real-time store information and congestion status based on the user's current location and surrounding information obtained from camera footage. For example, when a user asks, "What are the reviews for this cafe?", the server retrieves the latest review information and sends it to the device.

[0896] Input: Local information, user questions

[0897] Output: Dynamic store information and review information sent to the device

[0898] Step 9:

[0899] The server uses an emotion engine to analyze the user's facial expressions and tone of voice, and based on the analysis results, it provides the shortest route if the user is in a hurry, or information on rest spots if the user is relaxing.

[0900] Input: Emotion recognition data (facial expressions, voice tone)

[0901] Output: Optimized information delivery

[0902] In this way, users can receive accurate location and real-time navigation even in indoor environments, as well as personalized information based on their emotions.

[0903] (Application example 2)

[0904] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0905] The present invention aims to provide a system that accurately identifies a user's location and provides navigation and information, especially in indoor environments where GPS is unavailable, and also aims to solve the problem of improving the user experience by recognizing the user's emotions and optimizing the information provided based on those emotions.

[0906] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0907] In this invention, the server includes a means for acquiring video images, a means for identifying a location from the video images, a means for generating map information in real time, a means for interactively providing information based on the location information and questions, and a means for recognizing a user's emotions and optimizing the information. This enables highly accurate location identification and navigation even in indoor environments that do not support GPS, and further enables personalized information provision according to the user's emotions.

[0908] "Means for acquiring video images" refers to the function that allows users to capture video in real time using the camera on their smartphone or tablet.

[0909] "Means for identifying a position from a video image" refers to a function that analyzes the acquired video image and performs processing to identify the user's current position from characteristic points (walls, doors, signs, etc.).

[0910] "Means for generating map information in real time" refers to the function of the server generating and providing the latest map that reflects the user's location information based on the analysis results.

[0911] "Means for interactively providing information based on location information and questions" refers to a function that provides relevant information (store information, route information, etc.) in real time based on the user's current location and inquiries from the user.

[0912] "Means for recognizing the user's emotions and optimizing information" refers to the function of analyzing the user's emotions from their facial expressions and voice, and adjusting the information and navigation content provided based on those emotions.

[0913] This invention is a system that accurately identifies a user's location and provides navigation and information, especially in indoor environments where GPS is not available. It is also capable of recognizing the user's emotions and optimizing the information provided based on those emotions.

[0914] The main components of the system include a means for acquiring video images, a means for identifying a location from the video images, a means for generating map information in real time, a means for providing information interactively based on location information and questions, and a means for recognizing the user's emotions and optimizing the information.

[0915] Hardware and Software Configuration

[0916] 1. Acquiring video images

[0917] Users can capture indoor video images using mobile devices such as smartphones and tablets, using the device's built-in camera and image processing libraries such as OpenCV.

[0918] 2. Video data sent to the server

[0919] The acquired video data is sent from the device to the server. This sending process uses Requests, an HTTP request library.

[0920] 3. Location Identification and Map Generation

[0921] The server analyzes the video images sent and extracts indoor features (walls, doors, signs, etc.). Based on this, it locates the user's location by comparing it with existing map information. This analysis is performed using an image analysis engine.

[0922] The server generates the latest map information in real time and sends it to the terminal.

[0923] 4. Navigation and Information

[0924] The device displays the received map information and the user's current location, and the user can select a route to their destination. The route information is visualized, and the device guides the user to their destination. This guidance can provide multiple options, such as the shortest route or a route that avoids congestion.

[0925] 5. Emotion Recognition and Information Optimization

[0926] When a user provides video via a camera, the server analyzes the user's facial expressions and voice in real time using an emotion recognition engine.

[0927] Based on the recognized emotion, the information provided and navigation content are optimized. For example, if the user is in a hurry, the shortest route will be prioritized, and if the user is relaxed, information about rest spots will be provided.

[0928] Examples of specific examples and prompts

[0929] This system is extremely effective in large shopping malls and department stores, for example. When a user uses the system while walking around the mall, they can obtain real-time information on congestion and store reviews, along with navigation to their destination. Furthermore, the information provided is optimized according to the user's emotions, providing a more comfortable shopping experience.

[0930] Example prompt sentence:

[0931] "Your goal is to develop an application that identifies the user's location in a large shopping mall, analyzes camera footage to recognize emotions, and provides optimal store navigation based on those emotions."

[0932] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0933] Step 1:

[0934] A user captures indoor video using the camera on their smartphone or tablet. When the user launches the app and the camera acquires video, the device captures video in real time. The input of this step is the video captured by the camera, and the output is the captured video data.

[0935] Step 2:

[0936] The device sends the captured video data to the server. The device encodes the video data into a format such as JPEG and sends it to the server using an HTTP request library (e.g., Requests). The input to this step is the video data captured in step 1, and the output is the data sent to the server.

[0937] Step 3:

[0938] The server analyzes the received video data using its image analysis engine. The server decodes the transmitted data and extracts feature points (e.g., walls, doors, signs, etc.). The input of this step is the video data transmitted in step 2, and the output is the extracted feature point data.

[0939] Step 4:

[0940] The server performs location identification. The server compares the extracted feature points with existing map information to identify the user's current location. The input to this step is the feature point data and map information obtained in step 3, and the output is the identified user's location information.

[0941] Step 5:

[0942] The server generates map information in real time and sends it to the device. The server generates the latest map information based on the user's location information and sends it to the device. The input of this step is the identified location information, and the output is the generated map information.

[0943] Step 6:

[0944] The device displays the received map information and the user's current location. The device visually displays the received map information to the user, allowing the user to select a route to their destination. The input of this step is the map information received in step 5, and the output is the displayed map information and the user's selected route information.

[0945] Step 7:

[0946] The device navigates the user to the destination. The device provides multiple route options, such as the shortest route or a route that avoids congestion, and navigates along the route selected by the user. The input of this step is the route information selected by the user, and the output is the navigation information provided.

[0947] Step 8:

[0948] The server recognizes the user's emotions and optimizes the information. When the user provides camera footage, the server analyzes the user's facial expressions and voice in real time to recognize emotions. An emotion recognition engine is used for this analysis. The input for this step is camera footage and voice data, and the output is recognized emotional information.

[0949] Step 9:

[0950] The server optimizes information based on the recognized emotion. The navigation content and information provided are adjusted according to the recognized emotion. For example, if the user is in a hurry, the shortest route is prioritized, and if the user is relaxed, information about rest spots is provided. The input of this step is the recognized emotion information, and the output is optimized information.

[0951] Step 10:

[0952] The terminal displays the optimized information and provides it to the user. The terminal visually and audibly provides the optimized information received from the server to the user. The input of this step is the optimized information, and the output is the provided visual and audio feedback.

[0953] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0954] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0955] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0956] [Fourth embodiment]

[0957] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0958] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0959] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0960] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0961] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0962] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0963] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0964] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0965] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0966] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0967] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0968] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0969] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0970] In an embodiment of the present invention, a system is constructed that identifies a user's position, particularly in an indoor environment where GPS cannot be used, and provides navigation and store information. A specific embodiment of the system is described below.

[0971] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server. The server is equipped with a high-performance image analysis engine that can analyze the video image data and identify the user's current location.

[0972] By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to pinpoint the user's exact location. Based on this information, the server generates map information in real time and sends it to the device as feedback to the user.

[0973] The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation. In addition, the server provides dynamic information such as store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage.

[0974] For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays that information on the screen and provides visual and audio feedback to the user.

[0975] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[0976] According to the above-described embodiment, the system of the present invention can accurately identify the user's position and provide real-time navigation and useful information even in indoor environments where GPS cannot be used.

[0977] The processing flow will be explained below.

[0978] Step 1:

[0979] The user launches the smartphone camera app, which selects a dedicated navigation mode and the camera captures real-time images of the interior of the room.

[0980] Step 2:

[0981] The device processes the acquired video data in real time and sends it to the server. The data is divided at a fixed frame rate and sent as network packets.

[0982] Step 3:

[0983] The server analyzes the received video data frame by frame, and uses image analysis algorithms to extract feature points (e.g., walls, doors, signs) from the video.

[0984] Step 4:

[0985] The server identifies the user's current location based on feature points in the video, analyzes the relative positions of the feature points, and compares them with existing map data to determine the user's exact location.

[0986] Step 5:

[0987] The server transmits real-time generated map information and the user's current location information to the terminal, and the server also transmits surrounding map data related to the user's location.

[0988] Step 6:

[0989] The device displays the received map information and the user's current location. The user selects a destination on the app's interface, and the current location and route to the destination are displayed on the screen.

[0990] Step 7:

[0991] The device calculates and displays the optimal route based on the user's input, and the app provides users with multiple route options, such as the shortest route or a route that avoids congestion.

[0992] Step 8:

[0993] The user enters store information or other questions by text or voice, such as "Where is the nearest cafe?" or "Can you give me a review of this cafe?"

[0994] Step 9:

[0995] The server receives a question from the user, analyzes the question using AI, and uses natural language processing (NLP) to retrieve relevant information from a database.

[0996] Step 10:

[0997] The server sends the acquired information to the device, including details such as store location, opening hours, and reviews.

[0998] Step 11:

[0999] The device displays the received information to the user, and the app provides information to the user through text, images, and in some cases audio.

[1000] Step 12:

[1001] The user follows the navigation display in the app and begins moving towards their destination. The device updates their current location and provides real-time directions.

[1002] Step 13:

[1003] The device periodically sends video images to the server, which then analyzes them, providing accurate navigation from the user's current location until they reach their destination.

[1004] Through this series of processes, the system can accurately identify the user's location even in indoor environments, providing real-time navigation and interactive information.

[1005] Example 1

[1006] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1007] Conventional technologies have difficulty accurately determining a user's location in indoor environments where GPS is unavailable. Furthermore, there are problems with navigation and local information provision after location determination in real time. Furthermore, there is a lack of systems that can efficiently provide users with the optimal route to their destination.

[1008] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1009] In this invention, the server includes means for analyzing video data and extracting feature points, means for collating the feature points with map information to identify a location, and means for generating map information in real time, thereby enabling accurate identification of a user's location in an indoor environment and providing navigation information and dynamic store information in real time.

[1010] A "moving image" is an image that expresses movement by displaying multiple consecutive still images (usually 24 or more) per second.

[1011] "Motion image data" refers to data for recording or transferring motion images in digital format.

[1012] A "feature point" is an identifiable point in an image that serves as a reference for location specification and image recognition.

[1013] "Map information" refers to information that shows the geographical features and facility locations of a specific area.

[1014] "Real-time" is a term that refers to near-instant processing or reaction.

[1015] "Interactive" means that it allows for two-way interaction with the user.

[1016] "Location information" refers to information that indicates the current location of an object or person.

[1017] "Analysis" is the process of examining data or information in detail to uncover meanings and relationships.

[1018] "Matching" is the process of comparing different data or information to identify matches and connections.

[1019] A "route" is a route or path to a particular destination.

[1020] An "optimal route" is a route or path that is evaluated as being the most efficient under certain conditions.

[1021] "Dynamic store information" refers to information about a store that changes depending on the time or situation (e.g., occupancy status, latest reviews).

[1022] The present invention relates to a system for locating a user's position and providing navigation and store information, particularly in an indoor environment where GPS is unavailable. Specific embodiments of the system are described below.

[1023] Hardware and Software Configuration

[1024] Hardware

[1025] Smartphone / tablet (device): A device with a camera function and internet connection.

[1026] Server: A server capable of high-performance processing (e.g., a server provided by a cloud service provider).

[1027] software

[1028] Terminal application: A dedicated application that captures camera footage and sends it to a server in real time.

[1029] Image analysis engine: Software that analyzes video data using libraries such as OpenCV.

[1030] Database: A database for managing location information and store information (e.g., a relational database system).

[1031] System Overview

[1032] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server.

[1033] The server is equipped with a high-performance image analysis engine that can analyze video data to identify the user's current location. By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to identify the user's exact location.

[1034] The server generates map information in real time based on this information and sends it to the device as feedback to the user. The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation.

[1035] Furthermore, the server provides real-time dynamic information such as store information and congestion status based on the user's location information and surrounding information obtained from camera footage. For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server obtains the latest review information for that cafe and sends it to the device. The device displays that information on the screen and provides the user with visual and audio feedback.

[1036] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[1037] Specific examples

[1038] For example, if a user is in a large shopping mall, they can use their smartphone camera to capture video of the store. The application sends this video to a server, which analyzes the video data to determine the user's location. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays this information on the screen and provides the user with visual and audio feedback.

[1039] Prompt Sentence Examples

[1040] An example of a prompt sentence might be:

[1041] "Tell me about cafes in large shopping malls. I'd like to know recommended cafes and the latest reviews."

[1042] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1043] Step 1:

[1044] Users launch a dedicated application on their smartphone or tablet and use the camera to capture indoor video. The input is video from the device's camera, and the output is a real-time video preview. Users can accurately capture video by pointing the camera in the appropriate direction and checking the video preview on the device screen.

[1045] Step 2:

[1046] The terminal transmits the captured video data to the server in real time. The input is the video data captured by the terminal, and the output is the data transmitted to the server via the Internet. The terminal confirms the success of the transmission and displays the transmission status to the user.

[1047] Step 3:

[1048] The server analyzes the received video data and uses an image analysis engine (e.g., OpenCV) to extract indoor feature points (walls, doors, signs, etc.). The input is the video data sent to the server, and the output is the extracted feature point data. The server uses edge detection and feature point matching algorithms to extract the feature points.

[1049] Step 4:

[1050] The server compares the extracted feature points with pre-registered indoor map information to identify the user's current location. The input is the extracted feature points and map data, and the output is the coordinate information of the user's current location. The server uses algorithms such as KD tree and Hough transform for the comparison.

[1051] Step 5:

[1052] The server generates the necessary data based on the user's current location, the latest map information, and dynamic store information. The input is the user's current location, map data, and store data, and the output is map information and store information packaged in JSON format. The server sends this to the device in real time.

[1053] Step 6:

[1054] The device displays the received map information, the user's current location, and the destination. The input is JSON format data received from the server, and the output is a visual map display on the user interface. The device uses a map drawing library such as Google Maps API or Mapbox SDK to display the map.

[1055] Step 7:

[1056] The user selects a destination on the screen and sets the navigation route. The input is the user's destination selection operation, and the output is the navigation route information. The device displays the route and guides the user visually and audibly.

[1057] Step 8:

[1058] The server obtains dynamic information from the connected store database and sensor network and provides the necessary information to the user. The input is a data acquisition request from the store database, and the output is the latest store information and congestion status. The server sends this information to the terminal in real time, and the terminal displays it on the screen. Specifically, by providing the user with the latest store reviews and congestion status, the user can make better decisions.

[1059] These steps enable the system to accurately locate users in indoor environments where GPS is unavailable, providing real-time navigation and dynamic information provision.

[1060] (Application example 1)

[1061] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1062] Existing indoor navigation systems have difficulty pinpointing a user's exact location in environments where GPS is unavailable, and their ability to provide real-time information is limited. Another problem is that users cannot instantly obtain information by voice input. This makes it difficult to move around efficiently and use the system, especially in shopping malls and large facilities.

[1063] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1064] In this invention, the server includes a means for acquiring a moving image, a means for identifying a position from the moving image, a means for generating map information in real time, and a means for providing information interactively based on the position information and a voice input, thereby enabling the server to accurately identify a user's position even in an indoor environment where GPS is not available, and to provide information interactively at high speed via a voice input.

[1065] "Means for acquiring moving images" refers to devices or software that capture images of the user's surroundings in real time.

[1066] "Means for determining location from video images" refers to an algorithm or system for analyzing and determining a user's current location from captured video data.

[1067] "Means for generating map information in real time" refers to devices or programs that instantly create map information corresponding to the current environment based on the user's location information.

[1068] "Means for providing information interactively based on location information and voice input" refers to systems and technologies that respond to a user's current location information and voice questions, and provide relevant information on the spot.

[1069] "Means for analyzing camera footage and obtaining surrounding information and store information" refers to technology for analyzing footage captured by a camera and extracting detailed information about facilities and stores around the user.

[1070] "Audio and visual means" refers to interfaces and technologies for providing information to users through audio and screen displays.

[1071] To implement this invention, a smartphone, tablet, or other mobile device is used. When the user is in a shopping mall or large facility, the following system is used to provide indoor navigation and store information.

[1072] First, the user launches a dedicated app installed on their device and captures video of their surroundings through the camera. This video data is sent to the server in real time. The server then performs image analysis based on the received video data and determines the user's current location. The main technologies used for this are image analysis engines such as OpenCV and TensorFlow.

[1073] Specifically, the server extracts indoor features (e.g., walls, doors, signs, etc.) from camera footage and locates the location by comparing them with existing map information. Based on the identified location information, map information is generated in real time and fed back to the device. In addition, the system accepts voice input from the user and provides information interactively based on the location information and the question. Google Cloud Speech-to-Text and Text-to-Speech APIs are used for voice processing.

[1074] For example, if a user says, "Tell me a review of the cafe up ahead," the server will retrieve the location and review information of the nearest cafe and provide real-time audio and visual feedback, allowing the user to easily understand the congestion situation within the facility and detailed store information.

[1075] With the above system, the present invention can accurately locate a user's location even in indoor environments where GPS is not available, and provide real-time information and navigation. This convenience is particularly noticeable in large shopping malls and complex indoor facilities, providing users with comfortable and efficient mobility.

[1076] An example of a prompt sentence might be:

[1077] "When a user is searching for a cafe in a shopping mall, please explain the program that will identify their location using camera footage and provide information on the nearest cafe along with reviews."

[1078] The system of the present invention can greatly improve convenience at specific locations and dramatically improve the user experience.

[1079] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1080] Step 1:

[1081] The user launches a dedicated app on their smartphone and captures images of their surroundings through the camera. The input is real-time camera images, and the output is the captured image data. This allows the user's surroundings to be displayed in real time.

[1082] Step 2:

[1083] The device sends the captured video data to the server. The input is the video data from step 1, and the output is the data sent to the server. The device sends the video data using the HTTP or HTTPS protocol.

[1084] Step 3:

[1085] The server analyzes the received video data using an image analysis engine (e.g., OpenCV, TensorFlow) and extracts indoor feature points. The input is the transmitted video data, and the output is feature point data. The server identifies feature points such as walls, doors, and signs in the video.

[1086] Step 4:

[1087] The server matches the feature point data with existing map information to determine the user's current location. The input is feature point data and map information, and the output is the user's current location. The server determines the location using a database (e.g., MySQL, PostgreSQL).

[1088] Step 5:

[1089] The server generates map information in real time and sends it to the terminal. The input is the user's current location and map information, and the output is the generated map information. The server generates map information instantly and provides it to the terminal.

[1090] Step 6:

[1091] The terminal displays the received map information and location information on the screen. The input is the received map information and current location information, and the output is the map and current location displayed to the user. The terminal provides map information visually to the user.

[1092] Step 7:

[1093] The user asks a question by voice input. The input is a voice command such as "Please tell me a review of the cafe up ahead," and the output is the results of voice analysis. The user's voice information is analyzed by the device.

[1094] Step 8:

[1095] The device sends the analyzed voice input to the server and requests information based on the question. The input is the voice analysis result, and the output is a request to send to the server. The device converts the voice data into text and sends it to the server.

[1096] Step 9:

[1097] The server searches for relevant information based on the voice input and location information and sends it to the device. The input is the voice analysis results and location information, and the output is relevant information (e.g., cafe reviews). The server searches a database and provides the appropriate information.

[1098] Step 10:

[1099] The device provides the received information to the user via voice and visual display. The input is relevant information from the server, and the output is a voice response and visual display to the user. The device generates the voice response using Google Cloud Text-to-Speech.

[1100] Step 11:

[1101] The user receives audio and visual feedback from the device and acts based on the navigation and information. The input is audio and visual feedback, and the output is the user's actions. The user decides on their next action based on the displayed information.

[1102] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1103] In an embodiment of the present invention, a system is constructed to identify a user's location, particularly in an indoor environment where GPS is not available, and provide navigation and store information. Furthermore, the present invention combines an emotion engine that recognizes the user's emotions and optimizes the information provided based on the emotions.

[1104] The system starts when a user uses a device such as a smartphone or tablet to capture indoor scenes with a camera. The device then launches an appropriate application and begins capturing video images in real time. The captured video image data is then sent from the device to a server. The server is equipped with a high-performance image analysis engine that can analyze the video image data and identify the user's current location.

[1105] By analyzing the camera footage, the server extracts indoor features (e.g., walls, doors, signs, etc.) and compares them with existing map information to pinpoint the user's exact location. Based on this information, the server generates map information in real time and sends it to the device as feedback to the user.

[1106] The device displays the received map information, the user's current location, and the destination. When the user selects a route to the destination, the device visualizes the route information and begins navigation. In addition, the server provides dynamic information such as store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage.

[1107] For example, if a user is in a large shopping mall, the server analyzes information about each store in the mall through camera footage and sends that information to the device. When the user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays that information on the screen and provides visual and audio feedback to the user.

[1108] Once the user's location is identified, the server calculates the optimal route to the user's destination and sends the route information to the device. The device then provides appropriate guidance to help the user move along the displayed route. This guidance can also provide multiple route options (e.g., the shortest route, a route that avoids congestion, etc.).

[1109] Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions, and by analyzing the user's facial expressions and tone of voice, it grasps their emotions in real time. For example, if the user looks impatient, the system will determine that they are in a hurry and will prioritize displaying the shortest route. Also, if the user is relaxing, it will provide information on rest spots and cafes where they can relax.

[1110] The navigation display and audio guidance provided can also be adjusted based on the user's emotions. For example, if the user is stressed, the audio guidance can be reassuring and avoid complex instructions. Conversely, if the user is excited, the audio guidance can be more entertaining, providing detailed information and more options.

[1111] By combining these functions, the system of the present invention can accurately locate the user's location even in indoor environments where GPS is unavailable, providing real-time navigation and interactive information, and can also provide personalized services based on the user's emotions, providing a more convenient and user-friendly experience.

[1112] The processing flow will be explained below.

[1113] Step 1:

[1114] The user launches the smartphone camera app, which selects a dedicated navigation mode and the camera captures real-time images of the interior of the room.

[1115] Step 2:

[1116] The device processes the acquired video data in real time and sends it to the server. The data is divided at a fixed frame rate and sent as network packets.

[1117] Step 3:

[1118] The server analyzes the received video data frame by frame, and uses image analysis algorithms to extract feature points (e.g., walls, doors, signs) from the video.

[1119] Step 4:

[1120] The server identifies the user's current location based on feature points in the video, analyzes the relative positions of the feature points, and compares them with existing map data to determine the user's exact location.

[1121] Step 5:

[1122] The server transmits real-time generated map information and the user's current location information to the terminal, and the server also transmits surrounding map data related to the user's location.

[1123] Step 6:

[1124] The device displays the received map information and the user's current location. The user selects a destination on the app's interface, and the current location and route to the destination are displayed on the screen.

[1125] Step 7:

[1126] The device calculates and displays the optimal route based on the user's input, and the app provides users with multiple route options, such as the shortest route or a route that avoids congestion.

[1127] Step 8:

[1128] The user enters store information or other questions by text or voice, such as "Where is the nearest cafe?" or "Can you give me a review of this cafe?"

[1129] Step 9:

[1130] The server receives a question from the user, analyzes the question using AI, and uses natural language processing (NLP) to retrieve relevant information from a database.

[1131] Step 10:

[1132] The server sends the acquired information to the device, including details such as store location, opening hours, and reviews.

[1133] Step 11:

[1134] The device displays the received information to the user, and the app provides information to the user through text, images, and in some cases audio.

[1135] Step 12:

[1136] The user follows the navigation display in the app and begins moving towards their destination. The device updates their current location and provides real-time directions.

[1137] Step 13:

[1138] The device periodically sends video images to the server, which then analyzes them, providing accurate navigation from the user's current location until they reach their destination.

[1139] Step 14:

[1140] The device analyzes the camera footage and the user's voice and uses an emotion engine to identify the user's emotional state. For example, it recognizes the user's current emotion by analyzing the user's facial expressions and tone of voice.

[1141] Step 15:

[1142] The server adjusts the information provided and navigation methods based on the emotion data sent from the emotion engine. For example, if it determines that the user is in a hurry, it will prioritize suggesting the shortest route.

[1143] Step 16:

[1144] The device will provide users with information and navigation based on their emotions. For example, if the user is feeling relaxed, it will provide information on rest spots and cafes.

[1145] Step 17:

[1146] When the user again inputs a question about store information or navigation, the server takes the emotional data into consideration and provides the most appropriate information.

[1147] Step 18:

[1148] The server and device work together to continuously monitor the user's current location and emotional state, thereby constantly providing the user with the most appropriate information and navigation.

[1149] Example 2

[1150] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1151] In indoor environments, GPS is unavailable, making it difficult to pinpoint the user's exact location, and providing navigation and store information is also difficult.In addition, the inability to provide information that takes into account the user's emotions results in an inadequate user experience.

[1152] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring a moving image, a means for transmitting the moving image to the server, a means for identifying a location from the moving image, a means for generating map information in real time, a means for interactively providing information based on the location information and a question, and a means for recognizing a user's emotion and optimizing information. This makes it possible to identify the user's exact location even in an indoor environment, and to provide real-time navigation and personalized information.

[1153] The "means for acquiring moving images" is a function that allows a user to capture images of the interior of a room using the camera on a smartphone or tablet.

[1154] The "means for transmitting moving images to a server" is a communication function for transferring moving image data acquired by a terminal to a server in real time.

[1155] The "means for identifying a location from a video image" is a function that analyzes video image data using an image analysis engine installed in the server and identifies the user's current location.

[1156] "Means for generating map information in real time" refers to a function in which the server instantly creates the latest map information based on the user's location information and provides it to the user.

[1157] The "means for interactively providing information based on location information and questions" is a function that provides appropriate information in real time based on the user's current location and questions.

[1158] "Means for recognizing the user's emotions and optimizing information" refers to a function in which the server analyzes the user's facial expressions and tone of voice to understand their emotions and provide optimal information based on those emotions.

[1159] The "means for analyzing camera images and acquiring surrounding information" is a function for analyzing images acquired through a camera and identifying information about the environment surrounding the user.

[1160] The "means for providing the optimal route based on the user's current location" is a function that calculates and displays the most efficient route to the destination set by the user.

[1161] The present invention builds a system that identifies a user's location and provides navigation and store information, especially in indoor environments where GPS is not available. Furthermore, the present invention combines an emotion engine that recognizes the user's emotions and optimizes the information provided based on those emotions.

[1162] Hardware and software used

[1163] The main hardware of this system is the user's smartphone or tablet, which must be equipped with a high-performance camera. As for software, a dedicated application is installed on the device, which is responsible for capturing and transmitting video images.

[1164] The server is equipped with a high-performance image analysis engine (e.g., OpenCV, TensorFlow) that analyzes video data in real time. The server also has an emotion engine that recognizes the user's emotions and has the ability to analyze the user's facial expressions and tone of voice.

[1165] Specific processing and operations

[1166] The system starts with the user using a smartphone or tablet to take pictures of the indoor environment with a camera. The user then launches a dedicated application and starts capturing video images in real time. At this time, the device sends the video image data to a server via Wi-Fi or a mobile network.

[1167] The server analyzes the received video data and extracts indoor features (e.g., walls, doors, signs). It then compares this with existing map information to determine the user's current location. The server generates map information in real time based on this information and sends it to the device as feedback.

[1168] The device displays the received map information and the user's current location, as well as calculates and displays the optimal route to the user's selected destination. When the user selects "I want to go to the Apple Store," the device displays the route and guides the user along the route.

[1169] Furthermore, the server provides dynamic store information and congestion status in real time based on the user's location information and surrounding information obtained from camera footage. For example, when a user asks, "What are the reviews for this cafe?", the server retrieves the latest review information for that cafe and sends it to the device. The device then displays the information on its screen and provides visual and audio feedback to the user.

[1170] The emotion engine allows the server to analyze the user's facial expressions and tone of voice to understand their emotions in real time. For example, if the user looks anxious, the system will determine that they are in a hurry and prioritize the shortest route. Conversely, if the user is relaxed, it will provide information on rest spots and cafes where they can relax.

[1171] Examples of specific examples and prompts

[1172] When a user is in a large shopping mall, the information of each store is analyzed through camera footage, and the server sends that information to the device. For example, if a user selects "I want to go to the Apple Store," the device will display the shortest route. Similarly, if the user asks "What are the reviews of this cafe?", the device will display the latest reviews.

[1173] An example prompt is:

[1174] "Please give us a review of the cafe."

[1175] "What's the best route to get to the nearest restroom?"

[1176] "How do I get to the cinema?"

[1177] This allows users to easily locate and navigate indoors, as well as receive personalized information based on their emotions.

[1178] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1179] Step 1:

[1180] A user launches a dedicated application on a smartphone or tablet and activates the camera. This application is designed to allow users to take pictures of indoor scenes. When the user presses the capture button, the device begins capturing video images in real time.

[1181] Input: User initiates camera and takes a photo

[1182] Output: Real-time video data

[1183] Step 2:

[1184] The device transmits the acquired video data to a server via Wi-Fi or mobile networks in real time, minimizing time lag.

[1185] Input: Real-time video data

[1186] Output: Send video data to the server

[1187] Step 3:

[1188] The server uses a high-performance image analysis engine (e.g., OpenCV, TensorFlow) to analyze the received video data. The server detects indoor features (e.g., walls, doors, signs) and compares them with existing map information.

[1189] Input: Video data

[1190] Output: Current location data of the user

[1191] Step 4:

[1192] The server generates real-time map information based on the extracted feature points. This map information reflects the user's current location. The server then sends the generated map information to the device.

[1193] Input: current location data of the user

[1194] Output: Generate real-time map information and send it to the device

[1195] Step 5:

[1196] The device displays the received map information to the user. The map shows the user's current location and information about nearby stores. When the user selects a destination (e.g., an Apple Store), the device sends that information to the server.

[1197] Input: Real-time map information

[1198] Output: Map display and destination selection information sent to server

[1199] Step 6:

[1200] The server calculates the optimal route based on the user's current location and selected destination, and can provide multiple route options (e.g., shortest route, congestion avoidance route, etc.).

[1201] Input: User's current location data and destination selection information

[1202] Output: Calculate optimal route information and send it to the device

[1203] Step 7:

[1204] The terminal displays the optimal route information received from the server to the user, providing interactive navigation. The user follows the presented route and is guided to the destination.

[1205] Input: Optimal route information

[1206] Output: Display route information and provide navigation

[1207] Step 8:

[1208] The server provides real-time store information and congestion status based on the user's current location and surrounding information obtained from camera footage. For example, when a user asks, "What are the reviews for this cafe?", the server retrieves the latest review information and sends it to the device.

[1209] Input: Local information, user questions

[1210] Output: Dynamic store information and review information sent to the device

[1211] Step 9:

[1212] The server uses an emotion engine to analyze the user's facial expressions and tone of voice, and based on the analysis results, it provides the shortest route if the user is in a hurry, or information on rest spots if the user is relaxing.

[1213] Input: Emotion recognition data (facial expressions, voice tone)

[1214] Output: Optimized information delivery

[1215] In this way, users can receive accurate location and real-time navigation even in indoor environments, as well as personalized information based on their emotions.

[1216] (Application example 2)

[1217] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1218] The present invention aims to provide a system that accurately identifies a user's location and provides navigation and information, especially in indoor environments where GPS is unavailable, and also aims to solve the problem of improving the user experience by recognizing the user's emotions and optimizing the information provided based on those emotions.

[1219] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1220] In this invention, the server includes a means for acquiring video images, a means for identifying a location from the video images, a means for generating map information in real time, a means for interactively providing information based on the location information and questions, and a means for recognizing a user's emotions and optimizing the information. This enables highly accurate location identification and navigation even in indoor environments that do not support GPS, and further enables personalized information provision according to the user's emotions.

[1221] "Means for acquiring video images" refers to the function that allows users to capture video in real time using the camera on their smartphone or tablet.

[1222] "Means for identifying a position from a video image" refers to a function that analyzes the acquired video image and performs processing to identify the user's current position from characteristic points (walls, doors, signs, etc.).

[1223] "Means for generating map information in real time" refers to the function of the server generating and providing the latest map that reflects the user's location information based on the analysis results.

[1224] "Means for interactively providing information based on location information and questions" refers to a function that provides relevant information (store information, route information, etc.) in real time based on the user's current location and inquiries from the user.

[1225] "Means for recognizing the user's emotions and optimizing information" refers to the function of analyzing the user's emotions from their facial expressions and voice, and adjusting the information and navigation content provided based on those emotions.

[1226] This invention is a system that accurately identifies a user's location and provides navigation and information, especially in indoor environments where GPS is not available. It is also capable of recognizing the user's emotions and optimizing the information provided based on those emotions.

[1227] The main components of the system include a means for acquiring video images, a means for identifying a location from the video images, a means for generating map information in real time, a means for providing information interactively based on location information and questions, and a means for recognizing the user's emotions and optimizing the information.

[1228] Hardware and Software Configuration

[1229] 1. Acquiring video images

[1230] Users can capture indoor video images using mobile devices such as smartphones and tablets, using the device's built-in camera and image processing libraries such as OpenCV.

[1231] 2. Video data sent to the server

[1232] The acquired video data is sent from the device to the server. This sending process uses Requests, an HTTP request library.

[1233] 3. Location Identification and Map Generation

[1234] The server analyzes the video images sent and extracts indoor features (walls, doors, signs, etc.). Based on this, it locates the user's location by comparing it with existing map information. This analysis is performed using an image analysis engine.

[1235] The server generates the latest map information in real time and sends it to the terminal.

[1236] 4. Navigation and Information

[1237] The device displays the received map information and the user's current location, and the user can select a route to their destination. The route information is visualized, and the device guides the user to their destination. This guidance can provide multiple options, such as the shortest route or a route that avoids congestion.

[1238] 5. Emotion Recognition and Information Optimization

[1239] When a user provides video via a camera, the server analyzes the user's facial expressions and voice in real time using an emotion recognition engine.

[1240] Based on the recognized emotion, the information provided and navigation content are optimized. For example, if the user is in a hurry, the shortest route will be prioritized, and if the user is relaxed, information about rest spots will be provided.

[1241] Examples of specific examples and prompts

[1242] This system is extremely effective in large shopping malls and department stores, for example. When a user uses the system while walking around the mall, they can obtain real-time information on congestion and store reviews, along with navigation to their destination. Furthermore, the information provided is optimized according to the user's emotions, providing a more comfortable shopping experience.

[1243] Example prompt sentence:

[1244] "Your goal is to develop an application that identifies the user's location in a large shopping mall, analyzes camera footage to recognize emotions, and provides optimal store navigation based on those emotions."

[1245] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1246] Step 1:

[1247] A user captures indoor video using the camera on their smartphone or tablet. When the user launches the app and the camera acquires video, the device captures video in real time. The input of this step is the video captured by the camera, and the output is the captured video data.

[1248] Step 2:

[1249] The device sends the captured video data to the server. The device encodes the video data into a format such as JPEG and sends it to the server using an HTTP request library (e.g., Requests). The input to this step is the video data captured in step 1, and the output is the data sent to the server.

[1250] Step 3:

[1251] The server analyzes the received video data using its image analysis engine. The server decodes the transmitted data and extracts feature points (e.g., walls, doors, signs, etc.). The input of this step is the video data transmitted in step 2, and the output is the extracted feature point data.

[1252] Step 4:

[1253] The server performs location identification. The server compares the extracted feature points with existing map information to identify the user's current location. The input to this step is the feature point data and map information obtained in step 3, and the output is the identified user's location information.

[1254] Step 5:

[1255] The server generates map information in real time and sends it to the device. The server generates the latest map information based on the user's location information and sends it to the device. The input of this step is the identified location information, and the output is the generated map information.

[1256] Step 6:

[1257] The device displays the received map information and the user's current location. The device visually displays the received map information to the user, allowing the user to select a route to their destination. The input of this step is the map information received in step 5, and the output is the displayed map information and the user's selected route information.

[1258] Step 7:

[1259] The device navigates the user to the destination. The device provides multiple route options, such as the shortest route or a route that avoids congestion, and navigates along the route selected by the user. The input of this step is the route information selected by the user, and the output is the navigation information provided.

[1260] Step 8:

[1261] The server recognizes the user's emotions and optimizes the information. When the user provides camera footage, the server analyzes the user's facial expressions and voice in real time to recognize emotions. An emotion recognition engine is used for this analysis. The input for this step is camera footage and voice data, and the output is recognized emotional information.

[1262] Step 9:

[1263] The server optimizes information based on the recognized emotion. The navigation content and information provided are adjusted according to the recognized emotion. For example, if the user is in a hurry, the shortest route is prioritized, and if the user is relaxed, information about rest spots is provided. The input of this step is the recognized emotion information, and the output is optimized information.

[1264] Step 10:

[1265] The terminal displays the optimized information and provides it to the user. The terminal visually and audibly provides the optimized information received from the server to the user. The input of this step is the optimized information, and the output is the provided visual and audio feedback.

[1266] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1267] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1268] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1269] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1270] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1271] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1272] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1273] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1274] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1275] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1276] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1277] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1278] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1279] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1280] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1281] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1282] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1283] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1284] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1285] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1286] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1287] The following is further disclosed regarding the above embodiment.

[1288] (Claim 1)

[1289] A means for acquiring a moving image;

[1290] A means for identifying a position from a video image;

[1291] a means for generating map information in real time;

[1292] A means of providing interactive information based on location and questions;

[1293] A system including:

[1294] (Claim 2)

[1295] 10. The system of claim 1, further comprising means for analyzing camera footage to obtain information about the surroundings.

[1296] (Claim 3)

[1297] 10. The system of claim 1, further comprising means for providing an optimal route based on a current location of the user.

[1298] "Example 1"

[1299] (Claim 1)

[1300] A means for acquiring a moving image;

[1301] means for transmitting video data in real time;

[1302] means for analyzing video data and extracting feature points;

[1303] a means for identifying a location by matching the feature points with map information;

[1304] a means for generating map information in real time;

[1305] A means of providing interactive information based on location and questions;

[1306] A system including:

[1307] (Claim 2)

[1308] 10. The system of claim 1, further comprising means for analyzing the camera image and extracting surrounding feature points.

[1309] (Claim 3)

[1310] 10. The system of claim 1, further comprising means for calculating and providing an optimal route based on a current location of the user.

[1311] "Application Example 1"

[1312] (Claim 1)

[1313] A means for acquiring a moving image;

[1314] A means for identifying a position from a video image;

[1315] a means for generating map information in real time;

[1316] means for interactively providing information based on location information and voice input;

[1317] A system including:

[1318] (Claim 2)

[1319] 10. The system of claim 1, further comprising means for analyzing the camera image and obtaining surrounding information and store information.

[1320] (Claim 3)

[1321] 10. The system of claim 1, further comprising means for providing an optimal route audibly and visually based on the user's current location.

[1322] "Example 2: Combining Emotion Engines"

[1323] (Claim 1)

[1324] A means for acquiring a moving image;

[1325] means for transmitting the video to a server;

[1326] A means for identifying a position from a video image;

[1327] a means for generating map information in real time;

[1328] A means of providing interactive information based on location and questions;

[1329] A means for recognizing user emotions and optimizing information;

[1330] A system including:

[1331] (Claim 2)

[1332] 10. The system of claim 1, further comprising means for analyzing camera footage to obtain information about the surroundings.

[1333] (Claim 3)

[1334] 10. The system of claim 1, further comprising means for providing an optimal route based on a current location of the user.

[1335] "Application example 2 when combining emotion engines"

[1336] (Claim 1)

[1337] A means for acquiring a moving image;

[1338] A means for identifying a position from a video image;

[1339] a means for generating map information in real time;

[1340] A means of providing interactive information based on location and questions;

[1341] A means for recognizing user emotions and optimizing information;

[1342] A system including:

[1343] (Claim 2)

[1344] 10. The system of claim 1, further comprising means for analyzing camera footage to obtain information about the surroundings.

[1345] (Claim 3)

[1346] 10. The system of claim 1, further comprising means for providing an optimal route based on a current location of the user. [Explanation of symbols]

[1347] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring a moving image; A means for identifying a position from a video image; a means for generating map information in real time; A means of providing interactive information based on location and questions; A system including:

2. The system of claim 1 , further comprising means for analyzing camera footage to obtain information about the surroundings.

3. The system of claim 1 further comprising means for providing an optimal route based on a user's current location.

Images