System

Abstract

A system is provided.SOLUTION: A system including means for acquiring a video entering a field of view of a user by a camera attached to a viewing device worn by the user, means for preprocessing the acquired video data, means for performing image analysis using an artificial intelligence model for identifying a facility on the basis of the preprocessed video data, means for requesting facility information acquired on the basis of the image analysis from a server, and means for displaying detailed information of the facility returned from the server on a display of the viewing device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern urban life, many facilities are densely packed together, making it difficult for users to quickly and accurately grasp information about facilities around them. Furthermore, conventional smart devices require a great deal of effort to provide facility information, which can detract from the user experience. In particular, there is a lack of systems that can acquire and display facility information in real time on-site. To address these issues, a system is needed that can instantly recognize facility information within the user's field of vision and provide detailed information in real time. [Means for solving the problem]

[0005] The present invention provides a system including: a means for acquiring video images of a user's field of view using a camera attached to a visual device worn by a user; a means for preprocessing the acquired video data; a means for performing image analysis using an artificial intelligence model to identify facilities based on the preprocessed video data; a means for requesting facility information acquired based on the image analysis from a server; and a means for displaying detailed facility information returned from the server on a display of the visual device. This allows a user to instantly acquire information about facilities within their field of view and check the detailed information in real time. The preprocessing includes noise removal and resolution adjustment, and the artificial intelligence model can identify facility categories and specific facility names. This improves the user experience and enables facility information to be acquired and displayed quickly and efficiently.

[0006] "User" refers to the person who wears and uses the system.

[0007] "Visual devices" refer to devices worn by users that capture and display visual information from their surroundings, such as smart glasses.

[0008] "Camera" refers to an image capturing device built into or attached to a visual device.

[0009] "Video data" refers to digital data of images or videos captured by a camera.

[0010] "Preprocessing" refers to the process of removing noise and adjusting resolution on acquired video data to make it easier to analyze.

[0011] "Artificial intelligence model" refers to an algorithm that uses machine learning techniques to identify specific objects from video data.

[0012] "Image analysis" refers to the process of analyzing pre-processed video data using artificial intelligence models to extract specific information.

[0013] "Facility information" refers to data relating to a specific facility, such as information including the facility name and category.

[0014] A "server" refers to a remote computer system that provides data upon request over a network.

[0015] "Detailed information" refers to additional information such as the business hours, reputation, and special offers of the target facility.

[0016] "Display" refers to a video display device built into a visual device.

[0017] "Display" refers to the visual output of information on a display. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention aims to enable a user wearing a visual device to acquire and display information about surrounding facilities in real time. The specific processing content of the program and specific examples are described below.

[0040] First, the user wears the vision device. The camera in the vision device captures images of what is in the user's field of view in real time. This image is temporarily stored in the vision device.

[0041] The device then begins preprocessing the video data, which includes noise reduction and resolution adjustment, to improve the clarity of the image and subsequent analysis.

[0042] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[0043] Once the device identifies the facility information, it sends a request to the server based on that information. The request includes identification information such as the facility name and category.

[0044] The server searches the database based on the received request and retrieves detailed information about the facility, including opening hours, ratings, special offers, etc. The server then sends this information back to the terminal as a response.

[0045] The terminal analyzes the information received from the server and displays it in a format that can be understood by the user through the visual device. Specifically, detailed facility information is overlaid on the visual device's display, allowing the user to grasp detailed facility information in real time on-site.

[0046] Specific examples are given below.

[0047] Imagine a user is walking down the street and a cafe comes into view. At this time, the camera in the vision device captures video of the cafe. The device preprocesses the video and inputs it into the AI ​​model. The AI ​​model identifies the name of the cafe as "Café Delight." The device sends the identified information to the server, which retrieves "Café Delight's" business hours (8:00-20:00), rating (4.5 / 5), and special offers (10% off coupon available) from its database and sends them back to the device. Finally, the device displays this information on the vision device's display.

[0048] In this way, the present invention provides a system that allows a user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0049] The above is a specific description of the embodiment for carrying out the present invention.

[0050] The processing flow will be explained below.

[0051] Step 1:

[0052] The user wears the vision device. The camera in the vision device captures images of the user's field of view in real time. The image data is temporarily stored in the vision device.

[0053] Step 2:

[0054] The device begins pre-processing the captured video data, performing processes such as noise removal and resolution adjustment to make it easier for AI to analyze. This makes the video clearer and highlights the facility's unique features.

[0055] Step 3:

[0056] After preprocessing, the device inputs the video data into an artificial intelligence model. The AI ​​model analyzes the image and identifies the facility based on features such as the facility's exterior, sign, and logo contained in the video. For example, if a sign reads "Café Delight," it will perform character recognition.

[0057] Step 4:

[0058] The terminal sends a request to the server based on the identified facility information. The request includes information such as the name and category of the identified facility.

[0059] Step 5:

[0060] The server analyzes the received request and retrieves detailed information about the facility from the database, including opening hours, ratings, special offers, etc. The retrieved information is then sent back to the terminal as a response.

[0061] Step 6:

[0062] The device analyzes the detailed facility information received from the server and displays it on the visual device's display. Specifically, the facility's name, opening hours, ratings, and special offer information are overlaid in the user's field of vision.

[0063] Step 7:

[0064] The user can check the detailed information of the facility displayed on the visual device and take the necessary action. For example, when information about the cafe "Café Delight" is displayed, the user can check the opening hours and special offers and decide whether to visit.

[0065] The above is a detailed description of the specific processing steps of the program and the operations at each step.

[0066] Example 1

[0067] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0068] Conventional technologies lack a means for users to quickly and easily obtain detailed information about surrounding facilities in real time. This requires users to perform manual searches, which is not only inconvenient but also inefficient. Furthermore, conventional approaches are difficult to implement in situations where instantaneous on-site information is required. To address these issues, the present invention aims to provide a system that allows users to obtain facility information in real time using a visual device and instantly display detailed information.

[0069] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0070] In this invention, the server includes means for acquiring images of the user's field of view using a camera attached to a visual device worn by the user, means for preprocessing the acquired image data, means for performing image analysis using an artificial intelligence model for identifying facilities based on the preprocessed image data, means for requesting facility information acquired based on the image analysis from the server, and means for overlaying the detailed facility information returned from the server on the display of the visual device, thereby enabling the user to quickly acquire detailed information about surrounding facilities in real time through the visual device and efficiently utilize it.

[0071] A "user" is a person who utilizes the system of the present invention by wearing a visual device.

[0072] A "visual device" is a device worn by a user that includes a display and camera that displays to the field of view.

[0073] A "camera" is a photographing device attached to a visual device for capturing images within the user's field of vision.

[0074] "Video data" refers to digital data of images and videos captured by a camera.

[0075] "Preprocessing" refers to the step of performing processes such as noise removal and resolution adjustment on the video data.

[0076] An "artificial intelligence model" is an AI algorithm that uses pattern recognition technology to identify facilities contained within video footage.

[0077] "Image analysis" is the process of using artificial intelligence models to identify facility information from video data.

[0078] "Facility information" refers to data such as the name, category, and detailed information of the identified facility.

[0079] A "server" is a computer system that searches a database based on a request, obtains detailed facility information, and returns a response.

[0080] A "database" is an information aggregation system owned by a server that stores detailed facility information.

[0081] "Overlay display" refers to a function that displays information superimposed on the image displayed on the display of a visual device.

[0082] The present invention is a system that acquires information about surrounding facilities in real time while the user is wearing a visual device, and displays the details on the visual device. The processing contents of the program will be specifically described below.

[0083] First, the user wears a visual device. The device is equipped with a camera that captures images of the user's field of vision in real time. This image data is temporarily stored in the device.

[0084] Next, the device begins preprocessing the video data. This includes noise reduction and resolution adjustment. This makes the video clearer and improves the accuracy of subsequent analysis. This preprocessing typically uses an image processing library such as OpenCV.

[0085] Once preprocessed, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo. Frameworks such as TensorFlow and PyTorch are used for the AI ​​model.

[0086] Once the device identifies the facility information, it sends a request to the server based on that information. The request includes identifying information such as the facility's name and category. The server searches the database based on the received request and retrieves detailed information about the facility. This detailed information includes opening hours, ratings, special offers, and more. The server uses cloud services such as AWS and Google Cloud, and databases such as MySQL and PostgreSQL.

[0087] The acquired detailed facility information is sent back to the terminal as a response from the server. The terminal analyzes the information received from the server and displays it in a format that the user can understand through the visual device. Specifically, the detailed facility information is overlaid on the display of the visual device. This allows the user to grasp detailed facility information in real time on-site.

[0088] Here's a specific example: When a user is walking down the street and a cafe comes into view, the camera in the vision device captures video of the cafe. The device preprocesses the video and inputs it into an AI model. The AI ​​model identifies the name of the cafe as "Cafe Delight." The device sends the identified information to a server, which retrieves Cafe Delight's opening hours (8:00-20:00), rating (4.5 / 5), and special offers (10% off coupon available) from a database and sends them back to the device. Finally, the device displays this information on the vision device's display.

[0089] An example prompt is, "A user wears a visual device, and a camera captures video of the surroundings in real time. The video is preprocessed, and an AI model identifies the facility. Based on this facility information, a server provides detailed information, which is displayed on the visual device. As a specific example, please explain how detailed information about the cafe "Cafe Delight" is obtained and displayed to the user."

[0090] The above is a specific description of the embodiment of the present invention. With this system, a user can obtain information about surrounding facilities in real time and efficiently check detailed information.

[0091] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0092] Step 1:

[0093] The user wears a visual device, and a camera attached to the device captures images of what is in the user's field of view in real time.

[0094] Input: The image that the user sees.

[0095] Specific operation: The user wears the visual device on their head, and the camera continuously captures objects within their field of view.

[0096] Output: Captured video data.

[0097] Step 2:

[0098] The device pre-processes the captured video data, removing noise and adjusting the resolution to make the video clearer.

[0099] Input: The captured video data that is the output of step 1.

[0100] Data processing: Use an image processing library such as OpenCV to remove noise from the video data and improve its resolution.

[0101] Specific operation: The device filters the video data to reduce noise and adjust the image resolution appropriately.

[0102] Output: Pre-processed video data.

[0103] Step 3:

[0104] The device performs image analysis using artificial intelligence models to identify facilities based on pre-processed video data.

[0105] Input: The preprocessed video data that is the output of step 2.

[0106] Data Computing: Using AI frameworks such as TensorFlow and PyTorch, facilities are identified from video data.

[0107] Specific operation: Preprocessed video is input into the AI ​​model, and the facility's exterior, signs, logos, etc. are analyzed using a pattern recognition algorithm.

[0108] Output: Data with facility names and categories identified.

[0109] Step 4:

[0110] The device sends a request to the server based on the facility information it has identified, including identification information such as the facility name and category.

[0111] Input: Identified facility information, which is the output of Step 3.

[0112] Data processing: The facility identification information is converted into a data format for sending to the server as an HTTP request.

[0113] Specific operation: The terminal generates a request message containing facility information such as "Cafe Delight" and sends it to the server.

[0114] Output: The request data sent.

[0115] Step 5:

[0116] The server searches the database based on the received request and obtains detailed information about the facility.

[0117] Input: The request data sent in step 4.

[0118] Data Computation: Querying database systems such as MySQL or PostgreSQL to retrieve the requested facility details.

[0119] What happens: The server queries the database to get information about Cafe Delight, such as its hours of operation, ratings, and special offers.

[0120] Output: Detailed information about the retrieved facilities.

[0121] Step 6:

[0122] The terminal receives the detailed facility information returned from the server and displays it as an overlay on the display of the visual device.

[0123] Input: The retrieved facility details, which is the output of step 5.

[0124] Data processing: Preparing information in a format suitable for display on visual devices.

[0125] Specific operation: Detailed facility information (e.g., business hours 8:00-20:00, rating 4.5 / 5, 10% off coupon available) is sent to the user's visual device and overlaid on the screen.

[0126] Output: Facility details displayed on a visual aid.

[0127] The above are the specific processing steps of the program of this system, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0128] (Application example 1)

[0129] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] In modern society, users need to quickly and easily obtain information about nearby facilities while on the move. However, conventional methods require searching on a smartphone to obtain facility information, which is time-consuming and lacks real-time information. Furthermore, integrating information from different sources increases the burden on users, making it difficult to obtain information efficiently. Therefore, there is a need for an efficient system that allows users to instantly obtain detailed facility information on the spot using a visual device.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0132] In this invention, the server includes means for acquiring video within the user's field of view using a camera attached to a visual device worn by the user, means for preprocessing the acquired video data, means for performing image analysis using a generative AI model for identifying facilities based on the preprocessed video data, means for requesting facility information acquired based on the image analysis from the server via a communication device, and means for overlaying and displaying detailed facility information returned from the server on the display device of the visual device, thereby enabling the user to acquire information about surrounding facilities in real time and efficiently check the detailed information.

[0133] A "visual device" is a device that can display visual information on a display when worn by a user.

[0134] A "camera" is a device that is attached to a visual device and has the function of capturing images that come into the user's field of vision.

[0135] "Preprocessing" refers to processing such as removing noise from acquired video data and adjusting resolution to improve the accuracy of subsequent analysis.

[0136] "Generative AI model" means an artificial intelligence model used to analyze video data and identify the category and specific name of a facility.

[0137] A "communication device" is a device for transmitting identified facility information to a server and receiving a response from the server.

[0138] The "display device" is a display that is mounted on the visual device and has the function of overlaying detailed facility information to the user.

[0139] A "prompt sentence" is an instruction sentence used by the generative AI model to identify a facility.

[0140] The present invention provides a system that acquires and displays surrounding facility information in real time using a visual device worn by a user. This system is composed of an image capture device, a preprocessing module, a generative AI model, a communication device, and a display device mounted on the visual device.

[0141] The vision device is equipped with a camera for capturing images in real time. The camera captures the image of the user's field of view and sends it to a pre-processing module, where processing such as noise reduction and resolution adjustment of the image data is performed. This pre-processing improves the accuracy of subsequent image analysis.

[0142] The preprocessed video data is then input into a generative AI model, which uses pattern recognition technology to identify facilities within the video. Specifically, the model analyzes the facility's exterior, signage, logo, and other features to identify the facility's category and specific name. A prompt is then generated to assist the AI ​​model in the identification process.

[0143] The communication device is responsible for transmitting the identified facility information to the server. The server receives this information and searches a database to obtain detailed information about the facility, including opening hours, ratings, special offers, etc. The server then sends these details back to the visual device via the communication device.

[0144] Finally, the display device displays the information returned from the server to the user. Specifically, detailed facility information is overlaid on the display of the visual device, allowing the user to check detailed information about surrounding facilities in real time through the visual device.

[0145] As a specific example, consider the flow from when a user is walking down the street and a cafe comes into view, to when a camera captures video of the cafe. The preprocessing module preprocesses this video and inputs it into the generative AI model. The generative AI model identifies the cafe, and the communication device sends this information to the server. The server receives the returned detailed information about the cafe (opening hours, ratings, special offers), which it then overlays on the display of the visual device.

[0146] The following are examples of prompt statements:

[0147] User looks at "Cafe" and gets real-time information overlay showing operation hours (8:00-20:00), rating (4.5 / 5), and a special coupon (10% off).

[0148] The above is a specific description of the embodiment of the present invention, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0149] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0150] Step 1:

[0151] When a user walks around town, the camera of the vision device captures the image that comes into the user's field of view in real time. The input is the user's field of view, and the output is the captured image data.

[0152] Step 2:

[0153] The captured video data is processed by the pre-processing module. Specifically, noise removal and resolution adjustment are performed. The input is the captured video data, and the output is pre-processed video data that has been subjected to noise removal and resolution adjustment.

[0154] Step 3:

[0155] The preprocessed video data is input into the generative AI model to identify the facility. Specifically, the model analyzes the facility's exterior, signage, logo, and other features, and identifies the facility's category and specific name using prompt text. The input is the preprocessed video data, and the output is the facility's identification information (facility name, category).

[0156] Step 4:

[0157] The identified facility information is sent to the server by the communication device. Specifically, the facility name and category information are constructed as an HTTP request and sent to the server. The input at this time is the facility identification information, and the output is a request to the server.

[0158] Step 5:

[0159] The server searches the database based on the received request and retrieves detailed information about the facility (such as opening hours, ratings, special offers, etc.) The input is the request to the server, and the output is the detailed information about the facility.

[0160] Step 6:

[0161] The server returns the acquired facility details to the visual device via the communication device. Specifically, the facility details are constructed as an HTTP response and sent to the visual device. The input at this time is the facility details, and the output is the response from the server.

[0162] Step 7:

[0163] The display device of the visual device overlays the detailed facility information received from the server. Specifically, the detailed facility information is displayed on the display in an appropriate position according to the user's field of view. The input at this time is the response from the server, and the output is the detailed facility information overlaid on the user's field of view.

[0164] The above is a description of the specific processing steps in the system for implementing the present invention, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0165] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0166] The present invention is a system that allows a user to wear a visual device and acquire and display information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. The specific processing content of the program and concrete examples are explained below.

[0167] The user wears the vision device. The camera in the vision device captures images of what is in the user's field of view in real time. This image is temporarily stored in the vision device.

[0168] The device then begins preprocessing the video data, which includes noise reduction and resolution adjustment, to improve the clarity of the image and subsequent analysis.

[0169] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[0170] The visual device is equipped with an emotion engine to recognize the user's emotions. The emotion engine recognizes the user's emotional state by analyzing the user's voice, facial expressions, and biometric information (heart rate, skin galvanic response, etc.). For example, it can detect when the user is feeling stressed or curious.

[0171] The device sends a request to the server based on the facility information and emotion information. The request includes the name and category of the identified facility and the user's emotion information.

[0172] The server analyzes the received request and retrieves detailed information about the facility from the database, including opening hours, ratings, special offers, etc. The server then considers the user's emotional information and selects the information that best suits the user's current emotional state and returns it to the terminal.

[0173] The device analyzes the information received from the server and displays it in a format that the user can understand through their visual device. For example, if the user's emotions are positive, highly rated menu items and special offers will be highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information will be displayed.

[0174] Specific examples are given below.

[0175] Imagine a user is walking down the street and a cafe comes into view. The vision device's camera captures the image, and the device pre-processes the image. The AI ​​model identifies the name of the cafe as "Café Delight." The emotion engine analyzes the user's emotion and determines that the emotional state is positive, for example. The device sends the facility information and emotion information to the server, which then returns detailed information about "Café Delight" along with recommended menu items and special offers. Finally, the device displays this information on the vision device's display, allowing the user to obtain the most relevant information in real time.

[0176] In this way, the present invention is a system that allows users to obtain information about surrounding facilities in real time and provides optimal information based on their emotions, improving the user experience and providing more meaningful information.

[0177] The above is a specific description of the embodiment for carrying out the present invention.

[0178] The processing flow will be explained below.

[0179] Step 1:

[0180] The user wears the vision device. The camera in the vision device captures images of the user's field of view in real time. This image data is temporarily stored in the vision device.

[0181] Step 2:

[0182] The device begins pre-processing the captured video data, which includes noise removal and resolution adjustment, making the video clearer and easier for AI to analyze.

[0183] Step 3:

[0184] After preprocessing, the device inputs the video data into an artificial intelligence model, which then analyzes the images and identifies the facility based on its exterior, signage, logo, and other features contained in the video.

[0185] Step 4:

[0186] To recognize the user's emotions, the emotion engine of the vision device analyzes voice, facial expressions, and biometric information (e.g., heart rate, galvanic skin response). The emotion engine identifies the user's current emotional state.

[0187] Step 5:

[0188] The terminal sends a request to the server based on the facility information and the user's emotion information. The request includes the facility name, category, and emotion information.

[0189] Step 6:

[0190] The server analyzes the received request and retrieves detailed information about the facility (e.g., opening hours, ratings, special offers) from the database. The server also selects the most appropriate additional information based on the emotional information.

[0191] Step 7:

[0192] The server returns the acquired detailed information and selected additional information to the terminal as a response.

[0193] Step 8:

[0194] The device analyzes the information received from the server and displays it in a format that the user can perceive through their visual device. For example, if the user's emotions are positive, highly rated menu items and special offers will be highlighted, and if the user is showing signs of stress, suggestions for relaxation spots and coupon information will be displayed.

[0195] Step 9:

[0196] The user can check the detailed information of the facility and additional emotional information displayed on the visual device and take the necessary action. For example, when information about the cafe "Café Delight" is displayed, the user can check the opening hours and special offers and decide whether to visit.

[0197] The above is a detailed description of the specific processing steps of the program and the operations at each step.

[0198] Example 2

[0199] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0200] Conventional information acquisition systems using visual devices have difficulty providing optimal information according to the user's emotional state and environment. Furthermore, the acquired information is merely mechanical, which is insufficient for improving the user experience. Specifically, there is a lack of information presentation tailored to the user's stressful state or positive feelings toward a particular facility.

[0201] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0202] In this invention, the server includes means for acquiring video within the user's field of view, means for preprocessing the acquired video data, means for performing image analysis using an artificial intelligence model for identifying facilities based on the preprocessed video data, means including an emotion engine for analyzing the user's voice, facial expression, and biological information to recognize the user's emotional state, means for requesting facility information and emotion information acquired based on the image analysis and emotion analysis from the server, and means for displaying detailed facility information and optimal information based on the user's emotional state returned from the server on the display of the visual device, thereby making it possible to provide optimal facility information according to the user's emotional state and visual information.

[0203] A "user" is a person who wears and uses the visual device of the present invention.

[0204] A "vision device" is a device that includes a camera that captures images within the user's field of view and a display that displays that information to the user.

[0205] A "camera" is a device attached to a visual device that captures images in the user's field of view in real time.

[0206] "Video data" refers to image information captured by a camera and seen by a user.

[0207] "Preprocessing" refers to the process of removing noise and adjusting resolution of captured video data.

[0208] "Artificial Intelligence Model" means a machine learning model used to identify facilities using pre-processed video data.

[0209] "Image analysis" is a technology that uses an artificial intelligence model to extract the characteristics of a facility from video data and identify the facility.

[0210] An "emotion engine" is a technology that analyzes a user's voice, facial expression, and biological information to recognize the user's emotional state.

[0211] "Facility information" is detailed information such as the name and category of the identified facility.

[0212] A "server" is a computer system that retrieves information from a database based on a request and responds to a terminal.

[0213] A "display" is a device that is installed in a visual device and visually displays information obtained from a server or a terminal to a user.

[0214] "Biometric information" is data that indicates the user's physical condition, such as their heart rate and skin galvanic response.

[0215] A "request" is an inquiry that includes facility information and emotion information and is sent from a terminal to a server.

[0216] "Detailed information" is information such as opening hours, ratings, and special offers about a specific facility that is returned from the server.

[0217] The present invention is a system in which a user wears a visual device and acquires and displays information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[0218] Hardware and software used

[0219] 1. Visual Apparatus

[0220] The visual device worn by the user has a built-in camera and display. The camera captures images of the user's field of view in real time and temporarily stores them in the visual device's memory.

[0221] 2. Terminal

[0222] The device receives video data from the visual device, performs pre-processing and image analysis using artificial intelligence models, and also includes an emotion engine that recognizes the user's emotional state.

[0223] 3. Server

[0224] The server receives the request from the device, retrieves detailed information about the facility from the database, and returns the retrieved information to the device in an optimal format based on the user's emotional state.

[0225] Specific software used includes:

[0226] Artificial intelligence models: TensorFlow, PyTorch, etc.

[0227] Database: MySQL, PostgreSQL, etc.

[0228] Details of data processing and calculation

[0229] 1. Video capture by the user's visual device

[0230] The camera of the vision device captures images of what is in the user's field of view and temporarily stores this data within the vision device.

[0231] 2. Preprocessing of video data by the device

[0232] As soon as the device receives the video data from the vision device, it starts preprocessing, which includes noise removal and resolution adjustment, to enable subsequent image analysis to be performed with greater accuracy.

[0233] 3. Image analysis using AI models

[0234] The pre-processed video data is then fed into an AI model, which uses pattern recognition to identify the facility and identify its name and category based on its exterior, signage, logo, and other characteristics.

[0235] 4. Emotion analysis using an emotion engine

[0236] The emotion engine built into the visual device analyzes the user's voice, facial expressions, heart rate, and skin galvanic response to determine the user's emotional state.

[0237] 5. Sending a request to the server

[0238] The terminal integrates the identified facility information with the emotion information and sends a request to the server. This request includes the specific facility name, category, and emotion information.

[0239] 6. Server retrieves and returns details

[0240] The server analyzes the request received from the device, retrieves detailed information about the facility from the database, and then selects the most appropriate information based on the user's emotional state and returns it to the device.

[0241] 7. Displaying Information on a Terminal

[0242] The device receives detailed information about the facility from the server and displays it on the display of the visual device. If the user is in a positive emotional state, highly rated menu items and special offers are highlighted, while if the user is feeling stressed, suggestions for relaxation spots and coupon information are displayed.

[0243] Specific examples

[0244] Consider an example where a user is walking down the street and a cafe comes into view. The vision device's camera captures video of the cafe, and the device pre-processes the video. The AI ​​model identifies the cafe, and the emotion engine analyzes the user's emotions. For example, if the device determines that the user is in a positive emotional state, it sends the facility information and emotion information to the server. The server returns detailed information about the cafe, along with recommended menu items and special offers, which the device then displays on its screen. The user can check this information in real time and make the best choice.

[0245] Prompt Sentence Examples

[0246] User: When I use a visual aid to find a cafe, how exactly does the system process and display the information?

[0247] The above is a specific embodiment of the present invention, which allows the user to obtain information about surrounding facilities in real time, and furthermore, to obtain optimal information based on emotions.

[0248] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0249] Step 1:

[0250] The user wears the visual device.

[0251] How it works: The user puts on the vision device, activating it. The camera begins capturing video within the user's field of view.

[0252] Input: The image that the user sees.

[0253] Output: Captured video data.

[0254] Step 2:

[0255] The terminal receives the captured video data from the visual device.

[0256] Operation: The visual device's camera captures video data and sends it to the device, where it is temporarily saved.

[0257] Input: Captured video data.

[0258] Output: Received video data.

[0259] Step 3:

[0260] The terminal pre-processes the video data.

[0261] Operation: Performs noise reduction and resolution adjustment on the video data received by the device.

[0262] Input: Received video data.

[0263] Output: Pre-processed video data with noise removed and resolution adjusted.

[0264] Step 4:

[0265] The device inputs the preprocessed video data into an artificial intelligence model to identify the facility.

[0266] How it works: Preprocessed video data is fed into an AI model (e.g., TensorFlow, PyTorch) that uses pattern recognition techniques to identify facility names and categories.

[0267] Input: Preprocessed video data.

[0268] Output: Identified facility information (name, category).

[0269] Step 5:

[0270] The emotion engine of the visual device analyzes the user's emotions.

[0271] How it works: The emotion engine built into the visual device analyzes the user's voice, facial expressions, heart rate, and galvanic skin response to determine the user's emotional state.

[0272] Input: User's voice, facial expressions, heart rate, and galvanic skin response.

[0273] Output: The perceived emotional state of the user.

[0274] Step 6:

[0275] The terminal requests facility information and emotion information from the server.

[0276] Operation: The device compiles the identified facility information and analyzed emotion information and sends a request to the server.

[0277] Input: Identified facility information and emotion information.

[0278] Output: The request sent to the server.

[0279] Step 7:

[0280] The server receives the request, retrieves the details and sends them back to the device.

[0281] Operation: The server analyzes the received request and retrieves detailed information about the facility from its own database. Taking into account the user's emotional information, it selects the most appropriate information and returns it to the device.

[0282] Input: The request sent to the server.

[0283] Output: Detailed information about the selected facility.

[0284] Step 8:

[0285] The terminal displays the information on the visual device's display.

[0286] Operation: The device formats the facility information received from the server and displays it on the visual device's display. If the user is in a positive emotional state, highly rated menu items and special offers are highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information are displayed.

[0287] Input: Facility details received from the server.

[0288] Output: Information displayed on the visual device's display.

[0289] (Application example 2)

[0290] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0291] Conventional information provision systems using visual devices only allow users to visually obtain information about surrounding facilities, and have the problem of not being able to provide optimal information based on the user's current emotional state. Specifically, even if a user is feeling stressed, for example, they may not be provided with information about facilities where they can relax or special offers, which could result in a less meaningful shopping experience.

[0292] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0293] In this invention, the server includes means for selecting and returning optimal information taking into consideration the user's emotional information, means for preprocessing video data acquired by a camera attached to the visual device, and means for performing image analysis using an artificial intelligence model that identifies facilities based on the preprocessed video data. This allows the user to obtain recommended information based on their emotional state while obtaining information about surrounding facilities through the visual device in real time.

[0294] A "visual device" is a device equipped with a display and camera worn by the user, and has the function of capturing images within the user's field of vision and displaying the information.

[0295] A "camera" is an image capturing device attached to a visual device that captures images in the user's field of view in real time.

[0296] "Preprocessing" refers to processes such as removing noise from acquired video data and adjusting resolution to improve the accuracy of subsequent analysis.

[0297] An "artificial intelligence model" is a system that uses pattern recognition technology to identify facilities based on their exterior appearance, signage, logo, and other characteristics.

[0298] "Image analysis" refers to the process of identifying facilities using artificial intelligence models based on pre-processed video data.

[0299] "Emotional information" refers to data relating to the user's emotional state obtained by analyzing the user's voice, facial expression, and biometric information.

[0300] The "server" is a computer system that retrieves detailed facility information and recommended information from a database, and selects and returns the most appropriate information to the user, taking into account emotional information.

[0301] "Detailed information" refers to specific information about a facility, such as the facility's opening hours, ratings, and special offers.

[0302] An "emotion analysis engine" is a system that analyzes a user's voice, facial expression, and biometric information to identify their emotional state.

[0303] "Recommended information" refers to information on products, services, locations, etc. that are optimal for a user and are selected with consideration given to emotional information.

[0304] MODE FOR CARRYING OUT THE INVENTION

[0305] The present invention is a system in which a user wears a visual device and acquires and displays information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. A specific embodiment of this system will be described.

[0306] A user wears a visual device such as smart glasses, and a camera attached to the visual device captures images of what the user sees in real time. The video data captured by the camera is temporarily stored in the visual device. The device then begins preprocessing the video data. Preprocessing includes noise reduction and resolution adjustment, which improves the clarity of the image and the accuracy of subsequent analysis.

[0307] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[0308] Meanwhile, the visual device is equipped with an emotion engine to recognize the user's emotions. The emotion engine recognizes the user's emotional state by analyzing the user's voice, facial expressions, and biometric information (heart rate, skin galvanic response, etc.). For example, it can detect when the user is feeling stressed or curious.

[0309] Facility information and emotional information are sent from the device to the server as a request. The request includes the name and category of the identified facility, as well as the user's emotional information. The server analyzes the received request and retrieves detailed information about the facility from its database. This information includes opening hours, ratings, and special offers. The server then takes the emotional information into consideration and selects the information that best suits the user's current emotional state, which it then sends back to the device. For example, if the user's emotions are positive, the facility's highly rated menu items and special offers are highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information are also displayed.

[0310] As a specific example of use, imagine a user is walking through a shopping mall and the camera captures a sign for "Bookstore A." TensorFlow Lite identifies "Bookstore A" and performs sentiment analysis on the audio data. If it determines the sentiment is "positive," it sends "Bookstore A" and "positive" to the server. Information returned from the server, such as "discount information for bestselling books," is displayed on the smart glasses, and the user can see a pop-up that says "20% off bestselling books."

[0311] The following hardware and software are used to realize this system.

[0312] Hardware: Smart glasses (with camera), user device (smartphone, etc.)

[0313] Software: OpenCV (preprocessing), TensorFlow Lite (image analysis), Amazon Comprehend (sentiment analysis)

[0314] An example of a prompt is as follows:

[0315] example:

[0316] As the user walks through a shopping mall, the camera captures a sign for "Bookstore A." "Bookstore A" is identified using TensorFlow Lite. Sentiment analysis is performed on the audio data, and the result is determined to be "positive." "Bookstore A" and "positive" are sent to the server. The server returns "discount information for best-selling books." A pop-up message saying "20% off best-selling books" appears on the smart glasses.

[0317] As described above, the present invention is a system that allows users to obtain information about surrounding facilities in real time and provides optimal information based on their emotions, thereby improving the user experience and providing more meaningful information.

[0318] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0319] Step 1:

[0320] The user wears a visual device. The camera of this visual device captures the image of the user's field of view in real time. The input is the image of the user's field of view, and the output is raw image data temporarily stored in the visual device. Specifically, the camera constantly follows the user's field of view and continuously captures the image.

[0321] Step 2:

[0322] The device performs preprocessing on the captured video data. This preprocessing includes noise removal and resolution adjustment. The input is the captured raw video data, and the output is clear video data with noise removed and resolution adjusted. Specifically, the OpenCV library is used to filter noise and optimize resolution.

[0323] Step 3:

[0324] The preprocessed video data is input into an artificial intelligence model (generative AI model) installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. The input is the preprocessed video data, and the output is the name and category of the identified facility. Specifically, TensorFlow Lite is used to perform pattern recognition based on the facility's exterior, sign, and logo.

[0325] Step 4:

[0326] The device uses an emotion engine to analyze the user's emotions. The emotion engine analyzes the user's voice, facial expression, and biometric information (such as heart rate and galvanic skin response). The input is the user's voice, facial expression, and biometric information, and the output is data indicating the user's emotional state. Specifically, it collects and analyzes data from Amazon Comprehend and biometric sensors.

[0327] Step 5:

[0328] The device sends facility information and emotion information as a request to the server. The input is the name and category of the identified facility and the user's emotion information, and the output is the request data sent to the server. Specifically, the data is communicated to the server using a RESTful API.

[0329] Step 6:

[0330] The server analyzes the request data it receives. The server retrieves detailed information about the facility (such as opening hours, ratings, and special offers) from the database and selects the most appropriate recommendation information taking into account emotional information. The input is the request data sent to the server, and the output is detailed information about the facility and the most appropriate information for the user. Specifically, it extracts data from AWS RDS and filters the information as needed.

[0331] Step 7:

[0332] The server returns the selected optimal information to the terminal. The input is detailed information and recommended information about the selected facility, and the output is the response data sent to the terminal. Specifically, the data is sent to the terminal again using the RESTful API.

[0333] Step 8:

[0334] The terminal analyzes the information received from the server and displays it in a format that the user can understand through the visual device. The input is detailed facility information and recommended information returned from the server, and the output is the information displayed on the visual device's display. Specifically, the information is displayed visually using the Unity engine, etc.

[0335] Through the above processing steps, the user can obtain information about surrounding facilities in real time through the visual device, and can obtain optimal information based on their emotions.

[0336] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0337] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0338] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0339] [Second embodiment]

[0340] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0341] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0342] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0343] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0344] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0345] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0346] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0347] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0348] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0349] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0350] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0351] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0352] The present invention aims to enable a user wearing a visual device to acquire and display information about surrounding facilities in real time. The specific processing content of the program and specific examples are described below.

[0353] First, the user wears the vision device. The camera in the vision device captures images of what is in the user's field of view in real time. This image is temporarily stored in the vision device.

[0354] The device then begins preprocessing the video data, which includes noise reduction and resolution adjustment, to improve the clarity of the image and subsequent analysis.

[0355] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[0356] Once the device identifies the facility information, it sends a request to the server based on that information. The request includes identification information such as the facility name and category.

[0357] The server searches the database based on the received request and retrieves detailed information about the facility, including opening hours, ratings, special offers, etc. The server then sends this information back to the terminal as a response.

[0358] The terminal analyzes the information received from the server and displays it in a format that can be understood by the user through the visual device. Specifically, detailed facility information is overlaid on the visual device's display, allowing the user to grasp detailed facility information in real time on-site.

[0359] Specific examples are given below.

[0360] Imagine a user is walking down the street and a cafe comes into view. At this time, the camera in the vision device captures video of the cafe. The device preprocesses the video and inputs it into the AI ​​model. The AI ​​model identifies the name of the cafe as "Café Delight." The device sends the identified information to the server, which retrieves "Café Delight's" business hours (8:00-20:00), rating (4.5 / 5), and special offers (10% off coupon available) from its database and sends them back to the device. Finally, the device displays this information on the vision device's display.

[0361] In this way, the present invention provides a system that allows a user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0362] The above is a specific description of the embodiment for carrying out the present invention.

[0363] The processing flow will be explained below.

[0364] Step 1:

[0365] The user wears the vision device. The camera in the vision device captures images of the user's field of view in real time. The image data is temporarily stored in the vision device.

[0366] Step 2:

[0367] The device begins pre-processing the captured video data, performing processes such as noise removal and resolution adjustment to make it easier for AI to analyze. This makes the video clearer and highlights the facility's unique features.

[0368] Step 3:

[0369] After preprocessing, the device inputs the video data into an artificial intelligence model. The AI ​​model analyzes the image and identifies the facility based on features such as the facility's exterior, sign, and logo contained in the video. For example, if a sign reads "Café Delight," it will perform character recognition.

[0370] Step 4:

[0371] The terminal sends a request to the server based on the identified facility information. The request includes information such as the name and category of the identified facility.

[0372] Step 5:

[0373] The server analyzes the received request and retrieves detailed information about the facility from the database, including opening hours, ratings, special offers, etc. The retrieved information is then sent back to the terminal as a response.

[0374] Step 6:

[0375] The device analyzes the detailed facility information received from the server and displays it on the visual device's display. Specifically, the facility's name, opening hours, ratings, and special offer information are overlaid in the user's field of vision.

[0376] Step 7:

[0377] The user can check the detailed information of the facility displayed on the visual device and take the necessary action. For example, when information about the cafe "Café Delight" is displayed, the user can check the opening hours and special offers and decide whether to visit.

[0378] The above is a detailed description of the specific processing steps of the program and the operations at each step.

[0379] Example 1

[0380] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0381] Conventional technologies lack a means for users to quickly and easily obtain detailed information about surrounding facilities in real time. This requires users to perform manual searches, which is not only inconvenient but also inefficient. Furthermore, conventional approaches are difficult to implement in situations where instantaneous on-site information is required. To address these issues, the present invention aims to provide a system that allows users to obtain facility information in real time using a visual device and instantly display detailed information.

[0382] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0383] In this invention, the server includes means for acquiring images of the user's field of view using a camera attached to a visual device worn by the user, means for preprocessing the acquired image data, means for performing image analysis using an artificial intelligence model for identifying facilities based on the preprocessed image data, means for requesting facility information acquired based on the image analysis from the server, and means for overlaying the detailed facility information returned from the server on the display of the visual device, thereby enabling the user to quickly acquire detailed information about surrounding facilities in real time through the visual device and efficiently utilize it.

[0384] A "user" is a person who utilizes the system of the present invention by wearing a visual device.

[0385] A "visual device" is a device worn by a user that includes a display and camera that displays to the field of view.

[0386] A "camera" is a photographing device attached to a visual device for capturing images within the user's field of vision.

[0387] "Video data" refers to digital data of images and videos captured by a camera.

[0388] "Preprocessing" refers to the step of performing processes such as noise removal and resolution adjustment on the video data.

[0389] An "artificial intelligence model" is an AI algorithm that uses pattern recognition technology to identify facilities contained within video footage.

[0390] "Image analysis" is the process of using artificial intelligence models to identify facility information from video data.

[0391] "Facility information" refers to data such as the name, category, and detailed information of the identified facility.

[0392] A "server" is a computer system that searches a database based on a request, obtains detailed facility information, and returns a response.

[0393] A "database" is an information aggregation system owned by a server that stores detailed facility information.

[0394] "Overlay display" refers to a function that displays information superimposed on the image displayed on the display of a visual device.

[0395] The present invention is a system that acquires information about surrounding facilities in real time while the user is wearing a visual device, and displays the details on the visual device. The processing contents of the program will be specifically described below.

[0396] First, the user wears a visual device. The device is equipped with a camera that captures images of the user's field of vision in real time. This image data is temporarily stored in the device.

[0397] Next, the device begins preprocessing the video data. This includes noise reduction and resolution adjustment. This makes the video clearer and improves the accuracy of subsequent analysis. This preprocessing typically uses an image processing library such as OpenCV.

[0398] Once preprocessed, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo. Frameworks such as TensorFlow and PyTorch are used for the AI ​​model.

[0399] Once the device identifies the facility information, it sends a request to the server based on that information. The request includes identifying information such as the facility's name and category. The server searches the database based on the received request and retrieves detailed information about the facility. This detailed information includes opening hours, ratings, special offers, and more. The server uses cloud services such as AWS and Google Cloud, and databases such as MySQL and PostgreSQL.

[0400] The acquired detailed facility information is sent back to the terminal as a response from the server. The terminal analyzes the information received from the server and displays it in a format that the user can understand through the visual device. Specifically, the detailed facility information is overlaid on the display of the visual device. This allows the user to grasp detailed facility information in real time on-site.

[0401] Here's a specific example: When a user is walking down the street and a cafe comes into view, the camera in the vision device captures video of the cafe. The device preprocesses the video and inputs it into an AI model. The AI ​​model identifies the name of the cafe as "Cafe Delight." The device sends the identified information to a server, which retrieves Cafe Delight's opening hours (8:00-20:00), rating (4.5 / 5), and special offers (10% off coupon available) from a database and sends them back to the device. Finally, the device displays this information on the vision device's display.

[0402] An example prompt is, "A user wears a visual device, and a camera captures video of the surroundings in real time. The video is preprocessed, and an AI model identifies the facility. Based on this facility information, a server provides detailed information, which is displayed on the visual device. As a specific example, please explain how detailed information about the cafe "Cafe Delight" is obtained and displayed to the user."

[0403] The above is a specific description of the embodiment of the present invention. With this system, a user can obtain information about surrounding facilities in real time and efficiently check detailed information.

[0404] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0405] Step 1:

[0406] The user wears a visual device, and a camera attached to the device captures images of what is in the user's field of view in real time.

[0407] Input: The image that the user sees.

[0408] Specific operation: The user wears the visual device on their head, and the camera continuously captures objects within their field of view.

[0409] Output: Captured video data.

[0410] Step 2:

[0411] The device pre-processes the captured video data, removing noise and adjusting the resolution to make the video clearer.

[0412] Input: The captured video data that is the output of step 1.

[0413] Data processing: Use an image processing library such as OpenCV to remove noise from the video data and improve its resolution.

[0414] Specific operation: The device filters the video data to reduce noise and adjust the image resolution appropriately.

[0415] Output: Pre-processed video data.

[0416] Step 3:

[0417] The device performs image analysis using artificial intelligence models to identify facilities based on pre-processed video data.

[0418] Input: The preprocessed video data that is the output of step 2.

[0419] Data Computing: Using AI frameworks such as TensorFlow and PyTorch, facilities are identified from video data.

[0420] Specific operation: Preprocessed video is input into the AI ​​model, and the facility's exterior, signs, logos, etc. are analyzed using a pattern recognition algorithm.

[0421] Output: Data with facility names and categories identified.

[0422] Step 4:

[0423] The device sends a request to the server based on the facility information it has identified, including identification information such as the facility name and category.

[0424] Input: Identified facility information, which is the output of Step 3.

[0425] Data processing: The facility identification information is converted into a data format for sending to the server as an HTTP request.

[0426] Specific operation: The terminal generates a request message containing facility information such as "Cafe Delight" and sends it to the server.

[0427] Output: The request data sent.

[0428] Step 5:

[0429] The server searches the database based on the received request and obtains detailed information about the facility.

[0430] Input: The request data sent in step 4.

[0431] Data Computation: Querying database systems such as MySQL or PostgreSQL to retrieve the requested facility details.

[0432] What happens: The server queries the database to get information about Cafe Delight, such as its hours of operation, ratings, and special offers.

[0433] Output: Detailed information about the retrieved facilities.

[0434] Step 6:

[0435] The terminal receives the detailed facility information returned from the server and displays it as an overlay on the display of the visual device.

[0436] Input: The retrieved facility details, which is the output of step 5.

[0437] Data processing: Preparing information in a format suitable for display on visual devices.

[0438] Specific operation: Detailed facility information (e.g., business hours 8:00-20:00, rating 4.5 / 5, 10% off coupon available) is sent to the user's visual device and overlaid on the screen.

[0439] Output: Facility details displayed on a visual aid.

[0440] The above are the specific processing steps of the program of this system, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0441] (Application example 1)

[0442] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0443] In modern society, users need to quickly and easily obtain information about nearby facilities while on the move. However, conventional methods require searching on a smartphone to obtain facility information, which is time-consuming and lacks real-time information. Furthermore, integrating information from different sources increases the burden on users, making it difficult to obtain information efficiently. Therefore, there is a need for an efficient system that allows users to instantly obtain detailed facility information on the spot using a visual device.

[0444] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0445] In this invention, the server includes means for acquiring video within the user's field of view using a camera attached to a visual device worn by the user, means for preprocessing the acquired video data, means for performing image analysis using a generative AI model for identifying facilities based on the preprocessed video data, means for requesting facility information acquired based on the image analysis from the server via a communication device, and means for overlaying and displaying detailed facility information returned from the server on the display device of the visual device, thereby enabling the user to acquire information about surrounding facilities in real time and efficiently check the detailed information.

[0446] A "visual device" is a device that can display visual information on a display when worn by a user.

[0447] A "camera" is a device that is attached to a visual device and has the function of capturing images that come into the user's field of vision.

[0448] "Preprocessing" refers to processing such as removing noise from acquired video data and adjusting resolution to improve the accuracy of subsequent analysis.

[0449] "Generative AI model" means an artificial intelligence model used to analyze video data and identify the category and specific name of a facility.

[0450] A "communication device" is a device for transmitting identified facility information to a server and receiving a response from the server.

[0451] The "display device" is a display that is mounted on the visual device and has the function of overlaying detailed facility information to the user.

[0452] A "prompt sentence" is an instruction sentence used by the generative AI model to identify a facility.

[0453] The present invention provides a system that acquires and displays surrounding facility information in real time using a visual device worn by a user. This system is composed of an image capture device, a preprocessing module, a generative AI model, a communication device, and a display device mounted on the visual device.

[0454] The vision device is equipped with a camera for capturing images in real time. The camera captures the image of the user's field of view and sends it to a pre-processing module, where processing such as noise reduction and resolution adjustment of the image data is performed. This pre-processing improves the accuracy of subsequent image analysis.

[0455] The preprocessed video data is then input into a generative AI model, which uses pattern recognition technology to identify facilities within the video. Specifically, the model analyzes the facility's exterior, signage, logo, and other features to identify the facility's category and specific name. A prompt is then generated to assist the AI ​​model in the identification process.

[0456] The communication device is responsible for transmitting the identified facility information to the server. The server receives this information and searches a database to obtain detailed information about the facility, including opening hours, ratings, special offers, etc. The server then sends these details back to the visual device via the communication device.

[0457] Finally, the display device displays the information returned from the server to the user. Specifically, detailed facility information is overlaid on the display of the visual device, allowing the user to check detailed information about surrounding facilities in real time through the visual device.

[0458] As a specific example, consider the flow from when a user is walking down the street and a cafe comes into view, to when a camera captures video of the cafe. The preprocessing module preprocesses this video and inputs it into the generative AI model. The generative AI model identifies the cafe, and the communication device sends this information to the server. The server receives the returned detailed information about the cafe (opening hours, ratings, special offers), which it then overlays on the display of the visual device.

[0459] The following are examples of prompt statements:

[0460] User looks at "Cafe" and gets real-time information overlay showing operation hours (8:00-20:00), rating (4.5 / 5), and a special coupon (10% off).

[0461] The above is a specific description of the embodiment of the present invention, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0462] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0463] Step 1:

[0464] When a user walks around town, the camera of the vision device captures the image that comes into the user's field of view in real time. The input is the user's field of view, and the output is the captured image data.

[0465] Step 2:

[0466] The captured video data is processed by the pre-processing module. Specifically, noise removal and resolution adjustment are performed. The input is the captured video data, and the output is pre-processed video data that has been subjected to noise removal and resolution adjustment.

[0467] Step 3:

[0468] The preprocessed video data is input into the generative AI model to identify the facility. Specifically, the model analyzes the facility's exterior, signage, logo, and other features, and identifies the facility's category and specific name using prompt text. The input is the preprocessed video data, and the output is the facility's identification information (facility name, category).

[0469] Step 4:

[0470] The identified facility information is sent to the server by the communication device. Specifically, the facility name and category information are constructed as an HTTP request and sent to the server. The input at this time is the facility identification information, and the output is a request to the server.

[0471] Step 5:

[0472] The server searches the database based on the received request and retrieves detailed information about the facility (such as opening hours, ratings, special offers, etc.) The input is the request to the server, and the output is the detailed information about the facility.

[0473] Step 6:

[0474] The server returns the acquired facility details to the visual device via the communication device. Specifically, the facility details are constructed as an HTTP response and sent to the visual device. The input at this time is the facility details, and the output is the response from the server.

[0475] Step 7:

[0476] The display device of the visual device overlays the detailed facility information received from the server. Specifically, the detailed facility information is displayed on the display in an appropriate position according to the user's field of view. The input at this time is the response from the server, and the output is the detailed facility information overlaid on the user's field of view.

[0477] The above is a description of the specific processing steps in the system for implementing the present invention, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0478] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0479] The present invention is a system that allows a user to wear a visual device and acquire and display information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. The specific processing content of the program and concrete examples are explained below.

[0480] The user wears the vision device. The camera in the vision device captures images of what is in the user's field of view in real time. This image is temporarily stored in the vision device.

[0481] The device then begins preprocessing the video data, which includes noise reduction and resolution adjustment, to improve the clarity of the image and subsequent analysis.

[0482] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[0483] The visual device is equipped with an emotion engine to recognize the user's emotions. The emotion engine recognizes the user's emotional state by analyzing the user's voice, facial expressions, and biometric information (heart rate, skin galvanic response, etc.). For example, it can detect when the user is feeling stressed or curious.

[0484] The device sends a request to the server based on the facility information and emotion information. The request includes the name and category of the identified facility and the user's emotion information.

[0485] The server analyzes the received request and retrieves detailed information about the facility from the database, including opening hours, ratings, special offers, etc. The server then considers the user's emotional information and selects the information that best suits the user's current emotional state and returns it to the terminal.

[0486] The device analyzes the information received from the server and displays it in a format that the user can understand through their visual device. For example, if the user's emotions are positive, highly rated menu items and special offers will be highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information will be displayed.

[0487] Specific examples are given below.

[0488] Imagine a user is walking down the street and a cafe comes into view. The vision device's camera captures the image, and the device pre-processes the image. The AI ​​model identifies the name of the cafe as "Café Delight." The emotion engine analyzes the user's emotion and determines that the emotional state is positive, for example. The device sends the facility information and emotion information to the server, which then returns detailed information about "Café Delight" along with recommended menu items and special offers. Finally, the device displays this information on the vision device's display, allowing the user to obtain the most relevant information in real time.

[0489] In this way, the present invention is a system that allows users to obtain information about surrounding facilities in real time and provides optimal information based on their emotions, improving the user experience and providing more meaningful information.

[0490] The above is a specific description of the embodiment for carrying out the present invention.

[0491] The processing flow will be explained below.

[0492] Step 1:

[0493] The user wears the vision device. The camera in the vision device captures images of the user's field of view in real time. This image data is temporarily stored in the vision device.

[0494] Step 2:

[0495] The device begins pre-processing the captured video data, which includes noise removal and resolution adjustment, making the video clearer and easier for AI to analyze.

[0496] Step 3:

[0497] After preprocessing, the device inputs the video data into an artificial intelligence model, which then analyzes the images and identifies the facility based on its exterior, signage, logo, and other features contained in the video.

[0498] Step 4:

[0499] To recognize the user's emotions, the emotion engine of the vision device analyzes voice, facial expressions, and biometric information (e.g., heart rate, galvanic skin response). The emotion engine identifies the user's current emotional state.

[0500] Step 5:

[0501] The terminal sends a request to the server based on the facility information and the user's emotion information. The request includes the facility name, category, and emotion information.

[0502] Step 6:

[0503] The server analyzes the received request and retrieves detailed information about the facility (e.g., opening hours, ratings, special offers) from the database. The server also selects the most appropriate additional information based on the emotional information.

[0504] Step 7:

[0505] The server returns the acquired detailed information and selected additional information to the terminal as a response.

[0506] Step 8:

[0507] The device analyzes the information received from the server and displays it in a format that the user can perceive through their visual device. For example, if the user's emotions are positive, highly rated menu items and special offers will be highlighted, and if the user is showing signs of stress, suggestions for relaxation spots and coupon information will be displayed.

[0508] Step 9:

[0509] The user can check the detailed information of the facility and additional emotional information displayed on the visual device and take the necessary action. For example, when information about the cafe "Café Delight" is displayed, the user can check the opening hours and special offers and decide whether to visit.

[0510] The above is a detailed description of the specific processing steps of the program and the operations at each step.

[0511] Example 2

[0512] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0513] Conventional information acquisition systems using visual devices have difficulty providing optimal information according to the user's emotional state and environment. Furthermore, the acquired information is merely mechanical, which is insufficient for improving the user experience. Specifically, there is a lack of information presentation tailored to the user's stressful state or positive feelings toward a particular facility.

[0514] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0515] In this invention, the server includes means for acquiring video within the user's field of view, means for preprocessing the acquired video data, means for performing image analysis using an artificial intelligence model for identifying facilities based on the preprocessed video data, means including an emotion engine for analyzing the user's voice, facial expression, and biological information to recognize the user's emotional state, means for requesting facility information and emotion information acquired based on the image analysis and emotion analysis from the server, and means for displaying detailed facility information and optimal information based on the user's emotional state returned from the server on the display of the visual device, thereby making it possible to provide optimal facility information according to the user's emotional state and visual information.

[0516] A "user" is a person who wears and uses the visual device of the present invention.

[0517] A "vision device" is a device that includes a camera that captures images within the user's field of view and a display that displays that information to the user.

[0518] A "camera" is a device attached to a visual device that captures images in the user's field of view in real time.

[0519] "Video data" refers to image information captured by a camera and seen by a user.

[0520] "Preprocessing" refers to the process of removing noise and adjusting resolution of captured video data.

[0521] "Artificial Intelligence Model" means a machine learning model used to identify facilities using pre-processed video data.

[0522] "Image analysis" is a technology that uses an artificial intelligence model to extract the characteristics of a facility from video data and identify the facility.

[0523] An "emotion engine" is a technology that analyzes a user's voice, facial expression, and biological information to recognize the user's emotional state.

[0524] "Facility information" is detailed information such as the name and category of the identified facility.

[0525] A "server" is a computer system that retrieves information from a database based on a request and responds to a terminal.

[0526] A "display" is a device that is installed in a visual device and visually displays information obtained from a server or a terminal to a user.

[0527] "Biometric information" is data that indicates the user's physical condition, such as their heart rate and skin galvanic response.

[0528] A "request" is an inquiry that includes facility information and emotion information and is sent from a terminal to a server.

[0529] "Detailed information" is information such as opening hours, ratings, and special offers about a specific facility that is returned from the server.

[0530] The present invention is a system in which a user wears a visual device and acquires and displays information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[0531] Hardware and software used

[0532] 1. Visual Apparatus

[0533] The visual device worn by the user has a built-in camera and display. The camera captures images of the user's field of view in real time and temporarily stores them in the visual device's memory.

[0534] 2. Terminal

[0535] The device receives video data from the visual device, performs pre-processing and image analysis using artificial intelligence models, and also includes an emotion engine that recognizes the user's emotional state.

[0536] 3. Server

[0537] The server receives the request from the device, retrieves detailed information about the facility from the database, and returns the retrieved information to the device in an optimal format based on the user's emotional state.

[0538] Specific software used includes:

[0539] Artificial intelligence models: TensorFlow, PyTorch, etc.

[0540] Database: MySQL, PostgreSQL, etc.

[0541] Details of data processing and calculation

[0542] 1. Video capture by the user's visual device

[0543] The camera of the vision device captures images of what is in the user's field of view and temporarily stores this data within the vision device.

[0544] 2. Preprocessing of video data by the device

[0545] As soon as the device receives the video data from the vision device, it starts preprocessing, which includes noise removal and resolution adjustment, to enable subsequent image analysis to be performed with greater accuracy.

[0546] 3. Image analysis using AI models

[0547] The pre-processed video data is then fed into an AI model, which uses pattern recognition to identify the facility and identify its name and category based on its exterior, signage, logo, and other characteristics.

[0548] 4. Emotion analysis using an emotion engine

[0549] The emotion engine built into the visual device analyzes the user's voice, facial expressions, heart rate, and skin galvanic response to determine the user's emotional state.

[0550] 5. Sending a request to the server

[0551] The terminal integrates the identified facility information with the emotion information and sends a request to the server. This request includes the specific facility name, category, and emotion information.

[0552] 6. Server retrieves and returns details

[0553] The server analyzes the request received from the device, retrieves detailed information about the facility from the database, and then selects the most appropriate information based on the user's emotional state and returns it to the device.

[0554] 7. Displaying Information on a Terminal

[0555] The device receives detailed information about the facility from the server and displays it on the display of the visual device. If the user is in a positive emotional state, highly rated menu items and special offers are highlighted, while if the user is feeling stressed, suggestions for relaxation spots and coupon information are displayed.

[0556] Specific examples

[0557] Consider an example where a user is walking down the street and a cafe comes into view. The vision device's camera captures video of the cafe, and the device pre-processes the video. The AI ​​model identifies the cafe, and the emotion engine analyzes the user's emotions. For example, if the device determines that the user is in a positive emotional state, it sends the facility information and emotion information to the server. The server returns detailed information about the cafe, along with recommended menu items and special offers, which the device then displays on its screen. The user can check this information in real time and make the best choice.

[0558] Prompt Sentence Examples

[0559] User: When I use a visual aid to find a cafe, how exactly does the system process and display the information?

[0560] The above is a specific embodiment of the present invention, which allows the user to obtain information about surrounding facilities in real time, and furthermore, to obtain optimal information based on emotions.

[0561] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0562] Step 1:

[0563] The user wears the visual device.

[0564] How it works: The user puts on the vision device, activating it. The camera begins capturing video within the user's field of view.

[0565] Input: The image that the user sees.

[0566] Output: Captured video data.

[0567] Step 2:

[0568] The terminal receives the captured video data from the visual device.

[0569] Operation: The visual device's camera captures video data and sends it to the device, where it is temporarily saved.

[0570] Input: Captured video data.

[0571] Output: Received video data.

[0572] Step 3:

[0573] The terminal pre-processes the video data.

[0574] Operation: Performs noise reduction and resolution adjustment on the video data received by the device.

[0575] Input: Received video data.

[0576] Output: Pre-processed video data with noise removed and resolution adjusted.

[0577] Step 4:

[0578] The device inputs the preprocessed video data into an artificial intelligence model to identify the facility.

[0579] How it works: Preprocessed video data is fed into an AI model (e.g., TensorFlow, PyTorch) that uses pattern recognition techniques to identify facility names and categories.

[0580] Input: Preprocessed video data.

[0581] Output: Identified facility information (name, category).

[0582] Step 5:

[0583] The emotion engine of the visual device analyzes the user's emotions.

[0584] How it works: The emotion engine built into the visual device analyzes the user's voice, facial expressions, heart rate, and galvanic skin response to determine the user's emotional state.

[0585] Input: User's voice, facial expressions, heart rate, and galvanic skin response.

[0586] Output: The perceived emotional state of the user.

[0587] Step 6:

[0588] The terminal requests facility information and emotion information from the server.

[0589] Operation: The device compiles the identified facility information and analyzed emotion information and sends a request to the server.

[0590] Input: Identified facility information and emotion information.

[0591] Output: The request sent to the server.

[0592] Step 7:

[0593] The server receives the request, retrieves the details and sends them back to the device.

[0594] Operation: The server analyzes the received request and retrieves detailed information about the facility from its own database. Taking into account the user's emotional information, it selects the most appropriate information and returns it to the device.

[0595] Input: The request sent to the server.

[0596] Output: Detailed information about the selected facility.

[0597] Step 8:

[0598] The terminal displays the information on the visual device's display.

[0599] Operation: The device formats the facility information received from the server and displays it on the visual device's display. If the user is in a positive emotional state, highly rated menu items and special offers are highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information are displayed.

[0600] Input: Facility details received from the server.

[0601] Output: Information displayed on the visual device's display.

[0602] (Application example 2)

[0603] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0604] Conventional information provision systems using visual devices only allow users to visually obtain information about surrounding facilities, and have the problem of not being able to provide optimal information based on the user's current emotional state. Specifically, even if a user is feeling stressed, for example, they may not be provided with information about facilities where they can relax or special offers, which could result in a less meaningful shopping experience.

[0605] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0606] In this invention, the server includes means for selecting and returning optimal information taking into consideration the user's emotional information, means for preprocessing video data acquired by a camera attached to the visual device, and means for performing image analysis using an artificial intelligence model that identifies facilities based on the preprocessed video data. This allows the user to obtain recommended information based on their emotional state while obtaining information about surrounding facilities through the visual device in real time.

[0607] A "visual device" is a device equipped with a display and camera worn by the user, and has the function of capturing images within the user's field of vision and displaying the information.

[0608] A "camera" is an image capturing device attached to a visual device that captures images in the user's field of view in real time.

[0609] "Preprocessing" refers to processes such as removing noise from acquired video data and adjusting resolution to improve the accuracy of subsequent analysis.

[0610] An "artificial intelligence model" is a system that uses pattern recognition technology to identify facilities based on their exterior appearance, signage, logo, and other characteristics.

[0611] "Image analysis" refers to the process of identifying facilities using artificial intelligence models based on pre-processed video data.

[0612] "Emotional information" refers to data relating to the user's emotional state obtained by analyzing the user's voice, facial expression, and biometric information.

[0613] The "server" is a computer system that retrieves detailed facility information and recommended information from a database, and selects and returns the most appropriate information to the user, taking into account emotional information.

[0614] "Detailed information" refers to specific information about a facility, such as the facility's opening hours, ratings, and special offers.

[0615] An "emotion analysis engine" is a system that analyzes a user's voice, facial expression, and biometric information to identify their emotional state.

[0616] "Recommended information" refers to information on products, services, locations, etc. that are optimal for a user and are selected with consideration given to emotional information.

[0617] MODE FOR CARRYING OUT THE INVENTION

[0618] The present invention is a system in which a user wears a visual device and acquires and displays information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. A specific embodiment of this system will be described.

[0619] A user wears a visual device such as smart glasses, and a camera attached to the visual device captures images of what the user sees in real time. The video data captured by the camera is temporarily stored in the visual device. The device then begins preprocessing the video data. Preprocessing includes noise reduction and resolution adjustment, which improves the clarity of the image and the accuracy of subsequent analysis.

[0620] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[0621] Meanwhile, the visual device is equipped with an emotion engine to recognize the user's emotions. The emotion engine recognizes the user's emotional state by analyzing the user's voice, facial expressions, and biometric information (heart rate, skin galvanic response, etc.). For example, it can detect when the user is feeling stressed or curious.

[0622] Facility information and emotional information are sent from the device to the server as a request. The request includes the name and category of the identified facility, as well as the user's emotional information. The server analyzes the received request and retrieves detailed information about the facility from its database. This information includes opening hours, ratings, and special offers. The server then takes the emotional information into consideration and selects the information that best suits the user's current emotional state, which it then sends back to the device. For example, if the user's emotions are positive, the facility's highly rated menu items and special offers are highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information are also displayed.

[0623] As a specific example of use, imagine a user is walking through a shopping mall and the camera captures a sign for "Bookstore A." TensorFlow Lite identifies "Bookstore A" and performs sentiment analysis on the audio data. If it determines the sentiment is "positive," it sends "Bookstore A" and "positive" to the server. Information returned from the server, such as "discount information for bestselling books," is displayed on the smart glasses, and the user can see a pop-up that says "20% off bestselling books."

[0624] The following hardware and software are used to realize this system.

[0625] Hardware: Smart glasses (with camera), user device (smartphone, etc.)

[0626] Software: OpenCV (preprocessing), TensorFlow Lite (image analysis), Amazon Comprehend (sentiment analysis)

[0627] An example of a prompt is as follows:

[0628] example:

[0629] As the user walks through a shopping mall, the camera captures a sign for "Bookstore A." "Bookstore A" is identified using TensorFlow Lite. Sentiment analysis is performed on the audio data, and the result is determined to be "positive." "Bookstore A" and "positive" are sent to the server. The server returns "discount information for best-selling books." A pop-up message saying "20% off best-selling books" appears on the smart glasses.

[0630] As described above, the present invention is a system that allows users to obtain information about surrounding facilities in real time and provides optimal information based on their emotions, thereby improving the user experience and providing more meaningful information.

[0631] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0632] Step 1:

[0633] The user wears a visual device. The camera of this visual device captures the image of the user's field of view in real time. The input is the image of the user's field of view, and the output is raw image data temporarily stored in the visual device. Specifically, the camera constantly follows the user's field of view and continuously captures the image.

[0634] Step 2:

[0635] The device performs preprocessing on the captured video data. This preprocessing includes noise removal and resolution adjustment. The input is the captured raw video data, and the output is clear video data with noise removed and resolution adjusted. Specifically, the OpenCV library is used to filter noise and optimize resolution.

[0636] Step 3:

[0637] The preprocessed video data is input into an artificial intelligence model (generative AI model) installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. The input is the preprocessed video data, and the output is the name and category of the identified facility. Specifically, TensorFlow Lite is used to perform pattern recognition based on the facility's exterior, sign, and logo.

[0638] Step 4:

[0639] The device uses an emotion engine to analyze the user's emotions. The emotion engine analyzes the user's voice, facial expression, and biometric information (such as heart rate and galvanic skin response). The input is the user's voice, facial expression, and biometric information, and the output is data indicating the user's emotional state. Specifically, it collects and analyzes data from Amazon Comprehend and biometric sensors.

[0640] Step 5:

[0641] The device sends facility information and emotion information as a request to the server. The input is the name and category of the identified facility and the user's emotion information, and the output is the request data sent to the server. Specifically, the data is communicated to the server using a RESTful API.

[0642] Step 6:

[0643] The server analyzes the request data it receives. The server retrieves detailed information about the facility (such as opening hours, ratings, and special offers) from the database and selects the most appropriate recommendation information taking into account emotional information. The input is the request data sent to the server, and the output is detailed information about the facility and the most appropriate information for the user. Specifically, it extracts data from AWS RDS and filters the information as needed.

[0644] Step 7:

[0645] The server returns the selected optimal information to the terminal. The input is detailed information and recommended information about the selected facility, and the output is the response data sent to the terminal. Specifically, the data is sent to the terminal again using the RESTful API.

[0646] Step 8:

[0647] The terminal analyzes the information received from the server and displays it in a format that the user can understand through the visual device. The input is detailed facility information and recommended information returned from the server, and the output is the information displayed on the visual device's display. Specifically, the information is displayed visually using the Unity engine, etc.

[0648] Through the above processing steps, the user can obtain information about surrounding facilities in real time through the visual device, and can obtain optimal information based on their emotions.

[0649] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0650] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0651] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0652] [Third embodiment]

[0653] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0654] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0655] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0656] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0657] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0658] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0659] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0660] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0661] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0662] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0663] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0664] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0665] The present invention aims to enable a user wearing a visual device to acquire and display information about surrounding facilities in real time. The specific processing content of the program and specific examples are described below.

[0666] First, the user wears the vision device. The camera in the vision device captures images of what is in the user's field of view in real time. This image is temporarily stored in the vision device.

[0667] The device then begins preprocessing the video data, which includes noise reduction and resolution adjustment, to improve the clarity of the image and subsequent analysis.

[0668] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[0669] Once the device identifies the facility information, it sends a request to the server based on that information. The request includes identification information such as the facility name and category.

[0670] The server searches the database based on the received request and retrieves detailed information about the facility, including opening hours, ratings, special offers, etc. The server then sends this information back to the terminal as a response.

[0671] The terminal analyzes the information received from the server and displays it in a format that can be understood by the user through the visual device. Specifically, detailed facility information is overlaid on the visual device's display, allowing the user to grasp detailed facility information in real time on-site.

[0672] Specific examples are given below.

[0673] Imagine a user is walking down the street and a cafe comes into view. At this time, the camera in the vision device captures video of the cafe. The device preprocesses the video and inputs it into the AI ​​model. The AI ​​model identifies the name of the cafe as "Café Delight." The device sends the identified information to the server, which retrieves "Café Delight's" business hours (8:00-20:00), rating (4.5 / 5), and special offers (10% off coupon available) from its database and sends them back to the device. Finally, the device displays this information on the vision device's display.

[0674] In this way, the present invention provides a system that allows a user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0675] The above is a specific description of the embodiment for carrying out the present invention.

[0676] The processing flow will be explained below.

[0677] Step 1:

[0678] The user wears the vision device. The camera in the vision device captures images of the user's field of view in real time. The image data is temporarily stored in the vision device.

[0679] Step 2:

[0680] The device begins pre-processing the captured video data, performing processes such as noise removal and resolution adjustment to make it easier for AI to analyze. This makes the video clearer and highlights the facility's unique features.

[0681] Step 3:

[0682] After preprocessing, the device inputs the video data into an artificial intelligence model. The AI ​​model analyzes the image and identifies the facility based on features such as the facility's exterior, sign, and logo contained in the video. For example, if a sign reads "Café Delight," it will perform character recognition.

[0683] Step 4:

[0684] The terminal sends a request to the server based on the identified facility information. The request includes information such as the name and category of the identified facility.

[0685] Step 5:

[0686] The server analyzes the received request and retrieves detailed information about the facility from the database, including opening hours, ratings, special offers, etc. The retrieved information is then sent back to the terminal as a response.

[0687] Step 6:

[0688] The device analyzes the detailed facility information received from the server and displays it on the visual device's display. Specifically, the facility's name, opening hours, ratings, and special offer information are overlaid in the user's field of vision.

[0689] Step 7:

[0690] The user can check the detailed information of the facility displayed on the visual device and take the necessary action. For example, when information about the cafe "Café Delight" is displayed, the user can check the opening hours and special offers and decide whether to visit.

[0691] The above is a detailed description of the specific processing steps of the program and the operations at each step.

[0692] Example 1

[0693] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0694] Conventional technologies lack a means for users to quickly and easily obtain detailed information about surrounding facilities in real time. This requires users to perform manual searches, which is not only inconvenient but also inefficient. Furthermore, conventional approaches are difficult to implement in situations where instantaneous on-site information is required. To address these issues, the present invention aims to provide a system that allows users to obtain facility information in real time using a visual device and instantly display detailed information.

[0695] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0696] In this invention, the server includes means for acquiring images of the user's field of view using a camera attached to a visual device worn by the user, means for preprocessing the acquired image data, means for performing image analysis using an artificial intelligence model for identifying facilities based on the preprocessed image data, means for requesting facility information acquired based on the image analysis from the server, and means for overlaying the detailed facility information returned from the server on the display of the visual device, thereby enabling the user to quickly acquire detailed information about surrounding facilities in real time through the visual device and efficiently utilize it.

[0697] A "user" is a person who utilizes the system of the present invention by wearing a visual device.

[0698] A "visual device" is a device worn by a user that includes a display and camera that displays to the field of view.

[0699] A "camera" is a photographing device attached to a visual device for capturing images within the user's field of vision.

[0700] "Video data" refers to digital data of images and videos captured by a camera.

[0701] "Preprocessing" refers to the step of performing processes such as noise removal and resolution adjustment on the video data.

[0702] An "artificial intelligence model" is an AI algorithm that uses pattern recognition technology to identify facilities contained within video footage.

[0703] "Image analysis" is the process of using artificial intelligence models to identify facility information from video data.

[0704] "Facility information" refers to data such as the name, category, and detailed information of the identified facility.

[0705] A "server" is a computer system that searches a database based on a request, obtains detailed facility information, and returns a response.

[0706] A "database" is an information aggregation system owned by a server that stores detailed facility information.

[0707] "Overlay display" refers to a function that displays information superimposed on the image displayed on the display of a visual device.

[0708] The present invention is a system that acquires information about surrounding facilities in real time while the user is wearing a visual device, and displays the details on the visual device. The processing contents of the program will be specifically described below.

[0709] First, the user wears a visual device. The device is equipped with a camera that captures images of the user's field of vision in real time. This image data is temporarily stored in the device.

[0710] Next, the device begins preprocessing the video data. This includes noise reduction and resolution adjustment. This makes the video clearer and improves the accuracy of subsequent analysis. This preprocessing typically uses an image processing library such as OpenCV.

[0711] Once preprocessed, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo. Frameworks such as TensorFlow and PyTorch are used for the AI ​​model.

[0712] Once the device identifies the facility information, it sends a request to the server based on that information. The request includes identifying information such as the facility's name and category. The server searches the database based on the received request and retrieves detailed information about the facility. This detailed information includes opening hours, ratings, special offers, and more. The server uses cloud services such as AWS and Google Cloud, and databases such as MySQL and PostgreSQL.

[0713] The acquired detailed facility information is sent back to the terminal as a response from the server. The terminal analyzes the information received from the server and displays it in a format that the user can understand through the visual device. Specifically, the detailed facility information is overlaid on the display of the visual device. This allows the user to grasp detailed facility information in real time on-site.

[0714] Here's a specific example: When a user is walking down the street and a cafe comes into view, the camera in the vision device captures video of the cafe. The device preprocesses the video and inputs it into an AI model. The AI ​​model identifies the name of the cafe as "Cafe Delight." The device sends the identified information to a server, which retrieves Cafe Delight's opening hours (8:00-20:00), rating (4.5 / 5), and special offers (10% off coupon available) from a database and sends them back to the device. Finally, the device displays this information on the vision device's display.

[0715] An example prompt is, "A user wears a visual device, and a camera captures video of the surroundings in real time. The video is preprocessed, and an AI model identifies the facility. Based on this facility information, a server provides detailed information, which is displayed on the visual device. As a specific example, please explain how detailed information about the cafe "Cafe Delight" is obtained and displayed to the user."

[0716] The above is a specific description of the embodiment of the present invention. With this system, a user can obtain information about surrounding facilities in real time and efficiently check detailed information.

[0717] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0718] Step 1:

[0719] The user wears a visual device, and a camera attached to the device captures images of what is in the user's field of view in real time.

[0720] Input: The image that the user sees.

[0721] Specific operation: The user wears the visual device on their head, and the camera continuously captures objects within their field of view.

[0722] Output: Captured video data.

[0723] Step 2:

[0724] The device pre-processes the captured video data, removing noise and adjusting the resolution to make the video clearer.

[0725] Input: The captured video data that is the output of step 1.

[0726] Data processing: Use an image processing library such as OpenCV to remove noise from the video data and improve its resolution.

[0727] Specific operation: The device filters the video data to reduce noise and adjust the image resolution appropriately.

[0728] Output: Pre-processed video data.

[0729] Step 3:

[0730] The device performs image analysis using artificial intelligence models to identify facilities based on pre-processed video data.

[0731] Input: The preprocessed video data that is the output of step 2.

[0732] Data Computing: Using AI frameworks such as TensorFlow and PyTorch, facilities are identified from video data.

[0733] Specific operation: Preprocessed video is input into the AI ​​model, and the facility's exterior, signs, logos, etc. are analyzed using a pattern recognition algorithm.

[0734] Output: Data with facility names and categories identified.

[0735] Step 4:

[0736] The device sends a request to the server based on the facility information it has identified, including identification information such as the facility name and category.

[0737] Input: Identified facility information, which is the output of Step 3.

[0738] Data processing: The facility identification information is converted into a data format for sending to the server as an HTTP request.

[0739] Specific operation: The terminal generates a request message containing facility information such as "Cafe Delight" and sends it to the server.

[0740] Output: The request data sent.

[0741] Step 5:

[0742] The server searches the database based on the received request and obtains detailed information about the facility.

[0743] Input: The request data sent in step 4.

[0744] Data Computation: Querying database systems such as MySQL or PostgreSQL to retrieve the requested facility details.

[0745] What happens: The server queries the database to get information about Cafe Delight, such as its hours of operation, ratings, and special offers.

[0746] Output: Detailed information about the retrieved facilities.

[0747] Step 6:

[0748] The terminal receives the detailed facility information returned from the server and displays it as an overlay on the display of the visual device.

[0749] Input: The retrieved facility details, which is the output of step 5.

[0750] Data processing: Preparing information in a format suitable for display on visual devices.

[0751] Specific operation: Detailed facility information (e.g., business hours 8:00-20:00, rating 4.5 / 5, 10% off coupon available) is sent to the user's visual device and overlaid on the screen.

[0752] Output: Facility details displayed on a visual aid.

[0753] The above are the specific processing steps of the program of this system, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0754] (Application example 1)

[0755] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0756] In modern society, users need to quickly and easily obtain information about nearby facilities while on the move. However, conventional methods require searching on a smartphone to obtain facility information, which is time-consuming and lacks real-time information. Furthermore, integrating information from different sources increases the burden on users, making it difficult to obtain information efficiently. Therefore, there is a need for an efficient system that allows users to instantly obtain detailed facility information on the spot using a visual device.

[0757] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0758] In this invention, the server includes means for acquiring video within the user's field of view using a camera attached to a visual device worn by the user, means for preprocessing the acquired video data, means for performing image analysis using a generative AI model for identifying facilities based on the preprocessed video data, means for requesting facility information acquired based on the image analysis from the server via a communication device, and means for overlaying and displaying detailed facility information returned from the server on the display device of the visual device, thereby enabling the user to acquire information about surrounding facilities in real time and efficiently check the detailed information.

[0759] A "visual device" is a device that can display visual information on a display when worn by a user.

[0760] A "camera" is a device that is attached to a visual device and has the function of capturing images that come into the user's field of vision.

[0761] "Preprocessing" refers to processing such as removing noise from acquired video data and adjusting resolution to improve the accuracy of subsequent analysis.

[0762] "Generative AI model" means an artificial intelligence model used to analyze video data and identify the category and specific name of a facility.

[0763] A "communication device" is a device for transmitting identified facility information to a server and receiving a response from the server.

[0764] The "display device" is a display that is mounted on the visual device and has the function of overlaying detailed facility information to the user.

[0765] A "prompt sentence" is an instruction sentence used by the generative AI model to identify a facility.

[0766] The present invention provides a system that acquires and displays surrounding facility information in real time using a visual device worn by a user. This system is composed of an image capture device, a preprocessing module, a generative AI model, a communication device, and a display device mounted on the visual device.

[0767] The vision device is equipped with a camera for capturing images in real time. The camera captures the image of the user's field of view and sends it to a pre-processing module, where processing such as noise reduction and resolution adjustment of the image data is performed. This pre-processing improves the accuracy of subsequent image analysis.

[0768] The preprocessed video data is then input into a generative AI model, which uses pattern recognition technology to identify facilities within the video. Specifically, the model analyzes the facility's exterior, signage, logo, and other features to identify the facility's category and specific name. A prompt is then generated to assist the AI ​​model in the identification process.

[0769] The communication device is responsible for transmitting the identified facility information to the server. The server receives this information and searches a database to obtain detailed information about the facility, including opening hours, ratings, special offers, etc. The server then sends these details back to the visual device via the communication device.

[0770] Finally, the display device displays the information returned from the server to the user. Specifically, detailed facility information is overlaid on the display of the visual device, allowing the user to check detailed information about surrounding facilities in real time through the visual device.

[0771] As a specific example, consider the flow from when a user is walking down the street and a cafe comes into view, to when a camera captures video of the cafe. The preprocessing module preprocesses this video and inputs it into the generative AI model. The generative AI model identifies the cafe, and the communication device sends this information to the server. The server receives the returned detailed information about the cafe (opening hours, ratings, special offers), which it then overlays on the display of the visual device.

[0772] The following are examples of prompt statements:

[0773] User looks at "Cafe" and gets real-time information overlay showing operation hours (8:00-20:00), rating (4.5 / 5), and a special coupon (10% off).

[0774] The above is a specific description of the embodiment of the present invention, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0775] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0776] Step 1:

[0777] When a user walks around town, the camera of the vision device captures the image that comes into the user's field of view in real time. The input is the user's field of view, and the output is the captured image data.

[0778] Step 2:

[0779] The captured video data is processed by the pre-processing module. Specifically, noise removal and resolution adjustment are performed. The input is the captured video data, and the output is pre-processed video data that has been subjected to noise removal and resolution adjustment.

[0780] Step 3:

[0781] The preprocessed video data is input into the generative AI model to identify the facility. Specifically, the model analyzes the facility's exterior, signage, logo, and other features, and identifies the facility's category and specific name using prompt text. The input is the preprocessed video data, and the output is the facility's identification information (facility name, category).

[0782] Step 4:

[0783] The identified facility information is sent to the server by the communication device. Specifically, the facility name and category information are constructed as an HTTP request and sent to the server. The input at this time is the facility identification information, and the output is a request to the server.

[0784] Step 5:

[0785] The server searches the database based on the received request and retrieves detailed information about the facility (such as opening hours, ratings, special offers, etc.) The input is the request to the server, and the output is the detailed information about the facility.

[0786] Step 6:

[0787] The server returns the acquired facility details to the visual device via the communication device. Specifically, the facility details are constructed as an HTTP response and sent to the visual device. The input at this time is the facility details, and the output is the response from the server.

[0788] Step 7:

[0789] The display device of the visual device overlays the detailed facility information received from the server. Specifically, the detailed facility information is displayed on the display in an appropriate position according to the user's field of view. The input at this time is the response from the server, and the output is the detailed facility information overlaid on the user's field of view.

[0790] The above is a description of the specific processing steps in the system for implementing the present invention, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0791] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0792] The present invention is a system that allows a user to wear a visual device and acquire and display information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. The specific processing content of the program and concrete examples are explained below.

[0793] The user wears the vision device. The camera in the vision device captures images of what is in the user's field of view in real time. This image is temporarily stored in the vision device.

[0794] The device then begins preprocessing the video data, which includes noise reduction and resolution adjustment, to improve the clarity of the image and subsequent analysis.

[0795] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[0796] The visual device is equipped with an emotion engine to recognize the user's emotions. The emotion engine recognizes the user's emotional state by analyzing the user's voice, facial expressions, and biometric information (heart rate, skin galvanic response, etc.). For example, it can detect when the user is feeling stressed or curious.

[0797] The device sends a request to the server based on the facility information and emotion information. The request includes the name and category of the identified facility and the user's emotion information.

[0798] The server analyzes the received request and retrieves detailed information about the facility from the database, including opening hours, ratings, special offers, etc. The server then considers the user's emotional information and selects the information that best suits the user's current emotional state and returns it to the terminal.

[0799] The device analyzes the information received from the server and displays it in a format that the user can understand through their visual device. For example, if the user's emotions are positive, highly rated menu items and special offers will be highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information will be displayed.

[0800] Specific examples are given below.

[0801] Imagine a user is walking down the street and a cafe comes into view. The vision device's camera captures the image, and the device pre-processes the image. The AI ​​model identifies the name of the cafe as "Café Delight." The emotion engine analyzes the user's emotion and determines that the emotional state is positive, for example. The device sends the facility information and emotion information to the server, which then returns detailed information about "Café Delight" along with recommended menu items and special offers. Finally, the device displays this information on the vision device's display, allowing the user to obtain the most relevant information in real time.

[0802] In this way, the present invention is a system that allows users to obtain information about surrounding facilities in real time and provides optimal information based on their emotions, improving the user experience and providing more meaningful information.

[0803] The above is a specific description of the embodiment for carrying out the present invention.

[0804] The processing flow will be explained below.

[0805] Step 1:

[0806] The user wears the vision device. The camera in the vision device captures images of the user's field of view in real time. This image data is temporarily stored in the vision device.

[0807] Step 2:

[0808] The device begins pre-processing the captured video data, which includes noise removal and resolution adjustment, making the video clearer and easier for AI to analyze.

[0809] Step 3:

[0810] After preprocessing, the device inputs the video data into an artificial intelligence model, which then analyzes the images and identifies the facility based on its exterior, signage, logo, and other features contained in the video.

[0811] Step 4:

[0812] To recognize the user's emotions, the emotion engine of the vision device analyzes voice, facial expressions, and biometric information (e.g., heart rate, galvanic skin response). The emotion engine identifies the user's current emotional state.

[0813] Step 5:

[0814] The terminal sends a request to the server based on the facility information and the user's emotion information. The request includes the facility name, category, and emotion information.

[0815] Step 6:

[0816] The server analyzes the received request and retrieves detailed information about the facility (e.g., opening hours, ratings, special offers) from the database. The server also selects the most appropriate additional information based on the emotional information.

[0817] Step 7:

[0818] The server returns the acquired detailed information and selected additional information to the terminal as a response.

[0819] Step 8:

[0820] The device analyzes the information received from the server and displays it in a format that the user can perceive through their visual device. For example, if the user's emotions are positive, highly rated menu items and special offers will be highlighted, and if the user is showing signs of stress, suggestions for relaxation spots and coupon information will be displayed.

[0821] Step 9:

[0822] The user can check the detailed information of the facility and additional emotional information displayed on the visual device and take the necessary action. For example, when information about the cafe "Café Delight" is displayed, the user can check the opening hours and special offers and decide whether to visit.

[0823] The above is a detailed description of the specific processing steps of the program and the operations at each step.

[0824] Example 2

[0825] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0826] Conventional information acquisition systems using visual devices have difficulty providing optimal information according to the user's emotional state and environment. Furthermore, the acquired information is merely mechanical, which is insufficient for improving the user experience. Specifically, there is a lack of information presentation tailored to the user's stressful state or positive feelings toward a particular facility.

[0827] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0828] In this invention, the server includes means for acquiring video within the user's field of view, means for preprocessing the acquired video data, means for performing image analysis using an artificial intelligence model for identifying facilities based on the preprocessed video data, means including an emotion engine for analyzing the user's voice, facial expression, and biological information to recognize the user's emotional state, means for requesting facility information and emotion information acquired based on the image analysis and emotion analysis from the server, and means for displaying detailed facility information and optimal information based on the user's emotional state returned from the server on the display of the visual device, thereby making it possible to provide optimal facility information according to the user's emotional state and visual information.

[0829] A "user" is a person who wears and uses the visual device of the present invention.

[0830] A "vision device" is a device that includes a camera that captures images within the user's field of view and a display that displays that information to the user.

[0831] A "camera" is a device attached to a visual device that captures images in the user's field of view in real time.

[0832] "Video data" refers to image information captured by a camera and seen by a user.

[0833] "Preprocessing" refers to the process of removing noise and adjusting resolution of captured video data.

[0834] "Artificial Intelligence Model" means a machine learning model used to identify facilities using pre-processed video data.

[0835] "Image analysis" is a technology that uses an artificial intelligence model to extract the characteristics of a facility from video data and identify the facility.

[0836] An "emotion engine" is a technology that analyzes a user's voice, facial expression, and biological information to recognize the user's emotional state.

[0837] "Facility information" is detailed information such as the name and category of the identified facility.

[0838] A "server" is a computer system that retrieves information from a database based on a request and responds to a terminal.

[0839] A "display" is a device that is installed in a visual device and visually displays information obtained from a server or a terminal to a user.

[0840] "Biometric information" is data that indicates the user's physical condition, such as their heart rate and skin galvanic response.

[0841] A "request" is an inquiry that includes facility information and emotion information and is sent from a terminal to a server.

[0842] "Detailed information" is information such as opening hours, ratings, and special offers about a specific facility that is returned from the server.

[0843] The present invention is a system in which a user wears a visual device and acquires and displays information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[0844] Hardware and software used

[0845] 1. Visual Apparatus

[0846] The visual device worn by the user has a built-in camera and display. The camera captures images of the user's field of view in real time and temporarily stores them in the visual device's memory.

[0847] 2. Terminal

[0848] The device receives video data from the visual device, performs pre-processing and image analysis using artificial intelligence models, and also includes an emotion engine that recognizes the user's emotional state.

[0849] 3. Server

[0850] The server receives the request from the device, retrieves detailed information about the facility from the database, and returns the retrieved information to the device in an optimal format based on the user's emotional state.

[0851] Specific software used includes:

[0852] Artificial intelligence models: TensorFlow, PyTorch, etc.

[0853] Database: MySQL, PostgreSQL, etc.

[0854] Details of data processing and calculation

[0855] 1. Video capture by the user's visual device

[0856] The camera of the vision device captures images of what is in the user's field of view and temporarily stores this data within the vision device.

[0857] 2. Preprocessing of video data by the device

[0858] As soon as the device receives the video data from the vision device, it starts preprocessing, which includes noise removal and resolution adjustment, to enable subsequent image analysis to be performed with greater accuracy.

[0859] 3. Image analysis using AI models

[0860] The pre-processed video data is then fed into an AI model, which uses pattern recognition to identify the facility and identify its name and category based on its exterior, signage, logo, and other characteristics.

[0861] 4. Emotion analysis using an emotion engine

[0862] The emotion engine built into the visual device analyzes the user's voice, facial expressions, heart rate, and skin galvanic response to determine the user's emotional state.

[0863] 5. Sending a request to the server

[0864] The terminal integrates the identified facility information with the emotion information and sends a request to the server. This request includes the specific facility name, category, and emotion information.

[0865] 6. Server retrieves and returns details

[0866] The server analyzes the request received from the device, retrieves detailed information about the facility from the database, and then selects the most appropriate information based on the user's emotional state and returns it to the device.

[0867] 7. Displaying Information on a Terminal

[0868] The device receives detailed information about the facility from the server and displays it on the display of the visual device. If the user is in a positive emotional state, highly rated menu items and special offers are highlighted, while if the user is feeling stressed, suggestions for relaxation spots and coupon information are displayed.

[0869] Specific examples

[0870] Consider an example where a user is walking down the street and a cafe comes into view. The vision device's camera captures video of the cafe, and the device pre-processes the video. The AI ​​model identifies the cafe, and the emotion engine analyzes the user's emotions. For example, if the device determines that the user is in a positive emotional state, it sends the facility information and emotion information to the server. The server returns detailed information about the cafe, along with recommended menu items and special offers, which the device then displays on its screen. The user can check this information in real time and make the best choice.

[0871] Prompt Sentence Examples

[0872] User: When I use a visual aid to find a cafe, how exactly does the system process and display the information?

[0873] The above is a specific embodiment of the present invention, which allows the user to obtain information about surrounding facilities in real time, and furthermore, to obtain optimal information based on emotions.

[0874] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0875] Step 1:

[0876] The user wears the visual device.

[0877] How it works: The user puts on the vision device, activating it. The camera begins capturing video within the user's field of view.

[0878] Input: The image that the user sees.

[0879] Output: Captured video data.

[0880] Step 2:

[0881] The terminal receives the captured video data from the visual device.

[0882] Operation: The visual device's camera captures video data and sends it to the device, where it is temporarily saved.

[0883] Input: Captured video data.

[0884] Output: Received video data.

[0885] Step 3:

[0886] The terminal pre-processes the video data.

[0887] Operation: Performs noise reduction and resolution adjustment on the video data received by the device.

[0888] Input: Received video data.

[0889] Output: Pre-processed video data with noise removed and resolution adjusted.

[0890] Step 4:

[0891] The device inputs the preprocessed video data into an artificial intelligence model to identify the facility.

[0892] How it works: Preprocessed video data is fed into an AI model (e.g., TensorFlow, PyTorch) that uses pattern recognition techniques to identify facility names and categories.

[0893] Input: Preprocessed video data.

[0894] Output: Identified facility information (name, category).

[0895] Step 5:

[0896] The emotion engine of the visual device analyzes the user's emotions.

[0897] How it works: The emotion engine built into the visual device analyzes the user's voice, facial expressions, heart rate, and galvanic skin response to determine the user's emotional state.

[0898] Input: User's voice, facial expressions, heart rate, and galvanic skin response.

[0899] Output: The perceived emotional state of the user.

[0900] Step 6:

[0901] The terminal requests facility information and emotion information from the server.

[0902] Operation: The device compiles the identified facility information and analyzed emotion information and sends a request to the server.

[0903] Input: Identified facility information and emotion information.

[0904] Output: The request sent to the server.

[0905] Step 7:

[0906] The server receives the request, retrieves the details and sends them back to the device.

[0907] Operation: The server analyzes the received request and retrieves detailed information about the facility from its own database. Taking into account the user's emotional information, it selects the most appropriate information and returns it to the device.

[0908] Input: The request sent to the server.

[0909] Output: Detailed information about the selected facility.

[0910] Step 8:

[0911] The terminal displays the information on the visual device's display.

[0912] Operation: The device formats the facility information received from the server and displays it on the visual device's display. If the user is in a positive emotional state, highly rated menu items and special offers are highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information are displayed.

[0913] Input: Facility details received from the server.

[0914] Output: Information displayed on the visual device's display.

[0915] (Application example 2)

[0916] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0917] Conventional information provision systems using visual devices only allow users to visually obtain information about surrounding facilities, and have the problem of not being able to provide optimal information based on the user's current emotional state. Specifically, even if a user is feeling stressed, for example, they may not be provided with information about facilities where they can relax or special offers, which could result in a less meaningful shopping experience.

[0918] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0919] In this invention, the server includes means for selecting and returning optimal information taking into consideration the user's emotional information, means for preprocessing video data acquired by a camera attached to the visual device, and means for performing image analysis using an artificial intelligence model that identifies facilities based on the preprocessed video data. This allows the user to obtain recommended information based on their emotional state while obtaining information about surrounding facilities through the visual device in real time.

[0920] A "visual device" is a device equipped with a display and camera worn by the user, and has the function of capturing images within the user's field of vision and displaying the information.

[0921] A "camera" is an image capturing device attached to a visual device that captures images in the user's field of view in real time.

[0922] "Preprocessing" refers to processes such as removing noise from acquired video data and adjusting resolution to improve the accuracy of subsequent analysis.

[0923] An "artificial intelligence model" is a system that uses pattern recognition technology to identify facilities based on their exterior appearance, signage, logo, and other characteristics.

[0924] "Image analysis" refers to the process of identifying facilities using artificial intelligence models based on pre-processed video data.

[0925] "Emotional information" refers to data relating to the user's emotional state obtained by analyzing the user's voice, facial expression, and biometric information.

[0926] The "server" is a computer system that retrieves detailed facility information and recommended information from a database, and selects and returns the most appropriate information to the user, taking into account emotional information.

[0927] "Detailed information" refers to specific information about a facility, such as the facility's opening hours, ratings, and special offers.

[0928] An "emotion analysis engine" is a system that analyzes a user's voice, facial expression, and biometric information to identify their emotional state.

[0929] "Recommended information" refers to information on products, services, locations, etc. that are optimal for a user and are selected with consideration given to emotional information.

[0930] MODE FOR CARRYING OUT THE INVENTION

[0931] The present invention is a system in which a user wears a visual device and acquires and displays information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. A specific embodiment of this system will be described.

[0932] A user wears a visual device such as smart glasses, and a camera attached to the visual device captures images of what the user sees in real time. The video data captured by the camera is temporarily stored in the visual device. The device then begins preprocessing the video data. Preprocessing includes noise reduction and resolution adjustment, which improves the clarity of the image and the accuracy of subsequent analysis.

[0933] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[0934] Meanwhile, the visual device is equipped with an emotion engine to recognize the user's emotions. The emotion engine recognizes the user's emotional state by analyzing the user's voice, facial expressions, and biometric information (heart rate, skin galvanic response, etc.). For example, it can detect when the user is feeling stressed or curious.

[0935] Facility information and emotional information are sent from the device to the server as a request. The request includes the name and category of the identified facility, as well as the user's emotional information. The server analyzes the received request and retrieves detailed information about the facility from its database. This information includes opening hours, ratings, and special offers. The server then takes the emotional information into consideration and selects the information that best suits the user's current emotional state, which it then sends back to the device. For example, if the user's emotions are positive, the facility's highly rated menu items and special offers are highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information are also displayed.

[0936] As a specific example of use, imagine a user is walking through a shopping mall and the camera captures a sign for "Bookstore A." TensorFlow Lite identifies "Bookstore A" and performs sentiment analysis on the audio data. If it determines the sentiment is "positive," it sends "Bookstore A" and "positive" to the server. Information returned from the server, such as "discount information for bestselling books," is displayed on the smart glasses, and the user can see a pop-up that says "20% off bestselling books."

[0937] The following hardware and software are used to realize this system.

[0938] Hardware: Smart glasses (with camera), user device (smartphone, etc.)

[0939] Software: OpenCV (preprocessing), TensorFlow Lite (image analysis), Amazon Comprehend (sentiment analysis)

[0940] An example of a prompt is as follows:

[0941] example:

[0942] As the user walks through a shopping mall, the camera captures a sign for "Bookstore A." "Bookstore A" is identified using TensorFlow Lite. Sentiment analysis is performed on the audio data, and the result is determined to be "positive." "Bookstore A" and "positive" are sent to the server. The server returns "discount information for best-selling books." A pop-up message saying "20% off best-selling books" appears on the smart glasses.

[0943] As described above, the present invention is a system that allows users to obtain information about surrounding facilities in real time and provides optimal information based on their emotions, thereby improving the user experience and providing more meaningful information.

[0944] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0945] Step 1:

[0946] The user wears a visual device. The camera of this visual device captures the image of the user's field of view in real time. The input is the image of the user's field of view, and the output is raw image data temporarily stored in the visual device. Specifically, the camera constantly follows the user's field of view and continuously captures the image.

[0947] Step 2:

[0948] The device performs preprocessing on the captured video data. This preprocessing includes noise removal and resolution adjustment. The input is the captured raw video data, and the output is clear video data with noise removed and resolution adjusted. Specifically, the OpenCV library is used to filter noise and optimize resolution.

[0949] Step 3:

[0950] The preprocessed video data is input into an artificial intelligence model (generative AI model) installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. The input is the preprocessed video data, and the output is the name and category of the identified facility. Specifically, TensorFlow Lite is used to perform pattern recognition based on the facility's exterior, sign, and logo.

[0951] Step 4:

[0952] The device uses an emotion engine to analyze the user's emotions. The emotion engine analyzes the user's voice, facial expression, and biometric information (such as heart rate and galvanic skin response). The input is the user's voice, facial expression, and biometric information, and the output is data indicating the user's emotional state. Specifically, it collects and analyzes data from Amazon Comprehend and biometric sensors.

[0953] Step 5:

[0954] The device sends facility information and emotion information as a request to the server. The input is the name and category of the identified facility and the user's emotion information, and the output is the request data sent to the server. Specifically, the data is communicated to the server using a RESTful API.

[0955] Step 6:

[0956] The server analyzes the request data it receives. The server retrieves detailed information about the facility (such as opening hours, ratings, and special offers) from the database and selects the most appropriate recommendation information taking into account emotional information. The input is the request data sent to the server, and the output is detailed information about the facility and the most appropriate information for the user. Specifically, it extracts data from AWS RDS and filters the information as needed.

[0957] Step 7:

[0958] The server returns the selected optimal information to the terminal. The input is detailed information and recommended information about the selected facility, and the output is the response data sent to the terminal. Specifically, the data is sent to the terminal again using the RESTful API.

[0959] Step 8:

[0960] The terminal analyzes the information received from the server and displays it in a format that the user can understand through the visual device. The input is detailed facility information and recommended information returned from the server, and the output is the information displayed on the visual device's display. Specifically, the information is displayed visually using the Unity engine, etc.

[0961] Through the above processing steps, the user can obtain information about surrounding facilities in real time through the visual device, and can obtain optimal information based on their emotions.

[0962] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0963] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0964] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0965] [Fourth embodiment]

[0966] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0967] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0968] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0969] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0970] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0971] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0972] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0973] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0974] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0975] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0976] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0977] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0978] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0979] The present invention aims to enable a user wearing a visual device to acquire and display information about surrounding facilities in real time. The specific processing content of the program and specific examples are described below.

[0980] First, the user wears the vision device. The camera in the vision device captures images of what is in the user's field of view in real time. This image is temporarily stored in the vision device.

[0981] The device then begins preprocessing the video data, which includes noise reduction and resolution adjustment, to improve the clarity of the image and subsequent analysis.

[0982] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[0983] Once the device identifies the facility information, it sends a request to the server based on that information. The request includes identification information such as the facility name and category.

[0984] The server searches the database based on the received request and retrieves detailed information about the facility, including opening hours, ratings, special offers, etc. The server then sends this information back to the terminal as a response.

[0985] The terminal analyzes the information received from the server and displays it in a format that can be understood by the user through the visual device. Specifically, detailed facility information is overlaid on the visual device's display, allowing the user to grasp detailed facility information in real time on-site.

[0986] Specific examples are given below.

[0987] Imagine a user is walking down the street and a cafe comes into view. At this time, the camera in the vision device captures video of the cafe. The device preprocesses the video and inputs it into the AI ​​model. The AI ​​model identifies the name of the cafe as "Café Delight." The device sends the identified information to the server, which retrieves "Café Delight's" business hours (8:00-20:00), rating (4.5 / 5), and special offers (10% off coupon available) from its database and sends them back to the device. Finally, the device displays this information on the vision device's display.

[0988] In this way, the present invention provides a system that allows a user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[0989] The above is a specific description of the embodiment for carrying out the present invention.

[0990] The processing flow will be explained below.

[0991] Step 1:

[0992] The user wears the vision device. The camera in the vision device captures images of the user's field of view in real time. The image data is temporarily stored in the vision device.

[0993] Step 2:

[0994] The device begins pre-processing the captured video data, performing processes such as noise removal and resolution adjustment to make it easier for AI to analyze. This makes the video clearer and highlights the facility's unique features.

[0995] Step 3:

[0996] After preprocessing, the device inputs the video data into an artificial intelligence model. The AI ​​model analyzes the image and identifies the facility based on features such as the facility's exterior, sign, and logo contained in the video. For example, if a sign reads "Café Delight," it will perform character recognition.

[0997] Step 4:

[0998] The terminal sends a request to the server based on the identified facility information. The request includes information such as the name and category of the identified facility.

[0999] Step 5:

[1000] The server analyzes the received request and retrieves detailed information about the facility from the database, including opening hours, ratings, special offers, etc. The retrieved information is then sent back to the terminal as a response.

[1001] Step 6:

[1002] The device analyzes the detailed facility information received from the server and displays it on the visual device's display. Specifically, the facility's name, opening hours, ratings, and special offer information are overlaid in the user's field of vision.

[1003] Step 7:

[1004] The user can check the detailed information of the facility displayed on the visual device and take the necessary action. For example, when information about the cafe "Café Delight" is displayed, the user can check the opening hours and special offers and decide whether to visit.

[1005] The above is a detailed description of the specific processing steps of the program and the operations at each step.

[1006] Example 1

[1007] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1008] Conventional technologies lack a means for users to quickly and easily obtain detailed information about surrounding facilities in real time. This requires users to perform manual searches, which is not only inconvenient but also inefficient. Furthermore, conventional approaches are difficult to implement in situations where instantaneous on-site information is required. To address these issues, the present invention aims to provide a system that allows users to obtain facility information in real time using a visual device and instantly display detailed information.

[1009] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1010] In this invention, the server includes means for acquiring images of the user's field of view using a camera attached to a visual device worn by the user, means for preprocessing the acquired image data, means for performing image analysis using an artificial intelligence model for identifying facilities based on the preprocessed image data, means for requesting facility information acquired based on the image analysis from the server, and means for overlaying the detailed facility information returned from the server on the display of the visual device, thereby enabling the user to quickly acquire detailed information about surrounding facilities in real time through the visual device and efficiently utilize it.

[1011] A "user" is a person who utilizes the system of the present invention by wearing a visual device.

[1012] A "visual device" is a device worn by a user that includes a display and camera that displays to the field of view.

[1013] A "camera" is a photographing device attached to a visual device for capturing images within the user's field of vision.

[1014] "Video data" refers to digital data of images and videos captured by a camera.

[1015] "Preprocessing" refers to the step of performing processes such as noise removal and resolution adjustment on the video data.

[1016] An "artificial intelligence model" is an AI algorithm that uses pattern recognition technology to identify facilities contained within video footage.

[1017] "Image analysis" is the process of using artificial intelligence models to identify facility information from video data.

[1018] "Facility information" refers to data such as the name, category, and detailed information of the identified facility.

[1019] A "server" is a computer system that searches a database based on a request, obtains detailed facility information, and returns a response.

[1020] A "database" is an information aggregation system owned by a server that stores detailed facility information.

[1021] "Overlay display" refers to a function that displays information superimposed on the image displayed on the display of a visual device.

[1022] The present invention is a system that acquires information about surrounding facilities in real time while the user is wearing a visual device, and displays the details on the visual device. The processing contents of the program will be specifically described below.

[1023] First, the user wears a visual device. The device is equipped with a camera that captures images of the user's field of vision in real time. This image data is temporarily stored in the device.

[1024] Next, the device begins preprocessing the video data. This includes noise reduction and resolution adjustment. This makes the video clearer and improves the accuracy of subsequent analysis. This preprocessing typically uses an image processing library such as OpenCV.

[1025] Once preprocessed, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo. Frameworks such as TensorFlow and PyTorch are used for the AI ​​model.

[1026] Once the device identifies the facility information, it sends a request to the server based on that information. The request includes identifying information such as the facility's name and category. The server searches the database based on the received request and retrieves detailed information about the facility. This detailed information includes opening hours, ratings, special offers, and more. The server uses cloud services such as AWS and Google Cloud, and databases such as MySQL and PostgreSQL.

[1027] The acquired detailed facility information is sent back to the terminal as a response from the server. The terminal analyzes the information received from the server and displays it in a format that the user can understand through the visual device. Specifically, the detailed facility information is overlaid on the display of the visual device. This allows the user to grasp detailed facility information in real time on-site.

[1028] Here's a specific example: When a user is walking down the street and a cafe comes into view, the camera in the vision device captures video of the cafe. The device preprocesses the video and inputs it into an AI model. The AI ​​model identifies the name of the cafe as "Cafe Delight." The device sends the identified information to a server, which retrieves Cafe Delight's opening hours (8:00-20:00), rating (4.5 / 5), and special offers (10% off coupon available) from a database and sends them back to the device. Finally, the device displays this information on the vision device's display.

[1029] An example prompt is, "A user wears a visual device, and a camera captures video of the surroundings in real time. The video is preprocessed, and an AI model identifies the facility. Based on this facility information, a server provides detailed information, which is displayed on the visual device. As a specific example, please explain how detailed information about the cafe "Cafe Delight" is obtained and displayed to the user."

[1030] The above is a specific description of the embodiment of the present invention. With this system, a user can obtain information about surrounding facilities in real time and efficiently check detailed information.

[1031] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1032] Step 1:

[1033] The user wears a visual device, and a camera attached to the device captures images of what is in the user's field of view in real time.

[1034] Input: The image that the user sees.

[1035] Specific operation: The user wears the visual device on their head, and the camera continuously captures objects within their field of view.

[1036] Output: Captured video data.

[1037] Step 2:

[1038] The device pre-processes the captured video data, removing noise and adjusting the resolution to make the video clearer.

[1039] Input: The captured video data that is the output of step 1.

[1040] Data processing: Use an image processing library such as OpenCV to remove noise from the video data and improve its resolution.

[1041] Specific operation: The device filters the video data to reduce noise and adjust the image resolution appropriately.

[1042] Output: Pre-processed video data.

[1043] Step 3:

[1044] The device performs image analysis using artificial intelligence models to identify facilities based on pre-processed video data.

[1045] Input: The preprocessed video data that is the output of step 2.

[1046] Data Computing: Using AI frameworks such as TensorFlow and PyTorch, facilities are identified from video data.

[1047] Specific operation: Preprocessed video is input into the AI ​​model, and the facility's exterior, signs, logos, etc. are analyzed using a pattern recognition algorithm.

[1048] Output: Data with facility names and categories identified.

[1049] Step 4:

[1050] The device sends a request to the server based on the facility information it has identified, including identification information such as the facility name and category.

[1051] Input: Identified facility information, which is the output of Step 3.

[1052] Data processing: The facility identification information is converted into a data format for sending to the server as an HTTP request.

[1053] Specific operation: The terminal generates a request message containing facility information such as "Cafe Delight" and sends it to the server.

[1054] Output: The request data sent.

[1055] Step 5:

[1056] The server searches the database based on the received request and obtains detailed information about the facility.

[1057] Input: The request data sent in step 4.

[1058] Data Computation: Querying database systems such as MySQL or PostgreSQL to retrieve the requested facility details.

[1059] What happens: The server queries the database to get information about Cafe Delight, such as its hours of operation, ratings, and special offers.

[1060] Output: Detailed information about the retrieved facilities.

[1061] Step 6:

[1062] The terminal receives the detailed facility information returned from the server and displays it as an overlay on the display of the visual device.

[1063] Input: The retrieved facility details, which is the output of step 5.

[1064] Data processing: Preparing information in a format suitable for display on visual devices.

[1065] Specific operation: Detailed facility information (e.g., business hours 8:00-20:00, rating 4.5 / 5, 10% off coupon available) is sent to the user's visual device and overlaid on the screen.

[1066] Output: Facility details displayed on a visual aid.

[1067] The above are the specific processing steps of the program of this system, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[1068] (Application example 1)

[1069] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1070] In modern society, users need to quickly and easily obtain information about nearby facilities while on the move. However, conventional methods require searching on a smartphone to obtain facility information, which is time-consuming and lacks real-time information. Furthermore, integrating information from different sources increases the burden on users, making it difficult to obtain information efficiently. Therefore, there is a need for an efficient system that allows users to instantly obtain detailed facility information on the spot using a visual device.

[1071] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1072] In this invention, the server includes means for acquiring video within the user's field of view using a camera attached to a visual device worn by the user, means for preprocessing the acquired video data, means for performing image analysis using a generative AI model for identifying facilities based on the preprocessed video data, means for requesting facility information acquired based on the image analysis from the server via a communication device, and means for overlaying and displaying detailed facility information returned from the server on the display device of the visual device, thereby enabling the user to acquire information about surrounding facilities in real time and efficiently check the detailed information.

[1073] A "visual device" is a device that can display visual information on a display when worn by a user.

[1074] A "camera" is a device that is attached to a visual device and has the function of capturing images that come into the user's field of vision.

[1075] "Preprocessing" refers to processing such as removing noise from acquired video data and adjusting resolution to improve the accuracy of subsequent analysis.

[1076] "Generative AI model" means an artificial intelligence model used to analyze video data and identify the category and specific name of a facility.

[1077] A "communication device" is a device for transmitting identified facility information to a server and receiving a response from the server.

[1078] The "display device" is a display that is mounted on the visual device and has the function of overlaying detailed facility information to the user.

[1079] A "prompt sentence" is an instruction sentence used by the generative AI model to identify a facility.

[1080] The present invention provides a system that acquires and displays surrounding facility information in real time using a visual device worn by a user. This system is composed of an image capture device, a preprocessing module, a generative AI model, a communication device, and a display device mounted on the visual device.

[1081] The vision device is equipped with a camera for capturing images in real time. The camera captures the image of the user's field of view and sends it to a pre-processing module, where processing such as noise reduction and resolution adjustment of the image data is performed. This pre-processing improves the accuracy of subsequent image analysis.

[1082] The preprocessed video data is then input into a generative AI model, which uses pattern recognition technology to identify facilities within the video. Specifically, the model analyzes the facility's exterior, signage, logo, and other features to identify the facility's category and specific name. A prompt is then generated to assist the AI ​​model in the identification process.

[1083] The communication device is responsible for transmitting the identified facility information to the server. The server receives this information and searches a database to obtain detailed information about the facility, including opening hours, ratings, special offers, etc. The server then sends these details back to the visual device via the communication device.

[1084] Finally, the display device displays the information returned from the server to the user. Specifically, detailed facility information is overlaid on the display of the visual device, allowing the user to check detailed information about surrounding facilities in real time through the visual device.

[1085] As a specific example, consider the flow from when a user is walking down the street and a cafe comes into view, to when a camera captures video of the cafe. The preprocessing module preprocesses this video and inputs it into the generative AI model. The generative AI model identifies the cafe, and the communication device sends this information to the server. The server receives the returned detailed information about the cafe (opening hours, ratings, special offers), which it then overlays on the display of the visual device.

[1086] The following are examples of prompt statements:

[1087] User looks at "Cafe" and gets real-time information overlay showing operation hours (8:00-20:00), rating (4.5 / 5), and a special coupon (10% off).

[1088] The above is a specific description of the embodiment of the present invention, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[1089] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1090] Step 1:

[1091] When a user walks around town, the camera of the vision device captures the image that comes into the user's field of view in real time. The input is the user's field of view, and the output is the captured image data.

[1092] Step 2:

[1093] The captured video data is processed by the pre-processing module. Specifically, noise removal and resolution adjustment are performed. The input is the captured video data, and the output is pre-processed video data that has been subjected to noise removal and resolution adjustment.

[1094] Step 3:

[1095] The preprocessed video data is input into the generative AI model to identify the facility. Specifically, the model analyzes the facility's exterior, signage, logo, and other features, and identifies the facility's category and specific name using prompt text. The input is the preprocessed video data, and the output is the facility's identification information (facility name, category).

[1096] Step 4:

[1097] The identified facility information is sent to the server by the communication device. Specifically, the facility name and category information are constructed as an HTTP request and sent to the server. The input at this time is the facility identification information, and the output is a request to the server.

[1098] Step 5:

[1099] The server searches the database based on the received request and retrieves detailed information about the facility (such as opening hours, ratings, special offers, etc.) The input is the request to the server, and the output is the detailed information about the facility.

[1100] Step 6:

[1101] The server returns the acquired facility details to the visual device via the communication device. Specifically, the facility details are constructed as an HTTP response and sent to the visual device. The input at this time is the facility details, and the output is the response from the server.

[1102] Step 7:

[1103] The display device of the visual device overlays the detailed facility information received from the server. Specifically, the detailed facility information is displayed on the display in an appropriate position according to the user's field of view. The input at this time is the response from the server, and the output is the detailed facility information overlaid on the user's field of view.

[1104] The above is a description of the specific processing steps in the system for implementing the present invention, which allows the user to obtain information about surrounding facilities in real time and efficiently check detailed information.

[1105] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1106] The present invention is a system that allows a user to wear a visual device and acquire and display information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. The specific processing content of the program and concrete examples are explained below.

[1107] The user wears the vision device. The camera in the vision device captures images of what is in the user's field of view in real time. This image is temporarily stored in the vision device.

[1108] The device then begins preprocessing the video data, which includes noise reduction and resolution adjustment, to improve the clarity of the image and subsequent analysis.

[1109] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[1110] The visual device is equipped with an emotion engine to recognize the user's emotions. The emotion engine recognizes the user's emotional state by analyzing the user's voice, facial expressions, and biometric information (heart rate, skin galvanic response, etc.). For example, it can detect when the user is feeling stressed or curious.

[1111] The device sends a request to the server based on the facility information and emotion information. The request includes the name and category of the identified facility and the user's emotion information.

[1112] The server analyzes the received request and retrieves detailed information about the facility from the database, including opening hours, ratings, special offers, etc. The server then considers the user's emotional information and selects the information that best suits the user's current emotional state and returns it to the terminal.

[1113] The device analyzes the information received from the server and displays it in a format that the user can understand through their visual device. For example, if the user's emotions are positive, highly rated menu items and special offers will be highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information will be displayed.

[1114] Specific examples are given below.

[1115] Imagine a user is walking down the street and a cafe comes into view. The vision device's camera captures the image, and the device pre-processes the image. The AI ​​model identifies the name of the cafe as "Café Delight." The emotion engine analyzes the user's emotion and determines that the emotional state is positive, for example. The device sends the facility information and emotion information to the server, which then returns detailed information about "Café Delight" along with recommended menu items and special offers. Finally, the device displays this information on the vision device's display, allowing the user to obtain the most relevant information in real time.

[1116] In this way, the present invention is a system that allows users to obtain information about surrounding facilities in real time and provides optimal information based on their emotions, improving the user experience and providing more meaningful information.

[1117] The above is a specific description of the embodiment for carrying out the present invention.

[1118] The processing flow will be explained below.

[1119] Step 1:

[1120] The user wears the vision device. The camera in the vision device captures images of the user's field of view in real time. This image data is temporarily stored in the vision device.

[1121] Step 2:

[1122] The device begins pre-processing the captured video data, which includes noise removal and resolution adjustment, making the video clearer and easier for AI to analyze.

[1123] Step 3:

[1124] After preprocessing, the device inputs the video data into an artificial intelligence model, which then analyzes the images and identifies the facility based on its exterior, signage, logo, and other features contained in the video.

[1125] Step 4:

[1126] To recognize the user's emotions, the emotion engine of the vision device analyzes voice, facial expressions, and biometric information (e.g., heart rate, galvanic skin response). The emotion engine identifies the user's current emotional state.

[1127] Step 5:

[1128] The terminal sends a request to the server based on the facility information and the user's emotion information. The request includes the facility name, category, and emotion information.

[1129] Step 6:

[1130] The server analyzes the received request and retrieves detailed information about the facility (e.g., opening hours, ratings, special offers) from the database. The server also selects the most appropriate additional information based on the emotional information.

[1131] Step 7:

[1132] The server returns the acquired detailed information and selected additional information to the terminal as a response.

[1133] Step 8:

[1134] The device analyzes the information received from the server and displays it in a format that the user can perceive through their visual device. For example, if the user's emotions are positive, highly rated menu items and special offers will be highlighted, and if the user is showing signs of stress, suggestions for relaxation spots and coupon information will be displayed.

[1135] Step 9:

[1136] The user can check the detailed information of the facility and additional emotional information displayed on the visual device and take the necessary action. For example, when information about the cafe "Café Delight" is displayed, the user can check the opening hours and special offers and decide whether to visit.

[1137] The above is a detailed description of the specific processing steps of the program and the operations at each step.

[1138] Example 2

[1139] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1140] Conventional information acquisition systems using visual devices have difficulty providing optimal information according to the user's emotional state and environment. Furthermore, the acquired information is merely mechanical, which is insufficient for improving the user experience. Specifically, there is a lack of information presentation tailored to the user's stressful state or positive feelings toward a particular facility.

[1141] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1142] In this invention, the server includes means for acquiring video within the user's field of view, means for preprocessing the acquired video data, means for performing image analysis using an artificial intelligence model for identifying facilities based on the preprocessed video data, means including an emotion engine for analyzing the user's voice, facial expression, and biological information to recognize the user's emotional state, means for requesting facility information and emotion information acquired based on the image analysis and emotion analysis from the server, and means for displaying detailed facility information and optimal information based on the user's emotional state returned from the server on the display of the visual device, thereby making it possible to provide optimal facility information according to the user's emotional state and visual information.

[1143] A "user" is a person who wears and uses the visual device of the present invention.

[1144] A "vision device" is a device that includes a camera that captures images within the user's field of view and a display that displays that information to the user.

[1145] A "camera" is a device attached to a visual device that captures images in the user's field of view in real time.

[1146] "Video data" refers to image information captured by a camera and seen by a user.

[1147] "Preprocessing" refers to the process of removing noise and adjusting resolution of captured video data.

[1148] "Artificial Intelligence Model" means a machine learning model used to identify facilities using pre-processed video data.

[1149] "Image analysis" is a technology that uses an artificial intelligence model to extract the characteristics of a facility from video data and identify the facility.

[1150] An "emotion engine" is a technology that analyzes a user's voice, facial expression, and biological information to recognize the user's emotional state.

[1151] "Facility information" is detailed information such as the name and category of the identified facility.

[1152] A "server" is a computer system that retrieves information from a database based on a request and responds to a terminal.

[1153] A "display" is a device that is installed in a visual device and visually displays information obtained from a server or a terminal to a user.

[1154] "Biometric information" is data that indicates the user's physical condition, such as their heart rate and skin galvanic response.

[1155] A "request" is an inquiry that includes facility information and emotion information and is sent from a terminal to a server.

[1156] "Detailed information" is information such as opening hours, ratings, and special offers about a specific facility that is returned from the server.

[1157] The present invention is a system in which a user wears a visual device and acquires and displays information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[1158] Hardware and software used

[1159] 1. Visual Apparatus

[1160] The visual device worn by the user has a built-in camera and display. The camera captures images of the user's field of view in real time and temporarily stores them in the visual device's memory.

[1161] 2. Terminal

[1162] The device receives video data from the visual device, performs pre-processing and image analysis using artificial intelligence models, and also includes an emotion engine that recognizes the user's emotional state.

[1163] 3. Server

[1164] The server receives the request from the device, retrieves detailed information about the facility from the database, and returns the retrieved information to the device in an optimal format based on the user's emotional state.

[1165] Specific software used includes:

[1166] Artificial intelligence models: TensorFlow, PyTorch, etc.

[1167] Database: MySQL, PostgreSQL, etc.

[1168] Details of data processing and calculation

[1169] 1. Video capture by the user's visual device

[1170] The camera of the vision device captures images of what is in the user's field of view and temporarily stores this data within the vision device.

[1171] 2. Preprocessing of video data by the device

[1172] As soon as the device receives the video data from the vision device, it starts preprocessing, which includes noise removal and resolution adjustment, to enable subsequent image analysis to be performed with greater accuracy.

[1173] 3. Image analysis using AI models

[1174] The pre-processed video data is then fed into an AI model, which uses pattern recognition to identify the facility and identify its name and category based on its exterior, signage, logo, and other characteristics.

[1175] 4. Emotion analysis using an emotion engine

[1176] The emotion engine built into the visual device analyzes the user's voice, facial expressions, heart rate, and skin galvanic response to determine the user's emotional state.

[1177] 5. Sending a request to the server

[1178] The terminal integrates the identified facility information with the emotion information and sends a request to the server. This request includes the specific facility name, category, and emotion information.

[1179] 6. Server retrieves and returns details

[1180] The server analyzes the request received from the device, retrieves detailed information about the facility from the database, and then selects the most appropriate information based on the user's emotional state and returns it to the device.

[1181] 7. Displaying Information on a Terminal

[1182] The device receives detailed information about the facility from the server and displays it on the display of the visual device. If the user is in a positive emotional state, highly rated menu items and special offers are highlighted, while if the user is feeling stressed, suggestions for relaxation spots and coupon information are displayed.

[1183] Specific examples

[1184] Consider an example where a user is walking down the street and a cafe comes into view. The vision device's camera captures video of the cafe, and the device pre-processes the video. The AI ​​model identifies the cafe, and the emotion engine analyzes the user's emotions. For example, if the device determines that the user is in a positive emotional state, it sends the facility information and emotion information to the server. The server returns detailed information about the cafe, along with recommended menu items and special offers, which the device then displays on its screen. The user can check this information in real time and make the best choice.

[1185] Prompt Sentence Examples

[1186] User: When I use a visual aid to find a cafe, how exactly does the system process and display the information?

[1187] The above is a specific embodiment of the present invention, which allows the user to obtain information about surrounding facilities in real time, and furthermore, to obtain optimal information based on emotions.

[1188] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1189] Step 1:

[1190] The user wears the visual device.

[1191] How it works: The user puts on the vision device, activating it. The camera begins capturing video within the user's field of view.

[1192] Input: The image that the user sees.

[1193] Output: Captured video data.

[1194] Step 2:

[1195] The terminal receives the captured video data from the visual device.

[1196] Operation: The visual device's camera captures video data and sends it to the device, where it is temporarily saved.

[1197] Input: Captured video data.

[1198] Output: Received video data.

[1199] Step 3:

[1200] The terminal pre-processes the video data.

[1201] Operation: Performs noise reduction and resolution adjustment on the video data received by the device.

[1202] Input: Received video data.

[1203] Output: Pre-processed video data with noise removed and resolution adjusted.

[1204] Step 4:

[1205] The device inputs the preprocessed video data into an artificial intelligence model to identify the facility.

[1206] How it works: Preprocessed video data is fed into an AI model (e.g., TensorFlow, PyTorch) that uses pattern recognition techniques to identify facility names and categories.

[1207] Input: Preprocessed video data.

[1208] Output: Identified facility information (name, category).

[1209] Step 5:

[1210] The emotion engine of the visual device analyzes the user's emotions.

[1211] How it works: The emotion engine built into the visual device analyzes the user's voice, facial expressions, heart rate, and galvanic skin response to determine the user's emotional state.

[1212] Input: User's voice, facial expressions, heart rate, and galvanic skin response.

[1213] Output: The perceived emotional state of the user.

[1214] Step 6:

[1215] The terminal requests facility information and emotion information from the server.

[1216] Operation: The device compiles the identified facility information and analyzed emotion information and sends a request to the server.

[1217] Input: Identified facility information and emotion information.

[1218] Output: The request sent to the server.

[1219] Step 7:

[1220] The server receives the request, retrieves the details and sends them back to the device.

[1221] Operation: The server analyzes the received request and retrieves detailed information about the facility from its own database. Taking into account the user's emotional information, it selects the most appropriate information and returns it to the device.

[1222] Input: The request sent to the server.

[1223] Output: Detailed information about the selected facility.

[1224] Step 8:

[1225] The terminal displays the information on the visual device's display.

[1226] Operation: The device formats the facility information received from the server and displays it on the visual device's display. If the user is in a positive emotional state, highly rated menu items and special offers are highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information are displayed.

[1227] Input: Facility details received from the server.

[1228] Output: Information displayed on the visual device's display.

[1229] (Application example 2)

[1230] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1231] Conventional information provision systems using visual devices only allow users to visually obtain information about surrounding facilities, and have the problem of not being able to provide optimal information based on the user's current emotional state. Specifically, even if a user is feeling stressed, for example, they may not be provided with information about facilities where they can relax or special offers, which could result in a less meaningful shopping experience.

[1232] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1233] In this invention, the server includes means for selecting and returning optimal information taking into consideration the user's emotional information, means for preprocessing video data acquired by a camera attached to the visual device, and means for performing image analysis using an artificial intelligence model that identifies facilities based on the preprocessed video data. This allows the user to obtain recommended information based on their emotional state while obtaining information about surrounding facilities through the visual device in real time.

[1234] A "visual device" is a device equipped with a display and camera worn by the user, and has the function of capturing images within the user's field of vision and displaying the information.

[1235] A "camera" is an image capturing device attached to a visual device that captures images in the user's field of view in real time.

[1236] "Preprocessing" refers to processes such as removing noise from acquired video data and adjusting resolution to improve the accuracy of subsequent analysis.

[1237] An "artificial intelligence model" is a system that uses pattern recognition technology to identify facilities based on their exterior appearance, signage, logo, and other characteristics.

[1238] "Image analysis" refers to the process of identifying facilities using artificial intelligence models based on pre-processed video data.

[1239] "Emotional information" refers to data relating to the user's emotional state obtained by analyzing the user's voice, facial expression, and biometric information.

[1240] The "server" is a computer system that retrieves detailed facility information and recommended information from a database, and selects and returns the most appropriate information to the user, taking into account emotional information.

[1241] "Detailed information" refers to specific information about a facility, such as the facility's opening hours, ratings, and special offers.

[1242] An "emotion analysis engine" is a system that analyzes a user's voice, facial expression, and biometric information to identify their emotional state.

[1243] "Recommended information" refers to information on products, services, locations, etc. that are optimal for a user and are selected with consideration given to emotional information.

[1244] MODE FOR CARRYING OUT THE INVENTION

[1245] The present invention is a system in which a user wears a visual device and acquires and displays information about surrounding facilities in real time, and further combines it with an emotion engine that recognizes the user's emotions. A specific embodiment of this system will be described.

[1246] A user wears a visual device such as smart glasses, and a camera attached to the visual device captures images of what the user sees in real time. The video data captured by the camera is temporarily stored in the visual device. The device then begins preprocessing the video data. Preprocessing includes noise reduction and resolution adjustment, which improves the clarity of the image and the accuracy of subsequent analysis.

[1247] After preprocessing, the video data is input into an artificial intelligence model installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. This identification is achieved using pattern recognition technology based on features such as the facility's exterior, signage, and logo.

[1248] Meanwhile, the visual device is equipped with an emotion engine to recognize the user's emotions. The emotion engine recognizes the user's emotional state by analyzing the user's voice, facial expressions, and biometric information (heart rate, skin galvanic response, etc.). For example, it can detect when the user is feeling stressed or curious.

[1249] Facility information and emotional information are sent from the device to the server as a request. The request includes the name and category of the identified facility, as well as the user's emotional information. The server analyzes the received request and retrieves detailed information about the facility from its database. This information includes opening hours, ratings, and special offers. The server then takes the emotional information into consideration and selects the information that best suits the user's current emotional state, which it then sends back to the device. For example, if the user's emotions are positive, the facility's highly rated menu items and special offers are highlighted. If the user is feeling stressed, suggestions for relaxation spots and coupon information are also displayed.

[1250] As a specific example of use, imagine a user is walking through a shopping mall and the camera captures a sign for "Bookstore A." TensorFlow Lite identifies "Bookstore A" and performs sentiment analysis on the audio data. If it determines the sentiment is "positive," it sends "Bookstore A" and "positive" to the server. Information returned from the server, such as "discount information for bestselling books," is displayed on the smart glasses, and the user can see a pop-up that says "20% off bestselling books."

[1251] The following hardware and software are used to realize this system.

[1252] Hardware: Smart glasses (with camera), user device (smartphone, etc.)

[1253] Software: OpenCV (preprocessing), TensorFlow Lite (image analysis), Amazon Comprehend (sentiment analysis)

[1254] An example of a prompt is as follows:

[1255] example:

[1256] As the user walks through a shopping mall, the camera captures a sign for "Bookstore A." "Bookstore A" is identified using TensorFlow Lite. Sentiment analysis is performed on the audio data, and the result is determined to be "positive." "Bookstore A" and "positive" are sent to the server. The server returns "discount information for best-selling books." A pop-up message saying "20% off best-selling books" appears on the smart glasses.

[1257] As described above, the present invention is a system that allows users to obtain information about surrounding facilities in real time and provides optimal information based on their emotions, thereby improving the user experience and providing more meaningful information.

[1258] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1259] Step 1:

[1260] The user wears a visual device. The camera of this visual device captures the image of the user's field of view in real time. The input is the image of the user's field of view, and the output is raw image data temporarily stored in the visual device. Specifically, the camera constantly follows the user's field of view and continuously captures the image.

[1261] Step 2:

[1262] The device performs preprocessing on the captured video data. This preprocessing includes noise removal and resolution adjustment. The input is the captured raw video data, and the output is clear video data with noise removed and resolution adjusted. Specifically, the OpenCV library is used to filter noise and optimize resolution.

[1263] Step 3:

[1264] The preprocessed video data is input into an artificial intelligence model (generative AI model) installed on the device. The AI ​​model performs image analysis and identifies facilities contained within the video. The input is the preprocessed video data, and the output is the name and category of the identified facility. Specifically, TensorFlow Lite is used to perform pattern recognition based on the facility's exterior, sign, and logo.

[1265] Step 4:

[1266] The device uses an emotion engine to analyze the user's emotions. The emotion engine analyzes the user's voice, facial expression, and biometric information (such as heart rate and galvanic skin response). The input is the user's voice, facial expression, and biometric information, and the output is data indicating the user's emotional state. Specifically, it collects and analyzes data from Amazon Comprehend and biometric sensors.

[1267] Step 5:

[1268] The device sends facility information and emotion information as a request to the server. The input is the name and category of the identified facility and the user's emotion information, and the output is the request data sent to the server. Specifically, the data is communicated to the server using a RESTful API.

[1269] Step 6:

[1270] The server analyzes the request data it receives. The server retrieves detailed information about the facility (such as opening hours, ratings, and special offers) from the database and selects the most appropriate recommendation information taking into account emotional information. The input is the request data sent to the server, and the output is detailed information about the facility and the most appropriate information for the user. Specifically, it extracts data from AWS RDS and filters the information as needed.

[1271] Step 7:

[1272] The server returns the selected optimal information to the terminal. The input is detailed information and recommended information about the selected facility, and the output is the response data sent to the terminal. Specifically, the data is sent to the terminal again using the RESTful API.

[1273] Step 8:

[1274] The terminal analyzes the information received from the server and displays it in a format that the user can understand through the visual device. The input is detailed facility information and recommended information returned from the server, and the output is the information displayed on the visual device's display. Specifically, the information is displayed visually using the Unity engine, etc.

[1275] Through the above processing steps, the user can obtain information about surrounding facilities in real time through the visual device, and can obtain optimal information based on their emotions.

[1276] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1277] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1278] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1279] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1280] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1281] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1282] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1283] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1284] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1285] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1286] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1287] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1288] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1289] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1290] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1291] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1292] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1293] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1294] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1295] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1296] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1297] The following is further disclosed regarding the above embodiment.

[1298] (Claim 1)

[1299] means for capturing an image of the user's field of view by a camera attached to a visual device worn by the user;

[1300] means for pre-processing the acquired video data;

[1301] means for performing image analysis using an artificial intelligence model to identify facilities based on the preprocessed video data;

[1302] a means for requesting facility information acquired based on image analysis from a server;

[1303] means for displaying the detailed facility information returned from the server on a display of the visual device;

[1304] A system including:

[1305] (Claim 2)

[1306] The system of claim 1 , wherein the preprocessing includes noise removal and resolution adjustment.

[1307] (Claim 3)

[1308] 10. The system of claim 1, wherein the artificial intelligence model identifies a category of establishment and a specific establishment name.

[1309] "Example 1"

[1310] (Claim 1)

[1311] means for capturing an image of the user's field of view by a camera attached to a visual device worn by the user;

[1312] means for pre-processing the acquired video data;

[1313] means for performing image analysis using an artificial intelligence model to identify facilities based on the preprocessed video data;

[1314] a means for requesting facility information acquired based on image analysis from a server;

[1315] a means for overlaying and displaying the detailed facility information returned from the server on the display of the visual device;

[1316] A system including:

[1317] (Claim 2)

[1318] The system of claim 1 , wherein the preprocessing includes noise removal and resolution adjustment.

[1319] (Claim 3)

[1320] 10. The system of claim 1, wherein the artificial intelligence model identifies a category of establishment and a specific establishment name.

[1321] "Application Example 1"

[1322] (Claim 1)

[1323] means for capturing an image of the user's field of view by means of a photographing device attached to a visual device worn by the user;

[1324] means for pre-processing the acquired video data;

[1325] a means for performing image analysis using a generative AI model to identify facilities based on pre-processed video data; and

[1326] a means for requesting facility information acquired based on image analysis from a server via a communication device;

[1327] means for overlaying and displaying the detailed facility information returned from the server on the display device of the visual device;

[1328] A system including:

[1329] (Claim 2)

[1330] The system of claim 1, wherein the preprocessing includes noise removal and resolution adjustment, and further includes generating a prompt sentence to identify the facility.

[1331] (Claim 3)

[1332] 10. The system of claim 1, wherein the generative AI model identifies a facility category and a specific facility name and processes the information in a format suitable for a display device.

[1333] "Example 2: Combining Emotion Engines"

[1334] (Claim 1)

[1335] means for capturing an image of the user's field of view by a camera attached to a visual device worn by the user;

[1336] means for pre-processing the acquired video data;

[1337] means for performing image analysis using an artificial intelligence model to identify facilities based on the preprocessed video data;

[1338] means including an emotion engine for recognizing an emotional state of a user by analyzing the user's voice, facial expression, and biological information;

[1339] a means for requesting facility information and emotion information acquired based on image analysis and emotion analysis from a server;

[1340] a means for displaying on a display of the visual device the detailed information of the facility returned from the server and the optimal information based on the emotional state of the user;

[1341] A system including:

[1342] (Claim 2)

[1343] The system of claim 1 , wherein the preprocessing includes noise removal and resolution adjustment.

[1344] (Claim 3)

[1345] 10. The system of claim 1, wherein the artificial intelligence model identifies a category of establishment and a specific establishment name.

[1346] "Application example 2 when combining emotion engines"

[1347] (Claim 1)

[1348] means for capturing an image of the user's field of view by a camera attached to a visual device worn by the user;

[1349] means for pre-processing the acquired video data;

[1350] means for performing image analysis using an artificial intelligence model to identify facilities based on the preprocessed video data;

[1351] a means for requesting facility information and user emotion information acquired based on image analysis from a server;

[1352] a means for displaying detailed facility information returned from the server and recommended information based on the user's emotional state on a display of the visual device;

[1353] A system including:

[1354] (Claim 2)

[1355] The system of claim 1 , wherein the preprocessing includes noise removal and resolution adjustment.

[1356] (Claim 3)

[1357] 10. The system of claim 1, wherein the artificial intelligence model identifies a category of establishment and a specific establishment name.

[1358] (Claim 4)

[1359] 10. The system of claim 1, further comprising an emotion analysis engine that analyzes a user's voice, facial expression, and biometric information to identify their emotional state.

[1360] (Claim 5)

[1361] 2. The system according to claim 1, wherein the server selects and returns the most suitable information to the user in consideration of the emotional information. [Explanation of symbols]

[1362] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for capturing an image of the user's field of view by a camera attached to a visual device worn by the user; means for pre-processing the acquired video data; means for performing image analysis using an artificial intelligence model to identify facilities based on the preprocessed video data; a means for requesting facility information acquired based on image analysis from a server; means for displaying the detailed facility information returned from the server on a display of the visual device; A system including:

2. The system of claim 1 , wherein the preprocessing includes noise removal and resolution adjustment.

3. The system of claim 1 , wherein the artificial intelligence model identifies a category of establishment and a specific establishment name.

Images