system

Abstract

We provide the system. [Solution] A means of automatically acquiring information using a data collection device, A means of performing value assessment using an artificial intelligence model that analyzes acquired information, A means of registering the evaluated information on a market platform and making it available for sale, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention provides a means for a user to easily monetize information by automatically collecting data using a smart device, analyzing the data, and providing it in a value-added state. In particular, in the current situation of information overload, an object is to solve the problem of enabling efficient utilization of information collected from individual devices.

Means for Solving the Problems

[0005] This invention minimizes the effort required for data collection by equipping users' devices with the ability to automatically acquire information using a camera, microphone, and location information device. Furthermore, it uses an artificial intelligence model to analyze the acquired information and evaluate its value based on that information. The evaluated information is then registered on a market platform and automatically made available for sale. This mechanism realizes a system that integrates information collection, analysis, and provision.

[0006] "Data acquisition device" refers to hardware or software used to automatically acquire information.

[0007] An "artificial intelligence model" refers to an algorithm or machine learning system used to analyze information and evaluate its value.

[0008] A "marketplace platform" refers to a digital marketplace that allows users to register and sell evaluated information.

[0009] "Value assessment" refers to the process of determining the importance and usefulness of acquired information, either numerically or qualitatively.

[0010] "Feedback" refers to information and evaluations provided by users, which are used to improve the system. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5]This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, let's explain the terminology used in the following explanation.

[0014] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0017] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] This invention is realized by utilizing a user's smart device to automatically collect information and by using an artificial intelligence model to analyze the collected information. Specifically, data collection is performed by combining the camera, microphone, and location information device of the device. The device obtains the necessary permissions from the user through an application and collects data according to the instructions. The data is transmitted to an agent server in real time, where multiple artificial intelligence models operate to perform image recognition, voice analysis, and location information analysis.

[0033] The server evaluates the value of the analyzed information and automatically registers it on the market platform. On the market platform, the data is tagged to make it easily searchable and made available for other users to access and purchase. Information seekers can use maps and keywords to find the information they need, and then purchase it after confirming the desired information.

[0034] As a concrete example, let's say a traveler uses this system at a tourist destination. The user's smartphone takes photos of the tourist destination, records ambient sounds, and records location information. This data is uploaded to the server without the traveler's knowledge. An artificial intelligence model on the server analyzes this data, evaluates it as "tourist destination congestion information," and lists it on a market platform at an appropriate price. Subsequently, another user planning to visit the same tourist destination can purchase this information, allowing them to know the real-time congestion situation in advance.

[0035] The following describes the processing flow.

[0036] Step 1:

[0037] User: Install the app on your smartphone and grant permission to use the camera, microphone, and location information. Launch the app and select a data collection profile to complete the setup.

[0038] Step 2:

[0039] Device: At the set time, the camera is activated and photos of the surroundings are automatically taken. In addition, audio data is recorded using the microphone for a certain period of time, and location information is obtained.

[0040] Step 3:

[0041] Terminal: Prepares the collected data to be packaged together and tagged with time and location information.

[0042] Step 4:

[0043] Terminal: Checks network status and sends data packages to the agent server via a secure connection.

[0044] Step 5:

[0045] Server: The server inputs the received data package into an artificial intelligence model for analysis and performs analysis on each data point (photo, audio, location).

[0046] Step 6:

[0047] Server: Based on the analysis results, it determines the usefulness of the information and quantifies its value. Subsequently, it automatically sets the price for registering the information on the market platform.

[0048] Step 7:

[0049] Server: Registers information on the market platform and updates the database so that users can access it. At the same time, it adds tags to make the information easier to search.

[0050] Step 8:

[0051] User: Information seekers access the market platform and search for the information they need. After finding the desired information, they review the details and make a purchase.

[0052] Step 9:

[0053] Device: Downloads purchased information and makes it available to the user.

[0054] Step 10:

[0055] Server: Collects user feedback and uses it to continuously improve the functionality of artificial intelligence models and market platforms.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] Conventional information gathering and analysis systems had problems such as manual data collection from users and difficulty in immediately reflecting the market value of the data analysis results. Furthermore, the analyzed information was not processed in a way that made it easily searchable, hindering its effective use. In addition, it was difficult to optimize artificial intelligence models using user feedback.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for automatically acquiring information using the user's device, means for performing evaluation using an artificial intelligence model that analyzes the acquired video, audio, and location data, and means for registering the analyzed information on an electronic market and enabling its distribution. This enables automatic and efficient information collection and market reflection of the analysis results.

[0061] A "user's device" refers to an electronic device owned by an individual and used to acquire information.

[0062] "Means of automatically acquiring information" refers to methods that have the function of collecting data without requiring user intervention.

[0063] "Video, audio, and location data" refers to images and videos obtained from cameras, audio acquired from microphones, and geographical location data obtained from location information systems.

[0064] An "artificial intelligence model" is a computer program designed to analyze data and identify specific patterns or values.

[0065] "Means of evaluation" refers to methods that use artificial intelligence models to determine the usefulness and value of acquired data.

[0066] An "electronic marketplace" is an online platform where information and digital content are traded.

[0067] "Means that enable distribution" refers to methods that provide a function that allows others to access and purchase evaluated information in the electronic market.

[0068] "Methods for making information easier to find" refer to methods of assigning relevant tags and attributes to data so that users can quickly and easily find the information they need.

[0069] "Feedback" refers to the opinions and evaluations provided by users, which are used to improve the system.

[0070] "Methods for optimizing settings" refer to methods of adjusting the parameters and behavior of an artificial intelligence model based on collected feedback.

[0071] This invention is a system that utilizes multiple functions of a user's device to automatically acquire information, analyze it, and provide market value. It primarily uses smart devices, with specific applications installed on them performing the information acquisition. For example, it might capture video using the device's built-in camera, record audio using the microphone, and record location using a location information system.

[0072] After data is collected, the device transmits this data to the server in real time. The server uses TENSORFLOW® for image recognition and Google® Speech-to-Text for speech analysis. Furthermore, it utilizes GIS (Geographic Information System) to perform a detailed analysis of the collected location information. This analysis allows the data to be evaluated and its value determined.

[0073] Once the analysis is complete, the data is registered on the electronic marketplace by the server. The server uses ElasticSearch® to tag the data, making it easily searchable by other users. On the marketplace platform, users can access and purchase the information they need.

[0074] A concrete example is when this system operates while a user visits a tourist destination. The user's smart device takes pictures, collects audio, and records its location at the tourist spot. This data is sent to a server, analyzed, and provided to the market as information on the current congestion level of the tourist spot. Other users who purchase this system can use this information to plan their own visits.

[0075] An example of a prompt for a generative AI model is, "Generate real-time event information from the current image and audio data." This prompt allows the server to perform more detailed and application-specific information analysis.

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The user installs the application on their smart device and grants the necessary permissions. These permissions include access to the camera, microphone, and location information. Based on these permissions, the device automatically begins collecting data without any manual user intervention. Inputs are data obtained from the smart device's sensors, and outputs are camera images, microphone audio, and location information.

[0079] Step 2:

[0080] The device transmits collected images, audio, and location information to the server in real time. Specifically, the device encrypts this data using HTTPS and sends it securely to the server. The input is the raw data transmitted from the device, and the output is the unprocessed data stored on the server.

[0081] Step 3:

[0082] The server analyzes the received data. First, it uses TensorFlow to analyze the acquired images and recognize objects and scenes within them. Next, it uses Google Speech-to-Text to convert the audio data into text and extract important keywords. Furthermore, it uses GIS to analyze the location data on a map. The input is stored raw data, and the output is the analyzed data: images, text, and location information.

[0083] Step 4:

[0084] The server evaluates the value of the analyzed data. Evaluation criteria include data freshness, relevance, and rarity. Based on this evaluation, the server registers the data on an electronic marketplace. Specifically, it uses Elasticsearch to assign relevant tags to the information, improving searchability. The input is the analyzed data, and the output is the data registered on the marketplace platform.

[0085] Step 5:

[0086] Users can search for and purchase registered information through the online marketplace. In doing so, users will use keywords and map search functions to find the information they are looking for. The input is the user's search query accessing the marketplace, and the output is the details of the information selected by the user.

[0087] In this way, the process of automatically collecting, analyzing, and processing information into a marketable format at each step is repeated, making that information available for use by users and third parties.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] Providing residents and travelers with real-time and accurate environmental information remains a challenge in modern urban life. Specifically, it is necessary to efficiently collect information such as congestion levels and sound environments and provide it appropriately to residents and travelers. This challenge is important for alleviating urban congestion and facilitating smooth event participation, but conventional methods suffer from a time lag between information collection and provision.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes means for automatically acquiring environmental information using data collection means, means for performing value evaluation using an artificial intelligence model that analyzes the acquired environmental information, means for registering the evaluated environmental information on a data platform and making it available, and means for presenting the analysis results to residents and travelers in real time. This makes it possible to provide environmental information in real time and with high accuracy.

[0093] "Data collection means" refers to devices or equipment that have the function of automatically collecting environmental information. Specifically, this includes cameras, microphones, GPS functions, etc.

[0094] "Environmental information" refers to data that describes the physical and acoustic conditions of a city or specific area, including congestion levels and noise levels.

[0095] An "artificial intelligence model" refers to an algorithm or framework that analyzes collected environmental information to create valuable information. Image recognition and speech analysis are specific examples.

[0096] "Value assessment" refers to the process of determining the practical value of information based on information analyzed by an artificial intelligence model.

[0097] A "data platform" refers to an online system that stores collected environmental information and provides it in a format accessible to users.

[0098] "Presenting analysis results in real time" refers to a process where data collection and analysis are performed immediately, and the information is provided to users quickly.

[0099] This invention is a system for realizing real-time information provision in urban environments. The entire system mainly consists of data collection means, analysis means, and information provision means.

[0100] Smartphones and tablets are used as data collection devices. These devices are equipped with cameras, microphones, and GPS functions, and automatically collect environmental information. The collected data is transmitted in real time from the device to a server in the cloud.

[0101] On the server side, an analysis method using a generative AI model is in operation. Specifically, machine learning frameworks such as TensorFlow and PyTorch are used to analyze environmental information such as congestion levels and noise levels through image recognition and speech analysis. Based on the collected information, a value assessment is performed.

[0102] The analyzed environmental information is provided to users through a data platform. This allows residents and travelers to access the analysis results in real time through a dedicated application. Low-latency communication protocols are used to ensure real-time information delivery.

[0103] As a concrete example, when a user is moving through an urban area, their device takes photos of the surroundings, records audio, and records location information. This information is immediately sent to a server and analyzed through prompts such as "I want to know the current congestion level." Ultimately, other users can search for information in the form of "Please tell me about the congestion level and event information in urban areas this weekend. In particular, I would like to know the audio level and congestion level around popular event venues in detail," and view the detailed analysis results.

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The device acquires information about the user's surroundings. Specifically, it takes photos using the camera, records audio using the microphone, and collects location information using the GPS function. This data is temporarily stored within the device.

[0107] Step 2:

[0108] The device transmits collected environmental information to a server in the cloud. The transmitted data includes photographic data, audio data, and location data, which are sent using a secure communication protocol and are accurately time-stamped.

[0109] Step 3:

[0110] The server analyzes the received environmental information using a generating AI model. Photo data is used to evaluate congestion levels through an image recognition algorithm, and audio data is used for noise level analysis. Location data is used for tagging data based on geographical characteristics. As a result of the analysis, real-time environmental assessment information is generated.

[0111] Step 4:

[0112] The server registers the environmental assessment information obtained through analysis into the data platform. This information is tagged with geographical attributes and relevant keywords, making it searchable by users. It is then integrated into existing databases, enabling responses to searches from other users.

[0113] Step 5:

[0114] Users use a dedicated application to input information about their areas of interest as prompts. For example, they might request information in the form of, "Please tell me about the congestion and events in urban areas this weekend."

[0115] Step 6:

[0116] The server retrieves relevant information from the data platform based on user prompts and provides the user with the most relevant environmental assessment information. This information includes the most recent analysis results, updated in real time.

[0117] Step 7:

[0118] Users can view information provided by the server within the application and use it to adjust and plan their actions. The information is presented in a visually easy-to-understand format, and more detailed information is available as needed.

[0119] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0120] This invention is a system that automatically collects information using a smart device, analyzes that information with an artificial intelligence model to evaluate its value, and further combines it with an emotion engine that recognizes the user's emotions. The user starts by installing the application on their smart device and granting permissions for the camera, microphone, location information, and facial expression recognition.

[0121] The device recognizes the user's emotions by analyzing their facial expressions using its camera and capturing their voice tone using its microphone. Simultaneously, it periodically collects photos, audio, and location data of the environment. The collected data is sent from the device to an agent server, where a multi-layered artificial intelligence model performs analysis.

[0122] The server integrates user sentiment data and environmental data, and evaluates the value of the data based on this. Based on this evaluation, the data is registered on the market platform and made available for other users to purchase. On the market platform, a recommendation algorithm works to prioritize the presentation of information that matches the user's sentiment.

[0123] To give a specific example, when a user visits a shopping area, this system takes photos of the surroundings and recognizes the user's emotions of joy. This information is sent to a server, where the congestion level of the commercial facility and specific promotional information are analyzed in real time. As a result, highly relevant shopping information is presented to the user and sold on the market platform. In this process, the user's emotion data is used to adjust the priority of how individual pieces of information are viewed. In this way, an environment is created where users can quickly and efficiently obtain the information that is best suited to them.

[0124] The following describes the processing flow.

[0125] Step 1:

[0126] User: Install the app on your smart device and grant permissions for camera, microphone, location, and facial recognition. This will prepare your device for data collection.

[0127] Step 2:

[0128] Device: Uses a camera to intermittently capture images of the surrounding environment and the user's facial expressions. Simultaneously, it uses a microphone to monitor surrounding sounds and the user's voice, and acquires location information.

[0129] Step 3:

[0130] Device: Captured images, recorded audio, and location information are compiled into a data package and saved to temporary storage along with timestamps and tags.

[0131] Step 4:

[0132] Terminal: Sends data packages to the agent server via a secure communication channel. Data transmission occurs as soon as the device's network connection becomes available.

[0133] Step 5:

[0134] Server: Inputs the received data package into an artificial intelligence model for analysis. The data is divided into categories, and image recognition, speech analysis, and location information analysis are performed.

[0135] Step 6:

[0136] Server: Uses an emotion engine to recognize emotions from the user's facial expressions and voice. The recognized emotion data is associated with environmental data.

[0137] Step 7:

[0138] Server: Based on environmental and sentiment data, it evaluates the usefulness of information and quantifies its value. Based on this evaluation, it sets the price when registering the information on the market platform.

[0139] Step 8:

[0140] Server: Registers information on the market platform and adjusts the system so that information is presented in order of priority according to the user's emotional state.

[0141] Step 9:

[0142] User: Information seekers access the market platform and search for the information they need. Based on the user's sentiment data, information that is likely to be of interest is presented.

[0143] Step 10:

[0144] Device: Downloads selected information and makes it immediately available to the user. The user then makes decisions based on this information.

[0145] (Example 2)

[0146] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0147] In modern society, providing information tailored to the individual user's emotions and circumstances is extremely important. However, existing information delivery systems have difficulty prioritizing information that accurately reflects the user's emotional state. Furthermore, mechanisms for improving system accuracy through user feedback are not adequately in place. Therefore, there is a need to develop a system that effectively and efficiently provides the most suitable information for the user.

[0148] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0149] In this invention, the server includes means for using an emotion engine that analyzes voice tone and facial expressions to identify the user's emotional state, means for automatically collecting user environmental information and transmitting it to an agent server, and means for analyzing and integrating the emotion data and environmental data using a multi-layered artificial intelligence model. This makes it possible to prioritize and provide optimal information based on the user's emotions.

[0150] An "emotion engine" is a program that incorporates technology to analyze voice tone and facial expressions in order to identify the user's emotional state.

[0151] "Automatic data collection methods" refer to technologies that use a user's smart device to collect photos, audio, and location information of the environment at regular time intervals without human intervention.

[0152] An "agent server" is a computer system that receives data sent from a terminal and performs data analysis using a multi-layered artificial intelligence model.

[0153] A "multi-layered artificial intelligence model" is a learning model with multiple layers, where each layer extracts specific features and is used to perform advanced data analysis.

[0154] A "market platform" refers to a place where analyzed data and its evaluation results are registered, and other users can view and purchase the information.

[0155] A "recommendation algorithm" is a computational method that selects and prioritizes the presentation of highly relevant information based on the sentiment data of individual users.

[0156] The user begins by installing a dedicated application on their smart device. This application needs to obtain permissions for the camera, microphone, location information, and facial recognition. This completes the necessary setup.

[0157] The device, via an installed application, captures the user's face with a camera and analyzes their facial expressions in real time. This facial expression data is used to infer the user's emotional state. Furthermore, the device collects the user's voice using a microphone and analyzes their voice tone to gain further insights into their emotions. Location information and ambient sounds are also collected as needed.

[0158] The server receives the aforementioned data transmitted by the terminal and performs analysis using a multi-layered artificial intelligence model. The AI ​​model uses various analytics modules, including an emotion engine, to perform advanced analysis of the collected data. The data obtained from the analysis is integrated, and a value assessment is made based on the user's emotional state and environmental conditions.

[0159] The evaluation results are registered on the market platform. Here, a recommendation algorithm works to prioritize providing information that best matches the user's emotions. For example, if a user is in a shopping center, a system that identifies their smiling face will provide specific promotional information or new product announcements in real time.

[0160] In this way, a system is realized that utilizes user sentiment data to quickly provide appropriate information. An example of a prompt would be, "Select and present the most relevant content based on the user's sentiment data." Such prompts function as instructions for the system to select the most suitable information for the user.

[0161] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0162] Step 1:

[0163] The user installs the application on their smart device and grants the necessary permissions. These include permissions for the camera, microphone, location information, and facial recognition. Once permissions are granted, the system is ready to proceed to the next data collection stage.

[0164] Step 2:

[0165] The device uses a camera to capture the user's facial expressions in real time. The input is video of the user's face. This video data is then analyzed using a facial expression analysis algorithm to obtain output that converts subtle facial changes into emotional states. For example, emotions such as smiles and surprises are quantified and output.

[0166] Step 3:

[0167] The device uses a microphone to collect the user's voice. The voice data is taken in as input, and by analyzing its tone and speed, supplementary information about emotions is output. For example, it calculates indicators of tension or relaxation from the pitch and tempo of the voice.

[0168] Step 4:

[0169] The device obtains the user's current location through the smart device's location services. Location data is provided as input, and environmental characteristic information is output. This includes geographical features of the current location and information about nearby facilities.

[0170] Step 5:

[0171] The server receives facial expression, voice, and location data transmitted by the terminal. The data received as input is analyzed using a multi-layered artificial intelligence model. Through data processing and calculations, it generates an output that integrates the user's emotional state and environmental characteristics.

[0172] Step 6:

[0173] The server evaluates the market value of the information based on the generated analysis results. An evaluation report is created as output. This report includes recommendations based on the user's emotional state and is prepared for registration on the next market platform.

[0174] Step 7:

[0175] The server registers the evaluation report on the market platform. As an output, the platform displays products and information suitable for the user. The recommendation algorithm dynamically adjusts the priority of information based on sentiment data, enabling it to provide the user with the most relevant information.

[0176] (Application Example 2)

[0177] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0178] Providing users with the information they need in real time requires personalized information based on their emotions and circumstances. However, conventional information delivery systems have struggled to recognize users' emotions and prioritize information based on them. Furthermore, they have been unable to link the value assessment of information to users' emotions, resulting in a decrease in the efficiency of information delivery.

[0179] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0180] In this invention, the server includes means for automatically acquiring information using a data collection device, means for performing value evaluation using an artificial intelligence model that analyzes the acquired information, means for analyzing the user's emotions using an emotion recognition device that acquires the user's facial expressions and voice tone, and means including a recommendation device that preferentially provides relevant information based on the user's emotions. This enables the user to quickly and appropriately obtain information that is in line with their emotions and environment.

[0181] A "data collection device" is a device that automatically acquires various types of information, and is used to collect images, audio, location information, and so on.

[0182] An "artificial intelligence model" is a program consisting of mathematical and logical structures for analyzing acquired information and performing value assessments.

[0183] A "marketplace platform" is an online or offline space where evaluated information can be registered and offered or sold to other users.

[0184] An "emotion recognition device" is a device that analyzes a user's emotional state based on their facial expressions and voice tone.

[0185] A "recommendation system" is a system that prioritizes providing relevant information based on the user's emotions and circumstances.

[0186] The system that realizes this invention mainly consists of a data collection device, an emotion recognition device, an artificial intelligence model, and a recommendation device. The user installs a dedicated application on a smart device and grants the necessary permissions. The device utilizes built-in sensors to capture the user's facial expressions with a camera and acquire voice with a microphone, thereby recognizing emotions from facial expressions and voice tone. This information is transmitted to a server in the cloud.

[0187] The server analyzes the transmitted data using artificial intelligence models based on services such as Google Cloud AI and Amazon SageMaker, and performs value assessments based on the user's emotions and environment. Simultaneously, the server accesses a database stored in MongoDB Atlas to find information relevant to the user. Then, based on the understood emotions and assessed information, it uses GOOGLE FI® rebase Cloud Messaging to send appropriate information to the user via real-time push notifications.

[0188] As a concrete example, while a user is visiting a shopping mall, the device detects expressions of joy on the user's face and sends the data to the cloud. Based on the emotional data and location information within the mall, the server selects special offers and new product information from specific stores and pushes this information to the user. This allows users to receive information tailored to their interests and mood, enriching their shopping experience.

[0189] An example of a prompt to input into a generative AI model would be, "Estimate the user's purchase intent from their facial expressions and provide relevant product information." This would enable the system to provide highly personalized information based on the user's facial expressions.

[0190] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0191] Step 1:

[0192] The device acquires the user's facial expressions and voice. It collects facial images in real time via the camera and captures voice data using the microphone. At this stage, the input is the camera image and voice input, and the output is raw data for emotion recognition.

[0193] Step 2:

[0194] The terminal sends the acquired facial expressions and audio data to a data preprocessing module for noise reduction and standardization. This improves the accuracy of the data. The input in this step is raw data, and the output is processed data.

[0195] Step 3:

[0196] The terminal sends the processed data to a server in the cloud. The server receives the data and prepares it for input into the AI ​​model. The input is the processed data, and the output is the data to be input to the AI ​​model.

[0197] Step 4:

[0198] The server uses an artificial intelligence model to analyze the received data and estimate the user's emotional state. For example, it might use a facial recognition API or a voice analysis library. The input at this stage is data for the AI ​​model, and the output is the estimated emotional data.

[0199] Step 5:

[0200] The server uses estimated sentiment data to retrieve relevant product and promotional information from the database. It accesses MongoDB Atlas to select information that matches the user's sentiment. The input here is sentiment data and database information, and the output is a list of information for the user.

[0201] Step 6:

[0202] The server uses Google Firebase Cloud Messaging to push selected information to the device. Users receive this notification and can check information of interest in real time. The input is a list of information for the user, and the output is a notification message displayed on the device.

[0203] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0204] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0205] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0206] [Second Embodiment]

[0207] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0208] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0209] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0210] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0211] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0212] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0213] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0214] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0215] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0216] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0217] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0218] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0219] This invention is realized by utilizing a user's smart device to automatically collect information and by using an artificial intelligence model to analyze the collected information. Specifically, data collection is performed by combining the camera, microphone, and location information device of the device. The device obtains the necessary permissions from the user through an application and collects data according to the instructions. The data is transmitted to an agent server in real time, where multiple artificial intelligence models operate to perform image recognition, voice analysis, and location information analysis.

[0220] The server evaluates the value of the analyzed information and automatically registers it on the market platform. On the market platform, the data is tagged to make it easily searchable and made available for other users to access and purchase. Information seekers can use maps and keywords to find the information they need, and then purchase it after confirming the desired information.

[0221] As a concrete example, let's say a traveler uses this system at a tourist destination. The user's smartphone takes photos of the tourist destination, records ambient sounds, and records location information. This data is uploaded to the server without the traveler's knowledge. An artificial intelligence model on the server analyzes this data, evaluates it as "tourist destination congestion information," and lists it on a market platform at an appropriate price. Subsequently, another user planning to visit the same tourist destination can purchase this information, allowing them to know the real-time congestion situation in advance.

[0222] The following describes the processing flow.

[0223] Step 1:

[0224] User: Install the app on your smartphone and grant permission to use the camera, microphone, and location information. Launch the app and select a data collection profile to complete the setup.

[0225] Step 2:

[0226] Device: At the set time, the camera is activated and photos of the surroundings are automatically taken. In addition, audio data is recorded using the microphone for a certain period of time, and location information is obtained.

[0227] Step 3:

[0228] Terminal: Prepares the collected data to be packaged together and tagged with time and location information.

[0229] Step 4:

[0230] Terminal: Checks network status and sends data packages to the agent server via a secure connection.

[0231] Step 5:

[0232] Server: The server inputs the received data package into an artificial intelligence model for analysis and performs analysis on each data point (photo, audio, location).

[0233] Step 6:

[0234] Server: Based on the analysis results, it determines the usefulness of the information and quantifies its value. Subsequently, it automatically sets the price for registering the information on the market platform.

[0235] Step 7:

[0236] Server: Registers information on the market platform and updates the database so that users can access it. At the same time, it adds tags to make the information easier to search.

[0237] Step 8:

[0238] User: Information seekers access the market platform and search for the information they need. After finding the desired information, they review the details and make a purchase.

[0239] Step 9:

[0240] Device: Downloads purchased information and makes it available to the user.

[0241] Step 10:

[0242] Server: Collects user feedback and uses it to continuously improve the functionality of artificial intelligence models and market platforms.

[0243] (Example 1)

[0244] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0245] Conventional information gathering and analysis systems had problems such as manual data collection from users and difficulty in immediately reflecting the market value of the data analysis results. Furthermore, the analyzed information was not processed in a way that made it easily searchable, hindering its effective use. In addition, it was difficult to optimize artificial intelligence models using user feedback.

[0246] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0247] In this invention, the server includes means for automatically acquiring information using the user's device, means for performing evaluation using an artificial intelligence model that analyzes the acquired video, audio, and location data, and means for registering the analyzed information on an electronic market and enabling its distribution. This enables automatic and efficient information collection and market reflection of the analysis results.

[0248] A "user's device" refers to an electronic device owned by an individual and used to acquire information.

[0249] "Means of automatically acquiring information" refers to methods that have the function of collecting data without requiring user intervention.

[0250] "Video, audio, and location data" refers to images and videos obtained from cameras, audio acquired from microphones, and geographical location data obtained from location information systems.

[0251] An "artificial intelligence model" is a computer program designed to analyze data and identify specific patterns or values.

[0252] "Means of evaluation" refers to methods that use artificial intelligence models to determine the usefulness and value of acquired data.

[0253] An "electronic marketplace" is an online platform where information and digital content are traded.

[0254] "Means that enable distribution" refers to methods that provide a function that allows others to access and purchase evaluated information in the electronic market.

[0255] "Methods for making information easier to find" refer to methods of assigning relevant tags and attributes to data so that users can quickly and easily find the information they need.

[0256] "Feedback" refers to the opinions and evaluations provided by users, which are used to improve the system.

[0257] "Methods for optimizing settings" refer to methods of adjusting the parameters and behavior of an artificial intelligence model based on collected feedback.

[0258] This invention is a system that utilizes multiple functions of a user's device to automatically acquire information, analyze it, and provide market value. It primarily uses smart devices, with specific applications installed on them performing the information acquisition. For example, it might capture video using the device's built-in camera, record audio using the microphone, and record location using a location information system.

[0259] After data is collected, the device sends this data to the server in real time. The server uses TensorFlow for image recognition and Google Speech-to-Text for speech analysis. Furthermore, it utilizes GIS (Geographic Information System) to perform a detailed analysis of the collected location information. This analysis allows the data to be evaluated and its value determined.

[0260] Once the analysis is complete, the data is registered on the e-marketplace by the server. The server uses Elasticsearch to tag the data, making it easily searchable by other users. On the marketplace platform, users can access and purchase the information they need.

[0261] A concrete example is when this system operates while a user visits a tourist destination. The user's smart device takes pictures, collects audio, and records its location at the tourist spot. This data is sent to a server, analyzed, and provided to the market as information on the current congestion level of the tourist spot. Other users who purchase this system can use this information to plan their own visits.

[0262] An example of a prompt for a generative AI model is, "Generate real-time event information from the current image and audio data." This prompt allows the server to perform more detailed and application-specific information analysis.

[0263] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0264] Step 1:

[0265] The user installs the application on their smart device and grants the necessary permissions. These permissions include access to the camera, microphone, and location information. Based on these permissions, the device automatically begins collecting data without any manual user intervention. Inputs are data obtained from the smart device's sensors, and outputs are camera images, microphone audio, and location information.

[0266] Step 2:

[0267] The device transmits collected images, audio, and location information to the server in real time. Specifically, the device encrypts this data using HTTPS and sends it securely to the server. The input is the raw data transmitted from the device, and the output is the unprocessed data stored on the server.

[0268] Step 3:

[0269] The server analyzes the received data. First, it uses TensorFlow to analyze the acquired images and recognize objects and scenes within them. Next, it uses Google Speech-to-Text to convert the audio data into text and extract important keywords. Furthermore, it uses GIS to analyze the location data on a map. The input is stored raw data, and the output is the analyzed data: images, text, and location information.

[0270] Step 4:

[0271] The server evaluates the value of the analyzed data. Evaluation criteria include data freshness, relevance, and rarity. Based on this evaluation, the server registers the data on an electronic marketplace. Specifically, it uses Elasticsearch to assign relevant tags to the information, improving searchability. The input is the analyzed data, and the output is the data registered on the marketplace platform.

[0272] Step 5:

[0273] Users can search for and purchase registered information through the online marketplace. In doing so, users will use keywords and map search functions to find the information they are looking for. The input is the user's search query accessing the marketplace, and the output is the details of the information selected by the user.

[0274] In this way, the process of automatically collecting, analyzing, and processing information into a marketable format at each step is repeated, making that information available for use by users and third parties.

[0275] (Application Example 1)

[0276] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0277] Providing residents and travelers with real-time and accurate environmental information remains a challenge in modern urban life. Specifically, it is necessary to efficiently collect information such as congestion levels and sound environments and provide it appropriately to residents and travelers. This challenge is important for alleviating urban congestion and facilitating smooth event participation, but conventional methods suffer from a time lag between information collection and provision.

[0278] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0279] In this invention, the server includes means for automatically acquiring environmental information using data collection means, means for performing value evaluation using an artificial intelligence model that analyzes the acquired environmental information, means for registering the evaluated environmental information on a data platform and making it available, and means for presenting the analysis results to residents and travelers in real time. This makes it possible to provide environmental information in real time and with high accuracy.

[0280] "Data collection means" refers to devices or equipment that have the function of automatically collecting environmental information. Specifically, this includes cameras, microphones, GPS functions, etc.

[0281] "Environmental information" refers to data that describes the physical and acoustic conditions of a city or specific area, including congestion levels and noise levels.

[0282] An "artificial intelligence model" refers to an algorithm or framework that analyzes collected environmental information to create valuable information. Image recognition and speech analysis are specific examples.

[0283] "Value evaluation" refers to the process of judging the practical value of information based on the information analyzed by an artificial intelligence model.

[0284] "Data platform" refers to an online system for storing the collected environmental information and providing it in a form accessible to users.

[0285] "Presenting the analysis results in real time" refers to the process of immediately collecting and analyzing data and quickly providing information to users.

[0286] This invention is a system for realizing real-time information provision in an urban environment. The entire system is mainly composed of data collection means, analysis means, and information provision means.

[0287] Terminal devices such as smartphones and tablets are used as data collection means. These terminal devices are equipped with cameras, microphones, and GPS functions, and automatically collect environmental information. The collected data is transmitted in real time from the terminal to a server on the cloud.

[0288] On the server side, analysis means using a generative AI model operates. Specifically, machine learning frameworks such as TensorFlow and PyTorch are used to analyze environmental information such as traffic congestion and noise levels through image recognition and voice analysis. Based on this, value evaluation is performed on the collected information.

[0289] The analyzed environmental information is provided to users through the data platform. As a result, residents and travelers can refer to the analysis results in real time through a dedicated application. To ensure real-time performance, a low-latency communication protocol is utilized for information provision.

[0290] As a concrete example, when a user is moving through an urban area, their device takes photos of the surroundings, records audio, and records location information. This information is immediately sent to a server and analyzed through prompts such as "I want to know the current congestion level." Ultimately, other users can search for information in the form of "Please tell me about the congestion level and event information in urban areas this weekend. In particular, I would like to know the audio level and congestion level around popular event venues in detail," and view the detailed analysis results.

[0291] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0292] Step 1:

[0293] The device acquires information about the user's surroundings. Specifically, it takes photos using the camera, records audio using the microphone, and collects location information using the GPS function. This data is temporarily stored within the device.

[0294] Step 2:

[0295] The device transmits collected environmental information to a server in the cloud. The transmitted data includes photographic data, audio data, and location data, which are sent using a secure communication protocol and are accurately time-stamped.

[0296] Step 3:

[0297] The server analyzes the received environmental information using a generating AI model. Photo data is used to evaluate congestion levels through an image recognition algorithm, and audio data is used for noise level analysis. Location data is used for tagging data based on geographical characteristics. As a result of the analysis, real-time environmental assessment information is generated.

[0298] Step 4:

[0299] The server registers the environmental assessment information obtained through analysis into the data platform. This information is tagged with geographical attributes and relevant keywords, making it searchable by users. It is then integrated into existing databases, enabling responses to searches from other users.

[0300] Step 5:

[0301] Users use a dedicated application to input information about their areas of interest as prompts. For example, they might request information in the form of, "Please tell me about the congestion and events in urban areas this weekend."

[0302] Step 6:

[0303] The server retrieves relevant information from the data platform based on user prompts and provides the user with the most relevant environmental assessment information. This information includes the most recent analysis results, updated in real time.

[0304] Step 7:

[0305] Users can view information provided by the server within the application and use it to adjust and plan their actions. The information is presented in a visually easy-to-understand format, and more detailed information is available as needed.

[0306] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0307] The present invention is a system that automatically collects information using a smart device, analyzes the information with an artificial intelligence model to evaluate its value, and further combines an emotion engine that recognizes the user's emotions. The user starts by installing an application on the smart device and granting permissions for the camera, microphone, location information, and facial recognition.

[0308] The terminal uses the camera to analyze the user's facial expressions and the microphone to obtain the voice tone to recognize the user's emotions. In parallel, it regularly collects photos, sounds, and location data of the environment. The acquired data is sent from the terminal to an agent server, where a multi-layer artificial intelligence model performs analysis.

[0309] The server integrates the user's emotion data and environmental data and evaluates the value of the data based on this. Based on this evaluation result, the data is registered on the market platform and provided so that other users can purchase it. In the market platform, a recommendation algorithm works to preferentially present information that matches the user's emotions.

[0310] For a specific example, when the user visits a shopping area, this system takes photos of the surroundings and recognizes the user's happy emotions. This information is sent to the server, and the congestion situation of commercial facilities and specific promotion information are analyzed in real time. As a result, shopping information highly relevant to the user is presented and sold on the market platform. In this process, the user's emotion data is used to adjust the priority of viewing each piece of information. In this way, an environment is created where the user can quickly and efficiently obtain the optimal information for themselves.

[0311] The following describes the processing flow.

[0312] Step 1:

[0313] User: Install the app on your smart device and grant permissions for camera, microphone, location, and facial recognition. This will prepare your device for data collection.

[0314] Step 2:

[0315] Device: Uses a camera to intermittently capture images of the surrounding environment and the user's facial expressions. Simultaneously, it uses a microphone to monitor surrounding sounds and the user's voice, and acquires location information.

[0316] Step 3:

[0317] Device: Captured images, recorded audio, and location information are compiled into a data package and saved to temporary storage along with timestamps and tags.

[0318] Step 4:

[0319] Terminal: Sends data packages to the agent server via a secure communication channel. Data transmission occurs as soon as the device's network connection becomes available.

[0320] Step 5:

[0321] Server: Inputs the received data package into an artificial intelligence model for analysis. The data is divided into categories, and image recognition, speech analysis, and location information analysis are performed.

[0322] Step 6:

[0323] Server: Uses an emotion engine to recognize emotions from the user's facial expressions and voice. The recognized emotion data is associated with environmental data.

[0324] Step 7:

[0325] Server: Based on environmental and sentiment data, it evaluates the usefulness of information and quantifies its value. Based on this evaluation, it sets the price when registering the information on the market platform.

[0326] Step 8:

[0327] Server: Registers information on the market platform and adjusts the system so that information is presented in order of priority according to the user's emotional state.

[0328] Step 9:

[0329] User: Information seekers access the market platform and search for the information they need. Based on the user's sentiment data, information that is likely to be of interest is presented.

[0330] Step 10:

[0331] Device: Downloads selected information and makes it immediately available to the user. The user then makes decisions based on this information.

[0332] (Example 2)

[0333] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0334] In modern society, providing information tailored to the individual user's emotions and circumstances is extremely important. However, existing information delivery systems have difficulty prioritizing information that accurately reflects the user's emotional state. Furthermore, mechanisms for improving system accuracy through user feedback are not adequately in place. Therefore, there is a need to develop a system that effectively and efficiently provides the most suitable information for the user.

[0335] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0336] In this invention, the server includes means for using an emotion engine that analyzes voice tone and facial expressions to identify the user's emotional state, means for automatically collecting user environmental information and transmitting it to an agent server, and means for analyzing and integrating the emotion data and environmental data using a multi-layered artificial intelligence model. This makes it possible to prioritize and provide optimal information based on the user's emotions.

[0337] An "emotion engine" is a program that incorporates technology to analyze voice tone and facial expressions in order to identify the user's emotional state.

[0338] "Automatic data collection methods" refer to technologies that use a user's smart device to collect photos, audio, and location information of the environment at regular time intervals without human intervention.

[0339] An "agent server" is a computer system that receives data sent from a terminal and performs data analysis using a multi-layered artificial intelligence model.

[0340] A "multi-layered artificial intelligence model" is a learning model with multiple layers, where each layer extracts specific features and is used to perform advanced data analysis.

[0341] A "market platform" refers to a place where analyzed data and its evaluation results are registered, and other users can view and purchase the information.

[0342] A "recommendation algorithm" is a computational method that selects and prioritizes the presentation of highly relevant information based on the sentiment data of individual users.

[0343] The user begins by installing a dedicated application on their smart device. This application needs to obtain permissions for the camera, microphone, location information, and facial recognition. This completes the necessary setup.

[0344] The device, via an installed application, captures the user's face with a camera and analyzes their facial expressions in real time. This facial expression data is used to infer the user's emotional state. Furthermore, the device collects the user's voice using a microphone and analyzes their voice tone to gain further insights into their emotions. Location information and ambient sounds are also collected as needed.

[0345] The server receives the aforementioned data transmitted by the terminal and performs analysis using a multi-layered artificial intelligence model. The AI ​​model uses various analytics modules, including an emotion engine, to perform advanced analysis of the collected data. The data obtained from the analysis is integrated, and a value assessment is made based on the user's emotional state and environmental conditions.

[0346] The evaluation results are registered on the market platform. Here, a recommendation algorithm works to prioritize providing information that best matches the user's emotions. For example, if a user is in a shopping center, a system that identifies their smiling face will provide specific promotional information or new product announcements in real time.

[0347] In this way, a system is realized that utilizes user sentiment data to quickly provide appropriate information. An example of a prompt would be, "Select and present the most relevant content based on the user's sentiment data." Such prompts function as instructions for the system to select the most suitable information for the user.

[0348] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0349] Step 1:

[0350] The user installs the application on their smart device and grants the necessary permissions. These include permissions for the camera, microphone, location information, and facial recognition. Once permissions are granted, the system is ready to proceed to the next data collection stage.

[0351] Step 2:

[0352] The device uses a camera to capture the user's facial expressions in real time. The input is video of the user's face. This video data is then analyzed using a facial expression analysis algorithm to obtain output that converts subtle facial changes into emotional states. For example, emotions such as smiles and surprises are quantified and output.

[0353] Step 3:

[0354] The device uses a microphone to collect the user's voice. The voice data is taken in as input, and by analyzing its tone and speed, supplementary information about emotions is output. For example, it calculates indicators of tension or relaxation from the pitch and tempo of the voice.

[0355] Step 4:

[0356] The device obtains the user's current location through the smart device's location services. Location data is provided as input, and environmental characteristic information is output. This includes geographical features of the current location and information about nearby facilities.

[0357] Step 5:

[0358] The server receives facial expression, voice, and location data transmitted by the terminal. The data received as input is analyzed using a multi-layered artificial intelligence model. Through data processing and calculations, it generates an output that integrates the user's emotional state and environmental characteristics.

[0359] Step 6:

[0360] The server evaluates the market value of the information based on the generated analysis results. An evaluation report is created as output. This report includes recommendations based on the user's emotional state and is prepared for registration on the next market platform.

[0361] Step 7:

[0362] The server registers the evaluation report on the market platform. As an output, the platform displays products and information suitable for the user. The recommendation algorithm dynamically adjusts the priority of information based on sentiment data, enabling it to provide the user with the most relevant information.

[0363] (Application Example 2)

[0364] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0365] Providing users with the information they need in real time requires personalized information based on their emotions and circumstances. However, conventional information delivery systems have struggled to recognize users' emotions and prioritize information based on them. Furthermore, they have been unable to link the value assessment of information to users' emotions, resulting in a decrease in the efficiency of information delivery.

[0366] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0367] In this invention, the server includes means for automatically acquiring information using a data collection device, means for performing value evaluation using an artificial intelligence model that analyzes the acquired information, means for analyzing the user's emotions using an emotion recognition device that acquires the user's facial expressions and voice tone, and means including a recommendation device that preferentially provides relevant information based on the user's emotions. This enables the user to quickly and appropriately obtain information that is in line with their emotions and environment.

[0368] A "data collection device" is a device that automatically acquires various types of information, and is used to collect images, audio, location information, and so on.

[0369] An "artificial intelligence model" is a program consisting of mathematical and logical structures for analyzing acquired information and performing value assessments.

[0370] A "marketplace platform" is an online or offline space where evaluated information can be registered and offered or sold to other users.

[0371] An "emotion recognition device" is a device that analyzes a user's emotional state based on their facial expressions and voice tone.

[0372] A "recommendation system" is a system that prioritizes providing relevant information based on the user's emotions and circumstances.

[0373] The system that realizes this invention mainly consists of a data collection device, an emotion recognition device, an artificial intelligence model, and a recommendation device. The user installs a dedicated application on a smart device and grants the necessary permissions. The device utilizes built-in sensors to capture the user's facial expressions with a camera and acquire voice with a microphone, thereby recognizing emotions from facial expressions and voice tone. This information is transmitted to a server in the cloud.

[0374] The server analyzes the transmitted data using artificial intelligence models based on services such as Google Cloud AI and Amazon SageMaker, and performs value assessments based on the user's emotions and environment. Simultaneously, the server accesses a database stored in MongoDB Atlas to find information relevant to the user. Then, based on the understood emotions and assessed information, it uses Google Firebase Cloud Messaging to send appropriate information to the user via real-time push notifications.

[0375] As a concrete example, while a user is visiting a shopping mall, the device detects expressions of joy on the user's face and sends the data to the cloud. Based on the emotional data and location information within the mall, the server selects special offers and new product information from specific stores and pushes this information to the user. This allows users to receive information tailored to their interests and mood, enriching their shopping experience.

[0376] An example of a prompt to input into a generative AI model would be, "Estimate the user's purchase intent from their facial expressions and provide relevant product information." This would enable the system to provide highly personalized information based on the user's facial expressions.

[0377] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0378] Step 1:

[0379] The device acquires the user's facial expressions and voice. It collects facial images in real time via the camera and captures voice data using the microphone. At this stage, the input is the camera image and voice input, and the output is raw data for emotion recognition.

[0380] Step 2:

[0381] The terminal sends the acquired facial expressions and audio data to a data preprocessing module for noise reduction and standardization. This improves the accuracy of the data. The input in this step is raw data, and the output is processed data.

[0382] Step 3:

[0383] The terminal sends the processed data to a server in the cloud. The server receives the data and prepares it for input into the AI ​​model. The input is the processed data, and the output is the data to be input to the AI ​​model.

[0384] Step 4:

[0385] The server uses an artificial intelligence model to analyze the received data and estimate the user's emotional state. For example, it might use a facial recognition API or a voice analysis library. The input at this stage is data for the AI ​​model, and the output is the estimated emotional data.

[0386] Step 5:

[0387] The server uses estimated sentiment data to retrieve relevant product and promotional information from the database. It accesses MongoDB Atlas to select information that matches the user's sentiment. The input here is sentiment data and database information, and the output is a list of information for the user.

[0388] Step 6:

[0389] The server uses Google Firebase Cloud Messaging to push selected information to the device. Users receive this notification and can check information of interest in real time. The input is a list of information for the user, and the output is a notification message displayed on the device.

[0390] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0391] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0392] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0393] [Third Embodiment]

[0394] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0395] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0396] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0397] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0398] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0399] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0400] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0401] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0402] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0403] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0404] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0405] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0406] This invention is realized by utilizing a user's smart device to automatically collect information and by using an artificial intelligence model to analyze the collected information. Specifically, data collection is performed by combining the camera, microphone, and location information device of the device. The device obtains the necessary permissions from the user through an application and collects data according to the instructions. The data is transmitted to an agent server in real time, where multiple artificial intelligence models operate to perform image recognition, voice analysis, and location information analysis.

[0407] The server evaluates the value of the analyzed information and automatically registers it on the market platform. On the market platform, the data is tagged to make it easily searchable and made available for other users to access and purchase. Information seekers can use maps and keywords to find the information they need, and then purchase it after confirming the desired information.

[0408] As a concrete example, let's say a traveler uses this system at a tourist destination. The user's smartphone takes photos of the tourist destination, records ambient sounds, and records location information. This data is uploaded to the server without the traveler's knowledge. An artificial intelligence model on the server analyzes this data, evaluates it as "tourist destination congestion information," and lists it on a market platform at an appropriate price. Subsequently, another user planning to visit the same tourist destination can purchase this information, allowing them to know the real-time congestion situation in advance.

[0409] The following describes the processing flow.

[0410] Step 1:

[0411] User: Install the app on your smartphone and grant permission to use the camera, microphone, and location information. Launch the app and select a data collection profile to complete the setup.

[0412] Step 2:

[0413] Device: At the set time, the camera is activated and photos of the surroundings are automatically taken. In addition, audio data is recorded using the microphone for a certain period of time, and location information is obtained.

[0414] Step 3:

[0415] Terminal: Prepares the collected data to be packaged together and tagged with time and location information.

[0416] Step 4:

[0417] Terminal: Checks network status and sends data packages to the agent server via a secure connection.

[0418] Step 5:

[0419] Server: The server inputs the received data package into an artificial intelligence model for analysis and performs analysis on each data point (photo, audio, location).

[0420] Step 6:

[0421] Server: Based on the analysis results, it determines the usefulness of the information and quantifies its value. Subsequently, it automatically sets the price for registering the information on the market platform.

[0422] Step 7:

[0423] Server: Registers information on the market platform and updates the database so that users can access it. At the same time, it adds tags to make the information easier to search.

[0424] Step 8:

[0425] User: Information seekers access the market platform and search for the information they need. After finding the desired information, they review the details and make a purchase.

[0426] Step 9:

[0427] Device: Downloads purchased information and makes it available to the user.

[0428] Step 10:

[0429] Server: Collects user feedback and uses it to continuously improve the functionality of artificial intelligence models and market platforms.

[0430] (Example 1)

[0431] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0432] Conventional information gathering and analysis systems had problems such as manual data collection from users and difficulty in immediately reflecting the market value of the data analysis results. Furthermore, the analyzed information was not processed in a way that made it easily searchable, hindering its effective use. In addition, it was difficult to optimize artificial intelligence models using user feedback.

[0433] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0434] In this invention, the server includes means for automatically acquiring information using the user's device, means for performing evaluation using an artificial intelligence model that analyzes the acquired video, audio, and location data, and means for registering the analyzed information on an electronic market and enabling its distribution. This enables automatic and efficient information collection and market reflection of the analysis results.

[0435] A "user's device" refers to an electronic device owned by an individual and used to acquire information.

[0436] "Means of automatically acquiring information" refers to methods that have the function of collecting data without requiring user intervention.

[0437] "Video, audio, and location data" refers to images and videos obtained from cameras, audio acquired from microphones, and geographical location data obtained from location information systems.

[0438] An "artificial intelligence model" is a computer program designed to analyze data and identify specific patterns or values.

[0439] "Means of evaluation" refers to methods that use artificial intelligence models to determine the usefulness and value of acquired data.

[0440] An "electronic marketplace" is an online platform where information and digital content are traded.

[0441] "Means that enable distribution" refers to methods that provide a function that allows others to access and purchase evaluated information in the electronic market.

[0442] "Methods for making information easier to find" refer to methods of assigning relevant tags and attributes to data so that users can quickly and easily find the information they need.

[0443] "Feedback" refers to the opinions and evaluations provided by users, which are used to improve the system.

[0444] "Methods for optimizing settings" refer to methods of adjusting the parameters and behavior of an artificial intelligence model based on collected feedback.

[0445] This invention is a system that utilizes multiple functions of a user's device to automatically acquire information, analyze it, and provide market value. It primarily uses smart devices, with specific applications installed on them performing the information acquisition. For example, it might capture video using the device's built-in camera, record audio using the microphone, and record location using a location information system.

[0446] After data is collected, the device sends this data to the server in real time. The server uses TensorFlow for image recognition and Google Speech-to-Text for speech analysis. Furthermore, it utilizes GIS (Geographic Information System) to perform a detailed analysis of the collected location information. This analysis allows the data to be evaluated and its value determined.

[0447] Once the analysis is complete, the data is registered on the e-marketplace by the server. The server uses Elasticsearch to tag the data, making it easily searchable by other users. On the marketplace platform, users can access and purchase the information they need.

[0448] A concrete example is when this system operates while a user visits a tourist destination. The user's smart device takes pictures, collects audio, and records its location at the tourist spot. This data is sent to a server, analyzed, and provided to the market as information on the current congestion level of the tourist spot. Other users who purchase this system can use this information to plan their own visits.

[0449] An example of a prompt for a generative AI model is, "Generate real-time event information from the current image and audio data." This prompt allows the server to perform more detailed and application-specific information analysis.

[0450] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0451] Step 1:

[0452] The user installs the application on their smart device and grants the necessary permissions. These permissions include access to the camera, microphone, and location information. Based on these permissions, the device automatically begins collecting data without any manual user intervention. Inputs are data obtained from the smart device's sensors, and outputs are camera images, microphone audio, and location information.

[0453] Step 2:

[0454] The device transmits collected images, audio, and location information to the server in real time. Specifically, the device encrypts this data using HTTPS and sends it securely to the server. The input is the raw data transmitted from the device, and the output is the unprocessed data stored on the server.

[0455] Step 3:

[0456] The server analyzes the received data. First, it uses TensorFlow to analyze the acquired images and recognize objects and scenes within them. Next, it uses Google Speech-to-Text to convert the audio data into text and extract important keywords. Furthermore, it uses GIS to analyze the location data on a map. The input is stored raw data, and the output is the analyzed data: images, text, and location information.

[0457] Step 4:

[0458] The server evaluates the value of the analyzed data. Evaluation criteria include data freshness, relevance, and rarity. Based on this evaluation, the server registers the data on an electronic marketplace. Specifically, it uses Elasticsearch to assign relevant tags to the information, improving searchability. The input is the analyzed data, and the output is the data registered on the marketplace platform.

[0459] Step 5:

[0460] Users can search for and purchase registered information through the online marketplace. In doing so, users will use keywords and map search functions to find the information they are looking for. The input is the user's search query accessing the marketplace, and the output is the details of the information selected by the user.

[0461] In this way, the process of automatically collecting, analyzing, and processing information into a marketable format at each step is repeated, making that information available for use by users and third parties.

[0462] (Application Example 1)

[0463] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0464] Providing residents and travelers with real-time and accurate environmental information remains a challenge in modern urban life. Specifically, it is necessary to efficiently collect information such as congestion levels and sound environments and provide it appropriately to residents and travelers. This challenge is important for alleviating urban congestion and facilitating smooth event participation, but conventional methods suffer from a time lag between information collection and provision.

[0465] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0466] In this invention, the server includes means for automatically acquiring environmental information using data collection means, means for performing value evaluation using an artificial intelligence model that analyzes the acquired environmental information, means for registering the evaluated environmental information on a data platform and making it available, and means for presenting the analysis results to residents and travelers in real time. This makes it possible to provide environmental information in real time and with high accuracy.

[0467] "Data collection means" refers to devices or equipment that have the function of automatically collecting environmental information. Specifically, this includes cameras, microphones, GPS functions, etc.

[0468] "Environmental information" refers to data that describes the physical and acoustic conditions of a city or specific area, including congestion levels and noise levels.

[0469] An "artificial intelligence model" refers to an algorithm or framework that analyzes collected environmental information to create valuable information. Image recognition and speech analysis are specific examples.

[0470] "Value assessment" refers to the process of determining the practical value of information based on information analyzed by an artificial intelligence model.

[0471] A "data platform" refers to an online system that stores collected environmental information and provides it in a format accessible to users.

[0472] "Presenting analysis results in real time" refers to a process where data collection and analysis are performed immediately, and the information is provided to users quickly.

[0473] This invention is a system for realizing real-time information provision in urban environments. The entire system mainly consists of data collection means, analysis means, and information provision means.

[0474] Smartphones and tablets are used as data collection devices. These devices are equipped with cameras, microphones, and GPS functions, and automatically collect environmental information. The collected data is transmitted in real time from the device to a server in the cloud.

[0475] On the server side, an analysis method using a generative AI model is in operation. Specifically, machine learning frameworks such as TensorFlow and PyTorch are used to analyze environmental information such as congestion levels and noise levels through image recognition and speech analysis. Based on the collected information, a value assessment is performed.

[0476] The analyzed environmental information is provided to users through a data platform. This allows residents and travelers to access the analysis results in real time through a dedicated application. Low-latency communication protocols are used to ensure real-time information delivery.

[0477] As a concrete example, when a user is moving through an urban area, their device takes photos of the surroundings, records audio, and records location information. This information is immediately sent to a server and analyzed through prompts such as "I want to know the current congestion level." Ultimately, other users can search for information in the form of "Please tell me about the congestion level and event information in urban areas this weekend. In particular, I would like to know the audio level and congestion level around popular event venues in detail," and view the detailed analysis results.

[0478] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0479] Step 1:

[0480] The device acquires information about the user's surroundings. Specifically, it takes photos using the camera, records audio using the microphone, and collects location information using the GPS function. This data is temporarily stored within the device.

[0481] Step 2:

[0482] The device transmits collected environmental information to a server in the cloud. The transmitted data includes photographic data, audio data, and location data, which are sent using a secure communication protocol and are accurately time-stamped.

[0483] Step 3:

[0484] The server analyzes the received environmental information using a generating AI model. Photo data is used to evaluate congestion levels through an image recognition algorithm, and audio data is used for noise level analysis. Location data is used for tagging data based on geographical characteristics. As a result of the analysis, real-time environmental assessment information is generated.

[0485] Step 4:

[0486] The server registers the environmental assessment information obtained through analysis into the data platform. This information is tagged with geographical attributes and relevant keywords, making it searchable by users. It is then integrated into existing databases, enabling responses to searches from other users.

[0487] Step 5:

[0488] Users use a dedicated application to input information about their areas of interest as prompts. For example, they might request information in the form of, "Please tell me about the congestion and events in urban areas this weekend."

[0489] Step 6:

[0490] The server retrieves relevant information from the data platform based on user prompts and provides the user with the most relevant environmental assessment information. This information includes the most recent analysis results, updated in real time.

[0491] Step 7:

[0492] Users can view information provided by the server within the application and use it to adjust and plan their actions. The information is presented in a visually easy-to-understand format, and more detailed information is available as needed.

[0493] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0494] This invention is a system that automatically collects information using a smart device, analyzes that information with an artificial intelligence model to evaluate its value, and further combines it with an emotion engine that recognizes the user's emotions. The user starts by installing the application on their smart device and granting permissions for the camera, microphone, location information, and facial expression recognition.

[0495] The device recognizes the user's emotions by analyzing their facial expressions using its camera and capturing their voice tone using its microphone. Simultaneously, it periodically collects photos, audio, and location data of the environment. The collected data is sent from the device to an agent server, where a multi-layered artificial intelligence model performs analysis.

[0496] The server integrates user sentiment data and environmental data, and evaluates the value of the data based on this. Based on this evaluation, the data is registered on the market platform and made available for other users to purchase. On the market platform, a recommendation algorithm works to prioritize the presentation of information that matches the user's sentiment.

[0497] To give a specific example, when a user visits a shopping area, this system takes photos of the surroundings and recognizes the user's emotions of joy. This information is sent to a server, where the congestion level of the commercial facility and specific promotional information are analyzed in real time. As a result, highly relevant shopping information is presented to the user and sold on the market platform. In this process, the user's emotion data is used to adjust the priority of how individual pieces of information are viewed. In this way, an environment is created where users can quickly and efficiently obtain the information that is best suited to them.

[0498] The following describes the processing flow.

[0499] Step 1:

[0500] User: Install the app on your smart device and grant permissions for camera, microphone, location, and facial recognition. This will prepare your device for data collection.

[0501] Step 2:

[0502] Device: Uses a camera to intermittently capture images of the surrounding environment and the user's facial expressions. Simultaneously, it uses a microphone to monitor surrounding sounds and the user's voice, and acquires location information.

[0503] Step 3:

[0504] Device: Captured images, recorded audio, and location information are compiled into a data package and saved to temporary storage along with timestamps and tags.

[0505] Step 4:

[0506] Terminal: Sends data packages to the agent server via a secure communication channel. Data transmission occurs as soon as the device's network connection becomes available.

[0507] Step 5:

[0508] Server: Inputs the received data package into an artificial intelligence model for analysis. The data is divided into categories, and image recognition, speech analysis, and location information analysis are performed.

[0509] Step 6:

[0510] Server: Uses an emotion engine to recognize emotions from the user's facial expressions and voice. The recognized emotion data is associated with environmental data.

[0511] Step 7:

[0512] Server: Based on environmental and sentiment data, it evaluates the usefulness of information and quantifies its value. Based on this evaluation, it sets the price when registering the information on the market platform.

[0513] Step 8:

[0514] Server: Registers information on the market platform and adjusts the system so that information is presented in order of priority according to the user's emotional state.

[0515] Step 9:

[0516] User: Information seekers access the market platform and search for the information they need. Based on the user's sentiment data, information that is likely to be of interest is presented.

[0517] Step 10:

[0518] Device: Downloads selected information and makes it immediately available to the user. The user then makes decisions based on this information.

[0519] (Example 2)

[0520] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0521] In modern society, providing information tailored to the individual user's emotions and circumstances is extremely important. However, existing information delivery systems have difficulty prioritizing information that accurately reflects the user's emotional state. Furthermore, mechanisms for improving system accuracy through user feedback are not adequately in place. Therefore, there is a need to develop a system that effectively and efficiently provides the most suitable information for the user.

[0522] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0523] In this invention, the server includes means for using an emotion engine that analyzes voice tone and facial expressions to identify the user's emotional state, means for automatically collecting user environmental information and transmitting it to an agent server, and means for analyzing and integrating the emotion data and environmental data using a multi-layered artificial intelligence model. This makes it possible to prioritize and provide optimal information based on the user's emotions.

[0524] An "emotion engine" is a program that incorporates technology to analyze voice tone and facial expressions in order to identify the user's emotional state.

[0525] "Automatic data collection methods" refer to technologies that use a user's smart device to collect photos, audio, and location information of the environment at regular time intervals without human intervention.

[0526] An "agent server" is a computer system that receives data sent from a terminal and performs data analysis using a multi-layered artificial intelligence model.

[0527] A "multi-layered artificial intelligence model" is a learning model with multiple layers, where each layer extracts specific features and is used to perform advanced data analysis.

[0528] A "market platform" refers to a place where analyzed data and its evaluation results are registered, and other users can view and purchase the information.

[0529] A "recommendation algorithm" is a computational method that selects and prioritizes the presentation of highly relevant information based on the sentiment data of individual users.

[0530] The user begins by installing a dedicated application on their smart device. This application needs to obtain permissions for the camera, microphone, location information, and facial recognition. This completes the necessary setup.

[0531] The device, via an installed application, captures the user's face with a camera and analyzes their facial expressions in real time. This facial expression data is used to infer the user's emotional state. Furthermore, the device collects the user's voice using a microphone and analyzes their voice tone to gain further insights into their emotions. Location information and ambient sounds are also collected as needed.

[0532] The server receives the aforementioned data transmitted by the terminal and performs analysis using a multi-layered artificial intelligence model. The AI ​​model uses various analytics modules, including an emotion engine, to perform advanced analysis of the collected data. The data obtained from the analysis is integrated, and a value assessment is made based on the user's emotional state and environmental conditions.

[0533] The evaluation results are registered on the market platform. Here, a recommendation algorithm works to prioritize providing information that best matches the user's emotions. For example, if a user is in a shopping center, a system that identifies their smiling face will provide specific promotional information or new product announcements in real time.

[0534] In this way, a system is realized that utilizes user sentiment data to quickly provide appropriate information. An example of a prompt would be, "Select and present the most relevant content based on the user's sentiment data." Such prompts function as instructions for the system to select the most suitable information for the user.

[0535] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0536] Step 1:

[0537] The user installs the application on their smart device and grants the necessary permissions. These include permissions for the camera, microphone, location information, and facial recognition. Once permissions are granted, the system is ready to proceed to the next data collection stage.

[0538] Step 2:

[0539] The device uses a camera to capture the user's facial expressions in real time. The input is video of the user's face. This video data is then analyzed using a facial expression analysis algorithm to obtain output that converts subtle facial changes into emotional states. For example, emotions such as smiles and surprises are quantified and output.

[0540] Step 3:

[0541] The device uses a microphone to collect the user's voice. The voice data is taken in as input, and by analyzing its tone and speed, supplementary information about emotions is output. For example, it calculates indicators of tension or relaxation from the pitch and tempo of the voice.

[0542] Step 4:

[0543] The device obtains the user's current location through the smart device's location services. Location data is provided as input, and environmental characteristic information is output. This includes geographical features of the current location and information about nearby facilities.

[0544] Step 5:

[0545] The server receives facial expression, voice, and location data transmitted by the terminal. The data received as input is analyzed using a multi-layered artificial intelligence model. Through data processing and calculations, it generates an output that integrates the user's emotional state and environmental characteristics.

[0546] Step 6:

[0547] The server evaluates the market value of the information based on the generated analysis results. An evaluation report is created as output. This report includes recommendations based on the user's emotional state and is prepared for registration on the next market platform.

[0548] Step 7:

[0549] The server registers the evaluation report on the market platform. As an output, the platform displays products and information suitable for the user. The recommendation algorithm dynamically adjusts the priority of information based on sentiment data, enabling it to provide the user with the most relevant information.

[0550] (Application Example 2)

[0551] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0552] Providing users with the information they need in real time requires personalized information based on their emotions and circumstances. However, conventional information delivery systems have struggled to recognize users' emotions and prioritize information based on them. Furthermore, they have been unable to link the value assessment of information to users' emotions, resulting in a decrease in the efficiency of information delivery.

[0553] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0554] In this invention, the server includes means for automatically acquiring information using a data collection device, means for performing value evaluation using an artificial intelligence model that analyzes the acquired information, means for analyzing the user's emotions using an emotion recognition device that acquires the user's facial expressions and voice tone, and means including a recommendation device that preferentially provides relevant information based on the user's emotions. This enables the user to quickly and appropriately obtain information that is in line with their emotions and environment.

[0555] A "data collection device" is a device that automatically acquires various types of information, and is used to collect images, audio, location information, and so on.

[0556] An "artificial intelligence model" is a program consisting of mathematical and logical structures for analyzing acquired information and performing value assessments.

[0557] A "marketplace platform" is an online or offline space where evaluated information can be registered and offered or sold to other users.

[0558] An "emotion recognition device" is a device that analyzes a user's emotional state based on their facial expressions and voice tone.

[0559] A "recommendation system" is a system that prioritizes providing relevant information based on the user's emotions and circumstances.

[0560] The system that realizes this invention mainly consists of a data collection device, an emotion recognition device, an artificial intelligence model, and a recommendation device. The user installs a dedicated application on a smart device and grants the necessary permissions. The device utilizes built-in sensors to capture the user's facial expressions with a camera and acquire voice with a microphone, thereby recognizing emotions from facial expressions and voice tone. This information is transmitted to a server in the cloud.

[0561] The server analyzes the transmitted data using artificial intelligence models based on services such as Google Cloud AI and Amazon SageMaker, and performs value assessments based on the user's emotions and environment. Simultaneously, the server accesses a database stored in MongoDB Atlas to find information relevant to the user. Then, based on the understood emotions and assessed information, it uses Google Firebase Cloud Messaging to send appropriate information to the user via real-time push notifications.

[0562] As a concrete example, while a user is visiting a shopping mall, the device detects expressions of joy on the user's face and sends the data to the cloud. Based on the emotional data and location information within the mall, the server selects special offers and new product information from specific stores and pushes this information to the user. This allows users to receive information tailored to their interests and mood, enriching their shopping experience.

[0563] An example of a prompt to input into a generative AI model would be, "Estimate the user's purchase intent from their facial expressions and provide relevant product information." This would enable the system to provide highly personalized information based on the user's facial expressions.

[0564] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0565] Step 1:

[0566] The device acquires the user's facial expressions and voice. It collects facial images in real time via the camera and captures voice data using the microphone. At this stage, the input is the camera image and voice input, and the output is raw data for emotion recognition.

[0567] Step 2:

[0568] The terminal sends the acquired facial expressions and audio data to a data preprocessing module for noise reduction and standardization. This improves the accuracy of the data. The input in this step is raw data, and the output is processed data.

[0569] Step 3:

[0570] The terminal sends the processed data to a server in the cloud. The server receives the data and prepares it for input into the AI ​​model. The input is the processed data, and the output is the data to be input to the AI ​​model.

[0571] Step 4:

[0572] The server uses an artificial intelligence model to analyze the received data and estimate the user's emotional state. For example, it might use a facial recognition API or a voice analysis library. The input at this stage is data for the AI ​​model, and the output is the estimated emotional data.

[0573] Step 5:

[0574] The server uses estimated sentiment data to retrieve relevant product and promotional information from the database. It accesses MongoDB Atlas to select information that matches the user's sentiment. The input here is sentiment data and database information, and the output is a list of information for the user.

[0575] Step 6:

[0576] The server uses Google Firebase Cloud Messaging to push selected information to the device. Users receive this notification and can check information of interest in real time. The input is a list of information for the user, and the output is a notification message displayed on the device.

[0577] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0578] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0579] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0580] [Fourth Embodiment]

[0581] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0582] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0583] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0584] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0585] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0586] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0587] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0588] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0589] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0590] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0591] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0592] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0593] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0594] This invention is realized by utilizing a user's smart device to automatically collect information and by using an artificial intelligence model to analyze the collected information. Specifically, data collection is performed by combining the camera, microphone, and location information device of the device. The device obtains the necessary permissions from the user through an application and collects data according to the instructions. The data is transmitted to an agent server in real time, where multiple artificial intelligence models operate to perform image recognition, voice analysis, and location information analysis.

[0595] The server evaluates the value of the analyzed information and automatically registers it on the market platform. On the market platform, the data is tagged to make it easily searchable and made available for other users to access and purchase. Information seekers can use maps and keywords to find the information they need, and then purchase it after confirming the desired information.

[0596] As a concrete example, let's say a traveler uses this system at a tourist destination. The user's smartphone takes photos of the tourist destination, records ambient sounds, and records location information. This data is uploaded to the server without the traveler's knowledge. An artificial intelligence model on the server analyzes this data, evaluates it as "tourist destination congestion information," and lists it on a market platform at an appropriate price. Subsequently, another user planning to visit the same tourist destination can purchase this information, allowing them to know the real-time congestion situation in advance.

[0597] The following describes the processing flow.

[0598] Step 1:

[0599] User: Install the app on your smartphone and grant permission to use the camera, microphone, and location information. Launch the app and select a data collection profile to complete the setup.

[0600] Step 2:

[0601] Device: At the set time, the camera is activated and photos of the surroundings are automatically taken. In addition, audio data is recorded using the microphone for a certain period of time, and location information is obtained.

[0602] Step 3:

[0603] Terminal: Prepares the collected data to be packaged together and tagged with time and location information.

[0604] Step 4:

[0605] Terminal: Checks network status and sends data packages to the agent server via a secure connection.

[0606] Step 5:

[0607] Server: The server inputs the received data package into an artificial intelligence model for analysis and performs analysis on each data point (photo, audio, location).

[0608] Step 6:

[0609] Server: Based on the analysis results, it determines the usefulness of the information and quantifies its value. Subsequently, it automatically sets the price for registering the information on the market platform.

[0610] Step 7:

[0611] Server: Registers information on the market platform and updates the database so that users can access it. At the same time, it adds tags to make the information easier to search.

[0612] Step 8:

[0613] User: Information seekers access the market platform and search for the information they need. After finding the desired information, they review the details and make a purchase.

[0614] Step 9:

[0615] Device: Downloads purchased information and makes it available to the user.

[0616] Step 10:

[0617] Server: Collects user feedback and uses it to continuously improve the functionality of artificial intelligence models and market platforms.

[0618] (Example 1)

[0619] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0620] Conventional information gathering and analysis systems had problems such as manual data collection from users and difficulty in immediately reflecting the market value of the data analysis results. Furthermore, the analyzed information was not processed in a way that made it easily searchable, hindering its effective use. In addition, it was difficult to optimize artificial intelligence models using user feedback.

[0621] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0622] In this invention, the server includes means for automatically acquiring information using the user's device, means for performing evaluation using an artificial intelligence model that analyzes the acquired video, audio, and location data, and means for registering the analyzed information on an electronic market and enabling its distribution. This enables automatic and efficient information collection and market reflection of the analysis results.

[0623] A "user's device" refers to an electronic device owned by an individual and used to acquire information.

[0624] "Means of automatically acquiring information" refers to methods that have the function of collecting data without requiring user intervention.

[0625] "Video, audio, and location data" refers to images and videos obtained from cameras, audio acquired from microphones, and geographical location data obtained from location information systems.

[0626] An "artificial intelligence model" is a computer program designed to analyze data and identify specific patterns or values.

[0627] "Means of evaluation" refers to methods that use artificial intelligence models to determine the usefulness and value of acquired data.

[0628] An "electronic marketplace" is an online platform where information and digital content are traded.

[0629] "Means that enable distribution" refers to methods that provide a function that allows others to access and purchase evaluated information in the electronic market.

[0630] "Methods for making information easier to find" refer to methods of assigning relevant tags and attributes to data so that users can quickly and easily find the information they need.

[0631] "Feedback" refers to the opinions and evaluations provided by users, which are used to improve the system.

[0632] "Methods for optimizing settings" refer to methods of adjusting the parameters and behavior of an artificial intelligence model based on collected feedback.

[0633] This invention is a system that utilizes multiple functions of a user's device to automatically acquire information, analyze it, and provide market value. It primarily uses smart devices, with specific applications installed on them performing the information acquisition. For example, it might capture video using the device's built-in camera, record audio using the microphone, and record location using a location information system.

[0634] After data is collected, the device sends this data to the server in real time. The server uses TensorFlow for image recognition and Google Speech-to-Text for speech analysis. Furthermore, it utilizes GIS (Geographic Information System) to perform a detailed analysis of the collected location information. This analysis allows the data to be evaluated and its value determined.

[0635] Once the analysis is complete, the data is registered on the e-marketplace by the server. The server uses Elasticsearch to tag the data, making it easily searchable by other users. On the marketplace platform, users can access and purchase the information they need.

[0636] A concrete example is when this system operates while a user visits a tourist destination. The user's smart device takes pictures, collects audio, and records its location at the tourist spot. This data is sent to a server, analyzed, and provided to the market as information on the current congestion level of the tourist spot. Other users who purchase this system can use this information to plan their own visits.

[0637] An example of a prompt for a generative AI model is, "Generate real-time event information from the current image and audio data." This prompt allows the server to perform more detailed and application-specific information analysis.

[0638] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0639] Step 1:

[0640] The user installs the application on their smart device and grants the necessary permissions. These permissions include access to the camera, microphone, and location information. Based on these permissions, the device automatically begins collecting data without any manual user intervention. Inputs are data obtained from the smart device's sensors, and outputs are camera images, microphone audio, and location information.

[0641] Step 2:

[0642] The device transmits collected images, audio, and location information to the server in real time. Specifically, the device encrypts this data using HTTPS and sends it securely to the server. The input is the raw data transmitted from the device, and the output is the unprocessed data stored on the server.

[0643] Step 3:

[0644] The server analyzes the received data. First, it uses TensorFlow to analyze the acquired images and recognize objects and scenes within them. Next, it uses Google Speech-to-Text to convert the audio data into text and extract important keywords. Furthermore, it uses GIS to analyze the location data on a map. The input is stored raw data, and the output is the analyzed data: images, text, and location information.

[0645] Step 4:

[0646] The server evaluates the value of the analyzed data. Evaluation criteria include data freshness, relevance, and rarity. Based on this evaluation, the server registers the data on an electronic marketplace. Specifically, it uses Elasticsearch to assign relevant tags to the information, improving searchability. The input is the analyzed data, and the output is the data registered on the marketplace platform.

[0647] Step 5:

[0648] Users can search for and purchase registered information through the online marketplace. In doing so, users will use keywords and map search functions to find the information they are looking for. The input is the user's search query accessing the marketplace, and the output is the details of the information selected by the user.

[0649] In this way, the process of automatically collecting, analyzing, and processing information into a marketable format at each step is repeated, making that information available for use by users and third parties.

[0650] (Application Example 1)

[0651] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0652] Providing residents and travelers with real-time and accurate environmental information remains a challenge in modern urban life. Specifically, it is necessary to efficiently collect information such as congestion levels and sound environments and provide it appropriately to residents and travelers. This challenge is important for alleviating urban congestion and facilitating smooth event participation, but conventional methods suffer from a time lag between information collection and provision.

[0653] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0654] In this invention, the server includes means for automatically acquiring environmental information using data collection means, means for performing value evaluation using an artificial intelligence model that analyzes the acquired environmental information, means for registering the evaluated environmental information on a data platform and making it available, and means for presenting the analysis results to residents and travelers in real time. This makes it possible to provide environmental information in real time and with high accuracy.

[0655] "Data collection means" refers to devices or equipment that have the function of automatically collecting environmental information. Specifically, this includes cameras, microphones, GPS functions, etc.

[0656] "Environmental information" refers to data that describes the physical and acoustic conditions of a city or specific area, including congestion levels and noise levels.

[0657] An "artificial intelligence model" refers to an algorithm or framework that analyzes collected environmental information to create valuable information. Image recognition and speech analysis are specific examples.

[0658] "Value assessment" refers to the process of determining the practical value of information based on information analyzed by an artificial intelligence model.

[0659] A "data platform" refers to an online system that stores collected environmental information and provides it in a format accessible to users.

[0660] "Presenting analysis results in real time" refers to a process where data collection and analysis are performed immediately, and the information is provided to users quickly.

[0661] This invention is a system for realizing real-time information provision in urban environments. The entire system mainly consists of data collection means, analysis means, and information provision means.

[0662] Smartphones and tablets are used as data collection devices. These devices are equipped with cameras, microphones, and GPS functions, and automatically collect environmental information. The collected data is transmitted in real time from the device to a server in the cloud.

[0663] On the server side, an analysis method using a generative AI model is in operation. Specifically, machine learning frameworks such as TensorFlow and PyTorch are used to analyze environmental information such as congestion levels and noise levels through image recognition and speech analysis. Based on the collected information, a value assessment is performed.

[0664] The analyzed environmental information is provided to users through a data platform. This allows residents and travelers to access the analysis results in real time through a dedicated application. Low-latency communication protocols are used to ensure real-time information delivery.

[0665] As a concrete example, when a user is moving through an urban area, their device takes photos of the surroundings, records audio, and records location information. This information is immediately sent to a server and analyzed through prompts such as "I want to know the current congestion level." Ultimately, other users can search for information in the form of "Please tell me about the congestion level and event information in urban areas this weekend. In particular, I would like to know the audio level and congestion level around popular event venues in detail," and view the detailed analysis results.

[0666] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0667] Step 1:

[0668] The device acquires information about the user's surroundings. Specifically, it takes photos using the camera, records audio using the microphone, and collects location information using the GPS function. This data is temporarily stored within the device.

[0669] Step 2:

[0670] The device transmits collected environmental information to a server in the cloud. The transmitted data includes photographic data, audio data, and location data, which are sent using a secure communication protocol and are accurately time-stamped.

[0671] Step 3:

[0672] The server analyzes the received environmental information using a generating AI model. Photo data is used to evaluate congestion levels through an image recognition algorithm, and audio data is used for noise level analysis. Location data is used for tagging data based on geographical characteristics. As a result of the analysis, real-time environmental assessment information is generated.

[0673] Step 4:

[0674] The server registers the environmental assessment information obtained through analysis into the data platform. This information is tagged with geographical attributes and relevant keywords, making it searchable by users. It is then integrated into existing databases, enabling responses to searches from other users.

[0675] Step 5:

[0676] Users use a dedicated application to input information about their areas of interest as prompts. For example, they might request information in the form of, "Please tell me about the congestion and events in urban areas this weekend."

[0677] Step 6:

[0678] The server retrieves relevant information from the data platform based on user prompts and provides the user with the most relevant environmental assessment information. This information includes the most recent analysis results, updated in real time.

[0679] Step 7:

[0680] Users can view information provided by the server within the application and use it to adjust and plan their actions. The information is presented in a visually easy-to-understand format, and more detailed information is available as needed.

[0681] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0682] This invention is a system that automatically collects information using a smart device, analyzes that information with an artificial intelligence model to evaluate its value, and further combines it with an emotion engine that recognizes the user's emotions. The user starts by installing the application on their smart device and granting permissions for the camera, microphone, location information, and facial expression recognition.

[0683] The device recognizes the user's emotions by analyzing their facial expressions using its camera and capturing their voice tone using its microphone. Simultaneously, it periodically collects photos, audio, and location data of the environment. The collected data is sent from the device to an agent server, where a multi-layered artificial intelligence model performs analysis.

[0684] The server integrates user sentiment data and environmental data, and evaluates the value of the data based on this. Based on this evaluation, the data is registered on the market platform and made available for other users to purchase. On the market platform, a recommendation algorithm works to prioritize the presentation of information that matches the user's sentiment.

[0685] To give a specific example, when a user visits a shopping area, this system takes photos of the surroundings and recognizes the user's emotions of joy. This information is sent to a server, where the congestion level of the commercial facility and specific promotional information are analyzed in real time. As a result, highly relevant shopping information is presented to the user and sold on the market platform. In this process, the user's emotion data is used to adjust the priority of how individual pieces of information are viewed. In this way, an environment is created where users can quickly and efficiently obtain the information that is best suited to them.

[0686] The following describes the processing flow.

[0687] Step 1:

[0688] User: Install the app on your smart device and grant permissions for camera, microphone, location, and facial recognition. This will prepare your device for data collection.

[0689] Step 2:

[0690] Device: Uses a camera to intermittently capture images of the surrounding environment and the user's facial expressions. Simultaneously, it uses a microphone to monitor surrounding sounds and the user's voice, and acquires location information.

[0691] Step 3:

[0692] Device: Captured images, recorded audio, and location information are compiled into a data package and saved to temporary storage along with timestamps and tags.

[0693] Step 4:

[0694] Terminal: Sends data packages to the agent server via a secure communication channel. Data transmission occurs as soon as the device's network connection becomes available.

[0695] Step 5:

[0696] Server: Inputs the received data package into an artificial intelligence model for analysis. The data is divided into categories, and image recognition, speech analysis, and location information analysis are performed.

[0697] Step 6:

[0698] Server: Uses an emotion engine to recognize emotions from the user's facial expressions and voice. The recognized emotion data is associated with environmental data.

[0699] Step 7:

[0700] Server: Based on environmental and sentiment data, it evaluates the usefulness of information and quantifies its value. Based on this evaluation, it sets the price when registering the information on the market platform.

[0701] Step 8:

[0702] Server: Registers information on the market platform and adjusts the system so that information is presented in order of priority according to the user's emotional state.

[0703] Step 9:

[0704] User: Information seekers access the market platform and search for the information they need. Based on the user's sentiment data, information that is likely to be of interest is presented.

[0705] Step 10:

[0706] Device: Downloads selected information and makes it immediately available to the user. The user then makes decisions based on this information.

[0707] (Example 2)

[0708] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0709] In modern society, providing information tailored to the individual user's emotions and circumstances is extremely important. However, existing information delivery systems have difficulty prioritizing information that accurately reflects the user's emotional state. Furthermore, mechanisms for improving system accuracy through user feedback are not adequately in place. Therefore, there is a need to develop a system that effectively and efficiently provides the most suitable information for the user.

[0710] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0711] In this invention, the server includes means for using an emotion engine that analyzes voice tone and facial expressions to identify the user's emotional state, means for automatically collecting user environmental information and transmitting it to an agent server, and means for analyzing and integrating the emotion data and environmental data using a multi-layered artificial intelligence model. This makes it possible to prioritize and provide optimal information based on the user's emotions.

[0712] An "emotion engine" is a program that incorporates technology to analyze voice tone and facial expressions in order to identify the user's emotional state.

[0713] "Automatic data collection methods" refer to technologies that use a user's smart device to collect photos, audio, and location information of the environment at regular time intervals without human intervention.

[0714] An "agent server" is a computer system that receives data sent from a terminal and performs data analysis using a multi-layered artificial intelligence model.

[0715] A "multi-layered artificial intelligence model" is a learning model with multiple layers, where each layer extracts specific features and is used to perform advanced data analysis.

[0716] A "market platform" refers to a place where analyzed data and its evaluation results are registered, and other users can view and purchase the information.

[0717] A "recommendation algorithm" is a computational method that selects and prioritizes the presentation of highly relevant information based on the sentiment data of individual users.

[0718] The user begins by installing a dedicated application on their smart device. This application needs to obtain permissions for the camera, microphone, location information, and facial recognition. This completes the necessary setup.

[0719] The device, via an installed application, captures the user's face with a camera and analyzes their facial expressions in real time. This facial expression data is used to infer the user's emotional state. Furthermore, the device collects the user's voice using a microphone and analyzes their voice tone to gain further insights into their emotions. Location information and ambient sounds are also collected as needed.

[0720] The server receives the aforementioned data transmitted by the terminal and performs analysis using a multi-layered artificial intelligence model. The AI ​​model uses various analytics modules, including an emotion engine, to perform advanced analysis of the collected data. The data obtained from the analysis is integrated, and a value assessment is made based on the user's emotional state and environmental conditions.

[0721] The evaluation results are registered on the market platform. Here, a recommendation algorithm works to prioritize providing information that best matches the user's emotions. For example, if a user is in a shopping center, a system that identifies their smiling face will provide specific promotional information or new product announcements in real time.

[0722] In this way, a system is realized that utilizes user sentiment data to quickly provide appropriate information. An example of a prompt would be, "Select and present the most relevant content based on the user's sentiment data." Such prompts function as instructions for the system to select the most suitable information for the user.

[0723] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0724] Step 1:

[0725] The user installs the application on their smart device and grants the necessary permissions. These include permissions for the camera, microphone, location information, and facial recognition. Once permissions are granted, the system is ready to proceed to the next data collection stage.

[0726] Step 2:

[0727] The device uses a camera to capture the user's facial expressions in real time. The input is video of the user's face. This video data is then analyzed using a facial expression analysis algorithm to obtain output that converts subtle facial changes into emotional states. For example, emotions such as smiles and surprises are quantified and output.

[0728] Step 3:

[0729] The device uses a microphone to collect the user's voice. The voice data is taken in as input, and by analyzing its tone and speed, supplementary information about emotions is output. For example, it calculates indicators of tension or relaxation from the pitch and tempo of the voice.

[0730] Step 4:

[0731] The device obtains the user's current location through the smart device's location services. Location data is provided as input, and environmental characteristic information is output. This includes geographical features of the current location and information about nearby facilities.

[0732] Step 5:

[0733] The server receives facial expression, voice, and location data transmitted by the terminal. The data received as input is analyzed using a multi-layered artificial intelligence model. Through data processing and calculations, it generates an output that integrates the user's emotional state and environmental characteristics.

[0734] Step 6:

[0735] The server evaluates the market value of the information based on the generated analysis results. An evaluation report is created as output. This report includes recommendations based on the user's emotional state and is prepared for registration on the next market platform.

[0736] Step 7:

[0737] The server registers the evaluation report on the market platform. As an output, the platform displays products and information suitable for the user. The recommendation algorithm dynamically adjusts the priority of information based on sentiment data, enabling it to provide the user with the most relevant information.

[0738] (Application Example 2)

[0739] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0740] Providing users with the information they need in real time requires personalized information based on their emotions and circumstances. However, conventional information delivery systems have struggled to recognize users' emotions and prioritize information based on them. Furthermore, they have been unable to link the value assessment of information to users' emotions, resulting in a decrease in the efficiency of information delivery.

[0741] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0742] In this invention, the server includes means for automatically acquiring information using a data collection device, means for performing value evaluation using an artificial intelligence model that analyzes the acquired information, means for analyzing the user's emotions using an emotion recognition device that acquires the user's facial expressions and voice tone, and means including a recommendation device that preferentially provides relevant information based on the user's emotions. This enables the user to quickly and appropriately obtain information that is in line with their emotions and environment.

[0743] A "data collection device" is a device that automatically acquires various types of information, and is used to collect images, audio, location information, and so on.

[0744] An "artificial intelligence model" is a program consisting of mathematical and logical structures for analyzing acquired information and performing value assessments.

[0745] A "marketplace platform" is an online or offline space where evaluated information can be registered and offered or sold to other users.

[0746] An "emotion recognition device" is a device that analyzes a user's emotional state based on their facial expressions and voice tone.

[0747] A "recommendation system" is a system that prioritizes providing relevant information based on the user's emotions and circumstances.

[0748] The system that realizes this invention mainly consists of a data collection device, an emotion recognition device, an artificial intelligence model, and a recommendation device. The user installs a dedicated application on a smart device and grants the necessary permissions. The device utilizes built-in sensors to capture the user's facial expressions with a camera and acquire voice with a microphone, thereby recognizing emotions from facial expressions and voice tone. This information is transmitted to a server in the cloud.

[0749] The server analyzes the transmitted data using artificial intelligence models based on services such as Google Cloud AI and Amazon SageMaker, and performs value assessments based on the user's emotions and environment. Simultaneously, the server accesses a database stored in MongoDB Atlas to find information relevant to the user. Then, based on the understood emotions and assessed information, it uses Google Firebase Cloud Messaging to send appropriate information to the user via real-time push notifications.

[0750] As a concrete example, while a user is visiting a shopping mall, the device detects expressions of joy on the user's face and sends the data to the cloud. Based on the emotional data and location information within the mall, the server selects special offers and new product information from specific stores and pushes this information to the user. This allows users to receive information tailored to their interests and mood, enriching their shopping experience.

[0751] An example of a prompt to input into a generative AI model would be, "Estimate the user's purchase intent from their facial expressions and provide relevant product information." This would enable the system to provide highly personalized information based on the user's facial expressions.

[0752] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0753] Step 1:

[0754] The device acquires the user's facial expressions and voice. It collects facial images in real time via the camera and captures voice data using the microphone. At this stage, the input is the camera image and voice input, and the output is raw data for emotion recognition.

[0755] Step 2:

[0756] The terminal sends the acquired facial expressions and audio data to a data preprocessing module for noise reduction and standardization. This improves the accuracy of the data. The input in this step is raw data, and the output is processed data.

[0757] Step 3:

[0758] The terminal sends the processed data to a server in the cloud. The server receives the data and prepares it for input into the AI ​​model. The input is the processed data, and the output is the data to be input to the AI ​​model.

[0759] Step 4:

[0760] The server uses an artificial intelligence model to analyze the received data and estimate the user's emotional state. For example, it might use a facial recognition API or a voice analysis library. The input at this stage is data for the AI ​​model, and the output is the estimated emotional data.

[0761] Step 5:

[0762] The server uses estimated sentiment data to retrieve relevant product and promotional information from the database. It accesses MongoDB Atlas to select information that matches the user's sentiment. The input here is sentiment data and database information, and the output is a list of information for the user.

[0763] Step 6:

[0764] The server uses Google Firebase Cloud Messaging to push selected information to the device. Users receive this notification and can check information of interest in real time. The input is a list of information for the user, and the output is a notification message displayed on the device.

[0765] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0766] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0767] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0768] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0769] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0770] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0771] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0772] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0773] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0774] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0775] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0776] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0777] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0778] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0779] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0780] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0781] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0782] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0783] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0784] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0785] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0786] The following is further disclosed regarding the embodiments described above.

[0787] (Claim 1)

[0788] A means of automatically acquiring information using a data collection device,

[0789] A means of performing value assessment using an artificial intelligence model that analyzes acquired information,

[0790] A means of registering the evaluated information on a market platform and making it available for sale,

[0791] A system that includes this.

[0792] (Claim 2)

[0793] The system according to claim 1, further comprising means for assigning information as searchable attributes based on location data of acquired information.

[0794] (Claim 3)

[0795] The system according to claim 1, further comprising means for collecting user feedback and adjusting the parameters of an artificial intelligence model based thereon.

[0796] "Example 1"

[0797] (Claim 1)

[0798] A means of automatically acquiring information using the user's device,

[0799] A means of performing evaluation using an artificial intelligence model that analyzes acquired video, audio, and location data,

[0800] A means to register the analyzed information on an electronic market and enable its distribution,

[0801] A system that includes this.

[0802] (Claim 2)

[0803] The system according to claim 1, comprising means for assigning relevant features based on the analyzed information to make the information easier to search.

[0804] (Claim 3)

[0805] The system according to claim 1, comprising means for collecting user opinions and optimizing the settings of an artificial intelligence model based on those opinions.

[0806] "Application Example 1"

[0807] (Claim 1)

[0808] A means for automatically acquiring environmental information using data collection means,

[0809] A means of performing value assessment using an artificial intelligence model that analyzes acquired environmental information,

[0810] A means to register the evaluated environmental information on a data platform and make it available,

[0811] A means of presenting analysis results to residents and tourists in real time,

[0812] A system that includes this.

[0813] (Claim 2)

[0814] The system according to claim 1, further comprising means for assigning environmental information as searchable attributes based on geographical data of acquired environmental information, and presenting it in a form accessible to residents and travelers.

[0815] (Claim 3)

[0816] The system according to claim 1, comprising means for collecting feedback from users and adjusting the parameters of an artificial intelligence model based on that feedback, and providing useful information for residents and travelers.

[0817] "Example 2 of combining an emotion engine"

[0818] (Claim 1)

[0819] A means of using an emotion engine that analyzes voice tone and facial expressions to identify the user's emotional state,

[0820] A means for automatically collecting user environment information and sending it to an agent server,

[0821] A means of analyzing and integrating emotional and environmental data using a multi-layered artificial intelligence model,

[0822] A means of evaluating the market value of information based on the analysis results and registering the results on a market platform,

[0823] A means of prioritizing the presentation of information that matches the user's emotional state using a recommendation algorithm,

[0824] A system that includes this.

[0825] (Claim 2)

[0826] The system according to claim 1, comprising means for assigning information as searchable attributes based on location data of acquired information.

[0827] (Claim 3)

[0828] The system according to claim 1, further comprising means for collecting user feedback and adjusting the parameters of an artificial intelligence model based thereon.

[0829] "Application example 2 when combining with an emotional engine"

[0830] (Claim 1)

[0831] A means of automatically acquiring information using a data collection device,

[0832] A means of performing value assessment using an artificial intelligence model that analyzes acquired information,

[0833] A means of registering the evaluated information on a market platform and making it available for sale,

[0834] A means of analyzing a user's emotions using an emotion recognition device that acquires the user's facial expressions and voice tone,

[0835] A means including a recommendation device that prioritizes providing relevant information based on the user's emotions,

[0836] A system that includes this.

[0837] (Claim 2)

[0838] The system according to claim 1, further comprising, in addition to the means for assigning information as searchable attributes based on location data of acquired information, an attribute assignment means that takes into account user sentiment data.

[0839] (Claim 3)

[0840] The system according to claim 1, further comprising means for collecting user feedback and adjusting the parameters of an artificial intelligence model based thereon, as well as means for using user emotional data as feedback data. [Explanation of Symbols]

[0841] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of automatically acquiring information using a data collection device, A means of performing value assessment using an artificial intelligence model that analyzes acquired information, A means of registering the evaluated information on a market platform and making it available for sale, A system that includes this.

2. The system according to claim 1, further comprising means for assigning information as searchable attributes based on location data of acquired information.

3. The system according to claim 1, further comprising means for collecting user feedback and adjusting the parameters of an artificial intelligence model based thereon.

Images