System

Abstract

A system is provided.SOLUTION: The system includes a user terminal means for inputting a landscape image, a server means for receiving the input landscape image, an image recognition algorithm means for extracting a feature amount from the received landscape image, a database collation means for comparing the extracted feature amount with a collation database, a position specifying means for acquiring latitude and longitude information of the landscape image based on a collation result, an information transmission means for transmitting the acquired latitude and longitude information to the user terminal, and a map display means for displaying the latitude and longitude information acquired by the user terminal on a map.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While conventional image search technology can identify distinctive locations such as famous buildings and famous tourist spots, it has been difficult to identify the specific locations where images were taken of ordinary places or landscapes without any special features. Therefore, there is a need for technology that can accurately identify photo locations for a wide range of applications, such as checking memorable places and analyzing incidents, accidents, and disasters. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means: a system including a user terminal means for inputting a landscape image, a server means for receiving the input landscape image, an image recognition algorithm means for extracting features from the received landscape image, a database comparison means for comparing the extracted features with a comparison database, a location identification means for acquiring latitude and longitude information of the landscape image based on the comparison result, an information transmission means for transmitting the acquired latitude and longitude information to the user terminal, and a map display means for displaying the latitude and longitude information acquired by the user terminal on a map. This system enables a user to accurately identify the shooting location of any landscape image.

[0006] A "scenic image" is a still image that shows a particular geographic location or scene.

[0007] "User terminal" refers to a device used by a user to input, send, and receive landscape images, and includes smartphones, PCs, tablets, etc.

[0008] A "server" is a computer system that receives and processes scenic images sent from user terminals.

[0009] An "image recognition algorithm" is a program or method for extracting features from landscape images, such as those that use convolutional neural networks (CNNs).

[0010] "Features" are data that quantifies important information contained in a landscape image, and include elements such as edges, colors, and textures.

[0011] A "database" is a warehouse containing a large amount of image data used for matching, including image data with geographical information such as street views.

[0012] The "matching means" is a program or method for comparing the extracted features with image data in a database to find matching images.

[0013] "Latitude and longitude information" is two-dimensional coordinate data that indicates a specific geographical location, and consists of latitude and longitude used to indicate a location on Earth.

[0014] The "position specifying means" is a program or method for obtaining accurate latitude and longitude information of a scenic image based on the collation result.

[0015] The "information transmitting means" is a program or method for transmitting the acquired latitude and longitude information to the user terminal.

[0016] The "map display means" is a program or method for visually displaying the shooting location on a map based on the latitude and longitude information acquired by the user terminal. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is a system for identifying the location where a landscape image was taken, in which a user uploads the landscape image from a terminal, a server processes the image, obtains specific location information, and provides the information to the user. Specific embodiments of the system are described below.

[0039] When a user wants to specify a landscape image, he or she selects the landscape image using his or her own device (e.g., a smartphone or a PC) and uploads it to the system. The user device then sends the landscape image to the server in a predetermined format (e.g., JPEG, PNG). The image upload process can use a general file upload function.

[0040] The server receives landscape images sent from the user's device. The images are temporarily stored on the server, and then feature extraction is performed. Feature extraction uses advanced image recognition algorithms such as convolutional neural networks (CNNs). These algorithms analyze information such as edges, colors, and textures in the images to extract highly distinctive data.

[0041] The server then uses the extracted features to match the image against the Street View database. This database matching process is performed by matching with a large amount of Street View image data. The server then ranks the resulting candidate images based on similarity scores and selects the image that appears to be the best match. The server then obtains the location information (latitude and longitude) of the selected image.

[0042] The server then sends the acquired latitude and longitude information back to the user's device in a general-purpose format such as JSON. The user's device receives the latitude and longitude information sent from the server and enters it into a map display application (for example, Google Maps). At this time, a marker such as a pin is displayed on the map based on the acquired location information, allowing the user to visually confirm the location where the photo was taken.

[0043] For example, if a user takes a photo of an anonymous beautiful landscape from their travels, the system can pinpoint the exact location of the place, allowing the user to visually confirm the location on a map, revisit it, or share the information with others.

[0044] This system makes it possible to accurately pinpoint the location where an image was taken for a wide range of applications, including identifying memorable locations, analyzing incidents and accidents, and even academic research.

[0045] The processing flow will be explained below.

[0046] Step 1:

[0047] The user opens the system's web application from their device (smartphone or PC), clicks the upload button, and selects a landscape image.

[0048] Step 2:

[0049] The device sends the selected landscape image to a designated API endpoint to upload it to the system. The image data is sent in a standard image format such as JPEG or PNG.

[0050] Step 3:

[0051] The server receives the landscape images sent from the user terminal and stores the image data in temporary storage within the server.

[0052] Step 4:

[0053] The server then passes the stored landscape images to an image recognition algorithm, which extracts features from the images. The algorithm uses cutting-edge technologies such as convolutional neural networks (CNNs). The algorithm analyzes important information such as edges, colors, and textures in the images and digitizes them.

[0054] Step 5:

[0055] The server compares the extracted feature data with the Street View database, thereby searching for similar images from the massive Street View image database.

[0056] Step 6:

[0057] The server ranks the candidate images obtained as a result of the matching based on their similarity scores, selects the image with the highest similarity score, and obtains the latitude and longitude information corresponding to that image.

[0058] Step 7:

[0059] The server encodes the acquired latitude and longitude information in a standard format such as JSON and prepares to send it back to the user's terminal.

[0060] Step 8:

[0061] The terminal receives the latitude and longitude information sent from the server, analyzes the received data, and converts it into an appropriate format.

[0062] Step 9:

[0063] The device inputs the acquired latitude and longitude information into a map display application (e.g., Google Maps) and displays the identified shooting location on the map. A pin or marker is placed on the map to visually indicate the shooting location to the user.

[0064] Step 10:

[0065] The user can check the map displayed on the device screen and view information about the shooting location. The user can also check detailed location information and information about surrounding facilities.

[0066] Example 1

[0067] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0068] Identifying the location where a landscape image was taken is extremely difficult using conventional methods, requiring a great deal of time and effort. Furthermore, there are only a limited number of systems for accurately identifying specific location information, and few of them are easily accessible to general users. Therefore, there is a need for a system that allows users to easily and quickly identify the location where a landscape image was taken and obtain detailed location information.

[0069] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0070] In this invention, the server includes means for receiving a landscape image and storing it in temporary storage, means for extracting features from the received landscape image, and means for comparing the extracted features with a database, thereby enabling the location where the landscape image was taken to be accurately identified and the latitude and longitude information of that location to be obtained.

[0071] The "user terminal means" is a device that allows a user to input a landscape image and transmit the image to the server.

[0072] The "server means" is a device that receives and temporarily stores scenic images sent from a user terminal.

[0073] The "image recognition algorithm means" is an algorithm for extracting features from the received landscape image.

[0074] The "database matching means" is a means for matching the extracted feature amount with information in a database.

[0075] The "position specifying means" is a means for specifying the latitude and longitude information of the scenic image based on the result of the comparison.

[0076] The "information transmitting means" is a means for transmitting the acquired latitude and longitude information to the user terminal.

[0077] The "map display means" is a means for displaying the latitude and longitude information acquired by the user terminal on a map.

[0078] The present invention provides a system for identifying the location where a landscape image was taken. A user uploads a landscape image using their own terminal, and a server analyzes the image to identify the location where the image was taken and provides the location information to the user. A specific embodiment is described below.

[0079] System Overview

[0080] The main purpose of this system is to identify the location of landscape images taken by users. The user device sends the image in a specified format (e.g., JPEG, PNG) to the server, which then analyzes the image to obtain location information. The analysis uses advanced image recognition algorithms such as convolutional neural networks (CNN).

[0081] Specific processing

[0082] User terminal

[0083] The user terminal uses a general device such as a smartphone or PC to select a landscape image and upload it to the system. The image upload process uses a general file upload function.

[0084] server

[0085] The server receives landscape images sent from the user's device and stores them in temporary storage. It then uses an image recognition algorithm to extract features from the received images. This extraction uses generative AI models such as TensorFlow and PyTorch. Specifically, it analyzes information such as edges, colors, and textures within the images to extract highly distinctive data.

[0086] Database collation

[0087] The server compares the extracted features with existing data in a database, which contains a large amount of Street View image data. The server then ranks the results based on similarity scores and selects the best matching image. The location information (latitude and longitude) of the selected image is then obtained.

[0088] Information transmission and display

[0089] The server sends the acquired latitude and longitude information to the user's device using a general-purpose format such as JSON. The user's device receives the location information sent from the server and enters it into a map display application (e.g., Google Maps). This application displays a pin (marker) on the map, allowing the user to visually confirm the location where the photo was taken.

[0090] Specific examples

[0091] For example, if a user inputs a photo of an unknown beautiful landscape taken during a trip into the system, the system can identify the location where the photo was taken. By checking the identified location information on a map, the user can revisit the location or share the information with others.

[0092] Prompt Sentence Examples

[0093] An example of a prompt to input to a generative AI model might be, "I want to identify the exact location of a landscape I photographed while traveling. I uploaded this image. Please tell me where it was taken."

[0094] This system makes it possible to accurately pinpoint the location where an image was taken for a wide range of purposes, including identifying memorable places, analyzing incidents and accidents, and academic research.

[0095] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0096] Step 1:

[0097] The user selects a landscape image using their device. Open the file selection dialog on the device and select the desired landscape image. In this case, the image file must be in a common format (e.g., JPEG, PNG). The input is the landscape image file, and the output is the path to the image file selected by the user.

[0098] Step 2:

[0099] The user presses the upload button to send a landscape image to the server. The device browser generates an HTTP request to send the image file to the server. The input is the path of the selected image file, and the output is the HTTP request sent to the server.

[0100] Step 3:

[0101] The server receives the landscape image sent from the user device and saves it in temporary storage. The server processes the received image and saves the image file in a specified directory in the file system. The input is the HTTP request from the user device, and the output is the image file saved in the server's temporary storage.

[0102] Step 4:

[0103] The server reads the images stored in temporary storage and runs an image recognition algorithm to extract features. A convolutional neural network (CNN) is used to extract features such as edges, colors, and textures within the image. The input is the image file stored in temporary storage, and the output is the extracted feature data.

[0104] Step 5:

[0105] The server compares the extracted features with a database. The feature data is used to compare with the Street View database using SQL queries or machine learning algorithms. The input is the extracted feature data, and the output is the matching result (multiple candidate images and similarity scores).

[0106] Step 6:

[0107] The server selects the best matching image based on the matching results and obtains the location information (latitude and longitude) of that image. A scoring algorithm is used to rank the candidate images and obtain the location information of the best matching image. The input is the matching result, and the output is the identified latitude and longitude information.

[0108] Step 7:

[0109] The server converts the acquired latitude and longitude information into a general-purpose format such as JSON and sends it to the user device. The server generates an HTTP response and returns the location information to the device. The input is the specified latitude and longitude information, and the output is JSON data sent to the user device.

[0110] Step 8:

[0111] The user device receives the JSON data sent from the server and inputs it into the map display application. The device parses the JSON data to extract latitude and longitude information and displays a pin (marker) on the map using the map display API. The input is the received JSON data, and the output is a pin for the photo location displayed on the map.

[0112] (Application example 1)

[0113] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0114] Currently, there is no efficient system for quickly and accurately identifying the location of landscape images taken by many security personnel at the scene. As a result, responses to incidents and accidents are delayed and difficult to deal with. Furthermore, conventional systems have the problem that it takes time to identify location information, making it difficult to respond in real time. A new system is needed to solve these problems and improve the efficiency of security operations.

[0115] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0116] In this invention, the server includes a data processing device means for receiving input landscape images, an image recognition algorithm means for extracting features from the received landscape images, and a database matching means for comparing the extracted features with a matching database, thereby enabling security personnel to quickly and accurately identify the location information of landscape images taken on-site.

[0117] An "information processing device" is a terminal device used by a user to input a landscape image. This includes smartphones, tablets, personal computers, and the like.

[0118] The "data processing device" is a server or cloud computing system that receives and processes landscape images sent by users.

[0119] An "image recognition algorithm" is an algorithm for extracting features from received landscape images. A typical example is a convolutional neural network.

[0120] "Database matching" is the process of comparing the extracted features with an existing database, thereby identifying the location of the photo.

[0121] "Location identification" is the process of acquiring location information for a landscape image based on the matching results, including latitude and longitude information.

[0122] "Information transmission" is a process of transmitting the acquired location information to the user's information processing device.

[0123] "Map display" is a process of displaying location information acquired on the user's information processing device on a map. This is done using a map application.

[0124] The "interface" is a user interface that allows an information processing device to use input landscape images for security purposes. It provides a function for quickly reporting suspicious situations or incidents.

[0125] This invention is a system for identifying the location where a landscape image was taken, in which the landscape image is uploaded from a user's information processing device, the data processing device processes the image to obtain specific location information, and provides that information back to the user. This system is particularly suitable for security applications.

[0126] Hardware and Software Configuration

[0127] 1. Information processing equipment:

[0128] A general user device such as a smartphone, tablet, or PC. It is used by users to upload landscape images they have taken on-site.

[0129] 2. Data Processing Unit:

[0130] It consists of a server and a cloud computing system. This device receives and processes landscape images sent by users, temporarily storing the images, extracting features, and performing matching.

[0131] 3. Image Recognition Algorithm:

[0132] A typical example is a convolutional neural network (CNN) using TensorFlow. Features are extracted from the received landscape image and compared with a database.

[0133] 4. Matching database:

[0134] It can use the Street View image database of Google Cloud Vision API, match it with the extracted features, and obtain the location information of the best-matching candidate.

[0135] 5. Map display:

[0136] The acquired location information is displayed on the user's information processing device using the Google Maps API.

[0137] Processing Flow

[0138] 1. Upload your image:

[0139] The user takes a picture of the scenery using an information processing device and uploads it to the server through the app. React Native can be used.

[0140] 2. Receiving images and extracting features:

[0141] The server stores the received images in temporary storage and extracts features using a convolutional neural network (CNN).

[0142] 3. Database Matching:

[0143] The extracted features are compared with a matching database to identify the location information of the best-matching candidate.

[0144] 4. Obtaining and sending location information:

[0145] Location information is obtained based on the matching results and returned to the user's information processing device in JSON format or similar.

[0146] 5. Map display:

[0147] The user's information processing device displays the acquired location information on a map using the Google Maps API.

[0148] Specific examples

[0149] Assume that a security officer discovers a suspicious person at the scene, takes a picture of the person, and uploads it. The server processes the received image and obtains specific location information. The location information is sent to the user's information processing device and displayed on a map in real time, enabling a prompt response.

[0150] Prompt Sentence Examples

[0151] "Write a Python program to upload a specific landscape image and display a marker on a map based on the latitude and longitude of that location. This program should also include image feature extraction using TensorFlow and Street View matching using the Google Cloud Vision API."

[0152] The above is a specific embodiment for carrying out the invention. This system makes security operations more efficient and enables quick responses.

[0153] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0154] Step 1:

[0155] A user takes landscape images of the site using an information processing device (e.g., a smartphone), selects the images, and uploads them to the server through the system's application. The input is the captured landscape image, and the output is the completion of the upload process to the server.

[0156] Step 2:

[0157] The server receives the landscape image uploaded by the user. The specific operation of the server is to store the image in temporary storage. The input of this step is the landscape image sent by the user, and the output is the image stored in the temporary storage.

[0158] Step 3:

[0159] The server extracts features from the stored landscape images. This process uses a convolutional neural network (CNN) using TensorFlow. The input is the stored landscape image, and the output is the features extracted from the image. The features are data such as edges, colors, and textures within the image.

[0160] Step 4:

[0161] The server compares the extracted features with a matching database (e.g., the Street View image database of the Google Cloud Vision API). Specifically, it matches the existing landscape image features in the database and calculates the similarity. The input is the extracted features, and the output is a list of best-matching candidate images and their similarity scores.

[0162] Step 5:

[0163] The server selects the best-matching candidate image based on the similarity score and obtains its location information (latitude and longitude). The input is a list of candidate images from the matching results, and the output is the location information of the selected image. The location information is provided in latitude and longitude format.

[0164] Step 6:

[0165] The server returns the acquired location information to the information processing device. Specifically, it packages the location information in JSON format and sends it to the user's information processing device. The input is the acquired location information, and the output is the completion of transmission to the information processing device.

[0166] Step 7:

[0167] The user's information processing device displays the received location information on a map using the Google Maps API. Specifically, it displays a pin on the map based on the acquired latitude and longitude information, visually indicating the location where the landscape image was taken. The input is the location information sent from the server, and the output is the location information displayed on the map application.

[0168] The above are the specific processing steps and operation flow of the system that realizes this application example.

[0169] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0170] The present invention is a system that provides interactive feedback according to the user's emotional state by combining a system that identifies the shooting location of a landscape image with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0171] Users upload landscape images to the system using their own devices (e.g., smartphones or PCs). At this time, the user's device captures not only the landscape images but also the user's facial expressions and voice data, which allows the system to estimate the user's emotional state.

[0172] The device can send the user's emotional data along with the landscape image to the server. This emotional data is obtained as a result of analyzing facial expressions and voice. For example, if the user is happy when uploading an image, the facial expression data is sent to the server.

[0173] The server receives the landscape images and emotion data sent from the user's device and uses an image recognition algorithm to extract features from the landscape images. The extracted features are compared with the Street View database to identify the most matching image. At this time, the emotion engine analyzes the user's emotional state, and the results are also used for feedback.

[0174] For example, if a user uploads a landscape image and the emotion engine recognizes the user's joy or excitement, the server can provide positive feedback that includes not only the identified location but also related and background information about the location. For example, information like "This place has beautiful scenery and is visited by many tourists" could be added.

[0175] The server transmits the identified latitude and longitude information and feedback information based on the user's emotion to the user device. The user device inputs this information into a map display application and visually displays the shooting location on a map. Furthermore, customized information based on the user's emotion can also be displayed.

[0176] This system not only allows users to accurately identify the location where a landscape image was taken, but also provides interactive and rich feedback based on their emotions. For example, if the user is relaxed, detailed information about the location's history and cultural background may be displayed. On the other hand, if the user is excited, information about activities and events in the location may be provided, enriching the user's experience.

[0177] Systems equipped with additional functions based on this emotion engine can be used in a wide range of fields, including tourist guides, education, remote exploration, and incident and accident analysis. Providing information linked to the user's emotions will further increase the system's usefulness.

[0178] The processing flow will be explained below.

[0179] Step 1:

[0180] The user opens the system's web application from their device (smartphone or PC) and uploads a landscape image. The user opens the file selection window, selects the landscape image, and clicks the upload button.

[0181] Step 2:

[0182] The device sends the selected landscape image to a specific API endpoint, while capturing the user's facial expressions with a camera and collecting voice data, which are then processed by the emotion engine.

[0183] Step 3:

[0184] The emotion engine analyzes the captured facial and voice data to determine the user's emotional state, which may be expressed as, for example, joy, sadness, surprise, or the like.

[0185] Step 4:

[0186] The device sends the emotion data from the emotion engine along with the landscape image data to the server. The landscape image data is often sent in JPEG or PNG format, and the emotion data is sent in JSON format.

[0187] Step 5:

[0188] The server receives the landscape image data and emotion data sent from the user device. The landscape image is temporarily stored in storage and then passed to the image recognition algorithm.

[0189] Step 6:

[0190] The server uses an image recognition algorithm (e.g., a convolutional neural network) to extract features from the landscape image, such as edge, color, and texture data.

[0191] Step 7:

[0192] The server then matches the extracted features with the Street View database, comparing the features with images in the database and identifying the most similar images.

[0193] Step 8:

[0194] The server selects the image with the highest similarity from the images obtained as a result of the comparison, and obtains the latitude and longitude information corresponding to that image.

[0195] Step 9:

[0196] The server then encodes the acquired latitude and longitude information in a standard format such as JSON and sends it back to the user's device, along with a customized message based on the user's emotional data.

[0197] Step 10:

[0198] The device receives the latitude and longitude information and emotion-based messages sent from the server, analyzes the received data, and converts it into an appropriate format.

[0199] Step 11:

[0200] The device inputs the acquired latitude and longitude information into a map display application (e.g., Google Maps) and displays the identified shooting location on the map. At this time, customized messages and information based on the emotion data are also displayed.

[0201] Step 12:

[0202] The user sees a map and a customized message displayed on the device screen. For example, if the user is happy, positive information about the location (tourist attractions and background information) is also displayed. On the other hand, if the user is sad, a comforting message may be displayed.

[0203] As a result, a system that combines an emotion engine can provide interactive and comprehensive feedback that corresponds to the user's emotional state.

[0204] Example 2

[0205] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] Existing landscape image location identification systems only identify the location of landscape images taken by the user, but are unable to provide interactive feedback that takes into account the user's emotional state. As a result, the information available to the user is limited, resulting in a lackluster user experience. Furthermore, they are unable to provide information that takes into account the context of the user's emotional state, making it difficult to provide flexible feedback that can be adapted to multiple uses.

[0207] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes user terminal means for inputting a landscape image and emotion data, server means for receiving the input landscape image and emotion data, image recognition algorithm means for extracting features from the received landscape image, emotion analysis means for analyzing the user's emotional state from the received emotion data, database comparison means for comparing the extracted features with a comparison database, location identification means for acquiring latitude and longitude information of the landscape image based on the comparison result, feedback generation means for generating feedback information based on the latitude and longitude information acquired by the location identification means and the emotion data analyzed by the emotion analysis means, information transmission means for transmitting the generated latitude and longitude information and feedback information to the user terminal, and map display means for displaying the latitude and longitude information and feedback information acquired by the user terminal on a map. This allows the user to not only accurately identify the location where the landscape image was taken but also receive interactive and comprehensive feedback tailored to the user's emotional state.

[0208] A "user terminal" is an electronic device that a user uses to input and transmit scenery images and emotion data.

[0209] The "server" is a computer system that processes and analyzes landscape images and emotion data received from user devices, and generates location identification and feedback information.

[0210] An "image recognition algorithm" is an analytical method for extracting features from landscape images and comparing them with a database.

[0211] "Emotion analysis" is the process of recognizing and determining a user's emotional state from facial expressions and voice data.

[0212] "Database matching" is the process of comparing and matching the features extracted from a landscape image with an existing database.

[0213] "Location identification" is a process of obtaining latitude and longitude information based on the results of matching the scenery image.

[0214] "Feedback generation" refers to the act of generating customized feedback information to provide to a user based on the location-identified information and the results of sentiment analysis.

[0215] "Information transmission" refers to the process of transmitting the generated latitude and longitude information and feedback information to the user terminal.

[0216] "Map display" is a function that visually displays the acquired latitude and longitude information and feedback information on a map.

[0217] The present invention provides a system that provides interactive feedback according to the user's emotional state by combining a system that identifies the shooting location of a landscape image with an emotion analysis function that recognizes the user's emotions. Specific embodiments of the system are described below.

[0218] Users upload landscape images to the system using their own devices (e.g., smartphones or PCs). At this time, the user's device captures not only the landscape images but also the user's facial expressions and voice data, which allows the system to estimate the user's emotional state.

[0219] The device sends the captured emotion data (facial expressions and voice analysis results) along with the landscape image to the server. The communication protocol is HTTP / HTTPS, and the data is sent in JSON format.

[0220] The server processes the received landscape images and emotion data. For landscape images, it uses an image recognition algorithm to extract features. In this case, it uses a convolutional neural network (CNN), an artificial intelligence model (AI model). The extracted features are compared with a landscape database to identify the most similar images.

[0221] Next, the server uses an emotion analysis engine to analyze the user's emotional data. This analysis method uses a general emotion recognition API (e.g., emotion analysis algorithm). The analysis result determines the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0222] The server generates customized feedback for the user based on the results of image recognition and emotion analysis. Specifically, it generates a positive message based on the user's emotions and tourist information about the identified location, along with the latitude and longitude information of the location. The generated feedback information is then sent back to the user's device in JSON format.

[0223] The user terminal inputs the information received from the server into a map display application (e.g., a map application) and visually displays the shooting location of the landscape image on a map. It also displays customized feedback information according to the user's emotions.

[0224] For example, if a user uploads a photo of a beach from their smartphone and the sentiment analysis engine recognizes the user's joy, the server will generate customized feedback such as, "This place is a popular tourist spot with many people and offers beautiful sunrises." This information is sent to the user's device and visually displayed on a map.

[0225] Example prompt sentences

[0226] An example of a prompt is as follows:

[0227] "Upload a landscape image and generate positive feedback based on how I feel when I'm happy."

[0228] In this way, the system of the present invention allows users to not only accurately identify the location where a landscape image was taken, but also to receive interactive and rich feedback according to their emotions. This system can be used in a wide range of fields, including tourist guides, education, remote exploration, and incident and accident analysis.

[0229] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0230] Step 1:

[0231] Users take landscape images using their own devices (e.g., smartphones or PCs) and upload them to the system. The device captures the landscape images taken with the camera as well as the user's facial expression and voice data. The input data are the landscape images and the facial and voice data. This allows us to collect initial data for estimating the user's emotional state.

[0232] Step 2:

[0233] The user device sends the captured landscape image and emotion data to the server. The data is packaged in JSON format and sent to the server via HTTP / HTTPS protocol. The input data are the landscape image and emotion data, and the output data are the landscape image and emotion data sent to the server.

[0234] Step 3:

[0235] The server analyzes the landscape images received from the user terminal using an image recognition algorithm. Specifically, it uses an artificial intelligence model (e.g., a convolutional neural network) to extract features from the landscape images. The input data is the landscape image, and the output data is the extracted features. The image recognition algorithm performs extraction and analysis based on these features.

[0236] Step 4:

[0237] The server analyzes the received emotion data using an emotion analysis engine. The algorithm used is a general emotion recognition API (e.g., emotion analysis algorithm). The input data is emotion data, and the output data is the user's emotional state (e.g., joy, sadness, surprise, etc.). The analyzed emotion data serves as the basis for generating feedback.

[0238] Step 5:

[0239] The server compares the extracted features with a matching database. The database stores landscape image data, and by matching the features with this data, it identifies the image that best matches. The input data are the extracted features, and the output data are the identified latitude and longitude information.

[0240] Step 6:

[0241] The server generates feedback for the user based on the acquired latitude and longitude information and the emotion analysis results. Specifically, it generates tourist information for the identified location and positive messages based on the user's emotions. The input data are the latitude and longitude information and the emotion analysis results, and the output data is customized feedback information.

[0242] Step 7:

[0243] The server sends the generated latitude and longitude information and feedback information to the user terminal. The data is again packaged in JSON format and sent via HTTP / HTTPS protocol. The input data is the generated latitude and longitude information and feedback information, and the output data is the information sent to the user terminal.

[0244] Step 8:

[0245] The user terminal inputs the latitude and longitude information and feedback information received from the server into a map display application, and visually displays the shooting location of the scenic image on a map. Customized feedback information according to the user's emotions is also displayed. The input data is the latitude and longitude information and feedback information received from the server, and the output data is the map display and the feedback display.

[0246] As a concrete example, if a user uploads a photo of a beach from their smartphone and the sentiment analysis engine recognizes the user's joy, the server generates customized feedback such as "This place is a popular tourist spot with many people, and you can see a beautiful sunrise," and sends it to the user's device. The user's device can display this on a map, providing the user with positive feedback along with visual information.

[0247] Prompt Sentence Examples

[0248] "Upload a landscape image and generate positive feedback based on how I feel when I'm happy."

[0249] (Application example 2)

[0250] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0251] In conventional systems for identifying landscape images, users simply identify the location where the landscape was photographed. This results in only providing flat information that ignores the user's emotional state, making it difficult to provide an interactive and rich experience tailored to the scene. Furthermore, there is a need for a system that can provide not only landscape images but also information customized to the user's emotional state.

[0252] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes user device means for inputting a landscape image, server device means for receiving the input landscape image, image recognition algorithm means for extracting features from the received landscape image, database comparison means for comparing the extracted features with a comparison database, location identification means for acquiring location information of the landscape image based on the comparison result, information transmission means for transmitting the acquired location information to the user device, map display means for displaying the acquired location information on a map in the user terminal, emotion recognition means for recognizing the user's emotion, and feedback generation means for generating customized feedback information based on the emotion recognition means. This allows the user not only to identify the location where the landscape image was taken, but also to receive interactive and comprehensive feedback according to their emotional state.

[0253] A "user device" is a terminal that a user uses to input a scenic image or receive that information.

[0254] The "server device" is a computer system that receives and processes scenic images sent from user devices.

[0255] An "image recognition algorithm" is a method for extracting features from received landscape images and analyzing and identifying those images.

[0256] A "verification database" is a database for comparing the extracted features with existing data.

[0257] The "position specifying means" is a means for acquiring the position information of the scenery image based on the result of comparison with the collation database.

[0258] The "information transmitting means" is a means for transmitting the acquired location information to the user device.

[0259] The "map display means" is a means for displaying the location information acquired on the user device on a map.

[0260] The "emotion recognition means" is a means for recognizing the user's emotional state, and estimates emotions by analyzing facial expressions, voice, etc.

[0261] A "feedback generating means" is a means for generating customized feedback information based on a recognized emotional state.

[0262] The present invention provides a system for identifying the location where a landscape image was taken, as well as a system for recognizing a user's emotions and providing interactive feedback in response to the emotions. This system is implemented in the following manner.

[0263] First, a user takes a landscape image using a user device and uploads it to the system. At this time, the user device captures not only the landscape image but also the user's facial expression and voice data, and estimates the user's emotional state. Specifically, the system recognizes the user's facial expression and voice using the camera and microphone of a smartphone, tablet, or PC.

[0264] The server device receives the scenery image and emotion data sent from the user device. Then, it extracts features from the received scenery image using an image recognition algorithm (e.g., a convolutional neural network). This image recognition algorithm can use libraries such as OpenCV and TensorFlow.

[0265] The server then compares the extracted features with a matching database to identify the best matching image, which can be, for example, Google Street View or a proprietary landscape database.

[0266] The server device acquires location information for the landscape image based on the matching result and transmits this location information to the user device. This transmission can be performed using a communication method such as a REST API or WebSocket. The user device then inputs the acquired location information into a map display application (such as Google Maps) to visually display the shooting location on a map.

[0267] Meanwhile, the emotion recognition means analyzes the user's emotional state and generates customized feedback information based on the results. This feedback generation uses an emotion engine to provide feedback including tourist information and activity information according to the user's emotions. This process can utilize, for example, the Emotion API or a proprietary emotion recognition model.

[0268] Specific examples

[0269] In this example, a user uploads a photo of a beautiful lake and the system provides feedback such as, "This lake is a great place to enjoy the tranquility of nature. There are many tourists around."

[0270] In addition, if a user uploads a photo of an old castle, the emotion engine can read their excited state and the system can provide information such as, "Historical ceremonies and events are held regularly at this castle. The next event is XX."

[0271] Example of input prompt for generative AI model

[0272] "Create an interactive tourist guide based on the following information. Give us an example of a system that generates customized tourist feedback based on the user's images and emotional state."

[0273] The above is an embodiment of the present invention. This system not only allows users to accurately identify the location where a landscape image was taken, but also provides interactive feedback based on their emotions, providing a richer experience.

[0274] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0275] Step 1:

[0276] A user takes a landscape image and uploads it to a user device, which then captures the image along with the user's facial expression and voice data, which then become input data.

[0277] Step 2:

[0278] The user device sends a landscape image and the user's emotional data to the server. At this time, the user device encodes the image and emotional data into an appropriate format (e.g., JPEG, MP3) and sends it. The server receives and decompresses the image and emotional data to obtain them.

[0279] Step 3:

[0280] The server uses an image recognition algorithm to extract features from the received landscape image. Specifically, it uses an image recognition algorithm (for example, a convolutional neural network) to convert the input image into numerical data and extracts characteristic patterns and edges, which become the image features.

[0281] Step 4:

[0282] The server matches the extracted features with a matching database. Here, it compares the features with existing database entries to find the best match. For example, it uses a comparison algorithm such as cosine similarity. This is an important data processing step required for location estimation.

[0283] Step 5:

[0284] The server obtains the location information of the landscape image from the matching results. Specifically, it extracts the location information (e.g., latitude and longitude data) stored in the database and stores it in a specific variable. This becomes the obtained location information.

[0285] Step 6:

[0286] The server transmits the acquired location information to the user device. At this time, the server encodes the location information into an appropriate format (e.g., JSON format) and transmits it to the user device. The user device receives and decodes the information to acquire the location information.

[0287] Step 7:

[0288] The user device inputs the acquired location information into a map display application and visually displays the shooting location on a map, allowing the user to confirm the shooting location of the scenic image on the map.

[0289] Step 8:

[0290] The server analyzes the user's emotional state using emotion recognition means. Specifically, it analyzes input facial expressions and voice data to identify the user's emotions (e.g., joy, excitement, relaxation, etc.). It then estimates the user's emotional state by performing data calculations using emotion recognition algorithms.

[0291] Step 9:

[0292] The server generates customized feedback information based on the emotion recognition result. The feedback generating means selects appropriate tourist information and activity information according to the emotional state and creates feedback to be provided to the user.

[0293] Step 10:

[0294] The server sends the generated feedback information to the user device, encodes the feedback information into an appropriate format (e.g., text, image link), and sends it to the user device, which receives it, decodes it, and displays it to the user in visual and textual form.

[0295] Through each of these steps, users can not only identify the location where the landscape image was taken, but also receive interactive feedback based on their emotional state.

[0296] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0297] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0298] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0299] [Second embodiment]

[0300] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0301] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0302] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0303] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0304] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0305] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0306] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0307] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0308] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0309] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0310] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0311] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0312] The present invention is a system for identifying the location where a landscape image was taken, in which a user uploads the landscape image from a terminal, a server processes the image, obtains specific location information, and provides the information to the user. Specific embodiments of the system are described below.

[0313] When a user wants to specify a landscape image, he or she selects the landscape image using his or her own device (e.g., a smartphone or a PC) and uploads it to the system. The user device then sends the landscape image to the server in a predetermined format (e.g., JPEG, PNG). The image upload process can use a general file upload function.

[0314] The server receives landscape images sent from the user's device. The images are temporarily stored on the server, and then feature extraction is performed. Feature extraction uses advanced image recognition algorithms such as convolutional neural networks (CNNs). These algorithms analyze information such as edges, colors, and textures in the images to extract highly distinctive data.

[0315] The server then uses the extracted features to match the image against the Street View database. This database matching process is performed by matching with a large amount of Street View image data. The server then ranks the resulting candidate images based on similarity scores and selects the image that appears to be the best match. The server then obtains the location information (latitude and longitude) of the selected image.

[0316] The server then sends the acquired latitude and longitude information back to the user's device in a general-purpose format such as JSON. The user's device receives the latitude and longitude information sent from the server and enters it into a map display application (for example, Google Maps). At this time, a marker such as a pin is displayed on the map based on the acquired location information, allowing the user to visually confirm the location where the photo was taken.

[0317] For example, if a user takes a photo of an anonymous beautiful landscape from their travels, the system can pinpoint the exact location of the place, allowing the user to visually confirm the location on a map, revisit it, or share the information with others.

[0318] This system makes it possible to accurately pinpoint the location where an image was taken for a wide range of applications, including identifying memorable locations, analyzing incidents and accidents, and even academic research.

[0319] The processing flow will be explained below.

[0320] Step 1:

[0321] The user opens the system's web application from their device (smartphone or PC), clicks the upload button, and selects a landscape image.

[0322] Step 2:

[0323] The device sends the selected landscape image to a designated API endpoint to upload it to the system. The image data is sent in a standard image format such as JPEG or PNG.

[0324] Step 3:

[0325] The server receives the landscape images sent from the user terminal and stores the image data in temporary storage within the server.

[0326] Step 4:

[0327] The server then passes the stored landscape images to an image recognition algorithm, which extracts features from the images. The algorithm uses cutting-edge technologies such as convolutional neural networks (CNNs). The algorithm analyzes important information such as edges, colors, and textures in the images and digitizes them.

[0328] Step 5:

[0329] The server compares the extracted feature data with the Street View database, thereby searching for similar images from the massive Street View image database.

[0330] Step 6:

[0331] The server ranks the candidate images obtained as a result of the matching based on their similarity scores, selects the image with the highest similarity score, and obtains the latitude and longitude information corresponding to that image.

[0332] Step 7:

[0333] The server encodes the acquired latitude and longitude information in a standard format such as JSON and prepares to send it back to the user's terminal.

[0334] Step 8:

[0335] The terminal receives the latitude and longitude information sent from the server, analyzes the received data, and converts it into an appropriate format.

[0336] Step 9:

[0337] The device inputs the acquired latitude and longitude information into a map display application (e.g., Google Maps) and displays the identified shooting location on the map. A pin or marker is placed on the map to visually indicate the shooting location to the user.

[0338] Step 10:

[0339] The user can check the map displayed on the device screen and view information about the shooting location. The user can also check detailed location information and information about surrounding facilities.

[0340] Example 1

[0341] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0342] Identifying the location where a landscape image was taken is extremely difficult using conventional methods, requiring a great deal of time and effort. Furthermore, there are only a limited number of systems for accurately identifying specific location information, and few of them are easily accessible to general users. Therefore, there is a need for a system that allows users to easily and quickly identify the location where a landscape image was taken and obtain detailed location information.

[0343] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0344] In this invention, the server includes means for receiving a landscape image and storing it in temporary storage, means for extracting features from the received landscape image, and means for comparing the extracted features with a database, thereby enabling the location where the landscape image was taken to be accurately identified and the latitude and longitude information of that location to be obtained.

[0345] The "user terminal means" is a device that allows a user to input a landscape image and transmit the image to the server.

[0346] The "server means" is a device that receives and temporarily stores scenic images sent from a user terminal.

[0347] The "image recognition algorithm means" is an algorithm for extracting features from the received landscape image.

[0348] The "database matching means" is a means for matching the extracted feature amount with information in a database.

[0349] The "position specifying means" is a means for specifying the latitude and longitude information of the scenic image based on the result of the comparison.

[0350] The "information transmitting means" is a means for transmitting the acquired latitude and longitude information to the user terminal.

[0351] The "map display means" is a means for displaying the latitude and longitude information acquired by the user terminal on a map.

[0352] The present invention provides a system for identifying the location where a landscape image was taken. A user uploads a landscape image using their own terminal, and a server analyzes the image to identify the location where the image was taken and provides the location information to the user. A specific embodiment is described below.

[0353] System Overview

[0354] The main purpose of this system is to identify the location of landscape images taken by users. The user device sends the image in a specified format (e.g., JPEG, PNG) to the server, which then analyzes the image to obtain location information. The analysis uses advanced image recognition algorithms such as convolutional neural networks (CNN).

[0355] Specific processing

[0356] User terminal

[0357] The user terminal uses a general device such as a smartphone or PC to select a landscape image and upload it to the system. The image upload process uses a general file upload function.

[0358] server

[0359] The server receives landscape images sent from the user's device and stores them in temporary storage. It then uses an image recognition algorithm to extract features from the received images. This extraction uses generative AI models such as TensorFlow and PyTorch. Specifically, it analyzes information such as edges, colors, and textures within the images to extract highly distinctive data.

[0360] Database collation

[0361] The server compares the extracted features with existing data in a database, which contains a large amount of Street View image data. The server then ranks the results based on similarity scores and selects the best matching image. The location information (latitude and longitude) of the selected image is then obtained.

[0362] Information transmission and display

[0363] The server sends the acquired latitude and longitude information to the user's device using a general-purpose format such as JSON. The user's device receives the location information sent from the server and enters it into a map display application (e.g., Google Maps). This application displays a pin (marker) on the map, allowing the user to visually confirm the location where the photo was taken.

[0364] Specific examples

[0365] For example, if a user inputs a photo of an unknown beautiful landscape taken during a trip into the system, the system can identify the location where the photo was taken. By checking the identified location information on a map, the user can revisit the location or share the information with others.

[0366] Prompt Sentence Examples

[0367] An example of a prompt to input to a generative AI model might be, "I want to identify the exact location of a landscape I photographed while traveling. I uploaded this image. Please tell me where it was taken."

[0368] This system makes it possible to accurately pinpoint the location where an image was taken for a wide range of purposes, including identifying memorable places, analyzing incidents and accidents, and academic research.

[0369] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0370] Step 1:

[0371] The user selects a landscape image using their device. Open the file selection dialog on the device and select the desired landscape image. In this case, the image file must be in a common format (e.g., JPEG, PNG). The input is the landscape image file, and the output is the path to the image file selected by the user.

[0372] Step 2:

[0373] The user presses the upload button to send a landscape image to the server. The device browser generates an HTTP request to send the image file to the server. The input is the path of the selected image file, and the output is the HTTP request sent to the server.

[0374] Step 3:

[0375] The server receives the landscape image sent from the user device and saves it in temporary storage. The server processes the received image and saves the image file in a specified directory in the file system. The input is the HTTP request from the user device, and the output is the image file saved in the server's temporary storage.

[0376] Step 4:

[0377] The server reads the images stored in temporary storage and runs an image recognition algorithm to extract features. A convolutional neural network (CNN) is used to extract features such as edges, colors, and textures within the image. The input is the image file stored in temporary storage, and the output is the extracted feature data.

[0378] Step 5:

[0379] The server compares the extracted features with a database. The feature data is used to compare with the Street View database using SQL queries or machine learning algorithms. The input is the extracted feature data, and the output is the matching result (multiple candidate images and similarity scores).

[0380] Step 6:

[0381] The server selects the best matching image based on the matching results and obtains the location information (latitude and longitude) of that image. A scoring algorithm is used to rank the candidate images and obtain the location information of the best matching image. The input is the matching result, and the output is the identified latitude and longitude information.

[0382] Step 7:

[0383] The server converts the acquired latitude and longitude information into a general-purpose format such as JSON and sends it to the user device. The server generates an HTTP response and returns the location information to the device. The input is the specified latitude and longitude information, and the output is JSON data sent to the user device.

[0384] Step 8:

[0385] The user device receives the JSON data sent from the server and inputs it into the map display application. The device parses the JSON data to extract latitude and longitude information and displays a pin (marker) on the map using the map display API. The input is the received JSON data, and the output is a pin for the photo location displayed on the map.

[0386] (Application example 1)

[0387] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0388] Currently, there is no efficient system for quickly and accurately identifying the location of landscape images taken by many security personnel at the scene. As a result, responses to incidents and accidents are delayed and difficult to deal with. Furthermore, conventional systems have the problem that it takes time to identify location information, making it difficult to respond in real time. A new system is needed to solve these problems and improve the efficiency of security operations.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0390] In this invention, the server includes a data processing device means for receiving input landscape images, an image recognition algorithm means for extracting features from the received landscape images, and a database matching means for comparing the extracted features with a matching database, thereby enabling security personnel to quickly and accurately identify the location information of landscape images taken on-site.

[0391] An "information processing device" is a terminal device used by a user to input a landscape image. This includes smartphones, tablets, personal computers, and the like.

[0392] The "data processing device" is a server or cloud computing system that receives and processes landscape images sent by users.

[0393] An "image recognition algorithm" is an algorithm for extracting features from received landscape images. A typical example is a convolutional neural network.

[0394] "Database matching" is the process of comparing the extracted features with an existing database, thereby identifying the location of the photo.

[0395] "Location identification" is the process of acquiring location information for a landscape image based on the matching results, including latitude and longitude information.

[0396] "Information transmission" is a process of transmitting the acquired location information to the user's information processing device.

[0397] "Map display" is a process of displaying location information acquired on the user's information processing device on a map. This is done using a map application.

[0398] The "interface" is a user interface that allows an information processing device to use input landscape images for security purposes. It provides a function for quickly reporting suspicious situations or incidents.

[0399] This invention is a system for identifying the location where a landscape image was taken, in which the landscape image is uploaded from a user's information processing device, the data processing device processes the image to obtain specific location information, and provides that information back to the user. This system is particularly suitable for security applications.

[0400] Hardware and Software Configuration

[0401] 1. Information processing equipment:

[0402] A general user device such as a smartphone, tablet, or PC. It is used by users to upload landscape images they have taken on-site.

[0403] 2. Data Processing Unit:

[0404] It consists of a server and a cloud computing system. This device receives and processes landscape images sent by users, temporarily storing the images, extracting features, and performing matching.

[0405] 3. Image Recognition Algorithm:

[0406] A typical example is a convolutional neural network (CNN) using TensorFlow. Features are extracted from the received landscape image and compared with a database.

[0407] 4. Matching database:

[0408] It can use the Street View image database of Google Cloud Vision API, match it with the extracted features, and obtain the location information of the best-matching candidate.

[0409] 5. Map display:

[0410] The acquired location information is displayed on the user's information processing device using the Google Maps API.

[0411] Processing Flow

[0412] 1. Upload your image:

[0413] The user takes a picture of the scenery using an information processing device and uploads it to the server through the app. React Native can be used.

[0414] 2. Receiving images and extracting features:

[0415] The server stores the received images in temporary storage and extracts features using a convolutional neural network (CNN).

[0416] 3. Database Matching:

[0417] The extracted features are compared with a matching database to identify the location information of the best-matching candidate.

[0418] 4. Obtaining and sending location information:

[0419] Location information is obtained based on the matching results and returned to the user's information processing device in JSON format or similar.

[0420] 5. Map display:

[0421] The user's information processing device displays the acquired location information on a map using the Google Maps API.

[0422] Specific examples

[0423] Assume that a security officer discovers a suspicious person at the scene, takes a picture of the person, and uploads it. The server processes the received image and obtains specific location information. The location information is sent to the user's information processing device and displayed on a map in real time, enabling a prompt response.

[0424] Prompt Sentence Examples

[0425] "Write a Python program to upload a specific landscape image and display a marker on a map based on the latitude and longitude of that location. This program should also include image feature extraction using TensorFlow and Street View matching using the Google Cloud Vision API."

[0426] The above is a specific embodiment for carrying out the invention. This system makes security operations more efficient and enables quick responses.

[0427] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0428] Step 1:

[0429] A user takes landscape images of the site using an information processing device (e.g., a smartphone), selects the images, and uploads them to the server through the system's application. The input is the captured landscape image, and the output is the completion of the upload process to the server.

[0430] Step 2:

[0431] The server receives the landscape image uploaded by the user. The specific operation of the server is to store the image in temporary storage. The input of this step is the landscape image sent by the user, and the output is the image stored in the temporary storage.

[0432] Step 3:

[0433] The server extracts features from the stored landscape images. This process uses a convolutional neural network (CNN) using TensorFlow. The input is the stored landscape image, and the output is the features extracted from the image. The features are data such as edges, colors, and textures within the image.

[0434] Step 4:

[0435] The server compares the extracted features with a matching database (e.g., the Street View image database of the Google Cloud Vision API). Specifically, it matches the existing landscape image features in the database and calculates the similarity. The input is the extracted features, and the output is a list of best-matching candidate images and their similarity scores.

[0436] Step 5:

[0437] The server selects the best-matching candidate image based on the similarity score and obtains its location information (latitude and longitude). The input is a list of candidate images from the matching results, and the output is the location information of the selected image. The location information is provided in latitude and longitude format.

[0438] Step 6:

[0439] The server returns the acquired location information to the information processing device. Specifically, it packages the location information in JSON format and sends it to the user's information processing device. The input is the acquired location information, and the output is the completion of transmission to the information processing device.

[0440] Step 7:

[0441] The user's information processing device displays the received location information on a map using the Google Maps API. Specifically, it displays a pin on the map based on the acquired latitude and longitude information, visually indicating the location where the landscape image was taken. The input is the location information sent from the server, and the output is the location information displayed on the map application.

[0442] The above are the specific processing steps and operation flow of the system that realizes this application example.

[0443] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0444] The present invention is a system that provides interactive feedback according to the user's emotional state by combining a system that identifies the shooting location of a landscape image with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0445] Users upload landscape images to the system using their own devices (e.g., smartphones or PCs). At this time, the user's device captures not only the landscape images but also the user's facial expressions and voice data, which allows the system to estimate the user's emotional state.

[0446] The device can send the user's emotional data along with the landscape image to the server. This emotional data is obtained as a result of analyzing facial expressions and voice. For example, if the user is happy when uploading an image, the facial expression data is sent to the server.

[0447] The server receives the landscape images and emotion data sent from the user's device and uses an image recognition algorithm to extract features from the landscape images. The extracted features are compared with the Street View database to identify the most matching image. At this time, the emotion engine analyzes the user's emotional state, and the results are also used for feedback.

[0448] For example, if a user uploads a landscape image and the emotion engine recognizes the user's joy or excitement, the server can provide positive feedback that includes not only the identified location but also related and background information about the location. For example, information like "This place has beautiful scenery and is visited by many tourists" could be added.

[0449] The server transmits the identified latitude and longitude information and feedback information based on the user's emotion to the user device. The user device inputs this information into a map display application and visually displays the shooting location on a map. Furthermore, customized information based on the user's emotion can also be displayed.

[0450] This system not only allows users to accurately identify the location where a landscape image was taken, but also provides interactive and rich feedback based on their emotions. For example, if the user is relaxed, detailed information about the location's history and cultural background may be displayed. On the other hand, if the user is excited, information about activities and events in the location may be provided, enriching the user's experience.

[0451] Systems equipped with additional functions based on this emotion engine can be used in a wide range of fields, including tourist guides, education, remote exploration, and incident and accident analysis. Providing information linked to the user's emotions will further increase the system's usefulness.

[0452] The processing flow will be explained below.

[0453] Step 1:

[0454] The user opens the system's web application from their device (smartphone or PC) and uploads a landscape image. The user opens the file selection window, selects the landscape image, and clicks the upload button.

[0455] Step 2:

[0456] The device sends the selected landscape image to a specific API endpoint, while capturing the user's facial expressions with a camera and collecting voice data, which are then processed by the emotion engine.

[0457] Step 3:

[0458] The emotion engine analyzes the captured facial and voice data to determine the user's emotional state, which may be expressed as, for example, joy, sadness, surprise, or the like.

[0459] Step 4:

[0460] The device sends the emotion data from the emotion engine along with the landscape image data to the server. The landscape image data is often sent in JPEG or PNG format, and the emotion data is sent in JSON format.

[0461] Step 5:

[0462] The server receives the landscape image data and emotion data sent from the user device. The landscape image is temporarily stored in storage and then passed to the image recognition algorithm.

[0463] Step 6:

[0464] The server uses an image recognition algorithm (e.g., a convolutional neural network) to extract features from the landscape image, such as edge, color, and texture data.

[0465] Step 7:

[0466] The server then matches the extracted features with the Street View database, comparing the features with images in the database and identifying the most similar images.

[0467] Step 8:

[0468] The server selects the image with the highest similarity from the images obtained as a result of the comparison, and obtains the latitude and longitude information corresponding to that image.

[0469] Step 9:

[0470] The server then encodes the acquired latitude and longitude information in a standard format such as JSON and sends it back to the user's device, along with a customized message based on the user's emotional data.

[0471] Step 10:

[0472] The device receives the latitude and longitude information and emotion-based messages sent from the server, analyzes the received data, and converts it into an appropriate format.

[0473] Step 11:

[0474] The device inputs the acquired latitude and longitude information into a map display application (e.g., Google Maps) and displays the identified shooting location on the map. At this time, customized messages and information based on the emotion data are also displayed.

[0475] Step 12:

[0476] The user sees a map and a customized message displayed on the device screen. For example, if the user is happy, positive information about the location (tourist attractions and background information) is also displayed. On the other hand, if the user is sad, a comforting message may be displayed.

[0477] As a result, a system that combines an emotion engine can provide interactive and comprehensive feedback that corresponds to the user's emotional state.

[0478] Example 2

[0479] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0480] Existing landscape image location identification systems only identify the location of landscape images taken by the user, but are unable to provide interactive feedback that takes into account the user's emotional state. As a result, the information available to the user is limited, resulting in a lackluster user experience. Furthermore, they are unable to provide information that takes into account the context of the user's emotional state, making it difficult to provide flexible feedback that can be adapted to multiple uses.

[0481] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes user terminal means for inputting a landscape image and emotion data, server means for receiving the input landscape image and emotion data, image recognition algorithm means for extracting features from the received landscape image, emotion analysis means for analyzing the user's emotional state from the received emotion data, database comparison means for comparing the extracted features with a comparison database, location identification means for acquiring latitude and longitude information of the landscape image based on the comparison result, feedback generation means for generating feedback information based on the latitude and longitude information acquired by the location identification means and the emotion data analyzed by the emotion analysis means, information transmission means for transmitting the generated latitude and longitude information and feedback information to the user terminal, and map display means for displaying the latitude and longitude information and feedback information acquired by the user terminal on a map. This allows the user to not only accurately identify the location where the landscape image was taken but also receive interactive and comprehensive feedback tailored to the user's emotional state.

[0482] A "user terminal" is an electronic device that a user uses to input and transmit scenery images and emotion data.

[0483] The "server" is a computer system that processes and analyzes landscape images and emotion data received from user devices, and generates location identification and feedback information.

[0484] An "image recognition algorithm" is an analytical method for extracting features from landscape images and comparing them with a database.

[0485] "Emotion analysis" is the process of recognizing and determining a user's emotional state from facial expressions and voice data.

[0486] "Database matching" is the process of comparing and matching the features extracted from a landscape image with an existing database.

[0487] "Location identification" is a process of obtaining latitude and longitude information based on the results of matching the scenery image.

[0488] "Feedback generation" refers to the act of generating customized feedback information to provide to a user based on the location-identified information and the results of sentiment analysis.

[0489] "Information transmission" refers to the process of transmitting the generated latitude and longitude information and feedback information to the user terminal.

[0490] "Map display" is a function that visually displays the acquired latitude and longitude information and feedback information on a map.

[0491] The present invention provides a system that provides interactive feedback according to the user's emotional state by combining a system that identifies the shooting location of a landscape image with an emotion analysis function that recognizes the user's emotions. Specific embodiments of the system are described below.

[0492] Users upload landscape images to the system using their own devices (e.g., smartphones or PCs). At this time, the user's device captures not only the landscape images but also the user's facial expressions and voice data, which allows the system to estimate the user's emotional state.

[0493] The device sends the captured emotion data (facial expressions and voice analysis results) along with the landscape image to the server. The communication protocol is HTTP / HTTPS, and the data is sent in JSON format.

[0494] The server processes the received landscape images and emotion data. For landscape images, it uses an image recognition algorithm to extract features. In this case, it uses a convolutional neural network (CNN), an artificial intelligence model (AI model). The extracted features are compared with a landscape database to identify the most similar images.

[0495] Next, the server uses an emotion analysis engine to analyze the user's emotion data. This analysis method uses a general emotion recognition API (e.g., emotion analysis algorithm). The analysis result determines the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0496] The server generates customized feedback for the user based on the results of image recognition and emotion analysis. Specifically, it generates a positive message based on the user's emotions and tourist information about the identified location, along with the latitude and longitude information of the location. The generated feedback information is then sent back to the user's device in JSON format.

[0497] The user terminal inputs the information received from the server into a map display application (e.g., a map application) and visually displays the shooting location of the landscape image on a map. It also displays customized feedback information according to the user's emotions.

[0498] For example, if a user uploads a photo of a beach from their smartphone and the sentiment analysis engine recognizes the user's joy, the server will generate customized feedback such as, "This place is a popular tourist spot with many people and offers beautiful sunrises." This information is sent to the user's device and visually displayed on a map.

[0499] Example prompt sentences

[0500] An example of a prompt is as follows:

[0501] "Upload a landscape image and generate positive feedback based on how I feel when I'm happy."

[0502] In this way, the system of the present invention allows users to not only accurately identify the location where a landscape image was taken, but also to receive interactive and rich feedback according to their emotions. This system can be used in a wide range of fields, including tourist guides, education, remote exploration, and incident and accident analysis.

[0503] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0504] Step 1:

[0505] Users take landscape images using their own devices (e.g., smartphones or PCs) and upload them to the system. The device captures the landscape images taken with the camera as well as the user's facial expression and voice data. The input data are the landscape images and the facial and voice data. This allows us to collect initial data for estimating the user's emotional state.

[0506] Step 2:

[0507] The user device sends the captured landscape image and emotion data to the server. The data is packaged in JSON format and sent to the server via HTTP / HTTPS protocol. The input data are the landscape image and emotion data, and the output data are the landscape image and emotion data sent to the server.

[0508] Step 3:

[0509] The server analyzes the landscape images received from the user terminal using an image recognition algorithm. Specifically, it uses an artificial intelligence model (e.g., a convolutional neural network) to extract features from the landscape images. The input data is the landscape image, and the output data is the extracted features. The image recognition algorithm performs extraction and analysis based on these features.

[0510] Step 4:

[0511] The server analyzes the received emotion data using an emotion analysis engine. The algorithm used is a general emotion recognition API (e.g., emotion analysis algorithm). The input data is emotion data, and the output data is the user's emotional state (e.g., joy, sadness, surprise, etc.). The analyzed emotion data serves as the basis for generating feedback.

[0512] Step 5:

[0513] The server compares the extracted features with a matching database. The database stores landscape image data, and by matching the features with this data, it identifies the image that best matches. The input data are the extracted features, and the output data are the identified latitude and longitude information.

[0514] Step 6:

[0515] The server generates feedback for the user based on the acquired latitude and longitude information and the emotion analysis results. Specifically, it generates tourist information for the identified location and positive messages based on the user's emotions. The input data are the latitude and longitude information and the emotion analysis results, and the output data is customized feedback information.

[0516] Step 7:

[0517] The server sends the generated latitude and longitude information and feedback information to the user terminal. The data is again packaged in JSON format and sent via HTTP / HTTPS protocol. The input data is the generated latitude and longitude information and feedback information, and the output data is the information sent to the user terminal.

[0518] Step 8:

[0519] The user terminal inputs the latitude and longitude information and feedback information received from the server into a map display application, and visually displays the shooting location of the scenic image on a map. Customized feedback information according to the user's emotions is also displayed. The input data is the latitude and longitude information and feedback information received from the server, and the output data is the map display and the feedback display.

[0520] As a concrete example, if a user uploads a photo of a beach from their smartphone and the sentiment analysis engine recognizes the user's joy, the server generates customized feedback such as "This place is a popular tourist spot with many people, and you can see a beautiful sunrise," and sends it to the user's device. The user's device can display this on a map, providing the user with positive feedback along with visual information.

[0521] Prompt Sentence Examples

[0522] "Upload a landscape image and generate positive feedback based on how I feel when I'm happy."

[0523] (Application example 2)

[0524] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0525] In conventional systems for identifying landscape images, users simply identify the location where the landscape was photographed. This results in only providing flat information that ignores the user's emotional state, making it difficult to provide an interactive and rich experience tailored to the scene. Furthermore, there is a need for a system that can provide not only landscape images but also information customized to the user's emotional state.

[0526] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes user device means for inputting a landscape image, server device means for receiving the input landscape image, image recognition algorithm means for extracting features from the received landscape image, database comparison means for comparing the extracted features with a comparison database, location identification means for acquiring location information of the landscape image based on the comparison result, information transmission means for transmitting the acquired location information to the user device, map display means for displaying the acquired location information on a map in the user terminal, emotion recognition means for recognizing the user's emotion, and feedback generation means for generating customized feedback information based on the emotion recognition means. This allows the user not only to identify the location where the landscape image was taken, but also to receive interactive and comprehensive feedback according to their emotional state.

[0527] A "user device" is a terminal that a user uses to input a scenic image or receive that information.

[0528] The "server device" is a computer system that receives and processes scenic images sent from user devices.

[0529] An "image recognition algorithm" is a method for extracting features from received landscape images and analyzing and identifying those images.

[0530] A "verification database" is a database for comparing the extracted features with existing data.

[0531] The "position specifying means" is a means for acquiring the position information of the scenery image based on the result of comparison with the collation database.

[0532] The "information transmitting means" is a means for transmitting the acquired location information to the user device.

[0533] The "map display means" is a means for displaying the location information acquired on the user device on a map.

[0534] The "emotion recognition means" is a means for recognizing the user's emotional state, and estimates emotions by analyzing facial expressions, voice, etc.

[0535] A "feedback generating means" is a means for generating customized feedback information based on a recognized emotional state.

[0536] The present invention provides a system for identifying the location where a landscape image was taken, as well as a system for recognizing a user's emotions and providing interactive feedback in response to the emotions. This system is implemented in the following manner.

[0537] First, a user takes a landscape image using a user device and uploads it to the system. At this time, the user device captures not only the landscape image but also the user's facial expression and voice data, and estimates the user's emotional state. Specifically, the system recognizes the user's facial expression and voice using the camera and microphone of a smartphone, tablet, or PC.

[0538] The server device receives the scenery image and emotion data sent from the user device. Then, it extracts features from the received scenery image using an image recognition algorithm (e.g., a convolutional neural network). This image recognition algorithm can use libraries such as OpenCV and TensorFlow.

[0539] The server then compares the extracted features with a matching database to identify the best matching image, which can be, for example, Google Street View or a proprietary landscape database.

[0540] The server device acquires location information for the landscape image based on the matching result and transmits this location information to the user device. This transmission can be performed using a communication method such as a REST API or WebSocket. The user device then inputs the acquired location information into a map display application (such as Google Maps) to visually display the shooting location on a map.

[0541] Meanwhile, the emotion recognition means analyzes the user's emotional state and generates customized feedback information based on the results. This feedback generation uses an emotion engine to provide feedback including tourist information and activity information according to the user's emotions. This process can utilize, for example, the Emotion API or a proprietary emotion recognition model.

[0542] Specific examples

[0543] In this example, a user uploads a photo of a beautiful lake and the system provides feedback such as, "This lake is a great place to enjoy the tranquility of nature. There are many tourists around."

[0544] In addition, if a user uploads a photo of an old castle, the emotion engine can read their excited state and the system can provide information such as, "Historical ceremonies and events are held regularly at this castle. The next event will be XX."

[0545] Example of input prompt for generative AI model

[0546] "Create an interactive tourist guide based on the following information. Give us an example of a system that generates customized tourist feedback based on the user's images and emotional state."

[0547] The above is an embodiment of the present invention. This system not only allows users to accurately identify the location where a landscape image was taken, but also provides interactive feedback based on their emotions, providing a richer experience.

[0548] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0549] Step 1:

[0550] A user takes a landscape image and uploads it to a user device, which then captures the image along with the user's facial expression and voice data, which then become input data.

[0551] Step 2:

[0552] The user device sends a landscape image and the user's emotional data to the server. At this time, the user device encodes the image and emotional data into an appropriate format (e.g., JPEG, MP3) and sends it. The server receives and decompresses the image and emotional data to obtain them.

[0553] Step 3:

[0554] The server uses an image recognition algorithm to extract features from the received landscape image. Specifically, it uses an image recognition algorithm (for example, a convolutional neural network) to convert the input image into numerical data and extracts characteristic patterns and edges, which become the image features.

[0555] Step 4:

[0556] The server matches the extracted features with a matching database. Here, it compares the features with existing database entries to find the best match. For example, it uses a comparison algorithm such as cosine similarity. This is an important data processing step required for location estimation.

[0557] Step 5:

[0558] The server obtains the location information of the landscape image from the matching results. Specifically, it extracts the location information (e.g., latitude and longitude data) stored in the database and stores it in a specific variable. This becomes the obtained location information.

[0559] Step 6:

[0560] The server transmits the acquired location information to the user device. At this time, the server encodes the location information into an appropriate format (e.g., JSON format) and transmits it to the user device. The user device receives and decodes the information to acquire the location information.

[0561] Step 7:

[0562] The user device inputs the acquired location information into a map display application and visually displays the shooting location on a map, allowing the user to confirm the shooting location of the scenic image on the map.

[0563] Step 8:

[0564] The server analyzes the user's emotional state using emotion recognition means. Specifically, it analyzes input facial expressions and voice data to identify the user's emotions (e.g., joy, excitement, relaxation, etc.). It then estimates the user's emotional state by performing data calculations using emotion recognition algorithms.

[0565] Step 9:

[0566] The server generates customized feedback information based on the emotion recognition result. The feedback generating means selects appropriate tourist information and activity information according to the emotional state and creates feedback to be provided to the user.

[0567] Step 10:

[0568] The server sends the generated feedback information to the user device, encodes the feedback information into an appropriate format (e.g., text, image link), and sends it to the user device, which receives it, decodes it, and displays it to the user in visual and textual form.

[0569] Through each of these steps, users can not only identify the location where the landscape image was taken, but also receive interactive feedback based on their emotional state.

[0570] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0571] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0572] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0573] [Third embodiment]

[0574] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0575] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0576] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0577] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0578] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0579] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0580] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0581] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0582] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0583] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0584] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0585] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0586] The present invention is a system for identifying the location where a landscape image was taken, in which a user uploads the landscape image from a terminal, a server processes the image, obtains specific location information, and provides the information to the user. Specific embodiments of the system are described below.

[0587] When a user wants to specify a landscape image, he or she selects the landscape image using his or her own device (e.g., a smartphone or a PC) and uploads it to the system. The user device then sends the landscape image to the server in a predetermined format (e.g., JPEG, PNG). The image upload process can use a general file upload function.

[0588] The server receives landscape images sent from the user's device. The images are temporarily stored on the server, and then feature extraction is performed. Feature extraction uses advanced image recognition algorithms such as convolutional neural networks (CNNs). These algorithms analyze information such as edges, colors, and textures in the images to extract highly distinctive data.

[0589] The server then uses the extracted features to match the image against the Street View database. This database matching process is performed by matching with a large amount of Street View image data. The server then ranks the resulting candidate images based on similarity scores and selects the image that appears to be the best match. The server then obtains the location information (latitude and longitude) of the selected image.

[0590] The server then sends the acquired latitude and longitude information back to the user's device in a general-purpose format such as JSON. The user's device receives the latitude and longitude information sent from the server and enters it into a map display application (for example, Google Maps). At this time, a marker such as a pin is displayed on the map based on the acquired location information, allowing the user to visually confirm the location where the photo was taken.

[0591] For example, if a user takes a photo of an anonymous beautiful landscape from their travels, the system can pinpoint the exact location of the place, allowing the user to visually confirm the location on a map, revisit it, or share the information with others.

[0592] This system makes it possible to accurately pinpoint the location where an image was taken for a wide range of applications, including identifying memorable locations, analyzing incidents and accidents, and even academic research.

[0593] The processing flow will be explained below.

[0594] Step 1:

[0595] The user opens the system's web application from their device (smartphone or PC), clicks the upload button, and selects a landscape image.

[0596] Step 2:

[0597] The device sends the selected landscape image to a designated API endpoint to upload it to the system. The image data is sent in a standard image format such as JPEG or PNG.

[0598] Step 3:

[0599] The server receives the landscape images sent from the user terminal and stores the image data in temporary storage within the server.

[0600] Step 4:

[0601] The server then passes the stored landscape images to an image recognition algorithm, which extracts features from the images. The algorithm uses cutting-edge technologies such as convolutional neural networks (CNNs). The algorithm analyzes important information such as edges, colors, and textures in the images and digitizes them.

[0602] Step 5:

[0603] The server compares the extracted feature data with the Street View database, thereby searching for similar images from the massive Street View image database.

[0604] Step 6:

[0605] The server ranks the candidate images obtained as a result of the matching based on their similarity scores, selects the image with the highest similarity score, and obtains the latitude and longitude information corresponding to that image.

[0606] Step 7:

[0607] The server encodes the acquired latitude and longitude information in a standard format such as JSON and prepares to send it back to the user's terminal.

[0608] Step 8:

[0609] The terminal receives the latitude and longitude information sent from the server, analyzes the received data, and converts it into an appropriate format.

[0610] Step 9:

[0611] The device inputs the acquired latitude and longitude information into a map display application (e.g., Google Maps) and displays the identified shooting location on the map. A pin or marker is placed on the map to visually indicate the shooting location to the user.

[0612] Step 10:

[0613] The user can check the map displayed on the device screen and view information about the shooting location. The user can also check detailed location information and information about surrounding facilities.

[0614] Example 1

[0615] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0616] Identifying the location where a landscape image was taken is extremely difficult using conventional methods, requiring a great deal of time and effort. Furthermore, there are only a limited number of systems for accurately identifying specific location information, and few of them are easily accessible to general users. Therefore, there is a need for a system that allows users to easily and quickly identify the location where a landscape image was taken and obtain detailed location information.

[0617] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0618] In this invention, the server includes means for receiving a landscape image and storing it in temporary storage, means for extracting features from the received landscape image, and means for comparing the extracted features with a database, thereby enabling the location where the landscape image was taken to be accurately identified and the latitude and longitude information of that location to be obtained.

[0619] The "user terminal means" is a device that allows a user to input a landscape image and transmit the image to the server.

[0620] The "server means" is a device that receives and temporarily stores scenic images sent from a user terminal.

[0621] The "image recognition algorithm means" is an algorithm for extracting features from the received landscape image.

[0622] The "database matching means" is a means for matching the extracted feature amount with information in a database.

[0623] The "position specifying means" is a means for specifying the latitude and longitude information of the scenic image based on the result of the comparison.

[0624] The "information transmitting means" is a means for transmitting the acquired latitude and longitude information to the user terminal.

[0625] The "map display means" is a means for displaying the latitude and longitude information acquired by the user terminal on a map.

[0626] The present invention provides a system for identifying the location where a landscape image was taken. A user uploads a landscape image using their own terminal, and a server analyzes the image to identify the location where the image was taken and provides the location information to the user. A specific embodiment is described below.

[0627] System Overview

[0628] The main purpose of this system is to identify the location of landscape images taken by users. The user device sends the image in a specified format (e.g., JPEG, PNG) to the server, which then analyzes the image to obtain location information. The analysis uses advanced image recognition algorithms such as convolutional neural networks (CNN).

[0629] Specific processing

[0630] User terminal

[0631] The user terminal uses a general device such as a smartphone or PC to select a landscape image and upload it to the system. The image upload process uses a general file upload function.

[0632] server

[0633] The server receives landscape images sent from the user's device and stores them in temporary storage. It then uses an image recognition algorithm to extract features from the received images. This extraction uses generative AI models such as TensorFlow and PyTorch. Specifically, it analyzes information such as edges, colors, and textures within the images to extract highly distinctive data.

[0634] Database collation

[0635] The server compares the extracted features with existing data in a database, which contains a large amount of Street View image data. The server then ranks the results based on similarity scores and selects the best matching image. The location information (latitude and longitude) of the selected image is then obtained.

[0636] Information transmission and display

[0637] The server sends the acquired latitude and longitude information to the user's device using a general-purpose format such as JSON. The user's device receives the location information sent from the server and enters it into a map display application (e.g., Google Maps). This application displays a pin (marker) on the map, allowing the user to visually confirm the location where the photo was taken.

[0638] Specific examples

[0639] For example, if a user inputs a photo of an unknown beautiful landscape taken during a trip into the system, the system can identify the location where the photo was taken. By checking the identified location information on a map, the user can revisit the location or share the information with others.

[0640] Prompt Sentence Examples

[0641] An example of a prompt to input to a generative AI model might be, "I want to identify the exact location of a landscape I photographed while traveling. I uploaded this image. Please tell me where it was taken."

[0642] This system makes it possible to accurately pinpoint the location where an image was taken for a wide range of purposes, including identifying memorable places, analyzing incidents and accidents, and academic research.

[0643] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0644] Step 1:

[0645] The user selects a landscape image using their device. Open the file selection dialog on the device and select the desired landscape image. In this case, the image file must be in a common format (e.g., JPEG, PNG). The input is the landscape image file, and the output is the path to the image file selected by the user.

[0646] Step 2:

[0647] The user presses the upload button to send a landscape image to the server. The device browser generates an HTTP request to send the image file to the server. The input is the path of the selected image file, and the output is the HTTP request sent to the server.

[0648] Step 3:

[0649] The server receives the landscape image sent from the user device and saves it in temporary storage. The server processes the received image and saves the image file in a specified directory in the file system. The input is the HTTP request from the user device, and the output is the image file saved in the server's temporary storage.

[0650] Step 4:

[0651] The server reads the images stored in temporary storage and runs an image recognition algorithm to extract features. A convolutional neural network (CNN) is used to extract features such as edges, colors, and textures within the image. The input is the image file stored in temporary storage, and the output is the extracted feature data.

[0652] Step 5:

[0653] The server compares the extracted features with a database. The feature data is used to compare with the Street View database using SQL queries or machine learning algorithms. The input is the extracted feature data, and the output is the matching result (multiple candidate images and similarity scores).

[0654] Step 6:

[0655] The server selects the best matching image based on the matching results and obtains the location information (latitude and longitude) of that image. A scoring algorithm is used to rank the candidate images and obtain the location information of the best matching image. The input is the matching result, and the output is the identified latitude and longitude information.

[0656] Step 7:

[0657] The server converts the acquired latitude and longitude information into a general-purpose format such as JSON and sends it to the user device. The server generates an HTTP response and returns the location information to the device. The input is the specified latitude and longitude information, and the output is JSON data sent to the user device.

[0658] Step 8:

[0659] The user device receives the JSON data sent from the server and inputs it into the map display application. The device parses the JSON data to extract latitude and longitude information and displays a pin (marker) on the map using the map display API. The input is the received JSON data, and the output is a pin for the photo location displayed on the map.

[0660] (Application example 1)

[0661] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0662] Currently, there is no efficient system for quickly and accurately identifying the location of landscape images taken by many security personnel at the scene. As a result, responses to incidents and accidents are delayed and difficult to deal with. Furthermore, conventional systems have the problem that it takes time to identify location information, making it difficult to respond in real time. A new system is needed to solve these problems and improve the efficiency of security operations.

[0663] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0664] In this invention, the server includes a data processing device means for receiving input landscape images, an image recognition algorithm means for extracting features from the received landscape images, and a database matching means for comparing the extracted features with a matching database, thereby enabling security personnel to quickly and accurately identify the location information of landscape images taken on-site.

[0665] An "information processing device" is a terminal device used by a user to input a landscape image. This includes smartphones, tablets, personal computers, and the like.

[0666] The "data processing device" is a server or cloud computing system that receives and processes landscape images sent by users.

[0667] An "image recognition algorithm" is an algorithm for extracting features from received landscape images. A typical example is a convolutional neural network.

[0668] "Database matching" is the process of comparing the extracted features with an existing database, thereby identifying the location of the photo.

[0669] "Location identification" is the process of acquiring location information for a landscape image based on the matching results, including latitude and longitude information.

[0670] "Information transmission" is a process of transmitting the acquired location information to the user's information processing device.

[0671] "Map display" is a process of displaying location information acquired on the user's information processing device on a map. This is done using a map application.

[0672] The "interface" is a user interface that allows an information processing device to use input landscape images for security purposes. It provides a function for quickly reporting suspicious situations or incidents.

[0673] This invention is a system for identifying the location where a landscape image was taken, in which the landscape image is uploaded from a user's information processing device, the data processing device processes the image to obtain specific location information, and provides that information back to the user. This system is particularly suitable for security applications.

[0674] Hardware and Software Configuration

[0675] 1. Information processing equipment:

[0676] A general user device such as a smartphone, tablet, or PC. It is used by users to upload landscape images they have taken on-site.

[0677] 2. Data Processing Unit:

[0678] It consists of a server and a cloud computing system. This device receives and processes landscape images sent by users, temporarily storing the images, extracting features, and performing matching.

[0679] 3. Image Recognition Algorithm:

[0680] A typical example is a convolutional neural network (CNN) using TensorFlow. Features are extracted from the received landscape image and compared with a database.

[0681] 4. Matching database:

[0682] It can use the Street View image database of Google Cloud Vision API, match it with the extracted features, and obtain the location information of the best-matching candidate.

[0683] 5. Map display:

[0684] The acquired location information is displayed on the user's information processing device using the Google Maps API.

[0685] Processing Flow

[0686] 1. Upload your image:

[0687] The user takes a picture of the scenery using an information processing device and uploads it to the server through the app. React Native can be used.

[0688] 2. Receiving images and extracting features:

[0689] The server stores the received images in temporary storage and extracts features using a convolutional neural network (CNN).

[0690] 3. Database Matching:

[0691] The extracted features are compared with a matching database to identify the location information of the best-matching candidate.

[0692] 4. Obtaining and sending location information:

[0693] Location information is obtained based on the matching results and returned to the user's information processing device in JSON format or similar.

[0694] 5. Map display:

[0695] The user's information processing device displays the acquired location information on a map using the Google Maps API.

[0696] Specific examples

[0697] Assume that a security officer discovers a suspicious person at the scene, takes a picture of the person, and uploads it. The server processes the received image and obtains specific location information. The location information is sent to the user's information processing device and displayed on a map in real time, enabling a prompt response.

[0698] Prompt Sentence Examples

[0699] "Write a Python program to upload a specific landscape image and display a marker on a map based on the latitude and longitude of that location. This program should also include image feature extraction using TensorFlow and Street View matching using the Google Cloud Vision API."

[0700] The above is a specific embodiment for carrying out the invention. This system makes security operations more efficient and enables quick responses.

[0701] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0702] Step 1:

[0703] A user takes landscape images of the site using an information processing device (e.g., a smartphone), selects the images, and uploads them to the server through the system's application. The input is the captured landscape image, and the output is the completion of the upload process to the server.

[0704] Step 2:

[0705] The server receives the landscape image uploaded by the user. The specific operation of the server is to store the image in temporary storage. The input of this step is the landscape image sent by the user, and the output is the image stored in the temporary storage.

[0706] Step 3:

[0707] The server extracts features from the stored landscape images. This process uses a convolutional neural network (CNN) using TensorFlow. The input is the stored landscape image, and the output is the features extracted from the image. The features are data such as edges, colors, and textures within the image.

[0708] Step 4:

[0709] The server compares the extracted features with a matching database (e.g., the Street View image database of the Google Cloud Vision API). Specifically, it matches the existing landscape image features in the database and calculates the similarity. The input is the extracted features, and the output is a list of best-matching candidate images and their similarity scores.

[0710] Step 5:

[0711] The server selects the best-matching candidate image based on the similarity score and obtains its location information (latitude and longitude). The input is a list of candidate images from the matching results, and the output is the location information of the selected image. The location information is provided in latitude and longitude format.

[0712] Step 6:

[0713] The server returns the acquired location information to the information processing device. Specifically, it packages the location information in JSON format and sends it to the user's information processing device. The input is the acquired location information, and the output is the completion of transmission to the information processing device.

[0714] Step 7:

[0715] The user's information processing device displays the received location information on a map using the Google Maps API. Specifically, it displays a pin on the map based on the acquired latitude and longitude information, visually indicating the location where the landscape image was taken. The input is the location information sent from the server, and the output is the location information displayed on the map application.

[0716] The above are the specific processing steps and operation flow of the system that realizes this application example.

[0717] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0718] The present invention is a system that provides interactive feedback according to the user's emotional state by combining a system that identifies the shooting location of a landscape image with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0719] Users upload landscape images to the system using their own devices (e.g., smartphones or PCs). At this time, the user's device captures not only the landscape images but also the user's facial expressions and voice data, which allows the system to estimate the user's emotional state.

[0720] The device can send the user's emotional data along with the landscape image to the server. This emotional data is obtained as a result of analyzing facial expressions and voice. For example, if the user is happy when uploading an image, the facial expression data is sent to the server.

[0721] The server receives the landscape images and emotion data sent from the user's device and uses an image recognition algorithm to extract features from the landscape images. The extracted features are compared with the Street View database to identify the most matching image. At this time, the emotion engine analyzes the user's emotional state, and the results are also used for feedback.

[0722] For example, if a user uploads a landscape image and the emotion engine recognizes the user's joy or excitement, the server can provide positive feedback that includes not only the identified location but also related and background information about the location. For example, information like "This place has beautiful scenery and is visited by many tourists" could be added.

[0723] The server transmits the identified latitude and longitude information and feedback information based on the user's emotion to the user device. The user device inputs this information into a map display application and visually displays the shooting location on a map. Furthermore, customized information based on the user's emotion can also be displayed.

[0724] This system not only allows users to accurately identify the location where a landscape image was taken, but also provides interactive and rich feedback based on their emotions. For example, if the user is relaxed, detailed information about the location's history and cultural background may be displayed. On the other hand, if the user is excited, information about activities and events in the location may be provided, enriching the user's experience.

[0725] Systems equipped with additional functions based on this emotion engine can be used in a wide range of fields, including tourist guides, education, remote exploration, and incident and accident analysis. Providing information linked to the user's emotions will further increase the system's usefulness.

[0726] The processing flow will be explained below.

[0727] Step 1:

[0728] The user opens the system's web application from their device (smartphone or PC) and uploads a landscape image. The user opens the file selection window, selects the landscape image, and clicks the upload button.

[0729] Step 2:

[0730] The device sends the selected landscape image to a specific API endpoint, while capturing the user's facial expressions with a camera and collecting voice data, which are then processed by the emotion engine.

[0731] Step 3:

[0732] The emotion engine analyzes the captured facial and voice data to determine the user's emotional state, which may be expressed as, for example, joy, sadness, surprise, or the like.

[0733] Step 4:

[0734] The device sends the emotion data from the emotion engine along with the landscape image data to the server. The landscape image data is often sent in JPEG or PNG format, and the emotion data is sent in JSON format.

[0735] Step 5:

[0736] The server receives the landscape image data and emotion data sent from the user device. The landscape image is temporarily stored in storage and then passed to the image recognition algorithm.

[0737] Step 6:

[0738] The server uses an image recognition algorithm (e.g., a convolutional neural network) to extract features from the landscape image, such as edge, color, and texture data.

[0739] Step 7:

[0740] The server then matches the extracted features with the Street View database, comparing the features with images in the database and identifying the most similar images.

[0741] Step 8:

[0742] The server selects the image with the highest similarity from the images obtained as a result of the comparison, and obtains the latitude and longitude information corresponding to that image.

[0743] Step 9:

[0744] The server then encodes the acquired latitude and longitude information in a standard format such as JSON and sends it back to the user's device, along with a customized message based on the user's emotional data.

[0745] Step 10:

[0746] The device receives the latitude and longitude information and emotion-based messages sent from the server, analyzes the received data, and converts it into an appropriate format.

[0747] Step 11:

[0748] The device inputs the acquired latitude and longitude information into a map display application (e.g., Google Maps) and displays the identified shooting location on the map. At this time, customized messages and information based on the emotion data are also displayed.

[0749] Step 12:

[0750] The user sees a map and a customized message displayed on the device screen. For example, if the user is happy, positive information about the location (tourist attractions and background information) is also displayed. On the other hand, if the user is sad, a comforting message may be displayed.

[0751] As a result, a system that combines an emotion engine can provide interactive and comprehensive feedback that corresponds to the user's emotional state.

[0752] Example 2

[0753] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0754] Existing landscape image location identification systems only identify the location of landscape images taken by the user, but are unable to provide interactive feedback that takes into account the user's emotional state. As a result, the information available to the user is limited, resulting in a lackluster user experience. Furthermore, they are unable to provide information that takes into account the context of the user's emotional state, making it difficult to provide flexible feedback that can be adapted to multiple uses.

[0755] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes user terminal means for inputting a landscape image and emotion data, server means for receiving the input landscape image and emotion data, image recognition algorithm means for extracting features from the received landscape image, emotion analysis means for analyzing the user's emotional state from the received emotion data, database comparison means for comparing the extracted features with a comparison database, location identification means for acquiring latitude and longitude information of the landscape image based on the comparison result, feedback generation means for generating feedback information based on the latitude and longitude information acquired by the location identification means and the emotion data analyzed by the emotion analysis means, information transmission means for transmitting the generated latitude and longitude information and feedback information to the user terminal, and map display means for displaying the latitude and longitude information and feedback information acquired by the user terminal on a map. This allows the user to not only accurately identify the location where the landscape image was taken but also receive interactive and comprehensive feedback tailored to the user's emotional state.

[0756] A "user terminal" is an electronic device that a user uses to input and transmit scenery images and emotion data.

[0757] The "server" is a computer system that processes and analyzes landscape images and emotion data received from user devices, and generates location identification and feedback information.

[0758] An "image recognition algorithm" is an analytical method for extracting features from landscape images and comparing them with a database.

[0759] "Emotion analysis" is the process of recognizing and determining a user's emotional state from facial expressions and voice data.

[0760] "Database matching" is the process of comparing and matching the features extracted from a landscape image with an existing database.

[0761] "Location identification" is a process of obtaining latitude and longitude information based on the results of matching the scenery image.

[0762] "Feedback generation" refers to the act of generating customized feedback information to provide to a user based on the location-identified information and the results of sentiment analysis.

[0763] "Information transmission" refers to the process of transmitting the generated latitude and longitude information and feedback information to the user terminal.

[0764] "Map display" is a function that visually displays the acquired latitude and longitude information and feedback information on a map.

[0765] The present invention provides a system that provides interactive feedback according to the user's emotional state by combining a system that identifies the shooting location of a landscape image with an emotion analysis function that recognizes the user's emotions. Specific embodiments of the system are described below.

[0766] Users upload landscape images to the system using their own devices (e.g., smartphones or PCs). At this time, the user's device captures not only the landscape images but also the user's facial expressions and voice data, which allows the system to estimate the user's emotional state.

[0767] The device sends the captured emotion data (facial expressions and voice analysis results) along with the landscape image to the server. The communication protocol is HTTP / HTTPS, and the data is sent in JSON format.

[0768] The server processes the received landscape images and emotion data. For landscape images, it uses an image recognition algorithm to extract features. In this case, it uses a convolutional neural network (CNN), an artificial intelligence model (AI model). The extracted features are compared with a landscape database to identify the most similar images.

[0769] Next, the server uses an emotion analysis engine to analyze the user's emotional data. This analysis method uses a general emotion recognition API (e.g., emotion analysis algorithm). The analysis result determines the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0770] The server generates customized feedback for the user based on the results of image recognition and emotion analysis. Specifically, it generates a positive message based on the user's emotions and tourist information about the identified location, along with the latitude and longitude information of the location. The generated feedback information is then sent back to the user's device in JSON format.

[0771] The user terminal inputs the information received from the server into a map display application (e.g., a map application) and visually displays the shooting location of the landscape image on a map. It also displays customized feedback information according to the user's emotions.

[0772] For example, if a user uploads a photo of a beach from their smartphone and the sentiment analysis engine recognizes the user's joy, the server will generate customized feedback such as, "This place is a popular tourist spot with many people and offers beautiful sunrises." This information is sent to the user's device and visually displayed on a map.

[0773] Example prompt sentences

[0774] An example of a prompt is as follows:

[0775] "Upload a landscape image and generate positive feedback based on how I feel when I'm happy."

[0776] In this way, the system of the present invention allows users to not only accurately identify the location where a landscape image was taken, but also to receive interactive and rich feedback according to their emotions. This system can be used in a wide range of fields, including tourist guides, education, remote exploration, and incident and accident analysis.

[0777] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0778] Step 1:

[0779] Users take landscape images using their own devices (e.g., smartphones or PCs) and upload them to the system. The device captures the landscape images taken with the camera as well as the user's facial expression and voice data. The input data are the landscape images and the facial and voice data. This allows us to collect initial data for estimating the user's emotional state.

[0780] Step 2:

[0781] The user device sends the captured landscape image and emotion data to the server. The data is packaged in JSON format and sent to the server via HTTP / HTTPS protocol. The input data are the landscape image and emotion data, and the output data are the landscape image and emotion data sent to the server.

[0782] Step 3:

[0783] The server analyzes the landscape images received from the user terminal using an image recognition algorithm. Specifically, it uses an artificial intelligence model (e.g., a convolutional neural network) to extract features from the landscape images. The input data is the landscape image, and the output data is the extracted features. The image recognition algorithm performs extraction and analysis based on these features.

[0784] Step 4:

[0785] The server analyzes the received emotion data using an emotion analysis engine. The algorithm used is a general emotion recognition API (e.g., emotion analysis algorithm). The input data is emotion data, and the output data is the user's emotional state (e.g., joy, sadness, surprise, etc.). The analyzed emotion data serves as the basis for generating feedback.

[0786] Step 5:

[0787] The server compares the extracted features with a matching database. The database stores landscape image data, and by matching the features with this data, it identifies the image that best matches. The input data are the extracted features, and the output data are the identified latitude and longitude information.

[0788] Step 6:

[0789] The server generates feedback for the user based on the acquired latitude and longitude information and the emotion analysis results. Specifically, it generates tourist information for the identified location and positive messages based on the user's emotions. The input data are the latitude and longitude information and the emotion analysis results, and the output data is customized feedback information.

[0790] Step 7:

[0791] The server sends the generated latitude and longitude information and feedback information to the user terminal. The data is again packaged in JSON format and sent via HTTP / HTTPS protocol. The input data is the generated latitude and longitude information and feedback information, and the output data is the information sent to the user terminal.

[0792] Step 8:

[0793] The user terminal inputs the latitude and longitude information and feedback information received from the server into a map display application, and visually displays the shooting location of the scenic image on a map. Customized feedback information according to the user's emotions is also displayed. The input data is the latitude and longitude information and feedback information received from the server, and the output data is the map display and the feedback display.

[0794] As a concrete example, if a user uploads a photo of a beach from their smartphone and the sentiment analysis engine recognizes the user's joy, the server generates customized feedback such as "This place is a popular tourist spot with many people, and you can see a beautiful sunrise," and sends it to the user's device. The user's device can display this on a map, providing the user with positive feedback along with visual information.

[0795] Prompt Sentence Examples

[0796] "Upload a landscape image and generate positive feedback based on how I feel when I'm happy."

[0797] (Application example 2)

[0798] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0799] In conventional systems for identifying landscape images, users simply identify the location where the landscape was photographed. This results in only providing flat information that ignores the user's emotional state, making it difficult to provide an interactive and rich experience tailored to the scene. Furthermore, there is a need for a system that can provide not only landscape images but also information customized to the user's emotional state.

[0800] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes user device means for inputting a landscape image, server device means for receiving the input landscape image, image recognition algorithm means for extracting features from the received landscape image, database comparison means for comparing the extracted features with a comparison database, location identification means for acquiring location information of the landscape image based on the comparison result, information transmission means for transmitting the acquired location information to the user device, map display means for displaying the acquired location information on a map in the user terminal, emotion recognition means for recognizing the user's emotion, and feedback generation means for generating customized feedback information based on the emotion recognition means. This allows the user not only to identify the location where the landscape image was taken, but also to receive interactive and comprehensive feedback according to their emotional state.

[0801] A "user device" is a terminal that a user uses to input a scenic image or receive that information.

[0802] The "server device" is a computer system that receives and processes scenic images sent from user devices.

[0803] An "image recognition algorithm" is a method for extracting features from received landscape images and analyzing and identifying those images.

[0804] A "verification database" is a database for comparing the extracted features with existing data.

[0805] The "position specifying means" is a means for acquiring the position information of the scenery image based on the result of comparison with the collation database.

[0806] The "information transmitting means" is a means for transmitting the acquired location information to the user device.

[0807] The "map display means" is a means for displaying the location information acquired on the user device on a map.

[0808] The "emotion recognition means" is a means for recognizing the user's emotional state, and estimates emotions by analyzing facial expressions, voice, etc.

[0809] A "feedback generating means" is a means for generating customized feedback information based on a recognized emotional state.

[0810] The present invention provides a system for identifying the location where a landscape image was taken, as well as a system for recognizing a user's emotions and providing interactive feedback in response to the emotions. This system is implemented in the following manner.

[0811] First, a user takes a landscape image using a user device and uploads it to the system. At this time, the user device captures not only the landscape image but also the user's facial expression and voice data, and estimates the user's emotional state. Specifically, the system recognizes the user's facial expression and voice using the camera and microphone of a smartphone, tablet, or PC.

[0812] The server device receives the scenery image and emotion data sent from the user device. Then, it extracts features from the received scenery image using an image recognition algorithm (e.g., a convolutional neural network). This image recognition algorithm can use libraries such as OpenCV and TensorFlow.

[0813] The server then compares the extracted features with a matching database to identify the best matching image, which can be, for example, Google Street View or a proprietary landscape database.

[0814] The server device acquires location information for the landscape image based on the matching result and transmits this location information to the user device. This transmission can be performed using a communication method such as a REST API or WebSocket. The user device then inputs the acquired location information into a map display application (such as Google Maps) to visually display the shooting location on a map.

[0815] Meanwhile, the emotion recognition means analyzes the user's emotional state and generates customized feedback information based on the results. This feedback generation uses an emotion engine to provide feedback including tourist information and activity information according to the user's emotions. This process can utilize, for example, the Emotion API or a proprietary emotion recognition model.

[0816] Specific examples

[0817] In this example, a user uploads a photo of a beautiful lake and the system provides feedback such as, "This lake is a great place to enjoy the tranquility of nature. There are many tourists around."

[0818] In addition, if a user uploads a photo of an old castle, the emotion engine can read their excited state and the system can provide information such as, "Historical ceremonies and events are held regularly at this castle. The next event is XX."

[0819] Example of input prompt for generative AI model

[0820] "Create an interactive tourist guide based on the following information. Give us an example of a system that generates customized tourist feedback based on the user's images and emotional state."

[0821] The above is an embodiment of the present invention. This system not only allows users to accurately identify the location where a landscape image was taken, but also provides interactive feedback based on their emotions, providing a richer experience.

[0822] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0823] Step 1:

[0824] A user takes a landscape image and uploads it to a user device, which then captures the image along with the user's facial expression and voice data, which then become input data.

[0825] Step 2:

[0826] The user device sends a landscape image and the user's emotional data to the server. At this time, the user device encodes the image and emotional data into an appropriate format (e.g., JPEG, MP3) and sends it. The server receives and decompresses the image and emotional data to obtain them.

[0827] Step 3:

[0828] The server uses an image recognition algorithm to extract features from the received landscape image. Specifically, it uses an image recognition algorithm (for example, a convolutional neural network) to convert the input image into numerical data and extracts characteristic patterns and edges, which become the image features.

[0829] Step 4:

[0830] The server matches the extracted features with a matching database. Here, it compares the features with existing database entries to find the best match. For example, it uses a comparison algorithm such as cosine similarity. This is an important data processing step required for location estimation.

[0831] Step 5:

[0832] The server obtains the location information of the landscape image from the matching results. Specifically, it extracts the location information (e.g., latitude and longitude data) stored in the database and stores it in a specific variable. This becomes the obtained location information.

[0833] Step 6:

[0834] The server transmits the acquired location information to the user device. At this time, the server encodes the location information into an appropriate format (e.g., JSON format) and transmits it to the user device. The user device receives and decodes the information to acquire the location information.

[0835] Step 7:

[0836] The user device inputs the acquired location information into a map display application and visually displays the shooting location on a map, allowing the user to confirm the shooting location of the scenic image on the map.

[0837] Step 8:

[0838] The server analyzes the user's emotional state using emotion recognition means. Specifically, it analyzes input facial expressions and voice data to identify the user's emotions (e.g., joy, excitement, relaxation, etc.). It then estimates the user's emotional state by performing data calculations using emotion recognition algorithms.

[0839] Step 9:

[0840] The server generates customized feedback information based on the emotion recognition result. The feedback generating means selects appropriate tourist information and activity information according to the emotional state and creates feedback to be provided to the user.

[0841] Step 10:

[0842] The server sends the generated feedback information to the user device, encodes the feedback information into an appropriate format (e.g., text, image link), and sends it to the user device, which receives it, decodes it, and displays it to the user in visual and textual form.

[0843] Through each of these steps, users can not only identify the location where the landscape image was taken, but also receive interactive feedback based on their emotional state.

[0844] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0845] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0846] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0847] [Fourth embodiment]

[0848] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0849] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0850] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0851] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0852] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0853] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0854] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0855] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0856] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0857] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0858] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0859] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0860] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0861] The present invention is a system for identifying the location where a landscape image was taken, in which a user uploads the landscape image from a terminal, a server processes the image, obtains specific location information, and provides the information to the user. Specific embodiments of the system are described below.

[0862] When a user wants to specify a landscape image, he or she selects the landscape image using his or her own device (e.g., a smartphone or a PC) and uploads it to the system. The user device then sends the landscape image to the server in a predetermined format (e.g., JPEG, PNG). The image upload process can use a general file upload function.

[0863] The server receives landscape images sent from the user's device. The images are temporarily stored on the server, and then feature extraction is performed. Feature extraction uses advanced image recognition algorithms such as convolutional neural networks (CNNs). These algorithms analyze information such as edges, colors, and textures in the images to extract highly distinctive data.

[0864] The server then uses the extracted features to match the image against the Street View database. This database matching process is performed by matching with a large amount of Street View image data. The server then ranks the resulting candidate images based on similarity scores and selects the image that appears to be the best match. The server then obtains the location information (latitude and longitude) of the selected image.

[0865] The server then sends the acquired latitude and longitude information back to the user's device in a general-purpose format such as JSON. The user's device receives the latitude and longitude information sent from the server and enters it into a map display application (for example, Google Maps). At this time, a marker such as a pin is displayed on the map based on the acquired location information, allowing the user to visually confirm the location where the photo was taken.

[0866] For example, if a user takes a photo of an anonymous beautiful landscape from their travels, the system can pinpoint the exact location of the place, allowing the user to visually confirm the location on a map, revisit it, or share the information with others.

[0867] This system makes it possible to accurately pinpoint the location where an image was taken for a wide range of applications, including identifying memorable locations, analyzing incidents and accidents, and even academic research.

[0868] The processing flow will be explained below.

[0869] Step 1:

[0870] The user opens the system's web application from their device (smartphone or PC), clicks the upload button, and selects a landscape image.

[0871] Step 2:

[0872] The device sends the selected landscape image to a designated API endpoint to upload it to the system. The image data is sent in a standard image format such as JPEG or PNG.

[0873] Step 3:

[0874] The server receives the landscape images sent from the user terminal and stores the image data in temporary storage within the server.

[0875] Step 4:

[0876] The server then passes the stored landscape images to an image recognition algorithm, which extracts features from the images. The algorithm uses cutting-edge technologies such as convolutional neural networks (CNNs). The algorithm analyzes important information such as edges, colors, and textures in the images and digitizes them.

[0877] Step 5:

[0878] The server compares the extracted feature data with the Street View database, thereby searching for similar images from the massive Street View image database.

[0879] Step 6:

[0880] The server ranks the candidate images obtained as a result of the matching based on their similarity scores, selects the image with the highest similarity score, and obtains the latitude and longitude information corresponding to that image.

[0881] Step 7:

[0882] The server encodes the acquired latitude and longitude information in a standard format such as JSON and prepares to send it back to the user's terminal.

[0883] Step 8:

[0884] The terminal receives the latitude and longitude information sent from the server, analyzes the received data, and converts it into an appropriate format.

[0885] Step 9:

[0886] The device inputs the acquired latitude and longitude information into a map display application (e.g., Google Maps) and displays the identified shooting location on the map. A pin or marker is placed on the map to visually indicate the shooting location to the user.

[0887] Step 10:

[0888] The user can check the map displayed on the device screen and view information about the shooting location. The user can also check detailed location information and information about surrounding facilities.

[0889] Example 1

[0890] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0891] Identifying the location where a landscape image was taken is extremely difficult using conventional methods, requiring a great deal of time and effort. Furthermore, there are only a limited number of systems for accurately identifying specific location information, and few of them are easily accessible to general users. Therefore, there is a need for a system that allows users to easily and quickly identify the location where a landscape image was taken and obtain detailed location information.

[0892] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0893] In this invention, the server includes means for receiving a landscape image and storing it in temporary storage, means for extracting features from the received landscape image, and means for comparing the extracted features with a database, thereby enabling the location where the landscape image was taken to be accurately identified and the latitude and longitude information of that location to be obtained.

[0894] The "user terminal means" is a device that allows a user to input a landscape image and transmit the image to the server.

[0895] The "server means" is a device that receives and temporarily stores scenic images sent from a user terminal.

[0896] The "image recognition algorithm means" is an algorithm for extracting features from the received landscape image.

[0897] The "database matching means" is a means for matching the extracted feature amount with information in a database.

[0898] The "position specifying means" is a means for specifying the latitude and longitude information of the scenic image based on the result of the comparison.

[0899] The "information transmitting means" is a means for transmitting the acquired latitude and longitude information to the user terminal.

[0900] The "map display means" is a means for displaying the latitude and longitude information acquired by the user terminal on a map.

[0901] The present invention provides a system for identifying the location where a landscape image was taken. A user uploads a landscape image using their own terminal, and a server analyzes the image to identify the location where the image was taken and provides the location information to the user. A specific embodiment is described below.

[0902] System Overview

[0903] The main purpose of this system is to identify the location of landscape images taken by users. The user device sends the image in a specified format (e.g., JPEG, PNG) to the server, which then analyzes the image to obtain location information. The analysis uses advanced image recognition algorithms such as convolutional neural networks (CNN).

[0904] Specific processing

[0905] User terminal

[0906] The user terminal uses a general device such as a smartphone or PC to select a landscape image and upload it to the system. The image upload process uses a general file upload function.

[0907] server

[0908] The server receives landscape images sent from the user's device and stores them in temporary storage. It then uses an image recognition algorithm to extract features from the received images. This extraction uses generative AI models such as TensorFlow and PyTorch. Specifically, it analyzes information such as edges, colors, and textures within the images to extract highly distinctive data.

[0909] Database collation

[0910] The server compares the extracted features with existing data in a database, which contains a large amount of Street View image data. The server then ranks the results based on similarity scores and selects the best matching image. The location information (latitude and longitude) of the selected image is then obtained.

[0911] Information transmission and display

[0912] The server sends the acquired latitude and longitude information to the user's device using a general-purpose format such as JSON. The user's device receives the location information sent from the server and enters it into a map display application (e.g., Google Maps). This application displays a pin (marker) on the map, allowing the user to visually confirm the location where the photo was taken.

[0913] Specific examples

[0914] For example, if a user inputs a photo of an unknown beautiful landscape taken during a trip into the system, the system can identify the location where the photo was taken. By checking the identified location information on a map, the user can revisit the location or share the information with others.

[0915] Prompt Sentence Examples

[0916] An example of a prompt to input to a generative AI model might be, "I want to identify the exact location of a landscape I photographed while traveling. I uploaded this image. Please tell me where it was taken."

[0917] This system makes it possible to accurately pinpoint the location where an image was taken for a wide range of purposes, including identifying memorable places, analyzing incidents and accidents, and academic research.

[0918] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0919] Step 1:

[0920] The user selects a landscape image using their device. Open the file selection dialog on the device and select the desired landscape image. In this case, the image file must be in a common format (e.g., JPEG, PNG). The input is the landscape image file, and the output is the path to the image file selected by the user.

[0921] Step 2:

[0922] The user presses the upload button to send a landscape image to the server. The device browser generates an HTTP request to send the image file to the server. The input is the path of the selected image file, and the output is the HTTP request sent to the server.

[0923] Step 3:

[0924] The server receives the landscape image sent from the user device and saves it in temporary storage. The server processes the received image and saves the image file in a specified directory in the file system. The input is the HTTP request from the user device, and the output is the image file saved in the server's temporary storage.

[0925] Step 4:

[0926] The server reads the images stored in temporary storage and runs an image recognition algorithm to extract features. A convolutional neural network (CNN) is used to extract features such as edges, colors, and textures within the image. The input is the image file stored in temporary storage, and the output is the extracted feature data.

[0927] Step 5:

[0928] The server compares the extracted features with a database. The feature data is used to compare with the Street View database using SQL queries or machine learning algorithms. The input is the extracted feature data, and the output is the matching result (multiple candidate images and similarity scores).

[0929] Step 6:

[0930] The server selects the best matching image based on the matching results and obtains the location information (latitude and longitude) of that image. A scoring algorithm is used to rank the candidate images and obtain the location information of the best matching image. The input is the matching result, and the output is the identified latitude and longitude information.

[0931] Step 7:

[0932] The server converts the acquired latitude and longitude information into a general-purpose format such as JSON and sends it to the user device. The server generates an HTTP response and returns the location information to the device. The input is the specified latitude and longitude information, and the output is JSON data sent to the user device.

[0933] Step 8:

[0934] The user device receives the JSON data sent from the server and inputs it into the map display application. The device parses the JSON data to extract latitude and longitude information and displays a pin (marker) on the map using the map display API. The input is the received JSON data, and the output is a pin for the photo location displayed on the map.

[0935] (Application example 1)

[0936] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0937] Currently, there is no efficient system for quickly and accurately identifying the location of landscape images taken by many security personnel at the scene. As a result, responses to incidents and accidents are delayed and difficult to deal with. Furthermore, conventional systems have the problem that it takes time to identify location information, making it difficult to respond in real time. A new system is needed to solve these problems and improve the efficiency of security operations.

[0938] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0939] In this invention, the server includes a data processing device means for receiving input landscape images, an image recognition algorithm means for extracting features from the received landscape images, and a database matching means for comparing the extracted features with a matching database, thereby enabling security personnel to quickly and accurately identify the location information of landscape images taken on-site.

[0940] An "information processing device" is a terminal device used by a user to input a landscape image. This includes smartphones, tablets, personal computers, and the like.

[0941] The "data processing device" is a server or cloud computing system that receives and processes landscape images sent by users.

[0942] An "image recognition algorithm" is an algorithm for extracting features from received landscape images. A typical example is a convolutional neural network.

[0943] "Database matching" is the process of comparing the extracted features with an existing database, thereby identifying the location of the photo.

[0944] "Location identification" is the process of acquiring location information for a landscape image based on the matching results, including latitude and longitude information.

[0945] "Information transmission" is a process of transmitting the acquired location information to the user's information processing device.

[0946] "Map display" is a process of displaying location information acquired on the user's information processing device on a map. This is done using a map application.

[0947] The "interface" is a user interface that allows an information processing device to use input landscape images for security purposes. It provides a function for quickly reporting suspicious situations or incidents.

[0948] This invention is a system for identifying the location where a landscape image was taken, in which the landscape image is uploaded from a user's information processing device, the data processing device processes the image to obtain specific location information, and provides that information back to the user. This system is particularly suitable for security applications.

[0949] Hardware and Software Configuration

[0950] 1. Information processing equipment:

[0951] A general user device such as a smartphone, tablet, or PC. It is used by users to upload landscape images they have taken on-site.

[0952] 2. Data Processing Unit:

[0953] It consists of a server and a cloud computing system. This device receives and processes landscape images sent by users, temporarily storing the images, extracting features, and performing matching.

[0954] 3. Image Recognition Algorithm:

[0955] A typical example is a convolutional neural network (CNN) using TensorFlow. Features are extracted from the received landscape image and compared with a database.

[0956] 4. Matching database:

[0957] It can use the Street View image database of Google Cloud Vision API, match it with the extracted features, and obtain the location information of the best-matching candidate.

[0958] 5. Map display:

[0959] The acquired location information is displayed on the user's information processing device using the Google Maps API.

[0960] Processing Flow

[0961] 1. Upload your image:

[0962] The user takes a picture of the scenery using an information processing device and uploads it to the server through the app. React Native can be used.

[0963] 2. Receiving images and extracting features:

[0964] The server stores the received images in temporary storage and extracts features using a convolutional neural network (CNN).

[0965] 3. Database Matching:

[0966] The extracted features are compared with a matching database to identify the location information of the best-matching candidate.

[0967] 4. Obtaining and sending location information:

[0968] Location information is obtained based on the matching results and returned to the user's information processing device in JSON format or similar.

[0969] 5. Map display:

[0970] The user's information processing device displays the acquired location information on a map using the Google Maps API.

[0971] Specific examples

[0972] Assume that a security officer discovers a suspicious person at the scene, takes a picture of the person, and uploads it. The server processes the received image and obtains specific location information. The location information is sent to the user's information processing device and displayed on a map in real time, enabling a prompt response.

[0973] Prompt Sentence Examples

[0974] "Write a Python program to upload a specific landscape image and display a marker on a map based on the latitude and longitude of that location. This program should also include image feature extraction using TensorFlow and Street View matching using the Google Cloud Vision API."

[0975] The above is a specific embodiment for carrying out the invention. This system makes security operations more efficient and enables quick responses.

[0976] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0977] Step 1:

[0978] A user takes landscape images of the site using an information processing device (e.g., a smartphone), selects the images, and uploads them to the server through the system's application. The input is the captured landscape image, and the output is the completion of the upload process to the server.

[0979] Step 2:

[0980] The server receives the landscape image uploaded by the user. The specific operation of the server is to store the image in temporary storage. The input of this step is the landscape image sent by the user, and the output is the image stored in the temporary storage.

[0981] Step 3:

[0982] The server extracts features from the stored landscape images. This process uses a convolutional neural network (CNN) using TensorFlow. The input is the stored landscape image, and the output is the features extracted from the image. The features are data such as edges, colors, and textures within the image.

[0983] Step 4:

[0984] The server compares the extracted features with a matching database (e.g., the Street View image database of the Google Cloud Vision API). Specifically, it matches the existing landscape image features in the database and calculates the similarity. The input is the extracted features, and the output is a list of best-matching candidate images and their similarity scores.

[0985] Step 5:

[0986] The server selects the best-matching candidate image based on the similarity score and obtains its location information (latitude and longitude). The input is a list of candidate images from the matching results, and the output is the location information of the selected image. The location information is provided in latitude and longitude format.

[0987] Step 6:

[0988] The server returns the acquired location information to the information processing device. Specifically, it packages the location information in JSON format and sends it to the user's information processing device. The input is the acquired location information, and the output is the completion of transmission to the information processing device.

[0989] Step 7:

[0990] The user's information processing device displays the received location information on a map using the Google Maps API. Specifically, it displays a pin on the map based on the acquired latitude and longitude information, visually indicating the location where the landscape image was taken. The input is the location information sent from the server, and the output is the location information displayed on the map application.

[0991] The above are the specific processing steps and operation flow of the system that realizes this application example.

[0992] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0993] The present invention is a system that provides interactive feedback according to the user's emotional state by combining a system that identifies the shooting location of a landscape image with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0994] Users upload landscape images to the system using their own devices (e.g., smartphones or PCs). At this time, the user's device captures not only the landscape images but also the user's facial expressions and voice data, which allows the system to estimate the user's emotional state.

[0995] The device can send the user's emotional data along with the landscape image to the server. This emotional data is obtained as a result of analyzing facial expressions and voice. For example, if the user is happy when uploading an image, the facial expression data is sent to the server.

[0996] The server receives the landscape images and emotion data sent from the user's device and uses an image recognition algorithm to extract features from the landscape images. The extracted features are compared with the Street View database to identify the most matching image. At this time, the emotion engine analyzes the user's emotional state, and the results are also used for feedback.

[0997] For example, if a user uploads a landscape image and the emotion engine recognizes the user's joy or excitement, the server can provide positive feedback that includes not only the identified location but also related and background information about the location. For example, information like "This place has beautiful scenery and is visited by many tourists" could be added.

[0998] The server transmits the identified latitude and longitude information and feedback information based on the user's emotion to the user device. The user device inputs this information into a map display application and visually displays the shooting location on a map. Furthermore, customized information based on the user's emotion can also be displayed.

[0999] This system not only allows users to accurately identify the location where a landscape image was taken, but also provides interactive and rich feedback based on their emotions. For example, if the user is relaxed, detailed information about the location's history and cultural background may be displayed. On the other hand, if the user is excited, information about activities and events in the location may be provided, enriching the user's experience.

[1000] Systems equipped with additional functions based on this emotion engine can be used in a wide range of fields, including tourist guides, education, remote exploration, and incident and accident analysis. Providing information linked to the user's emotions will further increase the system's usefulness.

[1001] The processing flow will be explained below.

[1002] Step 1:

[1003] The user opens the system's web application from their device (smartphone or PC) and uploads a landscape image. The user opens the file selection window, selects the landscape image, and clicks the upload button.

[1004] Step 2:

[1005] The device sends the selected landscape image to a specific API endpoint, while capturing the user's facial expressions with a camera and collecting voice data, which are then processed by the emotion engine.

[1006] Step 3:

[1007] The emotion engine analyzes the captured facial and voice data to determine the user's emotional state, which may be expressed as, for example, joy, sadness, surprise, or the like.

[1008] Step 4:

[1009] The device sends the emotion data from the emotion engine along with the landscape image data to the server. The landscape image data is often sent in JPEG or PNG format, and the emotion data is sent in JSON format.

[1010] Step 5:

[1011] The server receives the landscape image data and emotion data sent from the user device. The landscape image is temporarily stored in storage and then passed to the image recognition algorithm.

[1012] Step 6:

[1013] The server uses an image recognition algorithm (e.g., a convolutional neural network) to extract features from the landscape image, such as edge, color, and texture data.

[1014] Step 7:

[1015] The server then matches the extracted features with the Street View database, comparing the features with images in the database and identifying the most similar images.

[1016] Step 8:

[1017] The server selects the image with the highest similarity from the images obtained as a result of the comparison, and obtains the latitude and longitude information corresponding to that image.

[1018] Step 9:

[1019] The server then encodes the acquired latitude and longitude information in a standard format such as JSON and sends it back to the user's device, along with a customized message based on the user's emotional data.

[1020] Step 10:

[1021] The device receives the latitude and longitude information and emotion-based messages sent from the server, analyzes the received data, and converts it into an appropriate format.

[1022] Step 11:

[1023] The device inputs the acquired latitude and longitude information into a map display application (e.g., Google Maps) and displays the identified shooting location on the map. At this time, customized messages and information based on the emotion data are also displayed.

[1024] Step 12:

[1025] The user sees a map and a customized message displayed on the device screen. For example, if the user is happy, positive information about the location (tourist attractions and background information) is also displayed. On the other hand, if the user is sad, a comforting message may be displayed.

[1026] As a result, a system that combines an emotion engine can provide interactive and comprehensive feedback that corresponds to the user's emotional state.

[1027] Example 2

[1028] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1029] Existing landscape image location identification systems only identify the location of landscape images taken by the user, but are unable to provide interactive feedback that takes into account the user's emotional state. As a result, the information available to the user is limited, resulting in a lackluster user experience. Furthermore, they are unable to provide information that takes into account the context of the user's emotional state, making it difficult to provide flexible feedback that can be adapted to multiple uses.

[1030] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes user terminal means for inputting a landscape image and emotion data, server means for receiving the input landscape image and emotion data, image recognition algorithm means for extracting features from the received landscape image, emotion analysis means for analyzing the user's emotional state from the received emotion data, database comparison means for comparing the extracted features with a comparison database, location identification means for acquiring latitude and longitude information of the landscape image based on the comparison result, feedback generation means for generating feedback information based on the latitude and longitude information acquired by the location identification means and the emotion data analyzed by the emotion analysis means, information transmission means for transmitting the generated latitude and longitude information and feedback information to the user terminal, and map display means for displaying the latitude and longitude information and feedback information acquired by the user terminal on a map. This allows the user to not only accurately identify the location where the landscape image was taken but also receive interactive and comprehensive feedback tailored to the user's emotional state.

[1031] A "user terminal" is an electronic device that a user uses to input and transmit scenery images and emotion data.

[1032] The "server" is a computer system that processes and analyzes landscape images and emotion data received from user devices, and generates location identification and feedback information.

[1033] An "image recognition algorithm" is an analytical method for extracting features from landscape images and comparing them with a database.

[1034] "Emotion analysis" is the process of recognizing and determining a user's emotional state from facial expressions and voice data.

[1035] "Database matching" is the process of comparing and matching the features extracted from a landscape image with an existing database.

[1036] "Location identification" is a process of obtaining latitude and longitude information based on the results of matching the scenery image.

[1037] "Feedback generation" refers to the act of generating customized feedback information to provide to a user based on the location-identified information and the results of sentiment analysis.

[1038] "Information transmission" refers to the process of transmitting the generated latitude and longitude information and feedback information to the user terminal.

[1039] "Map display" is a function that visually displays the acquired latitude and longitude information and feedback information on a map.

[1040] The present invention provides a system that provides interactive feedback according to the user's emotional state by combining a system that identifies the shooting location of a landscape image with an emotion analysis function that recognizes the user's emotions. Specific embodiments of the system are described below.

[1041] Users upload landscape images to the system using their own devices (e.g., smartphones or PCs). At this time, the user's device captures not only the landscape images but also the user's facial expressions and voice data, which allows the system to estimate the user's emotional state.

[1042] The device sends the captured emotion data (facial expressions and voice analysis results) along with the landscape image to the server. The communication protocol is HTTP / HTTPS, and the data is sent in JSON format.

[1043] The server processes the received landscape images and emotion data. For landscape images, it uses an image recognition algorithm to extract features. In this case, it uses a convolutional neural network (CNN), an artificial intelligence model (AI model). The extracted features are compared with a landscape database to identify the most similar images.

[1044] Next, the server uses an emotion analysis engine to analyze the user's emotional data. This analysis method uses a general emotion recognition API (e.g., emotion analysis algorithm). The analysis result determines the user's emotional state (e.g., joy, sadness, surprise, etc.).

[1045] The server generates customized feedback for the user based on the results of image recognition and emotion analysis. Specifically, it generates a positive message based on the user's emotions and tourist information about the identified location, along with the latitude and longitude information of the location. The generated feedback information is then sent back to the user's device in JSON format.

[1046] The user terminal inputs the information received from the server into a map display application (e.g., a map application) and visually displays the shooting location of the landscape image on a map. It also displays customized feedback information according to the user's emotions.

[1047] For example, if a user uploads a photo of a beach from their smartphone and the sentiment analysis engine recognizes the user's joy, the server will generate customized feedback such as, "This place is a popular tourist spot with many people and offers beautiful sunrises." This information is sent to the user's device and visually displayed on a map.

[1048] Example prompt sentences

[1049] An example of a prompt is as follows:

[1050] "Upload a landscape image and generate positive feedback based on how I feel when I'm happy."

[1051] In this way, the system of the present invention allows users to not only accurately identify the location where a landscape image was taken, but also to receive interactive and rich feedback according to their emotions. This system can be used in a wide range of fields, including tourist guides, education, remote exploration, and incident and accident analysis.

[1052] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1053] Step 1:

[1054] Users take landscape images using their own devices (e.g., smartphones or PCs) and upload them to the system. The device captures the landscape images taken with the camera as well as the user's facial expression and voice data. The input data are the landscape images and the facial and voice data. This allows us to collect initial data for estimating the user's emotional state.

[1055] Step 2:

[1056] The user device sends the captured landscape image and emotion data to the server. The data is packaged in JSON format and sent to the server via HTTP / HTTPS protocol. The input data are the landscape image and emotion data, and the output data are the landscape image and emotion data sent to the server.

[1057] Step 3:

[1058] The server analyzes the landscape images received from the user terminal using an image recognition algorithm. Specifically, it uses an artificial intelligence model (e.g., a convolutional neural network) to extract features from the landscape images. The input data is the landscape image, and the output data is the extracted features. The image recognition algorithm performs extraction and analysis based on these features.

[1059] Step 4:

[1060] The server analyzes the received emotion data using an emotion analysis engine. The algorithm used is a general emotion recognition API (e.g., emotion analysis algorithm). The input data is emotion data, and the output data is the user's emotional state (e.g., joy, sadness, surprise, etc.). The analyzed emotion data serves as the basis for generating feedback.

[1061] Step 5:

[1062] The server compares the extracted features with a matching database. The database stores landscape image data, and by matching the features with this data, it identifies the image that best matches. The input data are the extracted features, and the output data are the identified latitude and longitude information.

[1063] Step 6:

[1064] The server generates feedback for the user based on the acquired latitude and longitude information and the emotion analysis results. Specifically, it generates tourist information for the identified location and positive messages based on the user's emotions. The input data are the latitude and longitude information and the emotion analysis results, and the output data is customized feedback information.

[1065] Step 7:

[1066] The server sends the generated latitude and longitude information and feedback information to the user terminal. The data is again packaged in JSON format and sent via HTTP / HTTPS protocol. The input data is the generated latitude and longitude information and feedback information, and the output data is the information sent to the user terminal.

[1067] Step 8:

[1068] The user terminal inputs the latitude and longitude information and feedback information received from the server into a map display application, and visually displays the shooting location of the scenic image on a map. Customized feedback information according to the user's emotions is also displayed. The input data is the latitude and longitude information and feedback information received from the server, and the output data is the map display and the feedback display.

[1069] As a concrete example, if a user uploads a photo of a beach from their smartphone and the sentiment analysis engine recognizes the user's joy, the server generates customized feedback such as "This place is a popular tourist spot with many people, and you can see a beautiful sunrise," and sends it to the user's device. The user's device can display this on a map, providing the user with positive feedback along with visual information.

[1070] Prompt Sentence Examples

[1071] "Upload a landscape image and generate positive feedback based on how I feel when I'm happy."

[1072] (Application example 2)

[1073] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1074] In conventional systems for identifying landscape images, users simply identify the location where the landscape was photographed. This results in only providing flat information that ignores the user's emotional state, making it difficult to provide an interactive and rich experience tailored to the scene. Furthermore, there is a need for a system that can provide not only landscape images but also information customized to the user's emotional state.

[1075] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes user device means for inputting a landscape image, server device means for receiving the input landscape image, image recognition algorithm means for extracting features from the received landscape image, database comparison means for comparing the extracted features with a comparison database, location identification means for acquiring location information of the landscape image based on the comparison result, information transmission means for transmitting the acquired location information to the user device, map display means for displaying the acquired location information on a map in the user terminal, emotion recognition means for recognizing the user's emotion, and feedback generation means for generating customized feedback information based on the emotion recognition means. This allows the user not only to identify the location where the landscape image was taken, but also to receive interactive and comprehensive feedback according to their emotional state.

[1076] A "user device" is a terminal that a user uses to input a scenic image or receive that information.

[1077] The "server device" is a computer system that receives and processes scenic images sent from user devices.

[1078] An "image recognition algorithm" is a method for extracting features from received landscape images and analyzing and identifying those images.

[1079] A "verification database" is a database for comparing the extracted features with existing data.

[1080] The "position specifying means" is a means for acquiring the position information of the scenery image based on the result of comparison with the collation database.

[1081] The "information transmitting means" is a means for transmitting the acquired location information to the user device.

[1082] The "map display means" is a means for displaying the location information acquired on the user device on a map.

[1083] The "emotion recognition means" is a means for recognizing the user's emotional state, and estimates emotions by analyzing facial expressions, voice, etc.

[1084] A "feedback generating means" is a means for generating customized feedback information based on a recognized emotional state.

[1085] The present invention provides a system for identifying the location where a landscape image was taken, as well as a system for recognizing a user's emotions and providing interactive feedback in response to the emotions. This system is implemented in the following manner.

[1086] First, a user takes a landscape image using a user device and uploads it to the system. At this time, the user device captures not only the landscape image but also the user's facial expression and voice data, and estimates the user's emotional state. Specifically, the system recognizes the user's facial expression and voice using the camera and microphone of a smartphone, tablet, or PC.

[1087] The server device receives the scenery image and emotion data sent from the user device. Then, it extracts features from the received scenery image using an image recognition algorithm (e.g., a convolutional neural network). This image recognition algorithm can use libraries such as OpenCV and TensorFlow.

[1088] The server then compares the extracted features with a matching database to identify the best matching image, which can be, for example, Google Street View or a proprietary landscape database.

[1089] The server device acquires location information for the landscape image based on the matching result and transmits this location information to the user device. This transmission can be performed using a communication method such as a REST API or WebSocket. The user device then inputs the acquired location information into a map display application (such as Google Maps) to visually display the shooting location on a map.

[1090] Meanwhile, the emotion recognition means analyzes the user's emotional state and generates customized feedback information based on the results. This feedback generation uses an emotion engine to provide feedback including tourist information and activity information according to the user's emotions. This process can utilize, for example, the Emotion API or a proprietary emotion recognition model.

[1091] Specific examples

[1092] In this example, a user uploads a photo of a beautiful lake and the system provides feedback such as, "This lake is a great place to enjoy the tranquility of nature. There are many tourists around."

[1093] In addition, if a user uploads a photo of an old castle, the emotion engine can read their excited state and the system can provide information such as, "Historical ceremonies and events are held regularly at this castle. The next event is XX."

[1094] Example of input prompt for generative AI model

[1095] "Create an interactive tourist guide based on the following information. Give us an example of a system that generates customized tourist feedback based on the user's images and emotional state."

[1096] The above is an embodiment of the present invention. This system not only allows users to accurately identify the location where a landscape image was taken, but also provides interactive feedback based on their emotions, providing a richer experience.

[1097] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1098] Step 1:

[1099] A user takes a landscape image and uploads it to a user device, which then captures the image along with the user's facial expression and voice data, which then become input data.

[1100] Step 2:

[1101] The user device sends a landscape image and the user's emotional data to the server. At this time, the user device encodes the image and emotional data into an appropriate format (e.g., JPEG, MP3) and sends it. The server receives and decompresses the image and emotional data to obtain them.

[1102] Step 3:

[1103] The server uses an image recognition algorithm to extract features from the received landscape image. Specifically, it uses an image recognition algorithm (for example, a convolutional neural network) to convert the input image into numerical data and extracts characteristic patterns and edges, which become the image features.

[1104] Step 4:

[1105] The server matches the extracted features with a matching database. Here, it compares the features with existing database entries to find the best match. For example, it uses a comparison algorithm such as cosine similarity. This is an important data processing step required for location estimation.

[1106] Step 5:

[1107] The server obtains the location information of the landscape image from the matching results. Specifically, it extracts the location information (e.g., latitude and longitude data) stored in the database and stores it in a specific variable. This becomes the obtained location information.

[1108] Step 6:

[1109] The server transmits the acquired location information to the user device. At this time, the server encodes the location information into an appropriate format (e.g., JSON format) and transmits it to the user device. The user device receives and decodes the information to acquire the location information.

[1110] Step 7:

[1111] The user device inputs the acquired location information into a map display application and visually displays the shooting location on a map, allowing the user to confirm the shooting location of the scenic image on the map.

[1112] Step 8:

[1113] The server analyzes the user's emotional state using emotion recognition means. Specifically, it analyzes input facial expressions and voice data to identify the user's emotions (e.g., joy, excitement, relaxation, etc.). It then estimates the user's emotional state by performing data calculations using emotion recognition algorithms.

[1114] Step 9:

[1115] The server generates customized feedback information based on the emotion recognition result. The feedback generating means selects appropriate tourist information and activity information according to the emotional state and creates feedback to be provided to the user.

[1116] Step 10:

[1117] The server sends the generated feedback information to the user device, encodes the feedback information into an appropriate format (e.g., text, image link), and sends it to the user device, which receives it, decodes it, and displays it to the user in visual and textual form.

[1118] Through each of these steps, users can not only identify the location where the landscape image was taken, but also receive interactive feedback based on their emotional state.

[1119] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1120] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1121] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1122] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1123] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1124] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1125] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1126] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1127] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1128] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1129] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1130] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1131] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1132] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1133] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1134] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1135] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1136] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1137] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1138] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1139] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1140] The following is further disclosed regarding the above embodiment.

[1141] (Claim 1)

[1142] a user terminal means for inputting a landscape image;

[1143] a server means for receiving an input scenic image;

[1144] an image recognition algorithm means for extracting features from the received scenic image;

[1145] a database matching means for comparing the extracted feature quantity with a matching database;

[1146] a position specifying means for acquiring latitude and longitude information of the scenic image based on the comparison result;

[1147] an information transmitting means for transmitting the acquired latitude and longitude information to a user terminal;

[1148] a map display means for displaying the acquired latitude and longitude information on a map;

[1149] A system including:

[1150] (Claim 2)

[1151] 2. The system according to claim 1, wherein the input landscape image is collated with a street view database by the server means.

[1152] (Claim 3)

[1153] 10. The system of claim 1, wherein the image recognition algorithm means uses a convolutional neural network.

[1154] "Example 1"

[1155] (Claim 1)

[1156] a user terminal means for inputting a landscape image;

[1157] a server means for receiving input scenery images and storing them in temporary storage;

[1158] an image recognition algorithm means for extracting features from the received scenic image;

[1159] a database matching means for matching the extracted feature amount with a database;

[1160] a position specifying means for specifying latitude and longitude information of the scenic image based on the comparison result;

[1161] an information transmitting means for transmitting the acquired latitude and longitude information to a user terminal;

[1162] a means for inputting the latitude and longitude information acquired by the user terminal into a map display means and displaying it on a map;

[1163] A system including:

[1164] (Claim 2)

[1165] 2. The system according to claim 1, wherein the input landscape image is collated with a street view database by the server means.

[1166] (Claim 3)

[1167] 10. The system of claim 1, wherein the image recognition algorithm means uses a convolutional neural network.

[1168] "Application Example 1"

[1169] (Claim 1)

[1170] an information processing device for inputting a landscape image;

[1171] a data processing means for receiving an input scenic image;

[1172] an image recognition algorithm means for extracting features from the received scenic image;

[1173] a database matching means for comparing the extracted feature quantity with a matching database;

[1174] a position specifying means for acquiring position information of the scenic image based on the comparison result;

[1175] an information transmitting means for transmitting the acquired location information to the information processing device;

[1176] a map display means for displaying the location information acquired by the information processing device on a map;

[1177] an interface means for allowing the information processing device to use the input scenic image for security purposes;

[1178] A system including:

[1179] (Claim 2)

[1180] 2. The system according to claim 1, wherein the input scenic image is collated with the location information database by the data processing means.

[1181] (Claim 3)

[1182] 10. The system of claim 1, wherein the image recognition algorithm means uses a convolutional neural network.

[1183] "Example 2: Combining Emotion Engines"

[1184] (Claim 1)

[1185] a user terminal means for inputting a landscape image and emotion data;

[1186] a server means for receiving input scenery images and emotion data;

[1187] an image recognition algorithm means for extracting features from the received scenic image;

[1188] emotion analysis means for analyzing the user's emotional state from the received emotion data;

[1189] a database matching means for comparing the extracted feature quantity with a matching database;

[1190] a position specifying means for acquiring latitude and longitude information of the scenic image based on the comparison result;

[1191] feedback generating means for generating feedback information based on the latitude and longitude information acquired by the location specifying means and the emotion data analyzed by the emotion analyzing means;

[1192] an information transmitting means for transmitting the generated latitude and longitude information and feedback information to the user terminal;

[1193] a map display means for displaying the latitude and longitude information and feedback information acquired by the user terminal on a map;

[1194] A system including:

[1195] (Claim 2)

[1196] 2. The system according to claim 1, wherein the input landscape image is collated with a landscape database by the server means.

[1197] (Claim 3)

[1198] 10. The system of claim 1, wherein the image recognition algorithm means uses an artificial intelligence model.

[1199] "Application example 2 when combining emotion engines"

[1200] Rewriting of claims

[1201] (Claim 1)

[1202] a user device means for inputting a scenic image;

[1203] a server device means for receiving an input scenic image;

[1204] an image recognition algorithm means for extracting features from the received scenic image;

[1205] a database matching means for comparing the extracted feature quantity with a matching database;

[1206] a position specifying means for acquiring position information of the scenic image based on the comparison result;

[1207] an information transmitting means for transmitting the acquired location information to the user device;

[1208] a map display means for displaying the acquired location information on a map in the user terminal;

[1209] emotion recognition means for recognizing an emotion of a user;

[1210] feedback generating means for generating customized feedback information based on the emotion recognition means;

[1211] A system including:

[1212] (Claim 2)

[1213] 2. The system of claim 1, wherein the input scenic image is matched by the server device means with a database of road views.

[1214] (Claim 3)

[1215] 10. The system of claim 1, wherein the image recognition algorithm means uses a convolutional neural network. [Explanation of symbols]

[1216] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a user terminal means for inputting a landscape image; a server means for receiving an input scenic image; image recognition algorithm means for extracting features from the received scenic image; a database matching means for comparing the extracted feature quantity with a matching database; a position specifying means for acquiring latitude and longitude information of the scenic image based on the comparison result; an information transmitting means for transmitting the acquired latitude and longitude information to a user terminal; a map display means for displaying the acquired latitude and longitude information on a map; A system including:

2. 2. The system according to claim 1, wherein the input scenic image is collated with a street view database by the server means.

3. 2. The system of claim 1, wherein the image recognition algorithm means uses a convolutional neural network.

Images