system

Abstract

A system is provided.SOLUTION: A system comprising means for a user to capture a video using a mobile device, means for obtaining GPS location information corresponding to the video in real-time and tagging the video data, means for compressing and uploading the obtained video data and GPS data to a server, means for analyzing the received video data and GPS data at the server to generate digital twin data, and means for calculating a reward based on the amount and quality of data provided to the user and paying via a remittance means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern urban development and management require the creation of real-time, detailed digital twin data. Conventional methods require advanced technical equipment and specialized knowledge, and few systems allow general users to participate. This poses the problem of costly and time-consuming data collection, making it difficult to quickly generate highly accurate digital twins. The present invention aims to solve these problems by building a system that allows general users to easily provide data and by rewarding users, thereby improving the efficiency and accuracy of data collection. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides the following means. Specifically, it provides a means for a user to shoot video using a mobile device. The system includes means for acquiring GPS location information corresponding to the video in real time and tagging the video data, and means for compressing the acquired video data and GPS data and uploading it to a server. Furthermore, it includes means for analyzing the received video data and GPS data in the server and generating digital twin data, and means for calculating a reward based on the quantity and quality of the data provided to the user and paying it via a remittance means. This system allows general users to easily provide data and enables the efficient creation of highly accurate digital twin data. Furthermore, since users are paid a reward, it also increases their motivation to provide data.

[0006] "User" means an individual or entity that uses the System and provides data.

[0007] A "mobile device" is an electronic device that can be carried by a user and is capable of capturing video and acquiring location information.

[0008] "Moving image" is digital data that records a series of video frames over a period of time.

[0009] "GPS location information" is data indicating specific coordinates on the Earth obtained using the Global Positioning System.

[0010] "Real-time" is a term that indicates that data acquisition and processing occurs immediately, without delay.

[0011] "Tagging" is the process of adding GPS location information as auxiliary information related to video data.

[0012] "Compression" is the process of encoding and transforming data to reduce its size.

[0013] "Server" means a networked computer system that receives, analyzes, and stores uploaded data.

[0014] "Analysis" is the process of examining acquired data in detail to extract and evaluate specific information.

[0015] "Digital twin data" is a dataset that digitally recreates real-world objects and environments.

[0016] "Reward" is the economic compensation paid to a user for the data provided by the user.

[0017] A "remittance means" is an electronic remittance system for paying rewards to users. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] Installing the dedicated app and registering as a user

[0040] The user downloads and installs the dedicated app from the GOOGLE PLAY(R) Store or App Store.

[0041] After installation, launch the app and create an account by entering the required information such as your name, email address, and password.

[0042] The terminal sends this information to the server, which then creates a new account in its database.

[0043] Once the account is created, the server will send a confirmation message to the device.

[0044] Data collection and upload

[0045] The user launches the app and switches to video recording mode.

[0046] Tap the "Start Recording" button at the shooting location to capture a specific area of ​​the city.

[0047] As soon as the device starts recording video, it acquires GPS location information in real time.

[0048] While recording, the device tags the video data with GPS location information and saves it as metadata in the video file.

[0049] Once the recording is complete, the device compresses the video data and uploads it to the server.

[0050] Data analysis and digital twin generation

[0051] The server analyzes the received video data and GPS data.

[0052] The analysis process involves identifying specific locations in the scene from the video data and confirming location information based on GPS data.

[0053] The server generates digital twin data using specialized algorithms and machine learning models.

[0054] The generated digital twin data is quality checked and any necessary filtering processes are performed.

[0055] The digital twin data is then stored in a database.

[0056] Reward calculation and payment

[0057] After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user.

[0058] Based on the evaluation results, the amount of compensation for the user is calculated.

[0059] The server obtains the user's PayPay account information and transfers the calculated reward amount.

[0060] When the payment is completed, the server sends a payment completion notification to the user's terminal.

[0061] Specific examples

[0062] For example, consider the case where a user takes a video while walking around a city's tourist attractions with their smartphone. As soon as the device starts recording the video, it acquires location information and tags the data in real time. When the video recording is finished, the data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a digital twin of the tourist attraction. Once this process is complete, the server evaluates the user's contribution to the data provision, calculates a certain reward, and transfers it to the user's PayPay account. The user's device then displays a notification that the reward has been paid.

[0063] This concludes the description of an embodiment of the system of the present invention. The system allows users to easily provide data and provides a reward system for providing data, enabling efficient and highly accurate digital twin generation.

[0064] The processing flow will be explained below.

[0065] Step 1: User Registration

[0066] The user downloads and installs the dedicated app from the Google (registered trademark) Play Store or App Store.

[0067] A user launches the app and enters the required information, such as their name, email address, and password.

[0068] The terminal transmits this registration information to the server.

[0069] The server stores the received registration information in a database and creates a new account.

[0070] The server sends a notification to the device that account creation is complete.

[0071] Step 2: Record video and collect GPS information

[0072] The user launches the app and selects video recording mode.

[0073] The user taps the "Start Recording" button to begin recording video.

[0074] As soon as the device starts recording video, it will obtain GPS location information in real time.

[0075] The GPS information acquired by the device while recording is tagged to the video data and saved as metadata.

[0076] Step 3: Upload your data

[0077] The user ends video recording.

[0078] The device compresses the captured video data.

[0079] The device uploads the compressed video data and associated GPS metadata to a server.

[0080] The server acknowledges receipt and prepares the data for storage.

[0081] Step 4: Analyze the data and create a digital twin

[0082] The server analyzes the received video data and GPS data.

[0083] During the analysis process, the server identifies specific locations in the scene from the video data.

[0084] The server checks the location information based on GPS data and compares it with the video data.

[0085] The server uses specialized algorithms and machine learning models to generate digital twin data for the urban area.

[0086] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[0087] Step 5: Calculating and paying rewards

[0088] The server evaluates the quantity and quality of the data provided by the user.

[0089] The server calculates the amount of reward for the user based on the evaluation result.

[0090] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[0091] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[0092] Step 6: Notification and confirmation

[0093] The terminal displays the remittance completion notice received to the user.

[0094] The user checks the reward amount in their PayPay account.

[0095] The above are the specific steps of the program process, which allows users to easily provide data and receive rewards.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] Conventional digital twin generation systems have struggled to efficiently collect large amounts of data and generate highly accurate digital twins. Furthermore, users lacked incentives to provide data, reducing the operational efficiency of the system. Furthermore, the time and effort required for data quality checking and filtering was also an issue.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes: a means for a user to shoot video using a mobile terminal; a means for acquiring global positioning system (GPS) location information corresponding to the video in real time and tagging the video data; a means for compressing the acquired video data and GPS data and uploading it to the server; a means for analyzing the received video data and GPS data in the server and generating digital twin data; a means for quality checking and filtering the generated digital twin data; and a means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via a remittance means. This enables more efficient and accurate digital twin generation. It also increases the incentive for users to provide data, improving the operational efficiency of the entire system.

[0101] "User" refers to an individual or legal entity that provides data using a mobile terminal.

[0102] "Mobile device" refers to a portable computing device such as a smartphone or tablet.

[0103] "Means for capturing video" refers to a method for recording video data using a camera mounted on a mobile terminal.

[0104] "Global Positioning System location information" refers to terrestrial location data obtained using the Global Positioning System.

[0105] "Means of obtaining and tagging video data in real time" refers to a method of obtaining global positioning system location information immediately when a video is shot and adding that location information as metadata to the video data.

[0106] "Means for compressing and uploading the captured video data and global positioning system data to a server" refers to a method for compressing the captured video data and the captured global positioning system location information to reduce their size and transmitting them to a server over a network.

[0107] "Server" refers to the computer system used to analyze and store video data and global positioning system data.

[0108] "Digital twin data" refers to data that digitally reproduces a real-world object.

[0109] "Means for analyzing and generating digital twin data" refers to a method for using received video data and global positioning system data to create a digital model of a real-world object.

[0110] "Quality check and filtering measures" refers to methods for checking the quality of generated digital twin data and removing or correcting inappropriate data or errors.

[0111] "Means for calculating remuneration based on the quantity and quality of data provided and paying it through a remittance means" refers to a method for calculating remuneration based on the quantity and quality of data provided by a user and paying the remuneration to the user through an electronic payment system.

[0112] This invention includes a system that uses a mobile terminal to shoot video, tags the video with Global Positioning System (GPS) location information, uploads the video to a server, generates digital twin data, and finally pays rewards to users. Each step is described in detail below.

[0113] Installing the dedicated app and registering as a user

[0114] The user downloads and installs the dedicated app from the Google Play Store or App Store. After installation, the user launches the app and creates an account by entering required information such as name, email address, and password. The user then sends the account information to the server via their device, and the server creates a new account in its database. Once the account is created, the server sends a confirmation message to the device.

[0115] Data collection and upload

[0116] The user launches the app and switches to video recording mode. At the recording location, they tap the "Start Recording" button and record a specific area of ​​the city. The device starts recording video and simultaneously acquires GPS location information in real time. While recording, the device tags the video data with GPS location information and saves it as metadata in the video file. Once recording is complete, the device compresses the video data and uploads it to the server.

[0117] Data analysis and digital twin generation

[0118] The server analyzes the received video data and GPS data. During the analysis process, specific locations in the scene are identified from the video data and location information is confirmed based on the GPS data. The server generates digital twin data using dedicated algorithms and machine learning models (e.g., OpenCV or TENSORFLOW®). The generated digital twin data is then quality checked and any necessary filtering is performed. Finally, the digital twin data is stored in a database.

[0119] Reward calculation and payment

[0120] After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user. Based on the evaluation results, it calculates the amount of compensation for the user. The server obtains the user's electronic payment account information and transfers the calculated amount of compensation. Once the payment is complete, the server sends a payment completion notification to the user's device.

[0121] Specific examples

[0122] For example, if a user takes a video while walking around a city's tourist attractions with their smartphone, the device starts recording the video and simultaneously acquires location information, tagging the data in real time. Once the video recording is finished, the data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a digital twin of the tourist attraction. Once this process is complete, the server evaluates the user's contribution to the data provision, calculates a certain reward, and transfers it to an electronic payment account. The user's device then displays a notification that the reward has been paid.

[0123] Prompt Sentence Examples

[0124] Please tell me the procedure for creating a new account.

[0125] "Please explain how to tag video data with GPS location information in real time."

[0126] "What kind of data do I need to generate a digital twin and how do I analyze it?"

[0127] This system allows users to easily provide data and receive compensation for that data, making it possible to generate digital twins efficiently and with high accuracy.

[0128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0129] Step 1: Install the dedicated app

[0130] (Explanation) The user downloads and installs the dedicated app from the Google Play Store or App Store.

[0131] (Input) A user accesses the Google Play Store or App Store.

[0132] (Data processing / calculation) Search for dedicated apps through the store's search function and download the apps you find.

[0133] (Output) A dedicated app is installed on the user's device.

[0134] Step 2: Create an account

[0135] (Description) The user launches the installed app and creates an account by entering required information such as name, email address, and password.

[0136] (Input) The user enters required information into the app, such as their name, email address, and password.

[0137] (Data processing / calculation) The terminal formats the user's input data to send to the server and generates an HTTP request.

[0138] (Output) The server creates the new account in the database and sends a confirmation message to the terminal.

[0139] Step 3: Start recording

[0140] (Explanation) The user launches the app, switches to video recording mode, and taps the "Start Recording" button at the recording location.

[0141] (Input) The user taps the "Start Recording" button.

[0142] (Data processing / calculation) The device starts recording video and simultaneously obtains location information in real time from GPS.

[0143] (Output) The device tags the video data and acquired GPS information in real time and saves it as metadata.

[0144] Step 4: Compress and upload your video data

[0145] (Explanation) Once video recording is complete, the device compresses the video data and uploads it to the server.

[0146] (Input) Recorded video data and GPS location information.

[0147] (Data processing / calculation) The terminal compresses the video data using a codec such as H.264 and sends it to the server along with the location information via an HTTP POST request.

[0148] (Output) Compressed video data and GPS data are saved on the server.

[0149] Step 5: Analyze the data

[0150] (Description) The server analyzes the video data and GPS data received.

[0151] (Input) Compressed video data and GPS data received by the server.

[0152] (Data processing / calculation) The server uses analysis software such as OpenCV and TensorFlow to identify specific locations in the scene from the video data and confirm the corresponding location information.

[0153] (Output) The analyzed scene data and position information are obtained.

[0154] Step 6: Generate the digital twin

[0155] (Explanation) The server generates digital twin data using dedicated algorithms and machine learning models.

[0156] (Input) Analyzed video data and location information.

[0157] (Data processing / calculation) The server generates a digital twin using 3D modeling software such as Blender based on the scene identification results and location information.

[0158] (Output) The generated digital twin data is obtained.

[0159] Step 7: Quality check and filtering

[0160] (Description) Quality check and filter the generated digital twin data.

[0161] (Input) Generated digital twin data.

[0162] (Data processing / calculation) The server checks the quality of the generated data, filters out inappropriate data, and in some cases corrects it.

[0163] (Output) Quality-confirmed digital twin data is obtained.

[0164] Step 8: Save your data

[0165] (Explanation) The server stores the quality-confirmed digital twin data in a database.

[0166] (Input) Quality-confirmed digital twin data.

[0167] The (data processing / calculation) server inserts the digital twin data into a database system such as SQL Server or MySQL (registered trademark).

[0168] (Output) Digital twin data is stored in a database.

[0169] Step 9: Calculating rewards

[0170] (Explanation) After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user and calculates the reward amount.

[0171] (Input) Provided digital twin data and user information.

[0172] (Data processing / calculation) The server scores the data based on its quantity and quality and calculates the reward amount.

[0173] (Output) The calculated reward amount is obtained.

[0174] Step 10: Send your rewards

[0175] (Explanation) The server obtains the user's electronic payment account information and transfers the calculated reward amount.

[0176] (Input) Calculated reward amount and user's electronic payment account information.

[0177] (Data processing / calculation) The server uses the PayPay API to transfer the reward amount to the user's account.

[0178] (Output) Rewards are transferred to the user's electronic payment account.

[0179] Step 11: Payment completion notification

[0180] (Explanation) When payment is completed, the server sends a payment completion notification to the user's terminal.

[0181] (Input) Information on completed remittance.

[0182] The (data processing / calculation) server generates a notification message and sends a push notification to the user's device.

[0183] (Output) A notification of payment completion is displayed on the user's terminal.

[0184] (Application example 1)

[0185] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0186] In conventional logistics centers, it is difficult to monitor abnormalities (e.g., missing pallets, obstructions) in real time, hindering efficient operation. Another issue is the difficulty of generating an accurate digital twin and understanding the situation within the logistics facility. Furthermore, there is a lack of a system that appropriately rewards users based on the quality and quantity of data they provide.

[0187] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0188] In this invention, the server includes means for a user to shoot video using a mobile device, means for acquiring location information in real time and tagging the video data, means for compressing the acquired video data and location data and uploading it to the server, means for analyzing the received video data and location data and generating digital twin data, means for monitoring abnormalities in the logistics facility in real time using the digital twin data, and means for calculating compensation based on the quantity and quality of data provided to the user and paying the compensation via a remittance means. This makes it possible to monitor the situation in the logistics facility in real time and manage it efficiently.

[0189] "User" means an individual or entity that uses a mobile device to capture video and provide the data, along with location information, to the server.

[0190] A "mobile device" is a portable electronic device capable of capturing video and acquiring location information.

[0191] "Video data" refers to video image information captured by a mobile device.

[0192] "Location information" refers to data that indicates a specific location or coordinates using GPS or other positioning means.

[0193] "Tagging" is the process of linking metadata such as location information to video data.

[0194] "Compression" is a technique for reducing data volume.

[0195] A "server" is a computer system that receives, stores, and analyzes data over a network.

[0196] "Digital twin data" is data that accurately recreates physical objects and spaces in the real world.

[0197] An "abnormal location" is a location within a logistics facility where an abnormal condition or problem is occurring.

[0198] "Real-time" refers to a situation in which processing and information updates occur almost immediately.

[0199] "Reward" refers to the payment made based on the data provided by the user.

[0200] "Remittance method" refers to the payment method or technology used to send rewards to users.

[0201] The system of this invention is realized by a user taking video of the inside of a logistics facility using a mobile device, tagging the video data with location information, and uploading it to a server. Details and specific examples of each means are provided below.

[0202] A user takes a video using a mobile device

[0203] Users use a mobile device such as a smartphone or tablet to capture video of a specific area within a logistics facility. This mobile device must have a camera, GPS functionality, and an internet connection. Users install a dedicated application and start recording using the app.

[0204] Real-time location information acquisition and tagging of video data

[0205] The device acquires location information in real time using GPS or other positioning methods while shooting video. The acquired location information is tagged as metadata to the video data being shot. This process is realized using libraries such as OpenCV and Geopy.

[0206] Compression of video data and location data and upload to server

[0207] Once the recording is complete, the device compresses the video and location data and uploads it to a server over its internet connection, for example by sending an HTTP POST request using the Requests library.

[0208] Data analysis on the server and generation of a digital twin

[0209] The server analyzes the received video and location data. During this analysis process, specialized algorithms and machine learning models are used to generate a digital twin of the logistics facility. For example, it identifies abnormalities (missing pallets, obstructions, etc.). This data is used for real-time monitoring of the logistics facility.

[0210] Calculation and payment of rewards based on quantity and quality of data provided

[0211] The server calculates the reward based on the quantity and quality of the data provided by the user and pays the user. The payment method is, for example, an electronic payment system. This information is notified to the user's terminal.

[0212] Specific examples

[0213] Users walk around the logistics facility with their smartphones and record video. As the device starts recording, it simultaneously acquires location information, which is tagged in real time. After filming is complete, the compressed video data and location data are uploaded to a server. The server analyzes the received data and generates a digital twin of the logistics facility. Through this process, users receive appropriate compensation for the data they provide.

[0214] Prompt Sentence Examples

[0215] Examples of prompts for a generative AI model might include:

[0216] "Please explain how a smartphone can be used in a logistics facility to capture video of a specific area, add location information in real time, and upload the data to a server."

[0217] The above steps enable efficient and highly accurate generation of digital twins and real-time monitoring of abnormalities within logistics facilities.

[0218] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0219] Step 1:

[0220] The user installs a dedicated application on the mobile device.

[0221] Input: Basic information such as user name, email address, and password

[0222] Process: The user launches the application and enters basic information. The device sends this information to the server, which then creates a new account in the database.

[0223] Output: Account verification message is displayed on the terminal.

[0224] Step 2:

[0225] The user launches the app and switches to video recording mode.

[0226] Input: User initiated video recording mode

[0227] Processing: The device activates the camera and GPS functions and prepares to capture video and obtain location information.

[0228] Output: The video recording mode screen will be displayed.

[0229] Step 3:

[0230] The user starts shooting video in a specific area within the logistics facility.

[0231] Input: User taps the "Start Recording" button

[0232] Processing: The device captures video in real time and simultaneously acquires GPS location information. Each captured video frame is tagged with location information.

[0233] Output: A video file and its corresponding geotag will be generated.

[0234] Step 4:

[0235] After the recording is complete, the device compresses the video data and location data.

[0236] Input: Video data and location data after shooting is complete

[0237] Processing: The device efficiently compresses video and location data to reduce data size.

[0238] Output: Compressed video data and location data

[0239] Step 5:

[0240] The compressed data is uploaded to the server.

[0241] Input: Compressed video data and location data

[0242] Processing: The device sends the data to the server over the Internet using an HTTP POST request.

[0243] Output: A message is displayed confirming successful data transmission.

[0244] Step 6:

[0245] The server analyzes the received data and generates a digital twin of the logistics facility.

[0246] Input: Video data and location data sent from the device

[0247] Processing: The server uses video analysis algorithms and machine learning models to identify anomalies within the facility and generate a digital twin.

[0248] Output: Digital twin data and abnormality location identification results

[0249] Step 7:

[0250] Rewards are calculated and paid based on the quantity and quality of data provided.

[0251] Input: The amount and quality of data analyzed by the server

[0252] Processing: The server calculates the user's reward amount using a reward calculation algorithm and makes the payment through an electronic payment system.

[0253] Output: A notification will be displayed on the device that the reward amount has been deposited into the user's account.

[0254] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0255] Installing the dedicated app and registering as a user

[0256] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[0257] After installation, launch the app and create an account by entering the required information such as your name, email address, and password.

[0258] The terminal sends this information to the server, which then creates a new account in its database.

[0259] Once the account is created, the server will send a confirmation message to the device.

[0260] Video recording, GPS information collection, and emotion recognition

[0261] The user launches the app and selects video recording mode.

[0262] Tap the "Start Recording" button at the shooting location to capture a specific area of ​​the city.

[0263] As soon as the device starts recording video, it acquires GPS location information in real time.

[0264] The GPS information acquired by the device while recording is tagged to the video data and saved as metadata.

[0265] While the device is recording video, the emotion engine recognizes the user's emotions in real time through facial recognition and voice analysis.

[0266] The recognized emotion data is also saved along with the video data.

[0267] Uploading data

[0268] The user ends video recording.

[0269] The device compresses the captured video data and emotion data.

[0270] The device uploads the compressed video data, GPS metadata, and emotion data to a server.

[0271] The server acknowledges receipt and prepares the data for storage.

[0272] Data analysis and digital twin generation

[0273] The server analyzes the received video data, GPS data, and emotion data.

[0274] During the analysis process, the server identifies specific locations in the scene from the video data.

[0275] The server checks the location information based on GPS data and compares it with the video data.

[0276] The server generates digital twin data for the urban area using dedicated algorithms and machine learning models, taking into account emotional data.

[0277] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[0278] Reward calculation and payment

[0279] After the digital twin data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data.

[0280] Based on the evaluation results, the amount of compensation for the user is calculated.

[0281] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[0282] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[0283] Specific examples

[0284] For example, consider the case where a user takes a video while walking around a tourist spot in a city with their smartphone. As soon as the device starts recording the video, it acquires GPS location information and tags the data in real time. While recording the video, the device uses an emotion engine to recognize emotions from the user's facial expressions and voice, and saves these as data. Once recording is complete, this data is automatically compressed and uploaded to the server. The server immediately analyzes the received data and creates a digital twin of the tourist spot. Once this process is complete, the server evaluates the user's contribution in providing data and the content of the emotional data, calculates a certain reward, and transfers it to the user's PayPay account. A notification is then displayed on the user's device indicating that the reward has been paid.

[0285] This concludes the description of an embodiment of the system of the present invention. The system allows users to easily provide data and provides a reward system for providing data, enabling efficient and highly accurate digital twin generation. Furthermore, adding emotional data can create richer digital twins.

[0286] The processing flow will be explained below.

[0287] Step 1: User Registration

[0288] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[0289] A user launches the app and enters the required information, such as their name, email address, and password.

[0290] The terminal transmits this registration information to the server.

[0291] The server stores the received registration information in a database and creates a new account.

[0292] The server sends a notification to the device that account creation is complete.

[0293] Step 2: Record video and collect GPS information

[0294] The user launches the app and selects video recording mode.

[0295] The user taps the "Start Recording" button to begin recording video.

[0296] As soon as the device starts recording video, it will acquire GPS location information in real time.

[0297] The GPS information acquired by the device in real time is tagged to the video data and saved as metadata.

[0298] Step 3: Emotion Recognition

[0299] While the device is recording video, it activates the emotion engine to recognize the user's face.

[0300] The device analyzes the user's facial expressions and voice and generates emotional data in real time.

[0301] The device stores the generated emotion data together with the video data and GPS data.

[0302] Step 4: Upload your data

[0303] The user ends video recording.

[0304] The device compresses the captured video data and emotion data.

[0305] The device uploads the compressed video data, GPS metadata, and emotion data to a server.

[0306] The server acknowledges receipt and prepares the data for storage.

[0307] Step 5: Analyze the data and create a digital twin

[0308] The server analyzes the received video data, GPS data, and emotion data.

[0309] During the analysis process, the server identifies specific locations in the scene from the video data.

[0310] The server checks the location information based on GPS data and compares it with the video data.

[0311] The server generates digital twin data that reflects the user's emotions based on the emotional data.

[0312] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[0313] Step 6: Calculating and paying rewards

[0314] After the digital twin data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data.

[0315] The server calculates the amount of reward for the user based on the evaluation result.

[0316] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[0317] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[0318] Step 7: Notification and confirmation

[0319] The terminal displays the remittance completion notice received to the user.

[0320] The user checks the reward amount in their PayPay account.

[0321] These are the specific processing steps of a system that combines an emotion engine. Users can easily provide data, and by utilizing emotion data, we can expect to improve the accuracy of digital twins.

[0322] Example 2

[0323] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0324] Conventional technologies have struggled to efficiently process user-generated video and location information and generate virtual environments based on that data. Furthermore, because data analysis, including user emotional data, is not performed, it is not possible to provide more realistic and detailed virtual environments. Furthermore, the process of calculating and paying rewards to users is complicated, and there is a lack of mechanisms to increase user motivation. Therefore, there is a need to improve the quantity and quality of data provided by users.

[0325] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0326] In this invention, the server includes means for recognizing emotion data from a user's facial expressions and voice in real time and integrating it into video data, means for using a machine learning model to analyze the video data including the emotion data, and means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via electronic remittance. This makes it possible to add emotion data to video data captured by the user and generate a highly accurate virtual environment. Furthermore, by promptly paying an appropriate reward to the user, it is possible to increase the user's motivation and encourage them to provide more and higher-quality data.

[0327] A "portable communication terminal" refers to a device that can be carried by a user and has a communication function, and specifically includes a smartphone, a tablet, a portable computer, and the like.

[0328] "Location information" is data indicating a geographical location, including, for example, latitude and longitude information obtained using a GPS (Global Positioning System).

[0329] "Central processing unit" refers to a central device for data processing, such as a server or cloud computing service that receives, analyzes, stores, and transmits data.

[0330] "Virtual environment data" refers to data such as digital twins and 3D models generated based on real physical spaces, which allows the real world to be reproduced digitally.

[0331] "Emotion data" is data that represents the emotional state of the user analyzed based on facial expressions, voice, and other biological information.

[0332] A "machine learning model" refers to software that has algorithms that analyze large amounts of data, find patterns and regularities, and make predictions and classifications based on them.

[0333] "Electronic remittance instruments" refers to systems and services for sending and receiving money over the Internet, and specifically includes digital wallets and bank online remittance systems.

[0334] MODE FOR CARRYING OUT THE INVENTION

[0335] In this invention, a system is constructed in which users collect data using mobile communication terminals, and the server analyzes and processes the data to generate virtual environment data, and also calculates and pays rewards to users.

[0336] Installing the dedicated app and registering as a user

[0337] Users must download and install a dedicated app from the ANDROID (registered trademark) or iOS app store. After installation, they launch the app and create an account by entering required information such as their name, email address, and password. The communication device sends this information to the server, which then creates a new account in its database. Once the account is created, the server sends a confirmation message to the communication device.

[0338] Video capture, location collection, and emotion recognition

[0339] The user launches the dedicated app and selects video recording mode. At the recording location, they tap the "Start Recording" button and record a specific area. As soon as the communication device starts recording video, it acquires GPS location information in real time. The communication device tags the location information acquired while recording with the video data and saves it as metadata. While the communication device is recording video, it uses an emotion engine (e.g., Microsoft® Azure® Emotion API) to recognize the user's emotions in real time through facial recognition and voice analysis. The recognized emotion data is also saved along with the video data.

[0340] Uploading data

[0341] When the user finishes recording a video, the communication device compresses the video data and emotion data. The compressed video data, GPS metadata, and emotion data are uploaded to the server. The server confirms receipt and prepares to store the data.

[0342] Data analysis and virtual environment data generation

[0343] The server analyzes the uploaded video data, location data, and emotion data. During the analysis process, the server identifies specific locations in the scene from the video data. The server matches the video data based on the location data. The server generates virtual environment data for the urban area using a dedicated algorithm or machine learning model (e.g., TensorFlow model), taking emotion data into consideration. The server checks the quality of the generated virtual environment data and filters out inappropriate data.

[0344] Reward calculation and payment

[0345] After the virtual environment data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data. Based on the evaluation results, the server calculates a reward amount for the user. The server then obtains the user's electronic payment account information and prepares to transfer the calculated reward amount. The server transfers the reward amount to the electronic payment account and sends a transfer completion notification to the communication terminal.

[0346] Specific examples

[0347] For example, consider the case where a user takes a video while walking around a tourist spot in a city with a smartphone. As the communication device starts recording the video, it simultaneously acquires location information and tags the data in real time. While recording the video, the communication device uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize emotions from the user's facial expressions and voice, and saves these as data. Once the recording is finished, this data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a virtual environment of the tourist spot. Once this process is complete, the server evaluates the user's contribution in providing data and the content of the emotional data, calculates a certain reward, and transfers it to an electronic payment account. A notification is then displayed on the user's device indicating that the reward has been paid.

[0348] Prompt Sentence Examples

[0349] "Please explain the process of generating a virtual environment using emotion data and location information when photographing tourist spots in a city."

[0350] "Please explain the reward system for taking videos using a smartphone app and uploading the data."

[0351] In this way, users can easily provide data and receive rewards for it, enabling efficient and highly accurate generation of virtual environments. Furthermore, adding emotional data can create richer virtual environments.

[0352] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0353] Step 1:

[0354] The user downloads and installs the dedicated app.

[0355] Input: App store search query, app download request.

[0356] Output: A dedicated app is installed on the communication device.

[0357] Specific operation: The user opens the Google Play Store or App Store, searches for "Digital Twin Creation App," and taps the download button. The app is automatically downloaded and installed on the communication device.

[0358] Step 2:

[0359] The user enters the required information to create an account.

[0360] Input: User information such as name, email address, and password.

[0361] Output: The user information is sent to the server and a new account is created in the database.

[0362] Specific operation: The user launches the app, enters the required information such as name, email address, and password, and taps the "Register" button. The entered data is sent from the communication device to the server as an HTTP POST request. The server creates a new account in the database based on the data received.

[0363] Step 3:

[0364] The terminal sends the user information to the server, and the server sends a confirmation message.

[0365] Input: Request to send user information.

[0366] Output: A server confirmation message is sent to the terminal.

[0367] Specific operation: The communication terminal sends user information to the server, and after the server verifies the received data, it sends a confirmation message in JSON format to the terminal. The terminal receives this message and displays the confirmation message to the user.

[0368] Step 4:

[0369] The user selects the video recording mode and starts recording.

[0370] Input: User action (selecting video recording mode, tapping the start recording button).

[0371] Output: Video recording and GPS information acquisition begins.

[0372] Specific operation: The user selects "Video recording mode" in the app and taps the "Start recording" button on the screen. The communication device activates the camera and starts recording video. At the same time, the GPS module begins obtaining location information in real time.

[0373] Step 5:

[0374] The device tags the video data with location information and also acquires emotional data.

[0375] Input: Video frames being captured, real-time GPS data, and user face and voice data.

[0376] Output: Video data tagged with location and emotion data.

[0377] How it works: The communication device acquires location data (latitude and longitude) for each video frame and tags it as metadata in the video data. In parallel, an emotion engine (e.g., Microsoft Azure Emotion API) analyzes the user's face and voice and recognizes emotion data in real time. This data is then integrated into the video data.

[0378] Step 6:

[0379] The user stops recording the video, and the device compresses the data.

[0380] Input: User action (tapping the stop recording button).

[0381] Output: Compressed video data, location data, and emotion data.

[0382] Specific operation: The user taps the "Stop Recording" button. The communication device compresses the captured video data, location data, and emotion data. For example, it compiles the data in ZIP format.

[0383] Step 7:

[0384] The device uploads the data to the server.

[0385] Input: Compressed data file.

[0386] Output: The data is uploaded to the server.

[0387] Specific operation: The communication device uploads the compressed data file to the server using an HTTP POST request. The progress is displayed during the upload.

[0388] Step 8:

[0389] The server checks the received data.

[0390] Input: The uploaded data file.

[0391] Output: Data integrity verification message.

[0392] Specific operation: The server checks the received data. Specifically, it checks the integrity of the data and ensures consistency. Once the data is confirmed, the server sends a response message to the communication terminal.

[0393] Step 9:

[0394] The server analyzes the data and generates virtual environment data.

[0395] Input: Uploaded video data, location data, emotion data.

[0396] Output: Parsed data, generated virtual environment data.

[0397] How it works: The server uses Python scripts and TensorFlow models to analyze the received video data, location data, and emotion data. It identifies specific locations in the scene from the video data and matches them with the location data. It generates virtual environment data for the urban area using a specific algorithm, taking emotion data into account. The server checks the quality of the generated virtual environment data and filters out inappropriate data.

[0398] Step 10:

[0399] The server evaluates the user's data and calculates the reward.

[0400] Input: Quantity and quality of analyzed data, user-provided data.

[0401] Output: Calculated reward amount.

[0402] How it works: The server evaluates the quantity and quality of the data provided by users in the database. Evaluation criteria include the clarity of the data and the reliability of the sentiment data. Based on the evaluation results, a Python script calculates the reward amount.

[0403] Step 11:

[0404] The server transfers the reward to the user's electronic payment account.

[0405] Input: User's electronic payment account information, calculated reward amount.

[0406] Output: Remittance completion notification.

[0407] Specific operation: The server obtains the user's electronic payment account information and transfers the reward amount via API. Once the transfer is complete, a push notification is sent to the device.

[0408] (Application example 2)

[0409] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0410] Conventional digital twin generation systems only use video data and GPS data, making it difficult to accurately and realistically reflect user experiences and emotions. Furthermore, the calculation of rewards for users' data provision is based on limited indicators, resulting in issues of fairness and inaccuracy. Furthermore, the analysis of collected data and the generation of digital twins often relies on specific algorithms, preventing the use of advanced generative AI models.

[0411] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0412] In this invention, the server includes: means for allowing a user to shoot video using a mobile device, acquiring GPS location information in real time, and tagging the video data; means for acquiring the user's emotional data in real time using an emotion recognition engine while shooting the video, and tagging the video data; means for compressing the acquired video data, GPS data, and emotional data and uploading them to the server; means for analyzing the received video data, GPS data, and emotional data in the server and generating digital twin data using a generative AI model; and means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via an electronic payment method. This enables the creation of more realistic and rich digital twins that reflect the user's experiences and emotions, and achieves fair and accurate reward calculations.

[0413] A "mobile device" is an electronic device that a user can carry around, such as a smartphone or tablet.

[0414] "Means for capturing video" refers to a method or apparatus for recording video using the camera function on a mobile device.

[0415] "GPS location information" refers to information that uses the Global Positioning System to obtain real-time location data about a specific place or location.

[0416] "Tagging" refers to the operation of adding identifying information to data, and in this case means adding GPS location information and emotional data to video data.

[0417] An "emotion recognition engine" refers to software or algorithms that analyze a user's emotions in real time from their facial expressions, tone of voice, etc.

[0418] "Means for compressing and uploading to a server" refers to technology for converting the captured video data, GPS data, and emotion data into a small data format and transmitting it to a server via the Internet.

[0419] A "generative AI model" refers to a program or algorithm that uses artificial intelligence technology to analyze and synthesize complex data.

[0420] "Digital twin data" refers to a digital replica of a physical space or object, and in this case refers to data that recreates an urban area in a virtual space.

[0421] "Electronic payment means" refers to a method or system for electronically transferring rewards to users, and generally includes electronic money and digital wallets.

[0422] "Reward Calculation" refers to the process of calculating rewards to users based on the quantity and quality of data provided.

[0423] System Overview

[0424] This invention is a system that allows users to shoot videos using a mobile device and tag the videos with GPS location information and emotion data acquired in real time. This data is then compressed and uploaded to a server. The server then analyzes the received video data, GPS data, and emotion data and generates digital twin data using a generative AI model. The system also calculates a reward for the user based on the quantity and quality of the data provided and pays the reward via electronic payment methods.

[0425] Hardware and software used

[0426] Mobile devices (e.g. smartphones, tablets)

[0427] A device that can be carried by the user and has video recording, GPS, and emotion recognition functions.

[0428] Camera function (software)

[0429] Software for capturing video.

[0430] GPS module (hardware)

[0431] Module for obtaining geographical location information in real time.

[0432] Emotion recognition engine (software)

[0433] Software that analyzes facial expressions and tone of voice to recognize emotions.

[0434] Compression algorithm (software)

[0435] An algorithm for efficiently compressing captured data.

[0436] Communication module (hardware)

[0437] A module for sending data to a server over the Internet.

[0438] Server (hardware)

[0439] A device for analyzing the received data and generating a digital twin using a generative AI model.

[0440] Generative AI model (software)

[0441] Artificial intelligence techniques for analyzing and synthesizing complex data.

[0442] Electronic payment system (software)

[0443] A system for electronically transferring rewards to users (e.g., electronic money, digital wallets).

[0444] Example of operation

[0445] The user launches the dedicated app and starts recording video. As the user walks around, the mobile device's camera captures the video data, and the GPS module acquires location information in real time. At the same time, the emotion recognition engine extracts emotional data from the user's facial expressions and voice, and tags this data into the video. Once recording is complete, the device's on-device compression algorithm compresses the data and uploads it to the server via the communication module.

[0446] When the data arrives at the server, the server first acknowledges receipt of the data and then begins analysis. The received video data, GPS data, and emotion data are analyzed using a generative AI model to generate digital twin data. This digital twin data is rich in content, reflecting the user's emotions and location information. Based on the results of this analysis, the server calculates a reward for the user based on the quantity and quality of the data provided. The calculated reward is paid to the user using an electronic payment system.

[0447] Prompt Sentence Examples

[0448] "Generate a Python program for a smartphone app that allows users to take videos while walking around a specific area of ​​a city and tag them with emotional data in real time. Include how to upload the data to a server and implement a reward system."

[0449] In order to implement the present invention, it is important to combine these hardware and software to provide a user-friendly interface.

[0450] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0451] Program processing steps

[0452] Step 1:

[0453] The user launches the dedicated app and selects video recording mode.

[0454] Input: User's finger action (tap).

[0455] Output: Video recording preparation screen.

[0456] Specific operation: Through the mobile device interface, the user taps the app icon and selects "Video Recording" from the menu to activate the recording mode.

[0457] Step 2:

[0458] The user taps the "Start Recording" button at the shooting location to capture a specific area.

[0459] Input: User's finger action (tap).

[0460] Output: Recording screen, real-time video data.

[0461] Specific operation: The camera starts capturing video, and at the same time, the device continues to generate video data in real time.

[0462] Step 3:

[0463] As soon as the device starts recording video, it acquires GPS location information in real time.

[0464] Input: GPS sensor on mobile device.

[0465] Output: Real-time GPS location information.

[0466] Specific operation: The GPS module is activated and current coordinate information is captured in real time, so that location information is added to the video data at any time.

[0467] Step 4:

[0468] The terminal acquires the user's emotion data in real time using an emotion recognition engine.

[0469] Input: The user's facial expressions and voice.

[0470] Output: Recognized emotion data (e.g., joy, sadness, surprise, etc.).

[0471] How it works: Video and audio data captured by the camera is passed to the emotion recognition engine, where it is analyzed. As a result, an emotion tag for the user is generated.

[0472] Step 5:

[0473] The device compresses the acquired video data, GPS data, and emotion data and uploads them to a server.

[0474] Input: Video data, GPS data, emotion data.

[0475] Output: Compressed data, notification of completion of transmission to the server.

[0476] Specific operation: The compression algorithm compresses the data, and the communication module transmits the compressed data to the server over the Internet.

[0477] Step 6:

[0478] The server acknowledges the received data and then begins analyzing it.

[0479] Input: Compressed video data, GPS data, emotion data.

[0480] Output: Analyzed data, notification that the digital twin is ready to be generated.

[0481] What happens: The server decompresses the data and prepares the individual data streams as base data for analysis, so that each data stream can be split appropriately and analyzed.

[0482] Step 7:

[0483] The server generates digital twin data using the generative AI model.

[0484] Input: Analyzed video data, GPS data, emotion data.

[0485] Output: Digital twin data.

[0486] How it works: By inputting the analysis results into a generative AI model, complex data analysis and synthesis processes are performed to generate a digital twin of the urban area.

[0487] Step 8:

[0488] The server calculates a reward based on the quantity and quality of the data provided and pays the reward to the user via an electronic payment means.

[0489] Input: Data quantity and quality assessment results.

[0490] Output: Calculated reward amount, notification of transfer completion.

[0491] Specific operation: The server's evaluation algorithm calculates the reward amount based on the quantity and quality of data provided, the usefulness of the emotional data, etc., and transfers it to the user's account through an electronic payment system.

[0492] This makes it possible to create a more realistic and rich digital twin that reflects the user's experiences and emotions, resulting in fair and accurate reward calculations.

[0493] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0494] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0495] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0496] [Second embodiment]

[0497] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0498] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0499] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0500] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0501] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0502] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0503] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0504] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0505] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0506] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0507] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0508] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0509] Installing the dedicated app and registering as a user

[0510] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[0511] After installation, launch the app and create an account by entering the required information such as your name, email address, and password.

[0512] The terminal sends this information to the server, which then creates a new account in its database.

[0513] Once the account is created, the server will send a confirmation message to the device.

[0514] Data collection and upload

[0515] The user launches the app and switches to video recording mode.

[0516] Tap the "Start Recording" button at the shooting location to capture a specific area of ​​the city.

[0517] As soon as the device starts recording video, it acquires GPS location information in real time.

[0518] While recording, the device tags the video data with GPS location information and saves it as metadata in the video file.

[0519] Once the recording is complete, the device compresses the video data and uploads it to the server.

[0520] Data analysis and digital twin generation

[0521] The server analyzes the received video data and GPS data.

[0522] The analysis process involves identifying specific locations in the scene from the video data and confirming location information based on GPS data.

[0523] The server generates digital twin data using specialized algorithms and machine learning models.

[0524] The generated digital twin data is quality checked and any necessary filtering processes are performed.

[0525] The digital twin data is then stored in a database.

[0526] Reward calculation and payment

[0527] After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user.

[0528] Based on the evaluation results, the amount of compensation for the user is calculated.

[0529] The server obtains the user's PayPay account information and transfers the calculated reward amount.

[0530] When the payment is completed, the server sends a payment completion notification to the user's terminal.

[0531] Specific examples

[0532] For example, consider the case where a user takes a video while walking around a city's tourist attractions with their smartphone. As soon as the device starts recording the video, it acquires location information and tags the data in real time. When the video recording is finished, the data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a digital twin of the tourist attraction. Once this process is complete, the server evaluates the user's contribution to the data provision, calculates a certain reward, and transfers it to the user's PayPay account. The user's device then displays a notification that the reward has been paid.

[0533] This concludes the description of an embodiment of the system of the present invention. The system allows users to easily provide data and provides a reward system for providing data, enabling efficient and highly accurate digital twin generation.

[0534] The processing flow will be explained below.

[0535] Step 1: User Registration

[0536] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[0537] A user launches the app and enters the required information, such as their name, email address, and password.

[0538] The terminal transmits this registration information to the server.

[0539] The server stores the received registration information in a database and creates a new account.

[0540] The server sends a notification to the device that account creation is complete.

[0541] Step 2: Record video and collect GPS information

[0542] The user launches the app and selects video recording mode.

[0543] The user taps the "Start Recording" button to begin recording video.

[0544] As soon as the device starts recording video, it will obtain GPS location information in real time.

[0545] The GPS information acquired by the device while recording is tagged to the video data and saved as metadata.

[0546] Step 3: Upload your data

[0547] The user ends video recording.

[0548] The device compresses the captured video data.

[0549] The device uploads the compressed video data and associated GPS metadata to a server.

[0550] The server acknowledges receipt and prepares the data for storage.

[0551] Step 4: Analyze the data and create a digital twin

[0552] The server analyzes the received video data and GPS data.

[0553] During the analysis process, the server identifies specific locations in the scene from the video data.

[0554] The server checks the location information based on GPS data and compares it with the video data.

[0555] The server uses specialized algorithms and machine learning models to generate digital twin data for the urban area.

[0556] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[0557] Step 5: Calculating and paying rewards

[0558] The server evaluates the quantity and quality of the data provided by the user.

[0559] The server calculates the amount of reward for the user based on the evaluation result.

[0560] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[0561] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[0562] Step 6: Notification and confirmation

[0563] The terminal displays the remittance completion notice received to the user.

[0564] The user checks the reward amount in their PayPay account.

[0565] The above are the specific steps of the program process, which allows users to easily provide data and receive rewards.

[0566] Example 1

[0567] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0568] Conventional digital twin generation systems have struggled to efficiently collect large amounts of data and generate highly accurate digital twins. Furthermore, users lacked incentives to provide data, reducing the operational efficiency of the system. Furthermore, the time and effort required for data quality checking and filtering was also an issue.

[0569] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0570] In this invention, the server includes: a means for a user to shoot video using a mobile terminal; a means for acquiring global positioning system (GPS) location information corresponding to the video in real time and tagging the video data; a means for compressing the acquired video data and GPS data and uploading it to the server; a means for analyzing the received video data and GPS data in the server and generating digital twin data; a means for quality checking and filtering the generated digital twin data; and a means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via a remittance means. This enables more efficient and accurate digital twin generation. It also increases the incentive for users to provide data, improving the operational efficiency of the entire system.

[0571] "User" refers to an individual or legal entity that provides data using a mobile terminal.

[0572] "Mobile device" refers to a portable computing device such as a smartphone or tablet.

[0573] "Means for capturing video" refers to a method for recording video data using a camera mounted on a mobile terminal.

[0574] "Global Positioning System location information" refers to terrestrial location data obtained using the Global Positioning System.

[0575] "Means of obtaining and tagging video data in real time" refers to a method of obtaining global positioning system location information immediately when a video is shot and adding that location information as metadata to the video data.

[0576] "Means for compressing and uploading the captured video data and global positioning system data to a server" refers to a method for compressing the captured video data and the captured global positioning system location information to reduce their size and transmitting them to a server over a network.

[0577] "Server" refers to the computer system used to analyze and store video data and global positioning system data.

[0578] "Digital twin data" refers to data that digitally reproduces a real-world object.

[0579] "Means for analyzing and generating digital twin data" refers to a method for using received video data and global positioning system data to create a digital model of a real-world object.

[0580] "Quality check and filtering measures" refers to methods for checking the quality of generated digital twin data and removing or correcting inappropriate data or errors.

[0581] "Means for calculating remuneration based on the quantity and quality of data provided and paying it through a remittance means" refers to a method for calculating remuneration based on the quantity and quality of data provided by a user and paying the remuneration to the user through an electronic payment system.

[0582] This invention includes a system that uses a mobile terminal to shoot video, tags the video with Global Positioning System (GPS) location information, uploads the video to a server, generates digital twin data, and finally pays rewards to users. Each step is described in detail below.

[0583] Installing the dedicated app and registering as a user

[0584] The user downloads and installs the dedicated app from the Google Play Store or App Store. After installation, the user launches the app and creates an account by entering required information such as name, email address, and password. The user then sends the account information to the server via their device, and the server creates a new account in its database. Once the account is created, the server sends a confirmation message to the device.

[0585] Data collection and upload

[0586] The user launches the app and switches to video recording mode. At the recording location, they tap the "Start Recording" button and record a specific area of ​​the city. The device starts recording video and simultaneously acquires GPS location information in real time. While recording, the device tags the video data with GPS location information and saves it as metadata in the video file. Once recording is complete, the device compresses the video data and uploads it to the server.

[0587] Data analysis and digital twin generation

[0588] The server analyzes the received video data and GPS data. During the analysis process, specific locations in the scene are identified from the video data and location information is confirmed based on the GPS data. The server generates digital twin data using dedicated algorithms and machine learning models (e.g., OpenCV and TensorFlow). The generated digital twin data is then quality checked and any necessary filtering is performed. Finally, the digital twin data is stored in a database.

[0589] Reward calculation and payment

[0590] After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user. Based on the evaluation results, it calculates the amount of compensation for the user. The server obtains the user's electronic payment account information and transfers the calculated amount of compensation. Once the payment is complete, the server sends a payment completion notification to the user's device.

[0591] Specific examples

[0592] For example, if a user takes a video while walking around a city's tourist attractions with their smartphone, the device starts recording the video and simultaneously acquires location information, tagging the data in real time. Once the video recording is finished, the data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a digital twin of the tourist attraction. Once this process is complete, the server evaluates the user's contribution to the data provision, calculates a certain reward, and transfers it to an electronic payment account. The user's device then displays a notification that the reward has been paid.

[0593] Prompt Sentence Examples

[0594] Please tell me the procedure for creating a new account.

[0595] "Please explain how to tag video data with GPS location information in real time."

[0596] "What kind of data do I need to generate a digital twin and how do I analyze it?"

[0597] This system allows users to easily provide data and receive compensation for that data, making it possible to generate digital twins efficiently and with high accuracy.

[0598] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0599] Step 1: Install the dedicated app

[0600] (Explanation) The user downloads and installs the dedicated app from the Google Play Store or App Store.

[0601] (Input) A user accesses the Google Play Store or App Store.

[0602] (Data processing / calculation) Search for dedicated apps through the store's search function and download the apps you find.

[0603] (Output) A dedicated app is installed on the user's device.

[0604] Step 2: Create an account

[0605] (Description) The user launches the installed app and creates an account by entering required information such as name, email address, and password.

[0606] (Input) The user enters required information into the app, such as their name, email address, and password.

[0607] (Data processing / calculation) The terminal formats the user's input data to send to the server and generates an HTTP request.

[0608] (Output) The server creates the new account in the database and sends a confirmation message to the terminal.

[0609] Step 3: Start recording

[0610] (Explanation) The user launches the app, switches to video recording mode, and taps the "Start Recording" button at the recording location.

[0611] (Input) The user taps the "Start Recording" button.

[0612] (Data processing / calculation) The device starts recording video and simultaneously obtains location information in real time from GPS.

[0613] (Output) The device tags the video data and acquired GPS information in real time and saves it as metadata.

[0614] Step 4: Compress and upload your video data

[0615] (Explanation) Once video recording is complete, the device compresses the video data and uploads it to the server.

[0616] (Input) Recorded video data and GPS location information.

[0617] (Data processing / calculation) The terminal compresses the video data using a codec such as H.264 and sends it to the server along with the location information via an HTTP POST request.

[0618] (Output) Compressed video data and GPS data are saved on the server.

[0619] Step 5: Analyze the data

[0620] (Description) The server analyzes the video data and GPS data received.

[0621] (Input) Compressed video data and GPS data received by the server.

[0622] (Data processing / calculation) The server uses analysis software such as OpenCV and TensorFlow to identify specific locations in the scene from the video data and confirm the corresponding location information.

[0623] (Output) The analyzed scene data and position information are obtained.

[0624] Step 6: Generate the digital twin

[0625] (Explanation) The server generates digital twin data using dedicated algorithms and machine learning models.

[0626] (Input) Analyzed video data and location information.

[0627] (Data processing / calculation) The server generates a digital twin using 3D modeling software such as Blender based on the scene identification results and location information.

[0628] (Output) The generated digital twin data is obtained.

[0629] Step 7: Quality check and filtering

[0630] (Description) Quality check and filter the generated digital twin data.

[0631] (Input) Generated digital twin data.

[0632] (Data processing / calculation) The server checks the quality of the generated data, filters out inappropriate data, and in some cases corrects it.

[0633] (Output) Quality-confirmed digital twin data is obtained.

[0634] Step 8: Save your data

[0635] (Explanation) The server stores the quality-confirmed digital twin data in a database.

[0636] (Input) Quality-confirmed digital twin data.

[0637] (Data processing / calculation) The server inserts the digital twin data into a database system such as SQL Server or MySQL.

[0638] (Output) Digital twin data is stored in a database.

[0639] Step 9: Calculating rewards

[0640] (Explanation) After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user and calculates the reward amount.

[0641] (Input) Provided digital twin data and user information.

[0642] (Data processing / calculation) The server scores the data based on its quantity and quality and calculates the reward amount.

[0643] (Output) The calculated reward amount is obtained.

[0644] Step 10: Send your rewards

[0645] (Explanation) The server obtains the user's electronic payment account information and transfers the calculated reward amount.

[0646] (Input) Calculated reward amount and user's electronic payment account information.

[0647] (Data processing / calculation) The server uses the PayPay API to transfer the reward amount to the user's account.

[0648] (Output) Rewards are transferred to the user's electronic payment account.

[0649] Step 11: Payment completion notification

[0650] (Explanation) When payment is completed, the server sends a payment completion notification to the user's terminal.

[0651] (Input) Information on completed remittance.

[0652] The (data processing / calculation) server generates a notification message and sends a push notification to the user's device.

[0653] (Output) A notification of payment completion is displayed on the user's terminal.

[0654] (Application example 1)

[0655] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0656] In conventional logistics centers, it is difficult to monitor abnormalities (e.g., missing pallets, obstructions) in real time, hindering efficient operation. Another issue is the difficulty of generating an accurate digital twin and understanding the situation within the logistics facility. Furthermore, there is a lack of a system that appropriately rewards users based on the quality and quantity of data they provide.

[0657] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0658] In this invention, the server includes means for a user to shoot video using a mobile device, means for acquiring location information in real time and tagging the video data, means for compressing the acquired video data and location data and uploading it to the server, means for analyzing the received video data and location data and generating digital twin data, means for monitoring abnormalities in the logistics facility in real time using the digital twin data, and means for calculating compensation based on the quantity and quality of data provided to the user and paying the compensation via a remittance means. This makes it possible to monitor the situation in the logistics facility in real time and manage it efficiently.

[0659] "User" means an individual or entity that uses a mobile device to capture video and provide the data, along with location information, to the server.

[0660] A "mobile device" is a portable electronic device capable of capturing video and acquiring location information.

[0661] "Video data" refers to video image information captured by a mobile device.

[0662] "Location information" refers to data that indicates a specific location or coordinates using GPS or other positioning means.

[0663] "Tagging" is the process of linking metadata such as location information to video data.

[0664] "Compression" is a technique for reducing data volume.

[0665] A "server" is a computer system that receives, stores, and analyzes data over a network.

[0666] "Digital twin data" is data that accurately recreates physical objects and spaces in the real world.

[0667] An "abnormal location" is a location within a logistics facility where an abnormal condition or problem is occurring.

[0668] "Real-time" refers to a situation in which processing and information updates occur almost immediately.

[0669] "Reward" refers to the payment made based on the data provided by the user.

[0670] "Remittance method" refers to the payment method or technology used to send rewards to users.

[0671] The system of this invention is realized by a user taking video of the inside of a logistics facility using a mobile device, tagging the video data with location information, and uploading it to a server. Details and specific examples of each means are provided below.

[0672] A user takes a video using a mobile device

[0673] Users use a mobile device such as a smartphone or tablet to capture video of a specific area within a logistics facility. This mobile device must have a camera, GPS functionality, and an internet connection. Users install a dedicated application and start recording using the app.

[0674] Real-time location information acquisition and tagging of video data

[0675] The device acquires location information in real time using GPS or other positioning methods while shooting video. The acquired location information is tagged as metadata to the video data being shot. This process is realized using libraries such as OpenCV and Geopy.

[0676] Compression of video data and location data and upload to server

[0677] Once the recording is complete, the device compresses the video and location data and uploads it to a server over its internet connection, for example by sending an HTTP POST request using the Requests library.

[0678] Data analysis on the server and generation of a digital twin

[0679] The server analyzes the received video and location data. During this analysis process, specialized algorithms and machine learning models are used to generate a digital twin of the logistics facility. For example, it identifies abnormalities (missing pallets, obstructions, etc.). This data is used for real-time monitoring of the logistics facility.

[0680] Calculation and payment of rewards based on quantity and quality of data provided

[0681] The server calculates the reward based on the quantity and quality of the data provided by the user and pays the user. The payment method is, for example, an electronic payment system. This information is notified to the user's terminal.

[0682] Specific examples

[0683] Users walk around the logistics facility with their smartphones and record video. As the device starts recording, it simultaneously acquires location information, which is tagged in real time. After filming is complete, the compressed video data and location data are uploaded to a server. The server analyzes the received data and generates a digital twin of the logistics facility. Through this process, users receive appropriate compensation for the data they provide.

[0684] Prompt Sentence Examples

[0685] Examples of prompts for a generative AI model might include:

[0686] "Please explain how a smartphone can be used in a logistics facility to capture video of a specific area, add location information in real time, and upload the data to a server."

[0687] The above steps enable efficient and highly accurate generation of digital twins and real-time monitoring of abnormalities within logistics facilities.

[0688] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0689] Step 1:

[0690] The user installs a dedicated application on the mobile device.

[0691] Input: Basic information such as user name, email address, and password

[0692] Process: The user launches the application and enters basic information. The device sends this information to the server, which then creates a new account in the database.

[0693] Output: Account verification message is displayed on the terminal.

[0694] Step 2:

[0695] The user launches the app and switches to video recording mode.

[0696] Input: User initiated video recording mode

[0697] Processing: The device activates the camera and GPS functions and prepares to capture video and obtain location information.

[0698] Output: The video recording mode screen will be displayed.

[0699] Step 3:

[0700] The user starts shooting video in a specific area within the logistics facility.

[0701] Input: User taps the "Start Recording" button

[0702] Processing: The device captures video in real time and simultaneously acquires GPS location information. Each captured video frame is tagged with location information.

[0703] Output: A video file and its corresponding geotag will be generated.

[0704] Step 4:

[0705] After the recording is complete, the device compresses the video data and location data.

[0706] Input: Video data and location data after shooting is complete

[0707] Processing: The device efficiently compresses video and location data to reduce data size.

[0708] Output: Compressed video data and location data

[0709] Step 5:

[0710] The compressed data is uploaded to the server.

[0711] Input: Compressed video data and location data

[0712] Processing: The device sends the data to the server over the Internet using an HTTP POST request.

[0713] Output: A message is displayed confirming successful data transmission.

[0714] Step 6:

[0715] The server analyzes the received data and generates a digital twin of the logistics facility.

[0716] Input: Video data and location data sent from the device

[0717] Processing: The server uses video analysis algorithms and machine learning models to identify anomalies within the facility and generate a digital twin.

[0718] Output: Digital twin data and abnormality location identification results

[0719] Step 7:

[0720] Rewards are calculated and paid based on the quantity and quality of data provided.

[0721] Input: The amount and quality of data analyzed by the server

[0722] Processing: The server calculates the user's reward amount using a reward calculation algorithm and makes the payment through an electronic payment system.

[0723] Output: A notification will be displayed on the device that the reward amount has been deposited into the user's account.

[0724] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0725] Installing the dedicated app and registering as a user

[0726] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[0727] After installation, launch the app and create an account by entering the required information such as your name, email address, and password.

[0728] The terminal sends this information to the server, which then creates a new account in its database.

[0729] Once the account is created, the server will send a confirmation message to the device.

[0730] Video recording, GPS information collection, and emotion recognition

[0731] The user launches the app and selects video recording mode.

[0732] Tap the "Start Recording" button at the shooting location to capture a specific area of ​​the city.

[0733] As soon as the device starts recording video, it acquires GPS location information in real time.

[0734] The GPS information acquired by the device while recording is tagged to the video data and saved as metadata.

[0735] While the device is recording video, the emotion engine recognizes the user's emotions in real time through facial recognition and voice analysis.

[0736] The recognized emotion data is also saved along with the video data.

[0737] Uploading data

[0738] The user ends video recording.

[0739] The device compresses the captured video data and emotion data.

[0740] The device uploads the compressed video data, GPS metadata, and emotion data to a server.

[0741] The server acknowledges receipt and prepares the data for storage.

[0742] Data analysis and digital twin generation

[0743] The server analyzes the received video data, GPS data, and emotion data.

[0744] During the analysis process, the server identifies specific locations in the scene from the video data.

[0745] The server checks the location information based on GPS data and compares it with the video data.

[0746] The server generates digital twin data for the urban area using dedicated algorithms and machine learning models, taking into account emotional data.

[0747] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[0748] Reward calculation and payment

[0749] After the digital twin data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data.

[0750] Based on the evaluation results, the amount of compensation for the user is calculated.

[0751] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[0752] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[0753] Specific examples

[0754] For example, consider the case where a user takes a video while walking around a tourist spot in a city with their smartphone. As soon as the device starts recording the video, it acquires GPS location information and tags the data in real time. While recording the video, the device uses an emotion engine to recognize emotions from the user's facial expressions and voice, and saves these as data. Once recording is complete, this data is automatically compressed and uploaded to the server. The server immediately analyzes the received data and creates a digital twin of the tourist spot. Once this process is complete, the server evaluates the user's contribution in providing data and the content of the emotional data, calculates a certain reward, and transfers it to the user's PayPay account. A notification is then displayed on the user's device indicating that the reward has been paid.

[0755] This concludes the description of an embodiment of the system of the present invention. The system allows users to easily provide data and provides a reward system for providing data, enabling efficient and highly accurate digital twin generation. Furthermore, adding emotional data can create richer digital twins.

[0756] The processing flow will be explained below.

[0757] Step 1: User Registration

[0758] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[0759] A user launches the app and enters the required information, such as their name, email address, and password.

[0760] The terminal transmits this registration information to the server.

[0761] The server stores the received registration information in a database and creates a new account.

[0762] The server sends a notification to the device that account creation is complete.

[0763] Step 2: Record video and collect GPS information

[0764] The user launches the app and selects video recording mode.

[0765] The user taps the "Start Recording" button to begin recording video.

[0766] As soon as the device starts recording video, it will acquire GPS location information in real time.

[0767] The GPS information acquired by the device in real time is tagged to the video data and saved as metadata.

[0768] Step 3: Emotion Recognition

[0769] While the device is recording video, it activates the emotion engine to recognize the user's face.

[0770] The device analyzes the user's facial expressions and voice and generates emotional data in real time.

[0771] The device stores the generated emotion data together with the video data and GPS data.

[0772] Step 4: Upload your data

[0773] The user ends video recording.

[0774] The device compresses the captured video data and emotion data.

[0775] The device uploads the compressed video data, GPS metadata, and emotion data to a server.

[0776] The server acknowledges receipt and prepares the data for storage.

[0777] Step 5: Analyze the data and create a digital twin

[0778] The server analyzes the received video data, GPS data, and emotion data.

[0779] During the analysis process, the server identifies specific locations in the scene from the video data.

[0780] The server checks the location information based on GPS data and compares it with the video data.

[0781] The server generates digital twin data that reflects the user's emotions based on the emotional data.

[0782] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[0783] Step 6: Calculating and paying rewards

[0784] After the digital twin data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data.

[0785] The server calculates the amount of reward for the user based on the evaluation result.

[0786] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[0787] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[0788] Step 7: Notification and confirmation

[0789] The terminal displays the remittance completion notice received to the user.

[0790] The user checks the reward amount in their PayPay account.

[0791] These are the specific processing steps of a system that combines an emotion engine. Users can easily provide data, and by utilizing emotion data, we can expect to improve the accuracy of digital twins.

[0792] Example 2

[0793] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0794] Conventional technologies have struggled to efficiently process user-generated video and location information and generate virtual environments based on that data. Furthermore, because data analysis, including user emotional data, is not performed, it is not possible to provide more realistic and detailed virtual environments. Furthermore, the process of calculating and paying rewards to users is complicated, and there is a lack of mechanisms to increase user motivation. Therefore, there is a need to improve the quantity and quality of data provided by users.

[0795] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0796] In this invention, the server includes means for recognizing emotion data from a user's facial expressions and voice in real time and integrating it into video data, means for using a machine learning model to analyze the video data including the emotion data, and means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via electronic remittance. This makes it possible to add emotion data to video data captured by the user and generate a highly accurate virtual environment. Furthermore, by promptly paying an appropriate reward to the user, it is possible to increase the user's motivation and encourage them to provide more and higher-quality data.

[0797] A "portable communication terminal" refers to a device that can be carried by a user and has a communication function, and specifically includes a smartphone, a tablet, a portable computer, and the like.

[0798] "Location information" is data indicating a geographical location, including, for example, latitude and longitude information obtained using a GPS (Global Positioning System).

[0799] "Central processing unit" refers to a central device for data processing, such as a server or cloud computing service that receives, analyzes, stores, and transmits data.

[0800] "Virtual environment data" refers to data such as digital twins and 3D models generated based on real physical spaces, which allows the real world to be reproduced digitally.

[0801] "Emotion data" is data that represents the emotional state of the user analyzed based on facial expressions, voice, and other biological information.

[0802] A "machine learning model" refers to software that has algorithms that analyze large amounts of data, find patterns and regularities, and make predictions and classifications based on them.

[0803] "Electronic remittance instruments" refers to systems and services for sending and receiving money over the Internet, and specifically includes digital wallets and bank online remittance systems.

[0804] MODE FOR CARRYING OUT THE INVENTION

[0805] In this invention, a system is constructed in which users collect data using mobile communication terminals, and the server analyzes and processes the data to generate virtual environment data, and also calculates and pays rewards to users.

[0806] Installing the dedicated app and registering as a user

[0807] Users must download and install a dedicated app from the Android or iOS app store. After installation, they launch the app and create an account by entering required information such as their name, email address, and password. The communication device sends this information to the server, which then creates a new account in its database. Once the account is created, the server sends a confirmation message to the communication device.

[0808] Video capture, location collection, and emotion recognition

[0809] The user launches the dedicated app and selects video recording mode. At the recording location, they tap the "Start Recording" button and record a specific area. As soon as the communication device starts recording video, it acquires GPS location information in real time. The communication device tags the location information acquired while recording with the video data and saves it as metadata. While the communication device is recording video, it uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions in real time through facial recognition and voice analysis. The recognized emotion data is also saved along with the video data.

[0810] Uploading data

[0811] When the user finishes recording a video, the communication device compresses the video data and emotion data. The compressed video data, GPS metadata, and emotion data are uploaded to the server. The server confirms receipt and prepares to store the data.

[0812] Data analysis and virtual environment data generation

[0813] The server analyzes the uploaded video data, location data, and emotion data. During the analysis process, the server identifies specific locations in the scene from the video data. The server matches the video data based on the location data. The server generates virtual environment data for the urban area using a dedicated algorithm or machine learning model (e.g., TensorFlow model), taking emotion data into consideration. The server checks the quality of the generated virtual environment data and filters out inappropriate data.

[0814] Reward calculation and payment

[0815] After the virtual environment data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data. Based on the evaluation results, the server calculates a reward amount for the user. The server then obtains the user's electronic payment account information and prepares to transfer the calculated reward amount. The server transfers the reward amount to the electronic payment account and sends a transfer completion notification to the communication terminal.

[0816] Specific examples

[0817] For example, consider the case where a user takes a video while walking around a tourist spot in a city with a smartphone. As the communication device starts recording the video, it simultaneously acquires location information and tags the data in real time. While recording the video, the communication device uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize emotions from the user's facial expressions and voice, and saves these as data. Once the recording is finished, this data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a virtual environment of the tourist spot. Once this process is complete, the server evaluates the user's contribution in providing data and the content of the emotional data, calculates a certain reward, and transfers it to an electronic payment account. A notification is then displayed on the user's device indicating that the reward has been paid.

[0818] Prompt Sentence Examples

[0819] "Please explain the process of generating a virtual environment using emotion data and location information when photographing tourist spots in a city."

[0820] "Please explain the reward system for taking videos using a smartphone app and uploading the data."

[0821] In this way, users can easily provide data and receive rewards for it, enabling efficient and highly accurate generation of virtual environments. Furthermore, adding emotional data can create richer virtual environments.

[0822] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0823] Step 1:

[0824] The user downloads and installs the dedicated app.

[0825] Input: App store search query, app download request.

[0826] Output: A dedicated app is installed on the communication device.

[0827] Specific operation: The user opens the Google Play Store or App Store, searches for "Digital Twin Creation App," and taps the download button. The app is automatically downloaded and installed on the communication device.

[0828] Step 2:

[0829] The user enters the required information to create an account.

[0830] Input: User information such as name, email address, and password.

[0831] Output: The user information is sent to the server and a new account is created in the database.

[0832] Specific operation: The user launches the app, enters the required information such as name, email address, and password, and taps the "Register" button. The entered data is sent from the communication device to the server as an HTTP POST request. The server creates a new account in the database based on the data received.

[0833] Step 3:

[0834] The terminal sends the user information to the server, and the server sends a confirmation message.

[0835] Input: Request to send user information.

[0836] Output: A server confirmation message is sent to the terminal.

[0837] Specific operation: The communication terminal sends user information to the server, and after the server verifies the received data, it sends a confirmation message in JSON format to the terminal. The terminal receives this message and displays the confirmation message to the user.

[0838] Step 4:

[0839] The user selects the video recording mode and starts recording.

[0840] Input: User action (selecting video recording mode, tapping the start recording button).

[0841] Output: Video recording and GPS information acquisition begins.

[0842] Specific operation: The user selects "Video recording mode" in the app and taps the "Start recording" button on the screen. The communication device activates the camera and starts recording video. At the same time, the GPS module begins obtaining location information in real time.

[0843] Step 5:

[0844] The device tags the video data with location information and also acquires emotional data.

[0845] Input: Video frames being captured, real-time GPS data, and user face and voice data.

[0846] Output: Video data tagged with location and emotion data.

[0847] How it works: The communication device acquires location data (latitude and longitude) for each video frame and tags it as metadata in the video data. In parallel, an emotion engine (e.g., Microsoft Azure Emotion API) analyzes the user's face and voice and recognizes emotion data in real time. This data is then integrated into the video data.

[0848] Step 6:

[0849] The user stops recording the video, and the device compresses the data.

[0850] Input: User action (tapping the stop recording button).

[0851] Output: Compressed video data, location data, and emotion data.

[0852] Specific operation: The user taps the "Stop Recording" button. The communication device compresses the captured video data, location data, and emotion data. For example, it compiles the data in ZIP format.

[0853] Step 7:

[0854] The device uploads the data to the server.

[0855] Input: Compressed data file.

[0856] Output: The data is uploaded to the server.

[0857] Specific operation: The communication device uploads the compressed data file to the server using an HTTP POST request. The progress is displayed during the upload.

[0858] Step 8:

[0859] The server checks the received data.

[0860] Input: The uploaded data file.

[0861] Output: Data integrity verification message.

[0862] Specific operation: The server checks the received data. Specifically, it checks the integrity of the data and ensures consistency. Once the data is confirmed, the server sends a response message to the communication terminal.

[0863] Step 9:

[0864] The server analyzes the data and generates virtual environment data.

[0865] Input: Uploaded video data, location data, emotion data.

[0866] Output: Parsed data, generated virtual environment data.

[0867] How it works: The server uses Python scripts and TensorFlow models to analyze the received video data, location data, and emotion data. It identifies specific locations in the scene from the video data and matches them with the location data. It generates virtual environment data for the urban area using a specific algorithm, taking emotion data into account. The server checks the quality of the generated virtual environment data and filters out inappropriate data.

[0868] Step 10:

[0869] The server evaluates the user's data and calculates the reward.

[0870] Input: Quantity and quality of analyzed data, user-provided data.

[0871] Output: Calculated reward amount.

[0872] How it works: The server evaluates the quantity and quality of the data provided by users in the database. Evaluation criteria include the clarity of the data and the reliability of the sentiment data. Based on the evaluation results, a Python script calculates the reward amount.

[0873] Step 11:

[0874] The server transfers the reward to the user's electronic payment account.

[0875] Input: User's electronic payment account information, calculated reward amount.

[0876] Output: Remittance completion notification.

[0877] Specific operation: The server obtains the user's electronic payment account information and transfers the reward amount via API. Once the transfer is complete, a push notification is sent to the device.

[0878] (Application example 2)

[0879] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0880] Conventional digital twin generation systems only use video data and GPS data, making it difficult to accurately and realistically reflect user experiences and emotions. Furthermore, the calculation of rewards for users' data provision is based on limited indicators, resulting in issues of fairness and inaccuracy. Furthermore, the analysis of collected data and the generation of digital twins often relies on specific algorithms, preventing the use of advanced generative AI models.

[0881] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0882] In this invention, the server includes: means for allowing a user to shoot video using a mobile device, acquiring GPS location information in real time, and tagging the video data; means for acquiring the user's emotional data in real time using an emotion recognition engine while shooting the video, and tagging the video data; means for compressing the acquired video data, GPS data, and emotional data and uploading them to the server; means for analyzing the received video data, GPS data, and emotional data in the server and generating digital twin data using a generative AI model; and means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via an electronic payment method. This enables the creation of more realistic and rich digital twins that reflect the user's experiences and emotions, and achieves fair and accurate reward calculations.

[0883] A "mobile device" is an electronic device that a user can carry around, such as a smartphone or tablet.

[0884] "Means for capturing video" refers to a method or apparatus for recording video using the camera function on a mobile device.

[0885] "GPS location information" refers to information that uses the Global Positioning System to obtain real-time location data about a specific place or location.

[0886] "Tagging" refers to the operation of adding identifying information to data, and in this case means adding GPS location information and emotional data to video data.

[0887] An "emotion recognition engine" refers to software or algorithms that analyze a user's emotions in real time from their facial expressions, tone of voice, etc.

[0888] "Means for compressing and uploading to a server" refers to technology for converting the captured video data, GPS data, and emotion data into a small data format and transmitting it to a server via the Internet.

[0889] A "generative AI model" refers to a program or algorithm that uses artificial intelligence technology to analyze and synthesize complex data.

[0890] "Digital twin data" refers to a digital replica of a physical space or object, and in this case refers to data that recreates an urban area in a virtual space.

[0891] "Electronic payment means" refers to a method or system for electronically transferring rewards to users, and generally includes electronic money and digital wallets.

[0892] "Reward Calculation" refers to the process of calculating rewards to users based on the quantity and quality of data provided.

[0893] System Overview

[0894] This invention is a system that allows users to shoot videos using a mobile device and tag the videos with GPS location information and emotion data acquired in real time. This data is then compressed and uploaded to a server. The server then analyzes the received video data, GPS data, and emotion data and generates digital twin data using a generative AI model. The system also calculates a reward for the user based on the quantity and quality of the data provided and pays the reward via electronic payment methods.

[0895] Hardware and software used

[0896] Mobile devices (e.g. smartphones, tablets)

[0897] A device that can be carried by the user and has video recording, GPS, and emotion recognition functions.

[0898] Camera function (software)

[0899] Software for capturing video.

[0900] GPS module (hardware)

[0901] Module for obtaining geographical location information in real time.

[0902] Emotion recognition engine (software)

[0903] Software that analyzes facial expressions and tone of voice to recognize emotions.

[0904] Compression algorithm (software)

[0905] An algorithm for efficiently compressing captured data.

[0906] Communication module (hardware)

[0907] A module for sending data to a server over the Internet.

[0908] Server (hardware)

[0909] A device for analyzing the received data and generating a digital twin using a generative AI model.

[0910] Generative AI model (software)

[0911] Artificial intelligence techniques for analyzing and synthesizing complex data.

[0912] Electronic payment system (software)

[0913] A system for electronically transferring rewards to users (e.g., electronic money, digital wallets).

[0914] Example of operation

[0915] The user launches the dedicated app and starts recording video. As the user walks around, the mobile device's camera captures the video data, and the GPS module acquires location information in real time. At the same time, the emotion recognition engine extracts emotional data from the user's facial expressions and voice, and tags this data into the video. Once recording is complete, the device's on-device compression algorithm compresses the data and uploads it to the server via the communication module.

[0916] When the data arrives at the server, the server first acknowledges receipt of the data and then begins analysis. The received video data, GPS data, and emotion data are analyzed using a generative AI model to generate digital twin data. This digital twin data is rich in content, reflecting the user's emotions and location information. Based on the results of this analysis, the server calculates a reward for the user based on the quantity and quality of the data provided. The calculated reward is paid to the user using an electronic payment system.

[0917] Prompt Sentence Examples

[0918] "Generate a Python program for a smartphone app that allows users to take videos while walking around a specific area of ​​a city and tag them with emotional data in real time. Include how to upload the data to a server and implement a reward system."

[0919] In order to implement the present invention, it is important to combine these hardware and software to provide a user-friendly interface.

[0920] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0921] Program processing steps

[0922] Step 1:

[0923] The user launches the dedicated app and selects video recording mode.

[0924] Input: User's finger action (tap).

[0925] Output: Video recording preparation screen.

[0926] Specific operation: Through the mobile device interface, the user taps the app icon and selects "Video Recording" from the menu to activate the recording mode.

[0927] Step 2:

[0928] The user taps the "Start Recording" button at the shooting location to capture a specific area.

[0929] Input: User's finger action (tap).

[0930] Output: Recording screen, real-time video data.

[0931] Specific operation: The camera starts capturing video, and at the same time, the device continues to generate video data in real time.

[0932] Step 3:

[0933] As soon as the device starts recording video, it acquires GPS location information in real time.

[0934] Input: GPS sensor on mobile device.

[0935] Output: Real-time GPS location information.

[0936] Specific operation: The GPS module is activated and current coordinate information is captured in real time, so that location information is added to the video data at any time.

[0937] Step 4:

[0938] The terminal acquires the user's emotion data in real time using an emotion recognition engine.

[0939] Input: The user's facial expressions and voice.

[0940] Output: Recognized emotion data (e.g., joy, sadness, surprise, etc.).

[0941] How it works: Video and audio data captured by the camera is passed to the emotion recognition engine, where it is analyzed. As a result, an emotion tag for the user is generated.

[0942] Step 5:

[0943] The device compresses the acquired video data, GPS data, and emotion data and uploads them to a server.

[0944] Input: Video data, GPS data, emotion data.

[0945] Output: Compressed data, notification of completion of transmission to the server.

[0946] Specific operation: The compression algorithm compresses the data, and the communication module transmits the compressed data to the server over the Internet.

[0947] Step 6:

[0948] The server acknowledges the received data and then begins analyzing it.

[0949] Input: Compressed video data, GPS data, emotion data.

[0950] Output: Analyzed data, notification that the digital twin is ready to be generated.

[0951] What happens: The server decompresses the data and prepares the individual data streams as base data for analysis, so that each data stream can be split appropriately and analyzed.

[0952] Step 7:

[0953] The server generates digital twin data using the generative AI model.

[0954] Input: Analyzed video data, GPS data, emotion data.

[0955] Output: Digital twin data.

[0956] How it works: By inputting the analysis results into a generative AI model, complex data analysis and synthesis processes are performed to generate a digital twin of the urban area.

[0957] Step 8:

[0958] The server calculates a reward based on the quantity and quality of the data provided and pays the reward to the user via an electronic payment means.

[0959] Input: Data quantity and quality assessment results.

[0960] Output: Calculated reward amount, notification of transfer completion.

[0961] Specific operation: The server's evaluation algorithm calculates the reward amount based on the quantity and quality of data provided, the usefulness of the emotional data, etc., and transfers it to the user's account through an electronic payment system.

[0962] This makes it possible to create a more realistic and rich digital twin that reflects the user's experiences and emotions, resulting in fair and accurate reward calculations.

[0963] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0964] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0965] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0966] [Third embodiment]

[0967] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0968] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0969] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0970] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0971] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0972] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0973] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0974] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0975] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0976] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0977] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0978] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0979] Installing the dedicated app and registering as a user

[0980] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[0981] After installation, launch the app and create an account by entering the required information such as your name, email address, and password.

[0982] The terminal sends this information to the server, which then creates a new account in its database.

[0983] Once the account is created, the server will send a confirmation message to the device.

[0984] Data collection and upload

[0985] The user launches the app and switches to video recording mode.

[0986] Tap the "Start Recording" button at the shooting location to capture a specific area of ​​the city.

[0987] As soon as the device starts recording video, it acquires GPS location information in real time.

[0988] While recording, the device tags the video data with GPS location information and saves it as metadata in the video file.

[0989] Once the recording is complete, the device compresses the video data and uploads it to the server.

[0990] Data analysis and digital twin generation

[0991] The server analyzes the received video data and GPS data.

[0992] The analysis process involves identifying specific locations in the scene from the video data and confirming location information based on GPS data.

[0993] The server generates digital twin data using specialized algorithms and machine learning models.

[0994] The generated digital twin data is quality checked and any necessary filtering processes are performed.

[0995] The digital twin data is then stored in a database.

[0996] Reward calculation and payment

[0997] After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user.

[0998] Based on the evaluation results, the amount of compensation for the user is calculated.

[0999] The server obtains the user's PayPay account information and transfers the calculated reward amount.

[1000] When the payment is completed, the server sends a payment completion notification to the user's terminal.

[1001] Specific examples

[1002] For example, consider the case where a user takes a video while walking around a city's tourist attractions with their smartphone. As soon as the device starts recording the video, it acquires location information and tags the data in real time. When the video recording is finished, the data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a digital twin of the tourist attraction. Once this process is complete, the server evaluates the user's contribution to the data provision, calculates a certain reward, and transfers it to the user's PayPay account. The user's device then displays a notification that the reward has been paid.

[1003] This concludes the description of an embodiment of the system of the present invention. The system allows users to easily provide data and provides a reward system for providing data, enabling efficient and highly accurate digital twin generation.

[1004] The processing flow will be explained below.

[1005] Step 1: User Registration

[1006] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[1007] A user launches the app and enters the required information, such as their name, email address, and password.

[1008] The terminal transmits this registration information to the server.

[1009] The server stores the received registration information in a database and creates a new account.

[1010] The server sends a notification to the device that account creation is complete.

[1011] Step 2: Record video and collect GPS information

[1012] The user launches the app and selects video recording mode.

[1013] The user taps the "Start Recording" button to begin recording video.

[1014] As soon as the device starts recording video, it will obtain GPS location information in real time.

[1015] The GPS information acquired by the device while recording is tagged to the video data and saved as metadata.

[1016] Step 3: Upload your data

[1017] The user ends video recording.

[1018] The device compresses the captured video data.

[1019] The device uploads the compressed video data and associated GPS metadata to a server.

[1020] The server acknowledges receipt and prepares the data for storage.

[1021] Step 4: Analyze the data and create a digital twin

[1022] The server analyzes the received video data and GPS data.

[1023] During the analysis process, the server identifies specific locations in the scene from the video data.

[1024] The server checks the location information based on GPS data and compares it with the video data.

[1025] The server uses specialized algorithms and machine learning models to generate digital twin data for the urban area.

[1026] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[1027] Step 5: Calculating and paying rewards

[1028] The server evaluates the quantity and quality of the data provided by the user.

[1029] The server calculates the amount of reward for the user based on the evaluation result.

[1030] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[1031] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[1032] Step 6: Notification and confirmation

[1033] The terminal displays the remittance completion notice received to the user.

[1034] The user checks the reward amount in their PayPay account.

[1035] The above are the specific steps of the program process, which allows users to easily provide data and receive rewards.

[1036] Example 1

[1037] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1038] Conventional digital twin generation systems have struggled to efficiently collect large amounts of data and generate highly accurate digital twins. Furthermore, users lacked incentives to provide data, reducing the operational efficiency of the system. Furthermore, the time and effort required for data quality checking and filtering was also an issue.

[1039] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1040] In this invention, the server includes: a means for a user to shoot video using a mobile terminal; a means for acquiring global positioning system (GPS) location information corresponding to the video in real time and tagging the video data; a means for compressing the acquired video data and GPS data and uploading it to the server; a means for analyzing the received video data and GPS data in the server and generating digital twin data; a means for quality checking and filtering the generated digital twin data; and a means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via a remittance means. This enables more efficient and accurate digital twin generation. It also increases the incentive for users to provide data, improving the operational efficiency of the entire system.

[1041] "User" refers to an individual or legal entity that provides data using a mobile terminal.

[1042] "Mobile device" refers to a portable computing device such as a smartphone or tablet.

[1043] "Means for capturing video" refers to a method for recording video data using a camera mounted on a mobile terminal.

[1044] "Global Positioning System location information" refers to terrestrial location data obtained using the Global Positioning System.

[1045] "Means of obtaining and tagging video data in real time" refers to a method of obtaining global positioning system location information immediately when a video is shot and adding that location information as metadata to the video data.

[1046] "Means for compressing and uploading the captured video data and global positioning system data to a server" refers to a method for compressing the captured video data and the captured global positioning system location information to reduce their size and transmitting them to a server over a network.

[1047] "Server" refers to the computer system used to analyze and store video data and global positioning system data.

[1048] "Digital twin data" refers to data that digitally reproduces a real-world object.

[1049] "Means for analyzing and generating digital twin data" refers to a method for using received video data and global positioning system data to create a digital model of a real-world object.

[1050] "Quality check and filtering measures" refers to methods for checking the quality of generated digital twin data and removing or correcting inappropriate data or errors.

[1051] "Means for calculating remuneration based on the quantity and quality of data provided and paying it through a remittance means" refers to a method for calculating remuneration based on the quantity and quality of data provided by a user and paying the remuneration to the user through an electronic payment system.

[1052] This invention includes a system that uses a mobile terminal to shoot video, tags the video with Global Positioning System (GPS) location information, uploads the video to a server, generates digital twin data, and finally pays rewards to users. Each step is described in detail below.

[1053] Installing the dedicated app and registering as a user

[1054] The user downloads and installs the dedicated app from the Google Play Store or App Store. After installation, the user launches the app and creates an account by entering required information such as name, email address, and password. The user then sends the account information to the server via their device, and the server creates a new account in its database. Once the account is created, the server sends a confirmation message to the device.

[1055] Data collection and upload

[1056] The user launches the app and switches to video recording mode. At the recording location, they tap the "Start Recording" button and record a specific area of ​​the city. The device starts recording video and simultaneously acquires GPS location information in real time. While recording, the device tags the video data with GPS location information and saves it as metadata in the video file. Once recording is complete, the device compresses the video data and uploads it to the server.

[1057] Data analysis and digital twin generation

[1058] The server analyzes the received video data and GPS data. During the analysis process, specific locations in the scene are identified from the video data and location information is confirmed based on the GPS data. The server generates digital twin data using dedicated algorithms and machine learning models (e.g., OpenCV and TensorFlow). The generated digital twin data is then quality checked and any necessary filtering is performed. Finally, the digital twin data is stored in a database.

[1059] Reward calculation and payment

[1060] After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user. Based on the evaluation results, it calculates the amount of compensation for the user. The server obtains the user's electronic payment account information and transfers the calculated amount of compensation. Once the payment is complete, the server sends a payment completion notification to the user's device.

[1061] Specific examples

[1062] For example, if a user takes a video while walking around a city's tourist attractions with their smartphone, the device starts recording the video and simultaneously acquires location information, tagging the data in real time. Once the video recording is finished, the data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a digital twin of the tourist attraction. Once this process is complete, the server evaluates the user's contribution to the data provision, calculates a certain reward, and transfers it to an electronic payment account. The user's device then displays a notification that the reward has been paid.

[1063] Prompt Sentence Examples

[1064] Please tell me the procedure for creating a new account.

[1065] "Please explain how to tag video data with GPS location information in real time."

[1066] "What kind of data do I need to generate a digital twin and how do I analyze it?"

[1067] This system allows users to easily provide data and receive compensation for that data, making it possible to generate digital twins efficiently and with high accuracy.

[1068] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1069] Step 1: Install the dedicated app

[1070] (Explanation) The user downloads and installs the dedicated app from the Google Play Store or App Store.

[1071] (Input) A user accesses the Google Play Store or App Store.

[1072] (Data processing / calculation) Search for dedicated apps through the store's search function and download the apps you find.

[1073] (Output) A dedicated app is installed on the user's device.

[1074] Step 2: Create an account

[1075] (Description) The user launches the installed app and creates an account by entering required information such as name, email address, and password.

[1076] (Input) The user enters required information into the app, such as their name, email address, and password.

[1077] (Data processing / calculation) The terminal formats the user's input data to send to the server and generates an HTTP request.

[1078] (Output) The server creates the new account in the database and sends a confirmation message to the terminal.

[1079] Step 3: Start recording

[1080] (Explanation) The user launches the app, switches to video recording mode, and taps the "Start Recording" button at the recording location.

[1081] (Input) The user taps the "Start Recording" button.

[1082] (Data processing / calculation) The device starts recording video and simultaneously obtains location information in real time from GPS.

[1083] (Output) The device tags the video data and acquired GPS information in real time and saves it as metadata.

[1084] Step 4: Compress and upload your video data

[1085] (Explanation) Once video recording is complete, the device compresses the video data and uploads it to the server.

[1086] (Input) Recorded video data and GPS location information.

[1087] (Data processing / calculation) The terminal compresses the video data using a codec such as H.264 and sends it to the server along with the location information via an HTTP POST request.

[1088] (Output) Compressed video data and GPS data are saved on the server.

[1089] Step 5: Analyze the data

[1090] (Description) The server analyzes the video data and GPS data received.

[1091] (Input) Compressed video data and GPS data received by the server.

[1092] (Data processing / calculation) The server uses analysis software such as OpenCV and TensorFlow to identify specific locations in the scene from the video data and confirm the corresponding location information.

[1093] (Output) The analyzed scene data and position information are obtained.

[1094] Step 6: Generate the digital twin

[1095] (Explanation) The server generates digital twin data using dedicated algorithms and machine learning models.

[1096] (Input) Analyzed video data and location information.

[1097] (Data processing / calculation) The server generates a digital twin using 3D modeling software such as Blender based on the scene identification results and location information.

[1098] (Output) The generated digital twin data is obtained.

[1099] Step 7: Quality check and filtering

[1100] (Description) Quality check and filter the generated digital twin data.

[1101] (Input) Generated digital twin data.

[1102] (Data processing / calculation) The server checks the quality of the generated data, filters out inappropriate data, and in some cases corrects it.

[1103] (Output) Quality-confirmed digital twin data is obtained.

[1104] Step 8: Save your data

[1105] (Explanation) The server stores the quality-confirmed digital twin data in a database.

[1106] (Input) Quality-confirmed digital twin data.

[1107] (Data processing / calculation) The server inserts the digital twin data into a database system such as SQL Server or MySQL.

[1108] (Output) Digital twin data is stored in a database.

[1109] Step 9: Calculating rewards

[1110] (Explanation) After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user and calculates the reward amount.

[1111] (Input) Provided digital twin data and user information.

[1112] (Data processing / calculation) The server scores the data based on its quantity and quality and calculates the reward amount.

[1113] (Output) The calculated reward amount is obtained.

[1114] Step 10: Send your rewards

[1115] (Explanation) The server obtains the user's electronic payment account information and transfers the calculated reward amount.

[1116] (Input) Calculated reward amount and user's electronic payment account information.

[1117] (Data processing / calculation) The server uses the PayPay API to transfer the reward amount to the user's account.

[1118] (Output) Rewards are transferred to the user's electronic payment account.

[1119] Step 11: Payment completion notification

[1120] (Explanation) When payment is completed, the server sends a payment completion notification to the user's terminal.

[1121] (Input) Information on completed remittance.

[1122] The (data processing / calculation) server generates a notification message and sends a push notification to the user's device.

[1123] (Output) A notification of payment completion is displayed on the user's terminal.

[1124] (Application example 1)

[1125] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1126] In conventional logistics centers, it is difficult to monitor abnormalities (e.g., missing pallets, obstructions) in real time, hindering efficient operation. Another issue is the difficulty of generating an accurate digital twin and understanding the situation within the logistics facility. Furthermore, there is a lack of a system that appropriately rewards users based on the quality and quantity of data they provide.

[1127] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1128] In this invention, the server includes means for a user to shoot video using a mobile device, means for acquiring location information in real time and tagging the video data, means for compressing the acquired video data and location data and uploading it to the server, means for analyzing the received video data and location data and generating digital twin data, means for monitoring abnormalities in the logistics facility in real time using the digital twin data, and means for calculating compensation based on the quantity and quality of data provided to the user and paying the compensation via a remittance means. This makes it possible to monitor the situation in the logistics facility in real time and manage it efficiently.

[1129] "User" means an individual or entity that uses a mobile device to capture video and provide the data, along with location information, to the server.

[1130] A "mobile device" is a portable electronic device capable of capturing video and acquiring location information.

[1131] "Video data" refers to video image information captured by a mobile device.

[1132] "Location information" refers to data that indicates a specific location or coordinates using GPS or other positioning means.

[1133] "Tagging" is the process of linking metadata such as location information to video data.

[1134] "Compression" is a technique for reducing data volume.

[1135] A "server" is a computer system that receives, stores, and analyzes data over a network.

[1136] "Digital twin data" is data that accurately recreates physical objects and spaces in the real world.

[1137] An "abnormal location" is a location within a logistics facility where an abnormal condition or problem is occurring.

[1138] "Real-time" refers to a situation in which processing and information updates occur almost immediately.

[1139] "Reward" refers to the payment made based on the data provided by the user.

[1140] "Remittance method" refers to the payment method or technology used to send rewards to users.

[1141] The system of this invention is realized by a user taking video of the inside of a logistics facility using a mobile device, tagging the video data with location information, and uploading it to a server. Details and specific examples of each means are provided below.

[1142] A user takes a video using a mobile device

[1143] Users use a mobile device such as a smartphone or tablet to capture video of a specific area within a logistics facility. This mobile device must have a camera, GPS functionality, and an internet connection. Users install a dedicated application and start recording using the app.

[1144] Real-time location information acquisition and tagging of video data

[1145] The device acquires location information in real time using GPS or other positioning methods while shooting video. The acquired location information is tagged as metadata to the video data being shot. This process is realized using libraries such as OpenCV and Geopy.

[1146] Compression of video data and location data and upload to server

[1147] Once the recording is complete, the device compresses the video and location data and uploads it to a server over its internet connection, for example by sending an HTTP POST request using the Requests library.

[1148] Data analysis on the server and generation of a digital twin

[1149] The server analyzes the received video and location data. During this analysis process, specialized algorithms and machine learning models are used to generate a digital twin of the logistics facility. For example, it identifies abnormalities (missing pallets, obstructions, etc.). This data is used for real-time monitoring of the logistics facility.

[1150] Calculation and payment of rewards based on quantity and quality of data provided

[1151] The server calculates the reward based on the quantity and quality of the data provided by the user and pays the user. The payment method is, for example, an electronic payment system. This information is notified to the user's terminal.

[1152] Specific examples

[1153] Users walk around the logistics facility with their smartphones and record video. As the device starts recording, it simultaneously acquires location information, which is tagged in real time. After filming is complete, the compressed video data and location data are uploaded to a server. The server analyzes the received data and generates a digital twin of the logistics facility. Through this process, users receive appropriate compensation for the data they provide.

[1154] Prompt Sentence Examples

[1155] Examples of prompts for a generative AI model might include:

[1156] "Please explain how a smartphone can be used in a logistics facility to capture video of a specific area, add location information in real time, and upload the data to a server."

[1157] The above steps enable efficient and highly accurate generation of digital twins and real-time monitoring of abnormalities within logistics facilities.

[1158] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1159] Step 1:

[1160] The user installs a dedicated application on the mobile device.

[1161] Input: Basic information such as user name, email address, and password

[1162] Process: The user launches the application and enters basic information. The device sends this information to the server, which then creates a new account in the database.

[1163] Output: Account verification message is displayed on the terminal.

[1164] Step 2:

[1165] The user launches the app and switches to video recording mode.

[1166] Input: User initiated video recording mode

[1167] Processing: The device activates the camera and GPS functions and prepares to capture video and obtain location information.

[1168] Output: The video recording mode screen will be displayed.

[1169] Step 3:

[1170] The user starts shooting video in a specific area within the logistics facility.

[1171] Input: User taps the "Start Recording" button

[1172] Processing: The device captures video in real time and simultaneously acquires GPS location information. Each captured video frame is tagged with location information.

[1173] Output: A video file and its corresponding geotag will be generated.

[1174] Step 4:

[1175] After the recording is complete, the device compresses the video data and location data.

[1176] Input: Video data and location data after shooting is complete

[1177] Processing: The device efficiently compresses video and location data to reduce data size.

[1178] Output: Compressed video data and location data

[1179] Step 5:

[1180] The compressed data is uploaded to the server.

[1181] Input: Compressed video data and location data

[1182] Processing: The device sends the data to the server over the Internet using an HTTP POST request.

[1183] Output: A message is displayed confirming successful data transmission.

[1184] Step 6:

[1185] The server analyzes the received data and generates a digital twin of the logistics facility.

[1186] Input: Video data and location data sent from the device

[1187] Processing: The server uses video analysis algorithms and machine learning models to identify anomalies within the facility and generate a digital twin.

[1188] Output: Digital twin data and abnormality location identification results

[1189] Step 7:

[1190] Rewards are calculated and paid based on the quantity and quality of data provided.

[1191] Input: The amount and quality of data analyzed by the server

[1192] Processing: The server calculates the user's reward amount using a reward calculation algorithm and makes the payment through an electronic payment system.

[1193] Output: A notification will be displayed on the device that the reward amount has been deposited into the user's account.

[1194] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1195] Installing the dedicated app and registering as a user

[1196] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[1197] After installation, launch the app and create an account by entering the required information such as your name, email address, and password.

[1198] The terminal sends this information to the server, which then creates a new account in its database.

[1199] Once the account is created, the server will send a confirmation message to the device.

[1200] Video recording, GPS information collection, and emotion recognition

[1201] The user launches the app and selects video recording mode.

[1202] Tap the "Start Recording" button at the shooting location to capture a specific area of ​​the city.

[1203] As soon as the device starts recording video, it acquires GPS location information in real time.

[1204] The GPS information acquired by the device while recording is tagged to the video data and saved as metadata.

[1205] While the device is recording video, the emotion engine recognizes the user's emotions in real time through facial recognition and voice analysis.

[1206] The recognized emotion data is also saved along with the video data.

[1207] Uploading data

[1208] The user ends video recording.

[1209] The device compresses the captured video data and emotion data.

[1210] The device uploads the compressed video data, GPS metadata, and emotion data to a server.

[1211] The server acknowledges receipt and prepares the data for storage.

[1212] Data analysis and digital twin generation

[1213] The server analyzes the received video data, GPS data, and emotion data.

[1214] During the analysis process, the server identifies specific locations in the scene from the video data.

[1215] The server checks the location information based on GPS data and compares it with the video data.

[1216] The server generates digital twin data for the urban area using dedicated algorithms and machine learning models, taking into account emotional data.

[1217] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[1218] Reward calculation and payment

[1219] After the digital twin data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data.

[1220] Based on the evaluation results, the amount of compensation for the user is calculated.

[1221] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[1222] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[1223] Specific examples

[1224] For example, consider the case where a user takes a video while walking around a tourist spot in a city with their smartphone. As soon as the device starts recording the video, it acquires GPS location information and tags the data in real time. While recording the video, the device uses an emotion engine to recognize emotions from the user's facial expressions and voice, and saves these as data. Once recording is complete, this data is automatically compressed and uploaded to the server. The server immediately analyzes the received data and creates a digital twin of the tourist spot. Once this process is complete, the server evaluates the user's contribution in providing data and the content of the emotional data, calculates a certain reward, and transfers it to the user's PayPay account. A notification is then displayed on the user's device indicating that the reward has been paid.

[1225] This concludes the description of an embodiment of the system of the present invention. The system allows users to easily provide data and provides a reward system for providing data, enabling efficient and highly accurate digital twin generation. Furthermore, adding emotional data can create richer digital twins.

[1226] The processing flow will be explained below.

[1227] Step 1: User Registration

[1228] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[1229] A user launches the app and enters the required information, such as their name, email address, and password.

[1230] The terminal transmits this registration information to the server.

[1231] The server stores the received registration information in a database and creates a new account.

[1232] The server sends a notification to the device that account creation is complete.

[1233] Step 2: Record video and collect GPS information

[1234] The user launches the app and selects video recording mode.

[1235] The user taps the "Start Recording" button to begin recording video.

[1236] As soon as the device starts recording video, it will acquire GPS location information in real time.

[1237] The GPS information acquired by the device in real time is tagged to the video data and saved as metadata.

[1238] Step 3: Emotion Recognition

[1239] While the device is recording video, it activates the emotion engine to recognize the user's face.

[1240] The device analyzes the user's facial expressions and voice and generates emotional data in real time.

[1241] The device stores the generated emotion data together with the video data and GPS data.

[1242] Step 4: Upload your data

[1243] The user ends video recording.

[1244] The device compresses the captured video data and emotion data.

[1245] The device uploads the compressed video data, GPS metadata, and emotion data to a server.

[1246] The server acknowledges receipt and prepares the data for storage.

[1247] Step 5: Analyze the data and create a digital twin

[1248] The server analyzes the received video data, GPS data, and emotion data.

[1249] During the analysis process, the server identifies specific locations in the scene from the video data.

[1250] The server checks the location information based on GPS data and compares it with the video data.

[1251] The server generates digital twin data that reflects the user's emotions based on the emotional data.

[1252] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[1253] Step 6: Calculating and paying rewards

[1254] After the digital twin data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data.

[1255] The server calculates the amount of reward for the user based on the evaluation result.

[1256] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[1257] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[1258] Step 7: Notification and confirmation

[1259] The terminal displays the remittance completion notice received to the user.

[1260] The user checks the reward amount in their PayPay account.

[1261] These are the specific processing steps of a system that combines an emotion engine. Users can easily provide data, and by utilizing emotion data, we can expect to improve the accuracy of digital twins.

[1262] Example 2

[1263] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1264] Conventional technologies have struggled to efficiently process user-generated video and location information and generate virtual environments based on that data. Furthermore, because data analysis, including user emotional data, is not performed, it is not possible to provide more realistic and detailed virtual environments. Furthermore, the process of calculating and paying rewards to users is complicated, and there is a lack of mechanisms to increase user motivation. Therefore, there is a need to improve the quantity and quality of data provided by users.

[1265] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1266] In this invention, the server includes means for recognizing emotion data from a user's facial expressions and voice in real time and integrating it into video data, means for using a machine learning model to analyze the video data including the emotion data, and means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via electronic remittance. This makes it possible to add emotion data to video data captured by the user and generate a highly accurate virtual environment. Furthermore, by promptly paying an appropriate reward to the user, it is possible to increase the user's motivation and encourage them to provide more and higher-quality data.

[1267] A "portable communication terminal" refers to a device that can be carried by a user and has a communication function, and specifically includes a smartphone, a tablet, a portable computer, and the like.

[1268] "Location information" is data indicating a geographical location, including, for example, latitude and longitude information obtained using a GPS (Global Positioning System).

[1269] "Central processing unit" refers to a central device for data processing, such as a server or cloud computing service that receives, analyzes, stores, and transmits data.

[1270] "Virtual environment data" refers to data such as digital twins and 3D models generated based on real physical spaces, which allows the real world to be reproduced digitally.

[1271] "Emotion data" is data that represents the emotional state of the user analyzed based on facial expressions, voice, and other biological information.

[1272] A "machine learning model" refers to software that has algorithms that analyze large amounts of data, find patterns and regularities, and make predictions and classifications based on them.

[1273] "Electronic remittance instruments" refers to systems and services for sending and receiving money over the Internet, and specifically includes digital wallets and bank online remittance systems.

[1274] MODE FOR CARRYING OUT THE INVENTION

[1275] In this invention, a system is constructed in which users collect data using mobile communication terminals, and the server analyzes and processes the data to generate virtual environment data, and also calculates and pays rewards to users.

[1276] Installing the dedicated app and registering as a user

[1277] Users must download and install a dedicated app from the Android or iOS app store. After installation, they launch the app and create an account by entering required information such as their name, email address, and password. The communication device sends this information to the server, which then creates a new account in its database. Once the account is created, the server sends a confirmation message to the communication device.

[1278] Video capture, location collection, and emotion recognition

[1279] The user launches the dedicated app and selects video recording mode. At the recording location, they tap the "Start Recording" button and record a specific area. As soon as the communication device starts recording video, it acquires GPS location information in real time. The communication device tags the location information acquired while recording with the video data and saves it as metadata. While the communication device is recording video, it uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions in real time through facial recognition and voice analysis. The recognized emotion data is also saved along with the video data.

[1280] Uploading data

[1281] When the user finishes recording a video, the communication device compresses the video data and emotion data. The compressed video data, GPS metadata, and emotion data are uploaded to the server. The server confirms receipt and prepares to store the data.

[1282] Data analysis and virtual environment data generation

[1283] The server analyzes the uploaded video data, location data, and emotion data. During the analysis process, the server identifies specific locations in the scene from the video data. The server matches the video data based on the location data. The server generates virtual environment data for the urban area using a dedicated algorithm or machine learning model (e.g., TensorFlow model), taking emotion data into consideration. The server checks the quality of the generated virtual environment data and filters out inappropriate data.

[1284] Reward calculation and payment

[1285] After the virtual environment data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data. Based on the evaluation results, the server calculates a reward amount for the user. The server then obtains the user's electronic payment account information and prepares to transfer the calculated reward amount. The server transfers the reward amount to the electronic payment account and sends a transfer completion notification to the communication terminal.

[1286] Specific examples

[1287] For example, consider the case where a user takes a video while walking around a tourist spot in a city with a smartphone. As the communication device starts recording the video, it simultaneously acquires location information and tags the data in real time. While recording the video, the communication device uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize emotions from the user's facial expressions and voice, and saves these as data. Once the recording is finished, this data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a virtual environment of the tourist spot. Once this process is complete, the server evaluates the user's contribution in providing data and the content of the emotional data, calculates a certain reward, and transfers it to an electronic payment account. A notification is then displayed on the user's device indicating that the reward has been paid.

[1288] Prompt Sentence Examples

[1289] "Please explain the process of generating a virtual environment using emotion data and location information when photographing tourist spots in a city."

[1290] "Please explain the reward system for taking videos using a smartphone app and uploading the data."

[1291] In this way, users can easily provide data and receive rewards for it, enabling efficient and highly accurate generation of virtual environments. Furthermore, adding emotional data can create richer virtual environments.

[1292] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1293] Step 1:

[1294] The user downloads and installs the dedicated app.

[1295] Input: App store search query, app download request.

[1296] Output: A dedicated app is installed on the communication device.

[1297] Specific operation: The user opens the Google Play Store or App Store, searches for "Digital Twin Creation App," and taps the download button. The app is automatically downloaded and installed on the communication device.

[1298] Step 2:

[1299] The user enters the required information to create an account.

[1300] Input: User information such as name, email address, and password.

[1301] Output: The user information is sent to the server and a new account is created in the database.

[1302] Specific operation: The user launches the app, enters the required information such as name, email address, and password, and taps the "Register" button. The entered data is sent from the communication device to the server as an HTTP POST request. The server creates a new account in the database based on the data received.

[1303] Step 3:

[1304] The terminal sends the user information to the server, and the server sends a confirmation message.

[1305] Input: Request to send user information.

[1306] Output: A server confirmation message is sent to the terminal.

[1307] Specific operation: The communication terminal sends user information to the server, and after the server verifies the received data, it sends a confirmation message in JSON format to the terminal. The terminal receives this message and displays the confirmation message to the user.

[1308] Step 4:

[1309] The user selects the video recording mode and starts recording.

[1310] Input: User action (selecting video recording mode, tapping the start recording button).

[1311] Output: Video recording and GPS information acquisition begins.

[1312] Specific operation: The user selects "Video recording mode" in the app and taps the "Start recording" button on the screen. The communication device activates the camera and starts recording video. At the same time, the GPS module begins obtaining location information in real time.

[1313] Step 5:

[1314] The device tags the video data with location information and also acquires emotional data.

[1315] Input: Video frames being captured, real-time GPS data, and user face and voice data.

[1316] Output: Video data tagged with location and emotion data.

[1317] How it works: The communication device acquires location data (latitude and longitude) for each video frame and tags it as metadata in the video data. In parallel, an emotion engine (e.g., Microsoft Azure Emotion API) analyzes the user's face and voice and recognizes emotion data in real time. This data is then integrated into the video data.

[1318] Step 6:

[1319] The user stops recording the video, and the device compresses the data.

[1320] Input: User action (tapping the stop recording button).

[1321] Output: Compressed video data, location data, and emotion data.

[1322] Specific operation: The user taps the "Stop Recording" button. The communication device compresses the captured video data, location data, and emotion data. For example, it compiles the data in ZIP format.

[1323] Step 7:

[1324] The device uploads the data to the server.

[1325] Input: Compressed data file.

[1326] Output: The data is uploaded to the server.

[1327] Specific operation: The communication device uploads the compressed data file to the server using an HTTP POST request. The progress is displayed during the upload.

[1328] Step 8:

[1329] The server checks the received data.

[1330] Input: The uploaded data file.

[1331] Output: Data integrity verification message.

[1332] Specific operation: The server checks the received data. Specifically, it checks the integrity of the data and ensures consistency. Once the data is confirmed, the server sends a response message to the communication terminal.

[1333] Step 9:

[1334] The server analyzes the data and generates virtual environment data.

[1335] Input: Uploaded video data, location data, emotion data.

[1336] Output: Parsed data, generated virtual environment data.

[1337] How it works: The server uses Python scripts and TensorFlow models to analyze the received video data, location data, and emotion data. It identifies specific locations in the scene from the video data and matches them with the location data. It generates virtual environment data for the urban area using a specific algorithm, taking emotion data into account. The server checks the quality of the generated virtual environment data and filters out inappropriate data.

[1338] Step 10:

[1339] The server evaluates the user's data and calculates the reward.

[1340] Input: Quantity and quality of analyzed data, user-provided data.

[1341] Output: Calculated reward amount.

[1342] How it works: The server evaluates the quantity and quality of the data provided by users in the database. Evaluation criteria include the clarity of the data and the reliability of the sentiment data. Based on the evaluation results, a Python script calculates the reward amount.

[1343] Step 11:

[1344] The server transfers the reward to the user's electronic payment account.

[1345] Input: User's electronic payment account information, calculated reward amount.

[1346] Output: Remittance completion notification.

[1347] Specific operation: The server obtains the user's electronic payment account information and transfers the reward amount via API. Once the transfer is complete, a push notification is sent to the device.

[1348] (Application example 2)

[1349] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1350] Conventional digital twin generation systems only use video data and GPS data, making it difficult to accurately and realistically reflect user experiences and emotions. Furthermore, the calculation of rewards for users' data provision is based on limited indicators, resulting in issues of fairness and inaccuracy. Furthermore, the analysis of collected data and the generation of digital twins often relies on specific algorithms, preventing the use of advanced generative AI models.

[1351] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1352] In this invention, the server includes: means for allowing a user to shoot video using a mobile device, acquiring GPS location information in real time, and tagging the video data; means for acquiring the user's emotional data in real time using an emotion recognition engine while shooting the video, and tagging the video data; means for compressing the acquired video data, GPS data, and emotional data and uploading them to the server; means for analyzing the received video data, GPS data, and emotional data in the server and generating digital twin data using a generative AI model; and means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via an electronic payment method. This enables the creation of more realistic and rich digital twins that reflect the user's experiences and emotions, and achieves fair and accurate reward calculations.

[1353] A "mobile device" is an electronic device that a user can carry around, such as a smartphone or tablet.

[1354] "Means for capturing video" refers to a method or apparatus for recording video using the camera function on a mobile device.

[1355] "GPS location information" refers to information that uses the Global Positioning System to obtain real-time location data about a specific place or location.

[1356] "Tagging" refers to the operation of adding identifying information to data, and in this case means adding GPS location information and emotional data to video data.

[1357] An "emotion recognition engine" refers to software or algorithms that analyze a user's emotions in real time from their facial expressions, tone of voice, etc.

[1358] "Means for compressing and uploading to a server" refers to technology for converting the captured video data, GPS data, and emotion data into a small data format and transmitting it to a server via the Internet.

[1359] A "generative AI model" refers to a program or algorithm that uses artificial intelligence technology to analyze and synthesize complex data.

[1360] "Digital twin data" refers to a digital replica of a physical space or object, and in this case refers to data that recreates an urban area in a virtual space.

[1361] "Electronic payment means" refers to a method or system for electronically transferring rewards to users, and generally includes electronic money and digital wallets.

[1362] "Reward Calculation" refers to the process of calculating rewards to users based on the quantity and quality of data provided.

[1363] System Overview

[1364] This invention is a system that allows users to shoot videos using a mobile device and tag the videos with GPS location information and emotion data acquired in real time. This data is then compressed and uploaded to a server. The server then analyzes the received video data, GPS data, and emotion data and generates digital twin data using a generative AI model. The system also calculates a reward for the user based on the quantity and quality of the data provided and pays the reward via electronic payment methods.

[1365] Hardware and software used

[1366] Mobile devices (e.g. smartphones, tablets)

[1367] A device that can be carried by the user and has video recording, GPS, and emotion recognition functions.

[1368] Camera function (software)

[1369] Software for capturing video.

[1370] GPS module (hardware)

[1371] Module for obtaining geographical location information in real time.

[1372] Emotion recognition engine (software)

[1373] Software that analyzes facial expressions and tone of voice to recognize emotions.

[1374] Compression algorithm (software)

[1375] An algorithm for efficiently compressing captured data.

[1376] Communication module (hardware)

[1377] A module for sending data to a server over the Internet.

[1378] Server (hardware)

[1379] A device for analyzing the received data and generating a digital twin using a generative AI model.

[1380] Generative AI model (software)

[1381] Artificial intelligence techniques for analyzing and synthesizing complex data.

[1382] Electronic payment system (software)

[1383] A system for electronically transferring rewards to users (e.g., electronic money, digital wallets).

[1384] Example of operation

[1385] The user launches the dedicated app and starts recording video. As the user walks around, the mobile device's camera captures the video data, and the GPS module acquires location information in real time. At the same time, the emotion recognition engine extracts emotional data from the user's facial expressions and voice, and tags this data into the video. Once recording is complete, the device's on-device compression algorithm compresses the data and uploads it to the server via the communication module.

[1386] When the data arrives at the server, the server first acknowledges receipt of the data and then begins analysis. The received video data, GPS data, and emotion data are analyzed using a generative AI model to generate digital twin data. This digital twin data is rich in content, reflecting the user's emotions and location information. Based on the results of this analysis, the server calculates a reward for the user based on the quantity and quality of the data provided. The calculated reward is paid to the user using an electronic payment system.

[1387] Prompt Sentence Examples

[1388] "Generate a Python program for a smartphone app that allows users to take videos while walking around a specific area of ​​a city and tag them with emotional data in real time. Include how to upload the data to a server and implement a reward system."

[1389] In order to implement the present invention, it is important to combine these hardware and software to provide a user-friendly interface.

[1390] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1391] Program processing steps

[1392] Step 1:

[1393] The user launches the dedicated app and selects video recording mode.

[1394] Input: User's finger action (tap).

[1395] Output: Video recording preparation screen.

[1396] Specific operation: Through the mobile device interface, the user taps the app icon and selects "Video Recording" from the menu to activate the recording mode.

[1397] Step 2:

[1398] The user taps the "Start Recording" button at the shooting location to capture a specific area.

[1399] Input: User's finger action (tap).

[1400] Output: Recording screen, real-time video data.

[1401] Specific operation: The camera starts capturing video, and at the same time, the device continues to generate video data in real time.

[1402] Step 3:

[1403] As soon as the device starts recording video, it acquires GPS location information in real time.

[1404] Input: GPS sensor on mobile device.

[1405] Output: Real-time GPS location information.

[1406] Specific operation: The GPS module is activated and current coordinate information is captured in real time, so that location information is added to the video data at any time.

[1407] Step 4:

[1408] The terminal acquires the user's emotion data in real time using an emotion recognition engine.

[1409] Input: The user's facial expressions and voice.

[1410] Output: Recognized emotion data (e.g., joy, sadness, surprise, etc.).

[1411] How it works: Video and audio data captured by the camera is passed to the emotion recognition engine, where it is analyzed. As a result, an emotion tag for the user is generated.

[1412] Step 5:

[1413] The device compresses the acquired video data, GPS data, and emotion data and uploads them to a server.

[1414] Input: Video data, GPS data, emotion data.

[1415] Output: Compressed data, notification of completion of transmission to the server.

[1416] Specific operation: The compression algorithm compresses the data, and the communication module transmits the compressed data to the server over the Internet.

[1417] Step 6:

[1418] The server acknowledges the received data and then begins analyzing it.

[1419] Input: Compressed video data, GPS data, emotion data.

[1420] Output: Analyzed data, notification that the digital twin is ready to be generated.

[1421] What happens: The server decompresses the data and prepares the individual data streams as base data for analysis, so that each data stream can be split appropriately and analyzed.

[1422] Step 7:

[1423] The server generates digital twin data using the generative AI model.

[1424] Input: Analyzed video data, GPS data, emotion data.

[1425] Output: Digital twin data.

[1426] How it works: By inputting the analysis results into a generative AI model, complex data analysis and synthesis processes are performed to generate a digital twin of the urban area.

[1427] Step 8:

[1428] The server calculates a reward based on the quantity and quality of the data provided and pays the reward to the user via an electronic payment means.

[1429] Input: Data quantity and quality assessment results.

[1430] Output: Calculated reward amount, notification of transfer completion.

[1431] Specific operation: The server's evaluation algorithm calculates the reward amount based on the quantity and quality of data provided, the usefulness of the emotional data, etc., and transfers it to the user's account through an electronic payment system.

[1432] This makes it possible to create a more realistic and rich digital twin that reflects the user's experiences and emotions, resulting in fair and accurate reward calculations.

[1433] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1434] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1435] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1436] [Fourth embodiment]

[1437] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1438] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1439] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1440] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1441] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1442] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1443] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1444] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1445] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1446] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1447] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1448] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1449] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1450] Installing the dedicated app and registering as a user

[1451] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[1452] After installation, launch the app and create an account by entering the required information such as your name, email address, and password.

[1453] The terminal sends this information to the server, which then creates a new account in its database.

[1454] Once the account is created, the server will send a confirmation message to the device.

[1455] Data collection and upload

[1456] The user launches the app and switches to video recording mode.

[1457] Tap the "Start Recording" button at the shooting location to capture a specific area of ​​the city.

[1458] As soon as the device starts recording video, it acquires GPS location information in real time.

[1459] While recording, the device tags the video data with GPS location information and saves it as metadata in the video file.

[1460] Once the recording is complete, the device compresses the video data and uploads it to the server.

[1461] Data analysis and digital twin generation

[1462] The server analyzes the received video data and GPS data.

[1463] The analysis process involves identifying specific locations in the scene from the video data and confirming location information based on GPS data.

[1464] The server generates digital twin data using specialized algorithms and machine learning models.

[1465] The generated digital twin data is quality checked and any necessary filtering processes are performed.

[1466] The digital twin data is then stored in a database.

[1467] Reward calculation and payment

[1468] After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user.

[1469] Based on the evaluation results, the amount of compensation for the user is calculated.

[1470] The server obtains the user's PayPay account information and transfers the calculated reward amount.

[1471] When the payment is completed, the server sends a payment completion notification to the user's terminal.

[1472] Specific examples

[1473] For example, consider the case where a user takes a video while walking around a city's tourist attractions with their smartphone. As soon as the device starts recording the video, it acquires location information and tags the data in real time. When the video recording is finished, the data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a digital twin of the tourist attraction. Once this process is complete, the server evaluates the user's contribution to the data provision, calculates a certain reward, and transfers it to the user's PayPay account. The user's device then displays a notification that the reward has been paid.

[1474] This concludes the description of an embodiment of the system of the present invention. The system allows users to easily provide data and provides a reward system for providing data, enabling efficient and highly accurate digital twin generation.

[1475] The processing flow will be explained below.

[1476] Step 1: User Registration

[1477] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[1478] A user launches the app and enters the required information, such as their name, email address, and password.

[1479] The terminal transmits this registration information to the server.

[1480] The server stores the received registration information in a database and creates a new account.

[1481] The server sends a notification to the device that account creation is complete.

[1482] Step 2: Record video and collect GPS information

[1483] The user launches the app and selects video recording mode.

[1484] The user taps the "Start Recording" button to begin recording video.

[1485] As soon as the device starts recording video, it will obtain GPS location information in real time.

[1486] The GPS information acquired by the device while recording is tagged to the video data and saved as metadata.

[1487] Step 3: Upload your data

[1488] The user ends video recording.

[1489] The device compresses the captured video data.

[1490] The device uploads the compressed video data and associated GPS metadata to a server.

[1491] The server acknowledges receipt and prepares the data for storage.

[1492] Step 4: Analyze the data and create a digital twin

[1493] The server analyzes the received video data and GPS data.

[1494] During the analysis process, the server identifies specific locations in the scene from the video data.

[1495] The server checks the location information based on GPS data and compares it with the video data.

[1496] The server uses specialized algorithms and machine learning models to generate digital twin data for the urban area.

[1497] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[1498] Step 5: Calculating and paying rewards

[1499] The server evaluates the quantity and quality of the data provided by the user.

[1500] The server calculates the amount of reward for the user based on the evaluation result.

[1501] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[1502] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[1503] Step 6: Notification and confirmation

[1504] The terminal displays the remittance completion notice received to the user.

[1505] The user checks the reward amount in their PayPay account.

[1506] The above are the specific steps of the program process, which allows users to easily provide data and receive rewards.

[1507] Example 1

[1508] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1509] Conventional digital twin generation systems have struggled to efficiently collect large amounts of data and generate highly accurate digital twins. Furthermore, users lacked incentives to provide data, reducing the operational efficiency of the system. Furthermore, the time and effort required for data quality checking and filtering was also an issue.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1511] In this invention, the server includes: a means for a user to shoot video using a mobile terminal; a means for acquiring global positioning system (GPS) location information corresponding to the video in real time and tagging the video data; a means for compressing the acquired video data and GPS data and uploading it to the server; a means for analyzing the received video data and GPS data in the server and generating digital twin data; a means for quality checking and filtering the generated digital twin data; and a means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via a remittance means. This enables more efficient and accurate digital twin generation. It also increases the incentive for users to provide data, improving the operational efficiency of the entire system.

[1512] "User" refers to an individual or legal entity that provides data using a mobile terminal.

[1513] "Mobile device" refers to a portable computing device such as a smartphone or tablet.

[1514] "Means for capturing video" refers to a method for recording video data using a camera mounted on a mobile terminal.

[1515] "Global Positioning System location information" refers to terrestrial location data obtained using the Global Positioning System.

[1516] "Means of obtaining and tagging video data in real time" refers to a method of obtaining global positioning system location information immediately when a video is shot and adding that location information as metadata to the video data.

[1517] "Means for compressing and uploading the captured video data and global positioning system data to a server" refers to a method for compressing the captured video data and the captured global positioning system location information to reduce their size and transmitting them to a server over a network.

[1518] "Server" refers to the computer system used to analyze and store video data and global positioning system data.

[1519] "Digital twin data" refers to data that digitally reproduces a real-world object.

[1520] "Means for analyzing and generating digital twin data" refers to a method for using received video data and global positioning system data to create a digital model of a real-world object.

[1521] "Quality check and filtering measures" refers to methods for checking the quality of generated digital twin data and removing or correcting inappropriate data or errors.

[1522] "Means for calculating remuneration based on the quantity and quality of data provided and paying it through a remittance means" refers to a method for calculating remuneration based on the quantity and quality of data provided by a user and paying the remuneration to the user through an electronic payment system.

[1523] This invention includes a system that uses a mobile terminal to shoot video, tags the video with Global Positioning System (GPS) location information, uploads the video to a server, generates digital twin data, and finally pays rewards to users. Each step is described in detail below.

[1524] Installing the dedicated app and registering as a user

[1525] The user downloads and installs the dedicated app from the Google Play Store or App Store. After installation, the user launches the app and creates an account by entering required information such as name, email address, and password. The user then sends the account information to the server via their device, and the server creates a new account in its database. Once the account is created, the server sends a confirmation message to the device.

[1526] Data collection and upload

[1527] The user launches the app and switches to video recording mode. At the recording location, they tap the "Start Recording" button and record a specific area of ​​the city. The device starts recording video and simultaneously acquires GPS location information in real time. While recording, the device tags the video data with GPS location information and saves it as metadata in the video file. Once recording is complete, the device compresses the video data and uploads it to the server.

[1528] Data analysis and digital twin generation

[1529] The server analyzes the received video data and GPS data. During the analysis process, specific locations in the scene are identified from the video data and location information is confirmed based on the GPS data. The server generates digital twin data using dedicated algorithms and machine learning models (e.g., OpenCV and TensorFlow). The generated digital twin data is then quality checked and any necessary filtering is performed. Finally, the digital twin data is stored in a database.

[1530] Reward calculation and payment

[1531] After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user. Based on the evaluation results, it calculates the amount of compensation for the user. The server obtains the user's electronic payment account information and transfers the calculated amount of compensation. Once the payment is complete, the server sends a payment completion notification to the user's device.

[1532] Specific examples

[1533] For example, if a user takes a video while walking around a city's tourist attractions with their smartphone, the device starts recording the video and simultaneously acquires location information, tagging the data in real time. Once the video recording is finished, the data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a digital twin of the tourist attraction. Once this process is complete, the server evaluates the user's contribution to the data provision, calculates a certain reward, and transfers it to an electronic payment account. The user's device then displays a notification that the reward has been paid.

[1534] Prompt Sentence Examples

[1535] Please tell me the procedure for creating a new account.

[1536] "Please explain how to tag video data with GPS location information in real time."

[1537] "What kind of data do I need to generate a digital twin and how do I analyze it?"

[1538] This system allows users to easily provide data and receive compensation for that data, making it possible to generate digital twins efficiently and with high accuracy.

[1539] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1540] Step 1: Install the dedicated app

[1541] (Explanation) The user downloads and installs the dedicated app from the Google Play Store or App Store.

[1542] (Input) A user accesses the Google Play Store or App Store.

[1543] (Data processing / calculation) Search for dedicated apps through the store's search function and download the apps you find.

[1544] (Output) A dedicated app is installed on the user's device.

[1545] Step 2: Create an account

[1546] (Description) The user launches the installed app and creates an account by entering required information such as name, email address, and password.

[1547] (Input) The user enters required information into the app, such as their name, email address, and password.

[1548] (Data processing / calculation) The terminal formats the user's input data to send to the server and generates an HTTP request.

[1549] (Output) The server creates the new account in the database and sends a confirmation message to the terminal.

[1550] Step 3: Start recording

[1551] (Explanation) The user launches the app, switches to video recording mode, and taps the "Start Recording" button at the recording location.

[1552] (Input) The user taps the "Start Recording" button.

[1553] (Data processing / calculation) The device starts recording video and simultaneously obtains location information in real time from GPS.

[1554] (Output) The device tags the video data and acquired GPS information in real time and saves it as metadata.

[1555] Step 4: Compress and upload your video data

[1556] (Explanation) Once video recording is complete, the device compresses the video data and uploads it to the server.

[1557] (Input) Recorded video data and GPS location information.

[1558] (Data processing / calculation) The terminal compresses the video data using a codec such as H.264 and sends it to the server along with the location information via an HTTP POST request.

[1559] (Output) Compressed video data and GPS data are saved on the server.

[1560] Step 5: Analyze the data

[1561] (Description) The server analyzes the video data and GPS data received.

[1562] (Input) Compressed video data and GPS data received by the server.

[1563] (Data processing / calculation) The server uses analysis software such as OpenCV and TensorFlow to identify specific locations in the scene from the video data and confirm the corresponding location information.

[1564] (Output) The analyzed scene data and position information are obtained.

[1565] Step 6: Generate the digital twin

[1566] (Explanation) The server generates digital twin data using dedicated algorithms and machine learning models.

[1567] (Input) Analyzed video data and location information.

[1568] (Data processing / calculation) The server generates a digital twin using 3D modeling software such as Blender based on the scene identification results and location information.

[1569] (Output) The generated digital twin data is obtained.

[1570] Step 7: Quality check and filtering

[1571] (Description) Quality check and filter the generated digital twin data.

[1572] (Input) Generated digital twin data.

[1573] (Data processing / calculation) The server checks the quality of the generated data, filters out inappropriate data, and in some cases corrects it.

[1574] (Output) Quality-confirmed digital twin data is obtained.

[1575] Step 8: Save your data

[1576] (Explanation) The server stores the quality-confirmed digital twin data in a database.

[1577] (Input) Quality-confirmed digital twin data.

[1578] (Data processing / calculation) The server inserts the digital twin data into a database system such as SQL Server or MySQL.

[1579] (Output) Digital twin data is stored in a database.

[1580] Step 9: Calculating rewards

[1581] (Explanation) After the digital twin data is stored in the database, the server evaluates the quantity and quality of the data provided by the user and calculates the reward amount.

[1582] (Input) Provided digital twin data and user information.

[1583] (Data processing / calculation) The server scores the data based on its quantity and quality and calculates the reward amount.

[1584] (Output) The calculated reward amount is obtained.

[1585] Step 10: Send your rewards

[1586] (Explanation) The server obtains the user's electronic payment account information and transfers the calculated reward amount.

[1587] (Input) Calculated reward amount and user's electronic payment account information.

[1588] (Data processing / calculation) The server uses the PayPay API to transfer the reward amount to the user's account.

[1589] (Output) Rewards are transferred to the user's electronic payment account.

[1590] Step 11: Payment completion notification

[1591] (Explanation) When payment is completed, the server sends a payment completion notification to the user's terminal.

[1592] (Input) Information on completed remittance.

[1593] The (data processing / calculation) server generates a notification message and sends a push notification to the user's device.

[1594] (Output) A notification of payment completion is displayed on the user's terminal.

[1595] (Application example 1)

[1596] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1597] In conventional logistics centers, it is difficult to monitor abnormalities (e.g., missing pallets, obstructions) in real time, hindering efficient operation. Another issue is the difficulty of generating an accurate digital twin and understanding the situation within the logistics facility. Furthermore, there is a lack of a system that appropriately rewards users based on the quality and quantity of data they provide.

[1598] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1599] In this invention, the server includes means for a user to shoot video using a mobile device, means for acquiring location information in real time and tagging the video data, means for compressing the acquired video data and location data and uploading it to the server, means for analyzing the received video data and location data and generating digital twin data, means for monitoring abnormalities in the logistics facility in real time using the digital twin data, and means for calculating compensation based on the quantity and quality of data provided to the user and paying the compensation via a remittance means. This makes it possible to monitor the situation in the logistics facility in real time and manage it efficiently.

[1600] "User" means an individual or entity that uses a mobile device to capture video and provide the data, along with location information, to the server.

[1601] A "mobile device" is a portable electronic device capable of capturing video and acquiring location information.

[1602] "Video data" refers to video image information captured by a mobile device.

[1603] "Location information" refers to data that indicates a specific location or coordinates using GPS or other positioning means.

[1604] "Tagging" is the process of linking metadata such as location information to video data.

[1605] "Compression" is a technique for reducing data volume.

[1606] A "server" is a computer system that receives, stores, and analyzes data over a network.

[1607] "Digital twin data" is data that accurately recreates physical objects and spaces in the real world.

[1608] An "abnormal location" is a location within a logistics facility where an abnormal condition or problem is occurring.

[1609] "Real-time" refers to a situation in which processing and information updates occur almost immediately.

[1610] "Reward" refers to the payment made based on the data provided by the user.

[1611] "Remittance method" refers to the payment method or technology used to send rewards to users.

[1612] The system of this invention is realized by a user taking video of the inside of a logistics facility using a mobile device, tagging the video data with location information, and uploading it to a server. Details and specific examples of each means are provided below.

[1613] A user takes a video using a mobile device

[1614] Users use a mobile device such as a smartphone or tablet to capture video of a specific area within a logistics facility. This mobile device must have a camera, GPS functionality, and an internet connection. Users install a dedicated application and start recording using the app.

[1615] Real-time location information acquisition and tagging of video data

[1616] The device acquires location information in real time using GPS or other positioning methods while shooting video. The acquired location information is tagged as metadata to the video data being shot. This process is realized using libraries such as OpenCV and Geopy.

[1617] Compression of video data and location data and upload to server

[1618] Once the recording is complete, the device compresses the video and location data and uploads it to a server over its internet connection, for example by sending an HTTP POST request using the Requests library.

[1619] Data analysis on the server and generation of a digital twin

[1620] The server analyzes the received video and location data. During this analysis process, specialized algorithms and machine learning models are used to generate a digital twin of the logistics facility. For example, it identifies abnormalities (missing pallets, obstructions, etc.). This data is used for real-time monitoring of the logistics facility.

[1621] Calculation and payment of rewards based on quantity and quality of data provided

[1622] The server calculates the reward based on the quantity and quality of the data provided by the user and pays the user. The payment method is, for example, an electronic payment system. This information is notified to the user's terminal.

[1623] Specific examples

[1624] Users walk around the logistics facility with their smartphones and record video. As the device starts recording, it simultaneously acquires location information, which is tagged in real time. After filming is complete, the compressed video data and location data are uploaded to a server. The server analyzes the received data and generates a digital twin of the logistics facility. Through this process, users receive appropriate compensation for the data they provide.

[1625] Prompt Sentence Examples

[1626] Examples of prompts for a generative AI model might include:

[1627] "Please explain how a smartphone can be used in a logistics facility to capture video of a specific area, add location information in real time, and upload the data to a server."

[1628] The above steps enable efficient and highly accurate generation of digital twins and real-time monitoring of abnormalities within logistics facilities.

[1629] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1630] Step 1:

[1631] The user installs a dedicated application on the mobile device.

[1632] Input: Basic information such as user name, email address, and password

[1633] Process: The user launches the application and enters basic information. The device sends this information to the server, which then creates a new account in the database.

[1634] Output: Account verification message is displayed on the terminal.

[1635] Step 2:

[1636] The user launches the app and switches to video recording mode.

[1637] Input: User initiated video recording mode

[1638] Processing: The device activates the camera and GPS functions and prepares to capture video and obtain location information.

[1639] Output: The video recording mode screen will be displayed.

[1640] Step 3:

[1641] The user starts shooting video in a specific area within the logistics facility.

[1642] Input: User taps the "Start Recording" button

[1643] Processing: The device captures video in real time and simultaneously acquires GPS location information. Each captured video frame is tagged with location information.

[1644] Output: A video file and its corresponding geotag will be generated.

[1645] Step 4:

[1646] After the recording is complete, the device compresses the video data and location data.

[1647] Input: Video data and location data after shooting is complete

[1648] Processing: The device efficiently compresses video and location data to reduce data size.

[1649] Output: Compressed video data and location data

[1650] Step 5:

[1651] The compressed data is uploaded to the server.

[1652] Input: Compressed video data and location data

[1653] Processing: The device sends the data to the server over the Internet using an HTTP POST request.

[1654] Output: A message is displayed confirming successful data transmission.

[1655] Step 6:

[1656] The server analyzes the received data and generates a digital twin of the logistics facility.

[1657] Input: Video data and location data sent from the device

[1658] Processing: The server uses video analysis algorithms and machine learning models to identify anomalies within the facility and generate a digital twin.

[1659] Output: Digital twin data and abnormality location identification results

[1660] Step 7:

[1661] Rewards are calculated and paid based on the quantity and quality of data provided.

[1662] Input: The amount and quality of data analyzed by the server

[1663] Processing: The server calculates the user's reward amount using a reward calculation algorithm and makes the payment through an electronic payment system.

[1664] Output: A notification will be displayed on the device that the reward amount has been deposited into the user's account.

[1665] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1666] Installing the dedicated app and registering as a user

[1667] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[1668] After installation, launch the app and create an account by entering the required information such as your name, email address, and password.

[1669] The terminal sends this information to the server, which then creates a new account in its database.

[1670] Once the account is created, the server will send a confirmation message to the device.

[1671] Video recording, GPS information collection, and emotion recognition

[1672] The user launches the app and selects video recording mode.

[1673] Tap the "Start Recording" button at the shooting location to capture a specific area of ​​the city.

[1674] As soon as the device starts recording video, it acquires GPS location information in real time.

[1675] The GPS information acquired by the device while recording is tagged to the video data and saved as metadata.

[1676] While the device is recording video, the emotion engine recognizes the user's emotions in real time through facial recognition and voice analysis.

[1677] The recognized emotion data is also saved along with the video data.

[1678] Uploading data

[1679] The user ends video recording.

[1680] The device compresses the captured video data and emotion data.

[1681] The device uploads the compressed video data, GPS metadata, and emotion data to a server.

[1682] The server acknowledges receipt and prepares the data for storage.

[1683] Data analysis and digital twin generation

[1684] The server analyzes the received video data, GPS data, and emotion data.

[1685] During the analysis process, the server identifies specific locations in the scene from the video data.

[1686] The server checks the location information based on GPS data and compares it with the video data.

[1687] The server generates digital twin data for the urban area using dedicated algorithms and machine learning models, taking into account emotional data.

[1688] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[1689] Reward calculation and payment

[1690] After the digital twin data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data.

[1691] Based on the evaluation results, the amount of compensation for the user is calculated.

[1692] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[1693] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[1694] Specific examples

[1695] For example, consider the case where a user takes a video while walking around a tourist spot in a city with their smartphone. As soon as the device starts recording the video, it acquires GPS location information and tags the data in real time. While recording the video, the device uses an emotion engine to recognize emotions from the user's facial expressions and voice, and saves these as data. Once recording is complete, this data is automatically compressed and uploaded to the server. The server immediately analyzes the received data and creates a digital twin of the tourist spot. Once this process is complete, the server evaluates the user's contribution in providing data and the content of the emotional data, calculates a certain reward, and transfers it to the user's PayPay account. A notification is then displayed on the user's device indicating that the reward has been paid.

[1696] This concludes the description of an embodiment of the system of the present invention. The system allows users to easily provide data and provides a reward system for providing data, enabling efficient and highly accurate digital twin generation. Furthermore, adding emotional data can create richer digital twins.

[1697] The processing flow will be explained below.

[1698] Step 1: User Registration

[1699] The user downloads and installs the dedicated app from the Google Play Store or App Store.

[1700] A user launches the app and enters the required information, such as their name, email address, and password.

[1701] The terminal transmits this registration information to the server.

[1702] The server stores the received registration information in a database and creates a new account.

[1703] The server sends a notification to the device that account creation is complete.

[1704] Step 2: Record video and collect GPS information

[1705] The user launches the app and selects video recording mode.

[1706] The user taps the "Start Recording" button to begin recording video.

[1707] As soon as the device starts recording video, it will acquire GPS location information in real time.

[1708] The GPS information acquired by the device in real time is tagged to the video data and saved as metadata.

[1709] Step 3: Emotion Recognition

[1710] While the device is recording video, it activates the emotion engine to recognize the user's face.

[1711] The device analyzes the user's facial expressions and voice and generates emotional data in real time.

[1712] The device stores the generated emotion data together with the video data and GPS data.

[1713] Step 4: Upload your data

[1714] The user ends video recording.

[1715] The device compresses the captured video data and emotion data.

[1716] The device uploads the compressed video data, GPS metadata, and emotion data to a server.

[1717] The server acknowledges receipt and prepares the data for storage.

[1718] Step 5: Analyze the data and create a digital twin

[1719] The server analyzes the received video data, GPS data, and emotion data.

[1720] During the analysis process, the server identifies specific locations in the scene from the video data.

[1721] The server checks the location information based on GPS data and compares it with the video data.

[1722] The server generates digital twin data that reflects the user's emotions based on the emotional data.

[1723] The server performs quality checks on the generated digital twin data and filters out inappropriate data.

[1724] Step 6: Calculating and paying rewards

[1725] After the digital twin data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data.

[1726] The server calculates the amount of reward for the user based on the evaluation result.

[1727] The server obtains the user's PayPay account information and prepares to transfer the calculated reward amount.

[1728] The server transfers the reward amount to the PayPay account and sends a notification of completion of the transfer to the terminal.

[1729] Step 7: Notification and confirmation

[1730] The terminal displays the remittance completion notice received to the user.

[1731] The user checks the reward amount in their PayPay account.

[1732] These are the specific processing steps of a system that combines an emotion engine. Users can easily provide data, and by utilizing emotion data, we can expect to improve the accuracy of digital twins.

[1733] Example 2

[1734] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1735] Conventional technologies have struggled to efficiently process user-generated video and location information and generate virtual environments based on that data. Furthermore, because data analysis, including user emotional data, is not performed, it is not possible to provide more realistic and detailed virtual environments. Furthermore, the process of calculating and paying rewards to users is complicated, and there is a lack of mechanisms to increase user motivation. Therefore, there is a need to improve the quantity and quality of data provided by users.

[1736] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1737] In this invention, the server includes means for recognizing emotion data from a user's facial expressions and voice in real time and integrating it into video data, means for using a machine learning model to analyze the video data including the emotion data, and means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via electronic remittance. This makes it possible to add emotion data to video data captured by the user and generate a highly accurate virtual environment. Furthermore, by promptly paying an appropriate reward to the user, it is possible to increase the user's motivation and encourage them to provide more and higher-quality data.

[1738] A "portable communication terminal" refers to a device that can be carried by a user and has a communication function, and specifically includes a smartphone, a tablet, a portable computer, and the like.

[1739] "Location information" is data indicating a geographical location, including, for example, latitude and longitude information obtained using a GPS (Global Positioning System).

[1740] "Central processing unit" refers to a central device for data processing, such as a server or cloud computing service that receives, analyzes, stores, and transmits data.

[1741] "Virtual environment data" refers to data such as digital twins and 3D models generated based on real physical spaces, which allows the real world to be reproduced digitally.

[1742] "Emotion data" is data that represents the emotional state of the user analyzed based on facial expressions, voice, and other biological information.

[1743] A "machine learning model" refers to software that has algorithms that analyze large amounts of data, find patterns and regularities, and make predictions and classifications based on them.

[1744] "Electronic remittance instruments" refers to systems and services for sending and receiving money over the Internet, and specifically includes digital wallets and bank online remittance systems.

[1745] MODE FOR CARRYING OUT THE INVENTION

[1746] In this invention, a system is constructed in which users collect data using mobile communication terminals, and the server analyzes and processes the data to generate virtual environment data, and also calculates and pays rewards to users.

[1747] Installing the dedicated app and registering as a user

[1748] Users must download and install a dedicated app from the Android or iOS app store. After installation, they launch the app and create an account by entering required information such as their name, email address, and password. The communication device sends this information to the server, which then creates a new account in its database. Once the account is created, the server sends a confirmation message to the communication device.

[1749] Video capture, location collection, and emotion recognition

[1750] The user launches the dedicated app and selects video recording mode. At the recording location, they tap the "Start Recording" button and record a specific area. As soon as the communication device starts recording video, it acquires GPS location information in real time. The communication device tags the location information acquired while recording with the video data and saves it as metadata. While the communication device is recording video, it uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions in real time through facial recognition and voice analysis. The recognized emotion data is also saved along with the video data.

[1751] Uploading data

[1752] When the user finishes recording a video, the communication device compresses the video data and emotion data. The compressed video data, GPS metadata, and emotion data are uploaded to the server. The server confirms receipt and prepares to store the data.

[1753] Data analysis and virtual environment data generation

[1754] The server analyzes the uploaded video data, location data, and emotion data. During the analysis process, the server identifies specific locations in the scene from the video data. The server matches the video data based on the location data. The server generates virtual environment data for the urban area using a dedicated algorithm or machine learning model (e.g., TensorFlow model), taking emotion data into consideration. The server checks the quality of the generated virtual environment data and filters out inappropriate data.

[1755] Reward calculation and payment

[1756] After the virtual environment data is stored in the database, the server performs a comprehensive evaluation of the data provided by the user, including the quantity and quality of the data and emotional data. Based on the evaluation results, the server calculates a reward amount for the user. The server then obtains the user's electronic payment account information and prepares to transfer the calculated reward amount. The server transfers the reward amount to the electronic payment account and sends a transfer completion notification to the communication terminal.

[1757] Specific examples

[1758] For example, consider the case where a user takes a video while walking around a tourist spot in a city with a smartphone. As the communication device starts recording the video, it simultaneously acquires location information and tags the data in real time. While recording the video, the communication device uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize emotions from the user's facial expressions and voice, and saves these as data. Once the recording is finished, this data is automatically compressed and uploaded to a server. The server immediately analyzes the received data and generates a virtual environment of the tourist spot. Once this process is complete, the server evaluates the user's contribution in providing data and the content of the emotional data, calculates a certain reward, and transfers it to an electronic payment account. A notification is then displayed on the user's device indicating that the reward has been paid.

[1759] Prompt Sentence Examples

[1760] "Please explain the process of generating a virtual environment using emotion data and location information when photographing tourist spots in a city."

[1761] "Please explain the reward system for taking videos using a smartphone app and uploading the data."

[1762] In this way, users can easily provide data and receive rewards for it, enabling efficient and highly accurate generation of virtual environments. Furthermore, adding emotional data can create richer virtual environments.

[1763] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1764] Step 1:

[1765] The user downloads and installs the dedicated app.

[1766] Input: App store search query, app download request.

[1767] Output: A dedicated app is installed on the communication device.

[1768] Specific operation: The user opens the Google Play Store or App Store, searches for "Digital Twin Creation App," and taps the download button. The app is automatically downloaded and installed on the communication device.

[1769] Step 2:

[1770] The user enters the required information to create an account.

[1771] Input: User information such as name, email address, and password.

[1772] Output: The user information is sent to the server and a new account is created in the database.

[1773] Specific operation: The user launches the app, enters the required information such as name, email address, and password, and taps the "Register" button. The entered data is sent from the communication device to the server as an HTTP POST request. The server creates a new account in the database based on the data received.

[1774] Step 3:

[1775] The terminal sends the user information to the server, and the server sends a confirmation message.

[1776] Input: Request to send user information.

[1777] Output: A server confirmation message is sent to the terminal.

[1778] Specific operation: The communication terminal sends user information to the server, and after the server verifies the received data, it sends a confirmation message in JSON format to the terminal. The terminal receives this message and displays the confirmation message to the user.

[1779] Step 4:

[1780] The user selects the video recording mode and starts recording.

[1781] Input: User action (selecting video recording mode, tapping the start recording button).

[1782] Output: Video recording and GPS information acquisition begins.

[1783] Specific operation: The user selects "Video recording mode" in the app and taps the "Start recording" button on the screen. The communication device activates the camera and starts recording video. At the same time, the GPS module begins obtaining location information in real time.

[1784] Step 5:

[1785] The device tags the video data with location information and also acquires emotional data.

[1786] Input: Video frames being captured, real-time GPS data, and user face and voice data.

[1787] Output: Video data tagged with location and emotion data.

[1788] How it works: The communication device acquires location data (latitude and longitude) for each video frame and tags it as metadata in the video data. In parallel, an emotion engine (e.g., Microsoft Azure Emotion API) analyzes the user's face and voice and recognizes emotion data in real time. This data is then integrated into the video data.

[1789] Step 6:

[1790] The user stops recording the video, and the device compresses the data.

[1791] Input: User action (tapping the stop recording button).

[1792] Output: Compressed video data, location data, and emotion data.

[1793] Specific operation: The user taps the "Stop Recording" button. The communication device compresses the captured video data, location data, and emotion data. For example, it compiles the data in ZIP format.

[1794] Step 7:

[1795] The device uploads the data to the server.

[1796] Input: Compressed data file.

[1797] Output: The data is uploaded to the server.

[1798] Specific operation: The communication device uploads the compressed data file to the server using an HTTP POST request. The progress is displayed during the upload.

[1799] Step 8:

[1800] The server checks the received data.

[1801] Input: The uploaded data file.

[1802] Output: Data integrity verification message.

[1803] Specific operation: The server checks the received data. Specifically, it checks the integrity of the data and ensures consistency. Once the data is confirmed, the server sends a response message to the communication terminal.

[1804] Step 9:

[1805] The server analyzes the data and generates virtual environment data.

[1806] Input: Uploaded video data, location data, emotion data.

[1807] Output: Parsed data, generated virtual environment data.

[1808] How it works: The server uses Python scripts and TensorFlow models to analyze the received video data, location data, and emotion data. It identifies specific locations in the scene from the video data and matches them with the location data. It generates virtual environment data for the urban area using a specific algorithm, taking emotion data into account. The server checks the quality of the generated virtual environment data and filters out inappropriate data.

[1809] Step 10:

[1810] The server evaluates the user's data and calculates the reward.

[1811] Input: Quantity and quality of analyzed data, user-provided data.

[1812] Output: Calculated reward amount.

[1813] How it works: The server evaluates the quantity and quality of the data provided by users in the database. Evaluation criteria include the clarity of the data and the reliability of the sentiment data. Based on the evaluation results, a Python script calculates the reward amount.

[1814] Step 11:

[1815] The server transfers the reward to the user's electronic payment account.

[1816] Input: User's electronic payment account information, calculated reward amount.

[1817] Output: Remittance completion notification.

[1818] Specific operation: The server obtains the user's electronic payment account information and transfers the reward amount via API. Once the transfer is complete, a push notification is sent to the device.

[1819] (Application example 2)

[1820] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1821] Conventional digital twin generation systems only use video data and GPS data, making it difficult to accurately and realistically reflect user experiences and emotions. Furthermore, the calculation of rewards for users' data provision is based on limited indicators, resulting in issues of fairness and inaccuracy. Furthermore, the analysis of collected data and the generation of digital twins often relies on specific algorithms, preventing the use of advanced generative AI models.

[1822] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1823] In this invention, the server includes: means for allowing a user to shoot video using a mobile device, acquiring GPS location information in real time, and tagging the video data; means for acquiring the user's emotional data in real time using an emotion recognition engine while shooting the video, and tagging the video data; means for compressing the acquired video data, GPS data, and emotional data and uploading them to the server; means for analyzing the received video data, GPS data, and emotional data in the server and generating digital twin data using a generative AI model; and means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via an electronic payment method. This enables the creation of more realistic and rich digital twins that reflect the user's experiences and emotions, and achieves fair and accurate reward calculations.

[1824] A "mobile device" is an electronic device that a user can carry around, such as a smartphone or tablet.

[1825] "Means for capturing video" refers to a method or apparatus for recording video using the camera function on a mobile device.

[1826] "GPS location information" refers to information that uses the Global Positioning System to obtain real-time location data about a specific place or location.

[1827] "Tagging" refers to the operation of adding identifying information to data, and in this case means adding GPS location information and emotional data to video data.

[1828] An "emotion recognition engine" refers to software or algorithms that analyze a user's emotions in real time from their facial expressions, tone of voice, etc.

[1829] "Means for compressing and uploading to a server" refers to technology for converting the captured video data, GPS data, and emotion data into a small data format and transmitting it to a server via the Internet.

[1830] A "generative AI model" refers to a program or algorithm that uses artificial intelligence technology to analyze and synthesize complex data.

[1831] "Digital twin data" refers to a digital replica of a physical space or object, and in this case refers to data that recreates an urban area in a virtual space.

[1832] "Electronic payment means" refers to a method or system for electronically transferring rewards to users, and generally includes electronic money and digital wallets.

[1833] "Reward Calculation" refers to the process of calculating rewards to users based on the quantity and quality of data provided.

[1834] System Overview

[1835] This invention is a system that allows users to shoot videos using a mobile device and tag the videos with GPS location information and emotion data acquired in real time. This data is then compressed and uploaded to a server. The server then analyzes the received video data, GPS data, and emotion data and generates digital twin data using a generative AI model. The system also calculates a reward for the user based on the quantity and quality of the data provided and pays the reward via electronic payment methods.

[1836] Hardware and software used

[1837] Mobile devices (e.g. smartphones, tablets)

[1838] A device that can be carried by the user and has video recording, GPS, and emotion recognition functions.

[1839] Camera function (software)

[1840] Software for capturing video.

[1841] GPS module (hardware)

[1842] Module for obtaining geographical location information in real time.

[1843] Emotion recognition engine (software)

[1844] Software that analyzes facial expressions and tone of voice to recognize emotions.

[1845] Compression algorithm (software)

[1846] An algorithm for efficiently compressing captured data.

[1847] Communication module (hardware)

[1848] A module for sending data to a server over the Internet.

[1849] Server (hardware)

[1850] A device for analyzing the received data and generating a digital twin using a generative AI model.

[1851] Generative AI model (software)

[1852] Artificial intelligence techniques for analyzing and synthesizing complex data.

[1853] Electronic payment system (software)

[1854] A system for electronically transferring rewards to users (e.g., electronic money, digital wallets).

[1855] Example of operation

[1856] The user launches the dedicated app and starts recording video. As the user walks around, the mobile device's camera captures the video data, and the GPS module acquires location information in real time. At the same time, the emotion recognition engine extracts emotional data from the user's facial expressions and voice, and tags this data into the video. Once recording is complete, the device's on-device compression algorithm compresses the data and uploads it to the server via the communication module.

[1857] When the data arrives at the server, the server first acknowledges receipt of the data and then begins analysis. The received video data, GPS data, and emotion data are analyzed using a generative AI model to generate digital twin data. This digital twin data is rich in content, reflecting the user's emotions and location information. Based on the results of this analysis, the server calculates a reward for the user based on the quantity and quality of the data provided. The calculated reward is paid to the user using an electronic payment system.

[1858] Prompt Sentence Examples

[1859] "Generate a Python program for a smartphone app that allows users to take videos while walking around a specific area of ​​a city and tag them with emotional data in real time. Include how to upload the data to a server and implement a reward system."

[1860] In order to implement the present invention, it is important to combine these hardware and software to provide a user-friendly interface.

[1861] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1862] Program processing steps

[1863] Step 1:

[1864] The user launches the dedicated app and selects video recording mode.

[1865] Input: User's finger action (tap).

[1866] Output: Video recording preparation screen.

[1867] Specific operation: Through the mobile device interface, the user taps the app icon and selects "Video Recording" from the menu to activate the recording mode.

[1868] Step 2:

[1869] The user taps the "Start Recording" button at the shooting location to capture a specific area.

[1870] Input: User's finger action (tap).

[1871] Output: Recording screen, real-time video data.

[1872] Specific operation: The camera starts capturing video, and at the same time, the device continues to generate video data in real time.

[1873] Step 3:

[1874] As soon as the device starts recording video, it acquires GPS location information in real time.

[1875] Input: GPS sensor on mobile device.

[1876] Output: Real-time GPS location information.

[1877] Specific operation: The GPS module is activated and current coordinate information is captured in real time, so that location information is added to the video data at any time.

[1878] Step 4:

[1879] The terminal acquires the user's emotion data in real time using an emotion recognition engine.

[1880] Input: The user's facial expressions and voice.

[1881] Output: Recognized emotion data (e.g., joy, sadness, surprise, etc.).

[1882] How it works: Video and audio data captured by the camera is passed to the emotion recognition engine, where it is analyzed. As a result, an emotion tag for the user is generated.

[1883] Step 5:

[1884] The device compresses the acquired video data, GPS data, and emotion data and uploads them to a server.

[1885] Input: Video data, GPS data, emotion data.

[1886] Output: Compressed data, notification of completion of transmission to the server.

[1887] Specific operation: The compression algorithm compresses the data, and the communication module transmits the compressed data to the server over the Internet.

[1888] Step 6:

[1889] The server acknowledges the received data and then begins analyzing it.

[1890] Input: Compressed video data, GPS data, emotion data.

[1891] Output: Analyzed data, notification that the digital twin is ready to be generated.

[1892] What happens: The server decompresses the data and prepares the individual data streams as base data for analysis, so that each data stream can be split appropriately and analyzed.

[1893] Step 7:

[1894] The server generates digital twin data using the generative AI model.

[1895] Input: Analyzed video data, GPS data, emotion data.

[1896] Output: Digital twin data.

[1897] How it works: By inputting the analysis results into a generative AI model, complex data analysis and synthesis processes are performed to generate a digital twin of the urban area.

[1898] Step 8:

[1899] The server calculates a reward based on the quantity and quality of the data provided and pays the reward to the user via an electronic payment means.

[1900] Input: Data quantity and quality assessment results.

[1901] Output: Calculated reward amount, notification of transfer completion.

[1902] Specific operation: The server's evaluation algorithm calculates the reward amount based on the quantity and quality of data provided, the usefulness of the emotional data, etc., and transfers it to the user's account through an electronic payment system.

[1903] This makes it possible to create a more realistic and rich digital twin that reflects the user's experiences and emotions, resulting in fair and accurate reward calculations.

[1904] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1905] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1906] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1907] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1908] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1909] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1910] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1911] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1912] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1913] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1914] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1915] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1916] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1917] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1918] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1919] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1920] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1921] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1922] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1923] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1924] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1925] The following is further disclosed regarding the above embodiment.

[1926] (Claim 1)

[1927] a means for a user to capture video using a mobile device;

[1928] A means of acquiring GPS location information corresponding to the video in real time and tagging the video data;

[1929] a means for compressing the acquired video data and GPS data and uploading them to a server;

[1930] A means for analyzing the received video data and GPS data in the server and generating digital twin data;

[1931] A means for calculating a reward based on the quantity and quality of data provided to the user and paying the reward via a remittance means;

[1932] A system including:

[1933] (Claim 2)

[1934] 10. The system of claim 1, wherein location information of the mobile device is obtained if the user gives permission.

[1935] (Claim 3)

[1936] 10. The system of claim 1, wherein the system synchronizes real-time GPS location information with video data.

[1937] "Example 1"

[1938] (Claim 1)

[1939] A means for a user to capture video using a mobile terminal;

[1940] means for obtaining global positioning system location information corresponding to the video in real time and tagging the video data;

[1941] means for compressing and uploading the captured video data and global positioning system data to a server;

[1942] A means for analyzing the received video data and global positioning system data in the server and generating digital twin data;

[1943] A means of quality checking and filtering the generated digital twin data; and

[1944] A means for calculating a reward based on the quantity and quality of data provided to the user and paying the reward via a remittance means;

[1945] A system including:

[1946] (Claim 2)

[1947] 10. The system of claim 1, wherein location information of the mobile terminal is obtained if the user gives permission.

[1948] (Claim 3)

[1949] 10. The system of claim 1, wherein the system synchronizes the video data with global positioning system location information obtained in real time.

[1950] "Application Example 1"

[1951] procedure:

[1952] 2. Extract the novel aspects of the technology from the explanation of the application examples.

[1953] 5. If there are proper nouns, proper names, personal names, or company names, convert them into general names and output them. Nouns are converted into superordinate expressions and output.

[1954] (Claim 1)

[1955] a means for a user to capture video using a mobile device;

[1956] A means of acquiring GPS location information corresponding to the video in real time and tagging the video data;

[1957] a means for compressing the acquired video data and GPS data and uploading them to a server;

[1958] A means for analyzing the received video data and GPS data in the server and generating digital twin data;

[1959] A means for calculating a reward based on the quantity and quality of data provided to the user and paying the reward via a remittance means;

[1960] A system including:

[1961] (Claim 2)

[1962] 10. The system of claim 1, wherein location information of the mobile device is obtained if the user gives permission.

[1963] (Claim 3)

[1964] 10. The system of claim 1, wherein the system synchronizes real-time GPS location information with video data.

[1965] Extracting novel parts of application examples:

[1966] Applying digital twin generation to logistics center monitoring

[1967] Real-time monitoring of abnormalities (e.g., missing pallets, obstructions)

[1968] Video recording of specific areas within the logistics center

[1969] Real-time monitoring of abnormalities

[1970] New inventions include:

[1971] Added a function to the logistics center monitoring system that allows users to take video and monitor abnormalities in real time.

[1972] New claim:

[1973] (Claim 1)

[1974] a means for a user to capture video using a mobile device;

[1975] A means for acquiring location information corresponding to the video in real time and tagging the video data;

[1976] means for compressing and uploading the captured video data and location data to a server;

[1977] A means for analyzing the video data and location data received in the server and generating digital twin data;

[1978] A means of monitoring abnormalities in logistics facilities in real time using digital twin data, and

[1979] A means for calculating a reward based on the quantity and quality of data provided to the user and paying the reward via a remittance means;

[1980] A system including:

[1981] (Claim 2)

[1982] 10. The system of claim 1, wherein location information of the mobile device is obtained if the user gives permission.

[1983] (Claim 3)

[1984] 10. The system of claim 1, wherein the system synchronizes location information acquired in real time with video data.

[1985] "Example 2: Combining Emotion Engines"

[1986] (Claim 1)

[1987] A means for a user to take a video using a mobile communication terminal;

[1988] A means for acquiring location information corresponding to the video in real time and tagging the video data;

[1989] means for compressing and uploading the acquired video data and location information data to a central processing unit;

[1990] a means for analyzing the received video data and location information data in a central processing unit and generating virtual environment data;

[1991] means for recognizing emotion data from a user's facial expressions and voice in real time and integrating the emotion data into video data;

[1992] a means for using a machine learning model to analyze video data including emotion data;

[1993] a means for calculating a reward based on the quantity and quality of data provided to the user and paying the reward via electronic funds transfer;

[1994] A system including:

[1995] (Claim 2)

[1996] 2. The system according to claim 1, wherein location information of the communication terminal is obtained when a user gives permission.

[1997] (Claim 3)

[1998] 10. The system of claim 1, wherein the system synchronizes location information acquired in real time with video data.

[1999] "Application example 2 when combining emotion engines"

[2000] (Claim 1)

[2001] a means for a user to capture video using a mobile device;

[2002] A means of acquiring GPS location information corresponding to the video in real time and tagging the video data;

[2003] A means for acquiring user emotion data in real time using an emotion recognition engine while shooting a video and tagging the video data;

[2004] a means for compressing the acquired video data, GPS data, and emotion data and uploading them to a server;

[2005] A means for analyzing the received video data, GPS data, and emotion data in a server and generating digital twin data using a generative AI model;

[2006] a means for calculating a reward based on the quantity and quality of the data provided to the user and paying the reward via an electronic payment means;

[2007] A system including:

[2008] (Claim 2)

[2009] 10. The system of claim 1, wherein location information of the mobile device is obtained if the user gives permission.

[2010] (Claim 3)

[2011] 10. The system of claim 1, wherein the system synchronizes real-time GPS location information with video data. [Explanation of symbols]

[2012] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for a user to capture video using a mobile device; A means of acquiring GPS location information corresponding to the video in real time and tagging the video data; a means for compressing the acquired video data and GPS data and uploading them to a server; A means for analyzing the received video data and GPS data in the server and generating digital twin data; A means for calculating a reward based on the quantity and quality of data provided to the user and paying the reward via a remittance means; A system including:

2. The system of claim 1 , wherein location information of the mobile device is obtained if the user gives permission.

3. 2. The system according to claim 1, wherein the GPS location information acquired in real time is synchronized with the video data.

Images