system

Abstract

This system aims to raise awareness among users about the dangers of using smartphones while walking and to promote safe walking habits. [Solution] A system including means for detecting user movement using an acceleration sensor, means for acquiring and monitoring changes in location information using a GPS sensor, means for displaying a warning message when it is determined that the user is operating a mobile device while moving, means for counting and recording the number of times the user is using their smartphone while walking, means for sending the counted number of times the user is using their smartphone while walking to a server at the end of the month, means for the server to aggregate the received data and record it for each user, means for sending another warning message to the user based on the number of times they are using their smartphone while walking, and means for awarding bonus points to users who do not use their smartphone while walking.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, with the spread of smartphones, so-called "walking smartphones", which are operated while walking, have become a social problem. Since a walking smartphone cannot check the surrounding situation, the risks of traffic accidents and collisions between pedestrians increase. However, at present, there are few effective countermeasures, and a method for preventing walking smartphones is required. In addition, there is a lack of a system for grasping how often each user uses a walking smartphone and giving warnings. Furthermore, it is desired to promote safe walking habits by providing appropriate incentives to users who do not use a walking smartphone.

Means for Solving the Problems

[0005] This invention is a system that detects when a user is using their smartphone while walking, displays a warning message, and counts and records the number of times this has happened. Specifically, it uses an accelerometer and a GPS sensor to monitor the user's movement and changes in location information, and displays a warning message when the user is operating their smartphone while walking. The number of times the user has used their smartphone while walking is stored on the device and sent to a server at the end of the month. The received data is aggregated on the server and recorded for each user. A follow-up warning message based on the number of times the user has used their smartphone while walking is then sent to each user. Furthermore, bonus points are awarded as an incentive to users who do not use their smartphone while walking. This system makes it possible to raise awareness among users about the dangers of using their smartphone while walking and promote safe walking habits.

[0006] An "accelerometer" is a sensor that detects the acceleration of an object and measures the changes in that acceleration.

[0007] A "GPS sensor" is a sensor that uses satellites to acquire location information on Earth.

[0008] A "user" refers to an individual using this system, or a person operating a mobile device.

[0009] A "mobile device" refers to a portable electronic device with communication capabilities, such as a mobile phone, smartphone, or tablet.

[0010] A "warning message" is a notification displayed to alert the user.

[0011] "Walking while using a smartphone" refers to the act of a user operating a mobile device while walking.

[0012] "Counting" refers to recording the number of times a particular event or action has occurred.

[0013] A "server" is a computer system used for storing and processing data.

[0014] "Aggregation" refers to the process of combining and organizing multiple data or pieces of information into a single, unified dataset or data.

[0015] "Bonus points" are rewards given to users for fulfilling specific actions or conditions. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12]It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention provides a system that detects when a user is walking while using their smartphone, issues a warning, and further collects the number of times this has happened, thereby raising awareness and providing incentives to the user. This system uses an accelerometer and GPS sensor installed in the device to detect walking while using a smartphone.

[0038] The device uses an accelerometer to detect whether the user is operating their smartphone while walking, and a GPS sensor to check for changes in location. For example, if the user is operating their smartphone while moving at regular intervals (e.g., every second), the device will determine this as walking while using a smartphone. Once this determination is made, a warning message will be displayed on the screen. A specific example of such a message might be displayed: "You are currently walking while using your smartphone. For your safety, please stop and operate your phone."

[0039] This system also counts the number of times a user walks while using their smartphone and records it as internal data. At the end of each month, this count data is sent to a server. For example, the device automatically sends the data on the number of times a user walks while using their smartphone to the server on the last day of the month.

[0040] The server aggregates the received data and records the number of times each user has used their smartphone while walking. For example, the server stores each user's smartphone usage count in a database and checks the aggregated results at the end of the month. Based on this data, a generative AI sends another warning message to the user. For example, a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," might be sent.

[0041] Furthermore, the server identifies users who did not use their smartphones while walking and rewards them with PayPay points as an incentive. Specifically, users who did not use their smartphones while walking will receive 10 PayPay points per day. This is achieved by extracting users from the database who have zero instances of using their smartphones while walking and then awarding points to those users.

[0042] As described above, the present invention detects a user's walking while using their smartphone in real time and issues an immediate warning. Furthermore, it is a system that collects data on a server at the end of the month and provides incentives to promote safe behavior such as not walking while using a smartphone.

[0043] For example, if a user uses their smartphone while walking five times in a day, the device counts each instance, and at the end of the month, it sends the total count to the server. Based on this data, the server warns the user, "You used your smartphone while walking five times this month," and notifies users who did not use their smartphone while walking, "You have been awarded 10 PayPay points for the day."

[0044] This system is an effective means of ensuring user safety and raising awareness of the dangers of operating a smartphone while walking.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The device activates its accelerometer and GPS sensor to monitor the user's movement and location. This involves measuring the user's movement with the accelerometer and checking for changes in location with the GPS sensor. Specifically, it uses AccelSensor().get_current_acceleration() and GPSSensor().get_current_position() to obtain current data.

[0048] Step 2:

[0049] The device acquires acceleration and location data again at regular intervals (e.g., every second) and determines if there are any changes. Specifically, time.sleep(1) is used to acquire and check data every second. This allows the system to determine if the user is continuously moving while operating the smartphone.

[0050] Step 3:

[0051] Based on the acquired data, the device will display a warning message on the screen if it determines that the user is using their phone while walking. For example, a message such as "You are currently using your phone while walking. Please stop and operate your phone for your safety." will be displayed using `display.show_message()`.

[0052] Step 4:

[0053] The device counts the number of times a user walks while using their phone as internal data and saves it to the device's storage area. For example, the count is incremented with `self.walk_count += 1` and saved with `local_storage.save("walk_count", self.walk_count)`.

[0054] Step 5:

[0055] At the end of the month, the device sends data on the number of times the user has walked while using their phone to the server. This includes a process that detects the end of the month and sends the data, such as `if datetime.datetime.now().day == datetime.date.today().replace(day=1).days_in_month: send_data_to_server()`.

[0056] Step 6:

[0057] The server receives data sent from the terminal and saves it to the database. Specifically, the data is saved using `database.save("user_walk_count", user_id, walk_count)`.

[0058] Step 7:

[0059] The server aggregates the data stored in the database and calculates the number of times each user walks while using their smartphone. For example, the aggregation is performed as follows: walk_count = database.get("user_walk_count", user_id).

[0060] Step 8:

[0061] Based on the aggregated results, the server generates and sends a warning message to each user corresponding to the number of times they have used their smartphone while walking. For example, it might send a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone."

[0062] Step 9:

[0063] The server extracts and identifies users who did not walk while using their smartphones. Specifically, it extracts these users from the database using `users_no_walking = database.query("SELECT user_id FROM users WHERE walk_count = 0")`.

[0064] Step 10:

[0065] The server awards bonus points to users who do not use their smartphones while walking and sends a notification message. For example, points are awarded using `for user_id in users_no_walking: grant_paypay_points(user_id, 10)` and a notification is sent stating, "10 PayPay points have been awarded for today."

[0066] (Example 1)

[0067] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0068] Using a smartphone while walking is a dangerous act for both the user and those around them, and can cause traffic accidents and falls. However, there is no system that can instantly detect when a user is operating a smartphone while walking and issue a warning. Furthermore, there is a lack of a system that collects data on users' smartphone-using behavior while walking and provides appropriate warnings and incentives based on that data. As a result, user awareness and improvement of behavior are insufficient.

[0069] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0070] In this invention, the server includes means for inputting a prompt sentence into a generating AI model based on received data and generating a warning message again, means for sending the generated warning message to the user based on the number of times the user has walked while using their smartphone, and means for awarding reward points to users who have not walked while using their smartphone. This enables users to recognize the dangers of walking while using their smartphone in real time and improve their behavior.

[0071] An "accelerometer" is a device used to detect the acceleration of an object and to measure the state of motion of that object.

[0072] A "GPS sensor" is a device that receives signals from satellites and determines the current location.

[0073] A "mobile device" is a portable electronic device that can be used while on the go, and includes smartphones, tablets, and other similar devices.

[0074] A "warning message" is a text or visual notification used to alert or warn a user.

[0075] "Counting" refers to the operation or process of counting the number of times a particular event has occurred.

[0076] A "server" is a computer system that provides various services and data processing to clients over a network.

[0077] A "database" is a software system that systematically stores data and manages it so that it can be easily searched and manipulated.

[0078] A "generative AI model" refers to an algorithm or program that uses artificial intelligence technology to automatically generate text and other content.

[0079] A "prompt statement" is an instruction or command given to a generative AI model, serving as a guideline for generating a specific output.

[0080] "Reward points" are incentives awarded to users who perform specific actions or meet certain conditions, and include those that can be used as monetary value or service benefits.

[0081] This invention is a system that detects the act of a user operating a smartphone while walking, commonly known as "walking while using a smartphone," and issues warnings and cautions in response. Furthermore, it records and aggregates the number of times a user walks while using a smartphone and provides incentives based on this information to promote improved user behavior.

[0082] The main components of the system are an accelerometer, GPS sensor, warning message display function, server, and generative AI model, all installed in the terminal.

[0083] Hardware and software to be used

[0084] 1. Accelerometer:

[0085] It is built into the device and detects the user's movement. Specifically, it is used to detect walking motion.

[0086] 2. GPS sensor:

[0087] It is built into the device and obtains the user's location information. By checking for changes in location information, it determines whether the user is moving.

[0088] 3. Terminal built-in software:

[0089] The system analyzes data from the accelerometer and GPS sensor to determine whether the user is operating their smartphone while walking. For example, if the user is moving and operating the screen every second, it is determined to be walking while using a smartphone.

[0090] 4. Warning message display function:

[0091] When walking while using a smartphone is detected, a warning message will be displayed on the screen. For example, a message such as "You are currently walking while using your smartphone. For your safety, please stop and operate your phone" will be displayed.

[0092] 5. Server:

[0093] The system receives and compiles data on the number of times users walk while using their smartphones, recording this data in a database for each user, and reviewing the results at the end of the month.

[0094] 6. Generative AI Models:

[0095] Based on the data collected by the server, prompts are entered to generate an appropriate warning message. The generated message is then sent to the user.

[0096] Example of a prompt message: "The user has used their smartphone while walking 50 times this month. Generate the following warning message: 'You have used your smartphone while walking 50 times this month. For your safety, please stop walking before using your smartphone.'"

[0097] 7. Incentive provision function:

[0098] The system extracts users from the database who have zero instances of walking while using their smartphones, and rewards these users with bonus points. For example, users who do not walk while using their smartphones will receive 10 bonus points per day.

[0099] Specific examples of system usage

[0100] When a user uses their smartphone while walking, the device's accelerometer and GPS sensor detect the movement and changes in location. The device recognizes this as walking while using a smartphone and displays a warning message on the screen: "You are currently using your smartphone while walking. For your safety, please stop and operate it."

[0101] Next, the system counts the number of times a user uses their phone while walking and records this data. At the end of the month, this data is sent to a server, which aggregates the data for each user. The server uses a generative AI model to generate a re-warning message and sends it to the user. A specific message such as, "User A used their phone while walking 5 times this month. For safety reasons, please stop walking before using your phone," is sent. In addition, users who do not use their phone while walking are rewarded with reward points.

[0102] In this way, the present invention is an effective method for ensuring user safety and raising awareness of the dangers of using a smartphone while walking, by detecting and warning against walking while using a smartphone in real time, and by providing incentives aimed at improving behavior.

[0103] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0104] Program processing flow

[0105] Step 1: Collect input data

[0106] The device collects data from its accelerometer and GPS sensor to detect whether the user is walking with their smartphone. The accelerometer detects movement, and the GPS sensor checks for changes in location.

[0107] Input: Acceleration data, GPS data

[0108] Output: Detected motion and position change data

[0109] Step 2: Detection of using a smartphone while walking

[0110] The device analyzes the collected sensor data to determine whether the user is operating their smartphone while moving at regular intervals. For example, if it detects movement and screen touches every second, it is determined to be walking while using a smartphone.

[0111] Input: Detected motion and position change data

[0112] Output: Result of detecting walking while using a smartphone

[0113] Specific action: The user walks 10 meters while touching the screen every second.

[0114] Step 3: Display of warning message

[0115] If the device detects that the user is using their phone while walking, it will immediately display a warning message on the screen. The message displayed will read, "You are currently using your phone while walking. For your safety, please stop and operate your phone."

[0116] Input: Result of detecting walking while using a smartphone

[0117] Output: Display of warning message

[0118] Specific action: The device displays the message, "You are currently using your phone while walking. For your safety, please stop and operate your phone."

[0119] Step 4: Count the number of times you use your phone while walking.

[0120] Each time the device detects that a user is walking while using their phone, it counts the number of times and records it in an internal database. This allows the device to accumulate data on how often a user has walked while using their phone.

[0121] Input: Result of detecting walking while using a smartphone

[0122] Output: Number of times people walked while using their smartphones were counted.

[0123] Specific operation: If detected 5 times in a day, the count is saved in internal memory.

[0124] Step 5: Send data on the number of times you walk while using your smartphone.

[0125] On the last day of the month, the device automatically sends accumulated data on the number of times users have walked while using their smartphone to the server.

[0126] Input: Data on the number of times people have walked while using their smartphones.

[0127] Output: Sending data to the server

[0128] Specific action: At 23:59 on the last day of the month, send data in the format "This month's number of times using a smartphone while walking is 5".

[0129] Step 6: Calculation

[0130] The server receives data sent from the terminals and records the number of times each user walks while using their smartphone in a database. The aggregated data is analyzed at the end of the month.

[0131] Input: Data on the number of times users have used their smartphones while walking.

[0132] Output: Aggregated database records

[0133] Specific action: The server records in the database that "User A used their smartphone while walking 5 times this month."

[0134] Step 7: Generate warning messages using a generative AI model.

[0135] The server inputs prompt messages into the AI ​​model, which then generates appropriate warning messages for each user.

[0136] Input: Aggregated data, prompt text

[0137] Output: Generated warning message

[0138] Specific prompt example: "The user has used their smartphone while walking 50 times this month. Please generate the following warning message: 'You have used your smartphone while walking 50 times this month. For your safety, please stop walking before using your smartphone.'"

[0139] Step 8: Sending a warning message

[0140] The server sends a warning message generated from the generated AI model to the user.

[0141] Input: Generated warning message

[0142] Output: Sending a message to the user

[0143] Specific action: A message is sent stating, "User A has used their smartphone while walking 5 times this month. For safety reasons, please stop walking before using your smartphone."

[0144] Step 9: Granting Incentives

[0145] The server extracts users from the database who have zero instances of using their smartphones while walking, and awards reward points to those users.

[0146] Input: Aggregated database list

[0147] Output: Reward points awarded

[0148] Specific action: A notification is sent stating, "User B has been awarded a total of 300 reward points."

[0149] The above outlines the specific processing steps of this system's program. This enables real-time detection, warning, and incentive provision for walking while using a smartphone, thereby ensuring user safety and promoting improvement of risky behaviors.

[0150] (Application Example 1)

[0151] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0152] The practice of using smartphones while walking, known as "walking while using a smartphone," increases the risk of traffic accidents and crime. However, currently, there is no system that not only displays warning messages but also collects data, analyzes user behavior, and issues appropriate warnings or provides incentives. Therefore, a more effective system is needed to ensure user safety and curb walking while using smartphones.

[0153] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0154] In this invention, the server includes means for sending prompt messages to the user and generating warning messages using a generation AI model, means for analyzing the user's tendency to walk while using a smartphone based on cumulative user data and generating corresponding warning content, and means for awarding bonus points to users who do not walk while using a smartphone. This makes it possible to significantly reduce the risks of walking while using a smartphone by analyzing user behavior in detail and providing appropriate warnings and incentives.

[0155] An "accelerometer" is a device that detects the acceleration of an object and is used in mobile devices to detect the user's movement.

[0156] A "GPS sensor" is a device that uses the Geographic Information System (GPS) to acquire location information and monitor changes in that location.

[0157] "Mobile devices" is a general term for electronic devices that can be carried and used, such as smartphones and tablets.

[0158] A "warning message" is a notification designed to alert a user to specific actions or situations.

[0159] "Means of recording" refers to systems and methods for storing data and information over long periods of time.

[0160] A "server" is a computer system used to store and process data on a network.

[0161] A "generative AI model" is an algorithm or model that uses machine learning and artificial intelligence technologies to automatically generate new information or messages.

[0162] A "prompt message" is a text message created by a generative AI model to encourage a specific action.

[0163] "Cumulative data" refers to the total amount of data accumulated over a certain period of time, as well as its records.

[0164] An "incentive" refers to a reward or perk offered to encourage a particular behavior.

[0165] "Means of analyzing trends" refers to methods and mechanisms for analyzing data and identifying specific patterns or trends.

[0166] "Bonus points" refer to additional points awarded to users who meet specific conditions.

[0167] This invention relates to a system that detects when a user is walking while using their smartphone, issues a warning, and further compiles the number of times this has happened, thereby providing awareness and incentives. The embodiments for carrying out this invention will be described in detail below.

[0168] composition

[0169] This system is primarily composed of the following hardware and software.

[0170] 1. Hardware

[0171] Mobile devices: Smartphones, tablets, etc.

[0172] Accelerometer sensor installed in mobile devices

[0173] GPS sensor installed in mobile devices

[0174] Servers that process and store data

[0175] 2. Software

[0176] Generative AI Models: Artificial intelligence algorithms for generating warning messages and prompt texts.

[0177] Sensor library for acquiring data from accelerometers and GPS sensors.

[0178] Python scripts and communication libraries (such as requests) for sending and analyzing data to and from the server.

[0179] operation

[0180] Detection of users walking while using their smartphones

[0181] 1. Mobile devices use accelerometers and GPS sensors to detect the user's movement.

[0182] 2. The accelerometer analyzes whether the user is walking.

[0183] 3. The GPS sensor monitors changes in location information.

[0184] If the device detects that a user is operating a mobile device while moving, it will display a warning message in real time. For example, the display might show a warning message such as, "You are currently using your phone while walking. Please stop and operate it for your safety."

[0185] Data recording and transmission

[0186] 1. The mobile device counts the number of times a person is using their smartphone while walking and records this as internal data.

[0187] 2. At the end of the month, the counted number of times a person was using their smartphone while walking is sent to the server.

[0188] Data aggregation and warning message generation

[0189] 1. The server aggregates the received data and records it for each user.

[0190] 2. Use a generative AI model to send a prompt message to the user and generate a warning message.

[0191] Specifically, the system analyzes user trends based on cumulative data of walking while using a smartphone and generates corresponding warning messages. An example of a prompt message is: "You have walked while using your smartphone XX times this month. For your safety, please stop walking before using your smartphone."

[0192] Granting of incentives

[0193] 1. The server will award bonus points to users who do not use their smartphones while walking. For example, users who do not use their smartphones while walking will receive 10 bonus points per day.

[0194] With the above configuration and operation, this invention can detect a user's walking while using a smartphone in real time, issue an immediate warning, and further provide incentives to promote safe behavior by aggregating the data at the end of the month.

[0195] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0196] Step 1:

[0197] Mobile devices detect user movement using accelerometers and GPS sensors. They acquire accelerometer and GPS data as input and analyze whether the user is walking. Based on this analysis, they obtain an output indicating that the user is moving.

[0198] Step 2:

[0199] The mobile device determines whether the user is operating their smartphone while moving. Based on acquired sensor data and screen operation status, it determines whether the user is walking while using their smartphone. Based on this determination, it outputs that the user is walking while using their smartphone.

[0200] Step 3:

[0201] The mobile device displays a warning message if it detects that the user is using their phone while walking. The input is the detection result for using a phone while walking, and a generation AI model is used to generate a prompt message. The output is a warning message that reads, "You are currently using your phone while walking. Please stop and operate it for your safety."

[0202] Step 4:

[0203] The mobile device counts the number of times a user is using their phone while walking and records this data internally. It takes the number of times walking while using a phone is detected as input and saves this information to internal memory. The stored data then outputs the number of times the user is using their phone while walking.

[0204] Step 5:

[0205] At the end of the month, the mobile device sends the counted number of times a user has walked while using their smartphone to the server. As input, it retrieves the count data stored internally and sends it to the server. As a result of this transmission, it receives output indicating that the count data has been saved on the server.

[0206] Step 6:

[0207] The server aggregates the received data and records it for each user. It receives data on the number of times a user has used their phone while walking as input and stores it in a database for each user. The stored data is then output as information on the number of times each user has used their phone while walking.

[0208] Step 7:

[0209] The server uses a generative AI model to send prompt messages to users and generate warning messages. It takes user data on the number of times they've used their phone while walking as input and generates a prompt message such as, "You've used your phone while walking XX times this month. For safety reasons, please stop before using your phone." This generated warning message is output.

[0210] Step 8:

[0211] The server awards bonus points to users who do not use their smartphones while walking. It retrieves user information for those who have used their smartphones while walking 0 times as input, and awards 10 bonus points per day to these users. The output is the result of this point awarding process.

[0212] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0213] This invention is a system that detects when a user is walking while using their smartphone, recognizes the user's emotions, and then issues a warning message. In addition to a basic walking / smartphone detection function using an accelerometer and GPS sensor, this system incorporates an emotion engine.

[0214] Detection of walking while using a smartphone and display of warning messages.

[0215] The device monitors the user's movement and changes in location using an accelerometer and a GPS sensor. Specifically, the accelerometer measures the user's movement, and the GPS sensor confirms changes in location. If the device determines that the user is operating their smartphone while moving at regular intervals, it considers this to be walking while using a smartphone. For example, if the user operates their smartphone while walking for one second, it will be detected.

[0216] Emotion recognition and message adjustment using an emotion engine.

[0217] If a user is walking while using their smartphone, the device activates its emotion engine to identify the user's current emotional state through facial recognition and voice analysis. For example, it uses the camera to analyze the user's facial expressions and estimates emotions from voice input. Based on this emotional information, the content and timing of warning messages are adjusted in real time.

[0218] For example, if a user is feeling stressed, the standard warning message will be changed to something like, "You seem tired today. For your safety, please take a short break before using your smartphone." This ensures that appropriate warnings are given, taking the user's feelings into consideration.

[0219] Counting and sending the number of times people walk while using their smartphones.

[0220] The device counts the number of times a user walks while using their smartphone and records this data internally. At the end of each month, this count data is sent to a server. For example, the device automatically sends the data on the number of times a user walks while using their smartphone to the server on the last day of the month.

[0221] Data aggregation by the server and a re-emphasis of warnings.

[0222] The server receives data sent from the terminal and stores it in a database. The received data is aggregated, and the number of times each user has used their smartphone while walking is calculated. Based on this, a generative AI sends another warning message to the user. For example, a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," is sent.

[0223] Bonus points awarded

[0224] Furthermore, the server identifies users who did not use their smartphones while walking and awards them bonus points as an incentive. For example, users who did not use their smartphones while walking receive 10 bonus points per day. This is achieved by extracting users with zero instances of using their smartphones while walking from the database and awarding points to those users.

[0225] Bonus point adjustment based on user emotions

[0226] The emotion engine also has a function that adjusts the bonus points a user receives according to their emotional state. For example, if a user is in a positive emotional state, they may be awarded more points than the standard amount.

[0227] As described above, the present invention is a system that detects a user's walking while using their smartphone in real time, provides appropriate warnings that take emotions into consideration, and offers incentives to promote safe behavior. For example, if a user walks while using their smartphone five times in a day, the emotion engine recognizes the user's state and displays an appropriate message, and sends this data to the server at the end of the month. The server aggregates the data and provides further warnings and bonus points.

[0228] The following describes the processing flow.

[0229] Step 1:

[0230] The device activates its accelerometer and GPS sensor to monitor the user's movement and location. The accelerometer measures the user's movement, and the GPS sensor confirms changes in position. Specifically, data is obtained using `AccelSensor().get_current_acceleration()` and `GPSSensor().get_current_position()`.

[0231] Step 2:

[0232] The device updates acceleration data and location data at regular intervals (for example, every second) to determine whether movement and changes in location meet certain conditions. Specifically, time.sleep(1) is used to acquire and check data every second.

[0233] Step 3:

[0234] If the device determines, based on the acquired data, that the user is using their smartphone while walking, it will display a warning message on the screen. For example, it might display a message such as, "You are currently using your smartphone while walking. Please stop and operate it for your safety," using display.show_message().

[0235] Step 4:

[0236] The device simultaneously activates an emotion engine and uses sensors such as cameras and microphones to detect the user's emotional state. For example, this may involve using a facial recognition camera and a voice recognition module to identify emotions from the user's facial expressions and voice.

[0237] Step 5:

[0238] The device adjusts the content and timing of warning messages in real time based on the user's emotions detected by the emotion engine. For example, if the user is feeling stressed, it will display a message such as, "You seem tired today. For your safety, please take a short break before using your smartphone."

[0239] Step 6:

[0240] The device counts the number of times the user is using their phone while walking as internal data and saves it to the device's storage area. For example, the count is incremented with `self.walk_count += 1` and the data is saved with `local_storage.save("walk_count", self.walk_count)`.

[0241] Step 7:

[0242] At the end of the month, the device sends the counted number of times a user has walked while using their phone to the server. Specifically, it uses `if datetime.datetime.now().day == datetime.date.today().replace(day=1).days_in_month: send_data_to_server()` to detect the end of the month and send the data.

[0243] Step 8:

[0244] The server receives data sent from the terminal and saves it to the database. The received data is saved using database.save("user_walk_count", user_id, walk_count).

[0245] Step 9:

[0246] The server aggregates the data and calculates the number of times each user walks while using their smartphone. For example, the data is extracted and aggregated using `walk_count = database.get("user_walk_count", user_id)`.

[0247] Step 10:

[0248] Based on the aggregated results, the server generates and sends a warning message to each user corresponding to the number of times they have used their smartphone while walking. For example, it might send a message such as, "You used your smartphone while walking 5 times this month. For your safety, please stop walking before using your smartphone."

[0249] Step 11:

[0250] The server identifies users who did not walk while using their smartphones and awards bonus points as an incentive to those identified. For example, it identifies target users using `users_no_walking = database.query("SELECT user_id FROM users WHERE walk_count = 0")` and awards points to those users.

[0251] Step 12:

[0252] The server uses an emotion engine to adjust bonus point allocation based on the user's emotions. Users with positive emotions may receive bonus points above the standard rate. For example, if a user expresses positive emotions, they may receive double the usual points.

[0253] (Example 2)

[0254] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0255] Conventional technologies lacked sufficient means to warn users of the dangers of walking while using a smartphone, and failed to provide warning messages that took into account the user's emotional state. Furthermore, the lack of a system to record the number of times users walked while using their smartphones and provide feedback to the user made it difficult to encourage improved behavior. In addition, there was no detailed system for providing incentives for refraining from walking while using a smartphone. As a result, it was difficult to continuously encourage safe behavior among users.

[0256] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0257] In this invention, the server includes means for detecting the user's movement using an acceleration sensor, means for acquiring and monitoring changes in location information using a location information acquisition device, means for displaying a warning message when it is determined that the user is operating a mobile device while moving, means for recognizing the user's emotions using facial recognition and voice analysis, means for adjusting the content of the warning message based on the recognized emotion information, means for counting and recording the number of times the user is using their smartphone while walking, means for transmitting the counted number of times the user is using their smartphone while walking to a recording device at the end of the month, means for the recording device to aggregate the received data and record it for each user, means for sending another warning message to the user based on the number of times the user is using their smartphone while walking, means for awarding reward points to users who do not use their smartphone while walking, and means for adjusting reward points based on the user's emotion information. This makes it possible to provide warning messages that take into account the user's emotional state, record and provide feedback on the number of times the user is using their smartphone while walking, and continuously encourage safe behavior by the user.

[0258] An "accelerometer" is a device that measures the user's speed and direction of movement.

[0259] A "location information acquisition device" is a device used to monitor and track a user's location.

[0260] A "mobile device" is an electronic device that a user can carry around and use for communication and information processing.

[0261] A "warning message" is a message displayed to alert the user.

[0262] "Facial recognition" is a technology that analyzes a user's facial expressions captured by a camera to identify their emotions.

[0263] "Voice analysis" is a technology that analyzes a user's voice collected through a microphone to identify their emotions.

[0264] "Emotional information" refers to data that indicates the user's current emotional state.

[0265] A "recording device" is a device used to store and save data.

[0266] "Reward points" are incentive points awarded to users for specific actions.

[0267] This invention is a system that detects when a user is operating a smartphone while walking, recognizes their emotions at that time, and issues an appropriate warning message. This system includes an accelerometer, a location information acquisition device, an emotion engine, and a server.

[0268] Hardware configuration

[0269] 1. An accelerometer (e.g., MPU-6050) is a device that measures the user's movement speed and direction.

[0270] 2. A location information acquisition device (e.g., U-blox NEO-6M) is a device for monitoring and tracking the user's location.

[0271] 3. A mobile device is an electronic device that a user can carry with them and performs communication and information processing, and includes a built-in camera and microphone.

[0272] 4. A server is a device for accumulating and storing data.

[0273] Software Configuration

[0274] 5. Facial recognition technology (e.g., OpenCV and dlib) is a technology that analyzes a user's facial expressions captured by a camera to identify their emotions.

[0275] 6. Speech analysis technology (for example, Google® Cloud Speech-to-Text) is a technology that analyzes a user's voice collected by a microphone to identify their emotions.

[0276] 7. The emotion engine is a software module that performs facial recognition and voice analysis to identify the user's emotional state.

[0277] System Functions

[0278] Detection of using a smartphone while walking

[0279] The device uses an accelerometer and a location information acquisition device to monitor the user's movement and changes in location. Next, the device analyzes the collected data at regular intervals and recognizes it as "walking while using a smartphone" if it determines that the user is operating their smartphone while walking.

[0280] Emotion recognition and adjustment of warning messages

[0281] When the user is recognized as walking while using the smartphone, the terminal activates the emotion engine. The terminal uses the camera and microphone to collect the user's expressions and voice, and analyzes them with the emotion engine. According to the user's emotional state, the terminal generates the content of an appropriate warning message and displays it on the display device of the terminal.

[0282] Specific example

[0283] For example, when the user is operating the smartphone on the platform of a station, the acceleration sensor (MPU - 6050) and the position information acquisition device (U - blox NEO - 6M) of the terminal detect the user's movement. If it is determined that the user is operating the smartphone while walking for one second, the terminal displays a warning message saying, "Walking while using the smartphone is dangerous. Please stop and then operate it." Also, if it is analyzed that the user is feeling stressed, the content is adjusted to something like, "You seem tired today. For safety, please take a little rest and then operate the smartphone."

[0284] Recording and sending the number of times of walking while using the smartphone

[0285] Every time the terminal detects walking while using the smartphone, it records the number of times in the internal database. On the last day of the month, the terminal sends the recorded number of times of walking while using the smartphone to the server.

[0286] Data aggregation by the server and resending of warning messages

[0287] The server receives the data on the number of times of walking while using the smartphone sent from the terminal and saves it in the database. The server aggregates the number of times of walking while using the smartphone for each user, generates a warning message again using the generated AI model, and sends it. As an example, a message like, "You walked while using the smartphone XX times this month. For safety, please use the smartphone after stopping." can be considered.

[0288] Granting of bonus points and adjustment according to emotion

[0289] The server extracts users from the database who have zero instances of walking while using their smartphones and awards them reward points as an incentive. Furthermore, it adjusts the reward points based on the user's emotional state. For example, users with a positive emotional state are awarded points above the standard rate.

[0290] Examples of prompts for generative AI models

[0291] The following are possible inputs for a generative AI model.

[0292] Let's assume the system detects that the user is walking while using their smartphone. Using data from the accelerometer and location information acquisition device, the system uses an emotion engine to recognize the user's emotions (through facial expressions and voice analysis). If the user is experiencing stress, generate an appropriate warning message.

[0293] Thus, the system of the present invention can continuously promote safe behavior by providing real-time alerts and incentives that take into account the user's emotional state.

[0294] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0295] Step 1:

[0296] The device detects that the user is operating a smartphone.

[0297] Input: User's smartphone operation (touchscreen use, app launching, etc.)

[0298] Data processing: None

[0299] Output: Smartphone operation detection signal

[0300] Specific operation: A system service on the device monitors the smartphone's operation events.

[0301] Step 2:

[0302] The terminal monitors the user's movement and location information using an acceleration sensor and a location information acquisition device.

[0303] Input: Data from the acceleration sensor (MPU-6050), data from the location information acquisition device (U-blox NEO-6M)

[0304] Data processing: Real-time analysis of acceleration data and location information

[0305] Output: Amount of change in movement and location information

[0306] Specific operation: The terminal collects acceleration data at regular intervals and acquires location information. The collected data is analyzed to confirm the presence or absence of movement and location changes.

[0307] Step 3:

[0308] The terminal determines whether the user is operating the smartphone while walking.

[0309] Input: Amount of change in movement and location information, detection signal of smartphone operation

[0310] Data processing: Analysis of data over a certain period (e.g., 1 second)

[0311] Output: Judgment result (true / false) of walking while using the smartphone [[ID=4​​​​​​​​​​​​​​

[0316] Data processing: None

[0317] Output: Command to display warning message

[0318] Specific action: The device displays a warning message on its screen (e.g., "Using your phone while walking is dangerous. Please stop before using it.").

[0319] Step 5:

[0320] The device activates an emotion engine when it detects someone walking while using their smartphone, and recognizes the user's emotions.

[0321] Input: Camera video (facial expression data), microphone audio (audio data)

[0322] Data processing: Analysis of facial expression data and voice data (using OpenCV, dlib, and Google Cloud Speech-to-Text).

[0323] Output: Emotional information (e.g., stress, joy, etc.)

[0324] Specific operation: The device captures the user's face with its camera and collects audio with its microphone. This data is then analyzed by an emotion engine to recognize the user's emotions.

[0325] Step 6:

[0326] The device adjusts the content of the warning message based on the emotional information it recognizes.

[0327] Input: Sentiment information, existing warning messages

[0328] Data processing: Generating messages based on emotional information

[0329] Output: Adjusted warning message

[0330] Specific action: For example, if the user is feeling stressed, a message will be displayed saying, "You seem tired today. For your safety, please take a short break before using your smartphone."

[0331] Step 7:

[0332] The device counts the number of times a user walks while using their smartphone and records the data in an internal database.

[0333] Input: Result of detecting walking while using a smartphone

[0334] Data processing: Incrementing the count, recording to the database.

[0335] Output: Updated number of times people have walked while using their smartphones.

[0336] Specific action: Each time walking while using a smartphone is detected, the recorded count increases by one.

[0337] Step 8:

[0338] At the end of the month, the device sends data on the number of times the user has walked while using their smartphone to the server.

[0339] Input: Data on the number of times people walk while using their smartphones.

[0340] Data processing: Data format conversion, application of transmission protocols.

[0341] Output: Sending data to the server

[0342] Specific operation: On the last day of the month, the device converts the counted number of times a user has walked while using their phone into an appropriate format and sends it to the server.

[0343] Step 9:

[0344] The server aggregates the received data and records it for each user.

[0345] Input: Data on the number of times people walk while using their smartphones.

[0346] Data processing: Data aggregation and recording for each user.

[0347] Output: Aggregated data on walking while using a smartphone

[0348] Specific operation: The server aggregates the received data for each user and stores it in the database.

[0349] Step 10:

[0350] The server sends another warning message to the user based on the number of times they have used their phone while walking.

[0351] Input: Aggregated data on walking while using a smartphone

[0352] Data processing: Generating warning messages using a generative AI model.

[0353] Output: Sending a re-warning message

[0354] Specific operation: Based on the number of times each user walks while using their smartphone, the server uses a generative AI model to generate an appropriate warning message and sends it to the user.

[0355] Step 11:

[0356] The server will award reward points to users who do not use their smartphones while walking.

[0357] Input: Aggregated data on walking while using a smartphone

[0358] Data processing: Calculation and awarding of reward points

[0359] Output: User accounts to which reward points have been awarded

[0360] Specific operation: The server calculates reward points as an incentive for users who have zero instances of using their smartphones while walking during the month, and adds them to the user's account.

[0361] Step 12:

[0362] The server adjusts reward points based on the user's sentiment information.

[0363] Input: Sentiment information, reward point data

[0364] Data processing: Adjusting reward points based on emotional information

[0365] Output: Adjusted reward points

[0366] Specific action: Users in a positive emotional state will be awarded points above the standard level, and their reward points will be adjusted accordingly.

[0367] (Application Example 2)

[0368] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0369] Conventional pedestrian detection systems have faced challenges in effectively detecting "walking while using a smartphone" behavior and issuing appropriate warnings. In particular, they lacked the technology to send messages that took into account the user's emotional state, making them ineffective in improving user responses and behavior. Furthermore, they lacked means of providing incentives to promote safe behavior.

[0370] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0371] In this invention, the server includes means for detecting the user's movement using an acceleration sensor, means for acquiring and monitoring changes in location information using a GPS sensor, means for displaying a warning message when it is determined that the user is operating a mobile device while moving, means for counting and recording the number of times the user is using their smartphone while walking, means for sending the counted number of times the user is using their smartphone while walking to the server at the end of the month, means for the server to aggregate the received data and record it for each user, means for sending another warning message to the user based on the number of times the user is using their smartphone while walking, means for awarding bonus points to users who do not use their smartphone while walking, means for analyzing the user's emotional state from their facial expressions and voice and adjusting the warning message accordingly, and means for adjusting bonus points according to the user's emotional state. This makes it possible to provide appropriate warning messages tailored to the user's current emotional state and to provide incentives to curb smartphone use while walking.

[0372] An "accelerometer" is a device used to detect the acceleration of a user's movements and actions.

[0373] A "GPS sensor" is a device used to acquire location information on Earth and monitor changes in that location.

[0374] A "mobile device" is an electronic device that a user carries and uses, and specifically refers to a smartphone.

[0375] A "warning message" is a notification designed to remind users to stop using their smartphones while walking.

[0376] "Means of counting" refers to the devices or functions necessary to record the number of times someone is walking while using their smartphone.

[0377] A "server" is a computer system that collects and manages data and sends warning messages and bonus points to users.

[0378] "Means of aggregation" refers to a function that allows the server to organize the data it receives and compile statistics for each user.

[0379] "Bonus points" are points awarded to users as an incentive, and are granted when certain conditions are met.

[0380] "Emotional state" refers to the user's psychological state at that time, as analyzed from their facial expressions and voice.

[0381] "Means of analyzing facial expressions and voice" refer to devices and software that read emotions from a user's facial movements and voice.

[0382] "Means of adjustment" refers to functions that change the content of messages or the amount of points awarded based on specific conditions.

[0383] This invention is a system that detects when a user is walking while using their smartphone, recognizes the user's emotions, and then issues a warning message. In addition to a basic walking / smartphone detection function using an accelerometer and GPS sensor, this system incorporates an emotion engine.

[0384] Terminal-side embodiment

[0385] The device monitors the user's movement and changes in location using an accelerometer and GPS sensor. Specifically, the accelerometer measures the user's movement, and the GPS sensor confirms changes in location. If the device determines that the user is operating their smartphone while moving at regular intervals, it considers this to be walking while using a smartphone. For example, if a user is walking while operating their smartphone for one second, this will be detected. At this time, an emotion engine is activated that uses the camera and microphone to analyze facial expressions and voice in order to recognize the user's emotional state. The emotion engine analyzes the user's facial expression and voice data in real time to identify the user's current emotional state.

[0386] Display of warning message

[0387] When a user is detected walking while using their smartphone, the emotion engine analyzes the user's emotional state and generates a warning message based on that analysis. For example, if the user is feeling stressed, a message such as, "You seem tired today. For your safety, please take a short break before using your smartphone," will be displayed. For other emotional states, a standard warning message will be displayed.

[0388] Counting the number of times people walk while using their smartphones and transmitting data.

[0389] The device counts the number of times a user walks while using their smartphone and records this data internally. At the end of each month, this data is automatically sent to the server. Specifically, on the last day of the month, the device sends the data on the number of times a user walks while using their smartphone to the server, which receives it and stores it in its database.

[0390] The server aggregates the data and issues another warning.

[0391] The server aggregates the data sent from the terminals and calculates the number of times each user has used their smartphone while walking. Based on this, the server sends another warning message to the user. For example, a message like, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," is possible.

[0392] Bonus points awarded

[0393] The server has a function to identify users who do not use their smartphones while walking and to award them bonus points as an incentive. It extracts users from the database who have zero instances of using their smartphones while walking and awards them 10 bonus points per day. Furthermore, if a user's emotional state is positive, they may be awarded more points than the standard amount.

[0394] Hardware and software to be used

[0395] To implement this system, a mobile device with a built-in accelerometer and GPS sensor is required, as well as facial recognition software and voice analysis software to act as the emotion engine. Specifically, a general-purpose facial recognition library and voice analysis tool can be used for emotion recognition. In addition, a database management system and data aggregation software are required on the server.

[0396] Examples of specific cases and prompt statements

[0397] As a concrete example, consider a system where an autonomous vehicle detects pedestrians wearing smart glasses and issues a warning if they are using their smartphones while walking. The following is an example of a prompt to be input to the generating AI model:

[0398] Example of a prompt:

[0399] Please provide a code example for building a system that detects when a user is walking while using their smartphone and sends the most appropriate warning message based on their emotions. Please include the following sections:

[0400] 1. Code to acquire data from the accelerometer and GPS sensor.

[0401] 2. Code that recognizes the user's emotions and adjusts the warning message accordingly.

[0402] 3. Code to send a warning message to the user.

[0403] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0404] Step 1:

[0405] The device acquires data from the accelerometer and GPS sensor.

[0406] Input: Device's accelerometer, GPS sensor

[0407] Data processing: Collection of acceleration data and location data.

[0408] Output: User movement and location change information

[0409] Specific operation: The device's built-in accelerometer measures the user's movement data (accelerometer data), and the GPS sensor acquires the user's location information. This data is collected and used as basic data necessary for detecting walking while using a smartphone.

[0410] Step 2:

[0411] The device detects when the user is operating their smartphone while on the move.

[0412] Input: Acceleration data, location data

[0413] Data processing: Comparison of movement patterns and device operations at regular intervals.

[0414] Output: Detection results for walking while using a smartphone

[0415] Specific operation: The device determines whether the user is operating their smartphone while moving, based on acceleration data and location data collected at regular intervals. If it is determined that the user is using their smartphone while walking, that information is sent to the next step.

[0416] Step 3:

[0417] The device activates an emotion engine that analyzes the user's facial expressions and voice.

[0418] Input: User facial expression data (camera), audio data (microphone)

[0419] Data processing: facial expression recognition, voice analysis

[0420] Output: User's emotional state

[0421] Specific operation: The system uses the camera and microphone built into the device to acquire user facial expression and voice data. By analyzing this data, the system identifies the user's emotional state. The emotion engine uses, for example, a facial recognition library or a voice analysis tool.

[0422] Step 4:

[0423] The device generates and displays warning messages based on the user's emotional state.

[0424] Input: Emotional state, detection results of walking while using a smartphone

[0425] Data processing: Adjusting message content based on emotional state.

[0426] Output: Warning message

[0427] Specific operation: The device adjusts the content of the warning message based on the detected emotional state. For example, if the user is feeling stressed, it will generate a gentle message. The generated message will be displayed on the device's screen.

[0428] Step 5:

[0429] The device counts and records the number of times you use your smartphone while walking.

[0430] Input: Detection results of walking while using a smartphone

[0431] Data processing: Counting the number of times a person walks while using their smartphone, and recording it in the internal data.

[0432] Output: Count data

[0433] Specific operation: The device counts the number of times a user is using their phone while walking and records this data in its internal storage. The recorded data is later sent to a server.

[0434] Step 6:

[0435] At the end of each month, the device sends the counted data to the server.

[0436] Input: Data on the number of times people walk while using their smartphones.

[0437] Data processing: Preparing and sending data for transmission.

[0438] Output: Data sent to the server

[0439] Specific operation: At the end of each month, the device automatically sends data on the number of times the user is using their smartphone while walking to the server. This transmission process uses network communication functionality.

[0440] Step 7:

[0441] The server aggregates the data it receives and records it for each user.

[0442] Input: Data on the number of times people walk while using their smartphones.

[0443] Data processing: Data aggregation, creation of user-specific statistical data.

[0444] Output: Data on the number of times users walk while using their smartphones.

[0445] Specific operation: The server stores data on the number of times a user walks while using their smartphone, received from the terminal, into a database and generates statistical data for each user. Based on this, individual user reports are created.

[0446] Step 8:

[0447] The server will send the warning message to the user again.

[0448] Input: User-specific data on the number of times a user walks while using their smartphone.

[0449] Data processing: Generating warning messages

[0450] Output: Warning message

[0451] Specific operation: The server generates a warning message again based on each user's data on the number of times they use their smartphone while walking, and sends it to the user. The message is sent via smartphone or email.

[0452] Step 9:

[0453] The server will award bonus points to users who do not use their smartphones while walking.

[0454] Input: User-specific data on the number of times a user walks while using their smartphone.

[0455] Data processing: Calculation and awarding of bonus points

[0456] Output: Bonus Points

[0457] Specific operation: The server calculates bonus points for users who have zero instances of using their smartphones while walking, and records this in the database. As an incentive, the bonus points are added to the user's account.

[0458] Step 10:

[0459] The server adjusts bonus points based on the user's emotional state.

[0460] Input: Emotional state data, bonus points

[0461] Data processing: Adjustment of bonus points

[0462] Output: Adjusted bonus points

[0463] Specific operation: The server considers the user's emotional state data and adjusts the bonus points awarded to users in a positive emotional state to be above the standard level. The adjusted points are recorded in the database.

[0464] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0465] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0466] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0467] [Second Embodiment]

[0468] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0469] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0470] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0471] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0472] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0473] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0474] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0475] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0476] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0477] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0478] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0479] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0480] This invention provides a system that detects when a user is walking while using their smartphone, issues a warning, and further collects the number of times this has happened, thereby raising awareness and providing incentives to the user. This system uses an accelerometer and GPS sensor installed in the device to detect walking while using a smartphone.

[0481] The device uses an accelerometer to detect whether the user is operating their smartphone while walking, and a GPS sensor to check for changes in location. For example, if the user is operating their smartphone while moving at regular intervals (e.g., every second), the device will determine this as walking while using a smartphone. Once this determination is made, a warning message will be displayed on the screen. A specific example of such a message might be displayed: "You are currently walking while using your smartphone. For your safety, please stop and operate your phone."

[0482] This system also counts the number of times a user walks while using their smartphone and records it as internal data. At the end of each month, this count data is sent to a server. For example, the device automatically sends the data on the number of times a user walks while using their smartphone to the server on the last day of the month.

[0483] The server aggregates the received data and records the number of times each user has used their smartphone while walking. For example, the server stores each user's smartphone usage count in a database and checks the aggregated results at the end of the month. Based on this data, a generative AI sends another warning message to the user. For example, a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," might be sent.

[0484] Furthermore, the server identifies users who did not use their smartphones while walking and rewards them with PayPay points as an incentive. Specifically, users who did not use their smartphones while walking will receive 10 PayPay points per day. This is achieved by extracting users from the database who have zero instances of using their smartphones while walking and then awarding points to those users.

[0485] As described above, the present invention detects a user's walking while using their smartphone in real time and issues an immediate warning. Furthermore, it is a system that collects data on a server at the end of the month and provides incentives to promote safe behavior such as not walking while using a smartphone.

[0486] For example, if a user uses their smartphone while walking five times in a day, the device counts each instance, and at the end of the month, it sends the total count to the server. Based on this data, the server warns the user, "You used your smartphone while walking five times this month," and notifies users who did not use their smartphone while walking, "You have been awarded 10 PayPay points for the day."

[0487] This system is an effective means of ensuring user safety and raising awareness of the dangers of operating a smartphone while walking.

[0488] The following describes the processing flow.

[0489] Step 1:

[0490] The device activates its accelerometer and GPS sensor to monitor the user's movement and location. This involves measuring the user's movement with the accelerometer and checking for changes in location with the GPS sensor. Specifically, it uses AccelSensor().get_current_acceleration() and GPSSensor().get_current_position() to obtain current data.

[0491] Step 2:

[0492] The device acquires acceleration and location data again at regular intervals (e.g., every second) and determines if there are any changes. Specifically, time.sleep(1) is used to acquire and check data every second. This allows the system to determine if the user is continuously moving while operating the smartphone.

[0493] Step 3:

[0494] Based on the acquired data, the device will display a warning message on the screen if it determines that the user is using their phone while walking. For example, a message such as "You are currently using your phone while walking. Please stop and operate your phone for your safety." will be displayed using `display.show_message()`.

[0495] Step 4:

[0496] The device counts the number of times a user walks while using their phone as internal data and saves it to the device's storage area. For example, the count is incremented with `self.walk_count += 1` and saved with `local_storage.save("walk_count", self.walk_count)`.

[0497] Step 5:

[0498] At the end of the month, the device sends data on the number of times the user has walked while using their phone to the server. This includes a process that detects the end of the month and sends the data, such as `if datetime.datetime.now().day == datetime.date.today().replace(day=1).days_in_month: send_data_to_server()`.

[0499] Step 6:

[0500] The server receives data sent from the terminal and saves it to the database. Specifically, the data is saved using `database.save("user_walk_count", user_id, walk_count)`.

[0501] Step 7:

[0502] The server aggregates the data stored in the database and calculates the number of times each user walks while using their smartphone. For example, the aggregation is performed as follows: walk_count = database.get("user_walk_count", user_id).

[0503] Step 8:

[0504] Based on the aggregated results, the server generates and sends a warning message to each user corresponding to the number of times they have used their smartphone while walking. For example, it might send a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone."

[0505] Step 9:

[0506] The server extracts and identifies users who did not walk while using their smartphones. Specifically, it extracts these users from the database using `users_no_walking = database.query("SELECT user_id FROM users WHERE walk_count = 0")`.

[0507] Step 10:

[0508] The server awards bonus points to users who do not use their smartphones while walking and sends a notification message. For example, points are awarded using `for user_id in users_no_walking: grant_paypay_points(user_id, 10)` and a notification is sent stating, "10 PayPay points have been awarded for today."

[0509] (Example 1)

[0510] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0511] Using a smartphone while walking is a dangerous act for both the user and those around them, and can cause traffic accidents and falls. However, there is no system that can instantly detect when a user is operating a smartphone while walking and issue a warning. Furthermore, there is a lack of a system that collects data on users' smartphone-using behavior while walking and provides appropriate warnings and incentives based on that data. As a result, user awareness and improvement of behavior are insufficient.

[0512] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0513] In this invention, the server includes means for inputting a prompt sentence into a generating AI model based on received data and generating a warning message again, means for sending the generated warning message to the user based on the number of times the user has walked while using their smartphone, and means for awarding reward points to users who have not walked while using their smartphone. This enables users to recognize the dangers of walking while using their smartphone in real time and improve their behavior.

[0514] An "accelerometer" is a device used to detect the acceleration of an object and to measure the state of motion of that object.

[0515] A "GPS sensor" is a device that receives signals from satellites and determines the current location.

[0516] A "mobile device" is a portable electronic device that can be used while on the go, and includes smartphones, tablets, and other similar devices.

[0517] A "warning message" is a text or visual notification used to alert or warn a user.

[0518] "Counting" refers to the operation or process of counting the number of times a particular event has occurred.

[0519] A "server" is a computer system that provides various services and data processing to clients over a network.

[0520] A "database" is a software system that systematically stores data and manages it so that it can be easily searched and manipulated.

[0521] A "generative AI model" refers to an algorithm or program that uses artificial intelligence technology to automatically generate text and other content.

[0522] A "prompt statement" is an instruction or command given to a generative AI model, serving as a guideline for generating a specific output.

[0523] "Reward points" are incentives awarded to users who perform specific actions or meet certain conditions, and include those that can be used as monetary value or service benefits.

[0524] This invention is a system that detects the act of a user operating a smartphone while walking, commonly known as "walking while using a smartphone," and issues warnings and cautions in response. Furthermore, it records and aggregates the number of times a user walks while using a smartphone and provides incentives based on this information to promote improved user behavior.

[0525] The main components of the system are an accelerometer, GPS sensor, warning message display function, server, and generative AI model, all installed in the terminal.

[0526] Hardware and software to be used

[0527] 1. Accelerometer:

[0528] It is built into the device and detects the user's movement. Specifically, it is used to detect walking motion.

[0529] 2. GPS sensor:

[0530] It is built into the device and obtains the user's location information. By checking for changes in location information, it determines whether the user is moving.

[0531] 3. Terminal built-in software:

[0532] The system analyzes data from the accelerometer and GPS sensor to determine whether the user is operating their smartphone while walking. For example, if the user is moving and operating the screen every second, it is determined to be walking while using a smartphone.

[0533] 4. Warning message display function:

[0534] When walking while using a smartphone is detected, a warning message will be displayed on the screen. For example, a message such as "You are currently walking while using your smartphone. For your safety, please stop and operate your phone" will be displayed.

[0535] 5. Server:

[0536] The system receives and compiles data on the number of times users walk while using their smartphones, recording this data in a database for each user, and reviewing the results at the end of the month.

[0537] 6. Generative AI Models:

[0538] Based on the data collected by the server, prompts are entered to generate an appropriate warning message. The generated message is then sent to the user.

[0539] Example of a prompt message: "The user has used their smartphone while walking 50 times this month. Generate the following warning message: 'You have used your smartphone while walking 50 times this month. For your safety, please stop walking before using your smartphone.'"

[0540] 7. Incentive provision function:

[0541] The system extracts users from the database who have zero instances of walking while using their smartphones, and rewards these users with bonus points. For example, users who do not walk while using their smartphones will receive 10 bonus points per day.

[0542] Specific examples of system usage

[0543] When a user uses their smartphone while walking, the device's accelerometer and GPS sensor detect the movement and changes in location. The device recognizes this as walking while using a smartphone and displays a warning message on the screen: "You are currently using your smartphone while walking. For your safety, please stop and operate it."

[0544] Next, the system counts the number of times a user uses their phone while walking and records this data. At the end of the month, this data is sent to a server, which aggregates the data for each user. The server uses a generative AI model to generate a re-warning message and sends it to the user. A specific message such as, "User A used their phone while walking 5 times this month. For safety reasons, please stop walking before using your phone," is sent. In addition, users who do not use their phone while walking are rewarded with reward points.

[0545] In this way, the present invention is an effective method for ensuring user safety and raising awareness of the dangers of using a smartphone while walking, by detecting and warning against walking while using a smartphone in real time, and by providing incentives aimed at improving behavior.

[0546] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0547] Program processing flow

[0548] Step 1: Collect input data

[0549] The device collects data from its accelerometer and GPS sensor to detect whether the user is walking with their smartphone. The accelerometer detects movement, and the GPS sensor checks for changes in location.

[0550] Input: Acceleration data, GPS data

[0551] Output: Detected motion and position change data

[0552] Step 2: Detection of using a smartphone while walking

[0553] The device analyzes the collected sensor data to determine whether the user is operating their smartphone while moving at regular intervals. For example, if it detects movement and screen touches every second, it is determined to be walking while using a smartphone.

[0554] Input: Detected motion and position change data

[0555] Output: Result of detecting walking while using a smartphone

[0556] Specific action: The user walks 10 meters while touching the screen every second.

[0557] Step 3: Display of warning message

[0558] If the device detects that the user is using their phone while walking, it will immediately display a warning message on the screen. The message displayed will read, "You are currently using your phone while walking. For your safety, please stop and operate your phone."

[0559] Input: Result of detecting walking while using a smartphone

[0560] Output: Display of warning message

[0561] Specific action: The device displays the message, "You are currently using your phone while walking. For your safety, please stop and operate your phone."

[0562] Step 4: Count the number of times you use your phone while walking.

[0563] Each time the device detects that a user is walking while using their phone, it counts the number of times and records it in an internal database. This allows the device to accumulate data on how often a user has walked while using their phone.

[0564] Input: Result of detecting walking while using a smartphone

[0565] Output: Number of times people walked while using their smartphones were counted.

[0566] Specific operation: If detected 5 times in a day, the count is saved in internal memory.

[0567] Step 5: Send data on the number of times you walk while using your smartphone.

[0568] On the last day of the month, the device automatically sends accumulated data on the number of times users have walked while using their smartphone to the server.

[0569] Input: Data on the number of times people have walked while using their smartphones.

[0570] Output: Sending data to the server

[0571] Specific action: At 23:59 on the last day of the month, send data in the format "This month's number of times using a smartphone while walking is 5".

[0572] Step 6: Calculation

[0573] The server receives data sent from the terminals and records the number of times each user walks while using their smartphone in a database. The aggregated data is analyzed at the end of the month.

[0574] Input: Data on the number of times users have used their smartphones while walking.

[0575] Output: Aggregated database records

[0576] Specific action: The server records in the database that "User A used their smartphone while walking 5 times this month."

[0577] Step 7: Generate warning messages using a generative AI model.

[0578] The server inputs prompt messages into the AI ​​model, which then generates appropriate warning messages for each user.

[0579] Input: Aggregated data, prompt text

[0580] Output: Generated warning message

[0581] Specific prompt example: "The user has used their smartphone while walking 50 times this month. Please generate the following warning message: 'You have used your smartphone while walking 50 times this month. For your safety, please stop walking before using your smartphone.'"

[0582] Step 8: Sending a warning message

[0583] The server sends a warning message generated from the generated AI model to the user.

[0584] Input: Generated warning message

[0585] Output: Sending a message to the user

[0586] Specific action: A message is sent stating, "User A has used their smartphone while walking 5 times this month. For safety reasons, please stop walking before using your smartphone."

[0587] Step 9: Granting Incentives

[0588] The server extracts users from the database who have zero instances of using their smartphones while walking, and awards reward points to those users.

[0589] Input: Aggregated database list

[0590] Output: Reward points awarded

[0591] Specific action: A notification is sent stating, "User B has been awarded a total of 300 reward points."

[0592] The above outlines the specific processing steps of this system's program. This enables real-time detection, warning, and incentive provision for walking while using a smartphone, thereby ensuring user safety and promoting improvement of risky behaviors.

[0593] (Application Example 1)

[0594] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0595] The practice of using smartphones while walking, known as "walking while using a smartphone," increases the risk of traffic accidents and crime. However, currently, there is no system that not only displays warning messages but also collects data, analyzes user behavior, and issues appropriate warnings or provides incentives. Therefore, a more effective system is needed to ensure user safety and curb walking while using smartphones.

[0596] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0597] In this invention, the server includes means for sending prompt messages to the user and generating warning messages using a generation AI model, means for analyzing the user's tendency to walk while using a smartphone based on cumulative user data and generating corresponding warning content, and means for awarding bonus points to users who do not walk while using a smartphone. This makes it possible to significantly reduce the risks of walking while using a smartphone by analyzing user behavior in detail and providing appropriate warnings and incentives.

[0598] An "accelerometer" is a device that detects the acceleration of an object and is used in mobile devices to detect the user's movement.

[0599] A "GPS sensor" is a device that uses the Geographic Information System (GPS) to acquire location information and monitor changes in that location.

[0600] "Mobile devices" is a general term for electronic devices that can be carried and used, such as smartphones and tablets.

[0601] A "warning message" is a notification designed to alert a user to specific actions or situations.

[0602] "Means of recording" refers to systems and methods for storing data and information over long periods of time.

[0603] A "server" is a computer system used to store and process data on a network.

[0604] A "generative AI model" is an algorithm or model that uses machine learning and artificial intelligence technologies to automatically generate new information or messages.

[0605] A "prompt message" is a text message created by a generative AI model to encourage a specific action.

[0606] "Cumulative data" refers to the total amount of data accumulated over a certain period of time, as well as its records.

[0607] An "incentive" refers to a reward or perk offered to encourage a particular behavior.

[0608] "Means of analyzing trends" refers to methods and mechanisms for analyzing data and identifying specific patterns or trends.

[0609] "Bonus points" refer to additional points awarded to users who meet specific conditions.

[0610] This invention relates to a system that detects when a user is walking while using their smartphone, issues a warning, and further compiles the number of times this has happened, thereby providing awareness and incentives. The embodiments for carrying out this invention will be described in detail below.

[0611] composition

[0612] This system is primarily composed of the following hardware and software.

[0613] 1. Hardware

[0614] Mobile devices: Smartphones, tablets, etc.

[0615] Accelerometer sensor installed in mobile devices

[0616] GPS sensor installed in mobile devices

[0617] Servers that process and store data

[0618] 2. Software

[0619] Generative AI Models: Artificial intelligence algorithms for generating warning messages and prompt texts.

[0620] Sensor library for acquiring data from accelerometers and GPS sensors.

[0621] Python scripts and communication libraries (such as requests) for sending and analyzing data to and from the server.

[0622] operation

[0623] Detection of users walking while using their smartphones

[0624] 1. Mobile devices use accelerometers and GPS sensors to detect the user's movement.

[0625] 2. The accelerometer analyzes whether the user is walking.

[0626] 3. The GPS sensor monitors changes in location information.

[0627] If the device detects that a user is operating a mobile device while moving, it will display a warning message in real time. For example, the display might show a warning message such as, "You are currently using your phone while walking. Please stop and operate it for your safety."

[0628] Data recording and transmission

[0629] 1. The mobile device counts the number of times a person is using their smartphone while walking and records this as internal data.

[0630] 2. At the end of the month, the counted number of times a person was using their smartphone while walking is sent to the server.

[0631] Data aggregation and warning message generation

[0632] 1. The server aggregates the received data and records it for each user.

[0633] 2. Use a generative AI model to send a prompt message to the user and generate a warning message.

[0634] Specifically, the system analyzes user trends based on cumulative data of walking while using a smartphone and generates corresponding warning messages. An example of a prompt message is: "You have walked while using your smartphone XX times this month. For your safety, please stop walking before using your smartphone."

[0635] Granting of incentives

[0636] 1. The server will award bonus points to users who do not use their smartphones while walking. For example, users who do not use their smartphones while walking will receive 10 bonus points per day.

[0637] With the above configuration and operation, this invention can detect a user's walking while using a smartphone in real time, issue an immediate warning, and further provide incentives to promote safe behavior by aggregating the data at the end of the month.

[0638] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0639] Step 1:

[0640] Mobile devices detect user movement using accelerometers and GPS sensors. They acquire accelerometer and GPS data as input and analyze whether the user is walking. Based on this analysis, they obtain an output indicating that the user is moving.

[0641] Step 2:

[0642] The mobile device determines whether the user is operating their smartphone while moving. Based on acquired sensor data and screen operation status, it determines whether the user is walking while using their smartphone. Based on this determination, it outputs that the user is walking while using their smartphone.

[0643] Step 3:

[0644] The mobile device displays a warning message if it detects that the user is using their phone while walking. The input is the detection result for using a phone while walking, and a generation AI model is used to generate a prompt message. The output is a warning message that reads, "You are currently using your phone while walking. Please stop and operate it for your safety."

[0645] Step 4:

[0646] The mobile device counts the number of times a user is using their phone while walking and records this data internally. It takes the number of times walking while using a phone is detected as input and saves this information to internal memory. The stored data then outputs the number of times the user is using their phone while walking.

[0647] Step 5:

[0648] At the end of the month, the mobile device sends the counted number of times a user has walked while using their smartphone to the server. As input, it retrieves the count data stored internally and sends it to the server. As a result of this transmission, it receives output indicating that the count data has been saved on the server.

[0649] Step 6:

[0650] The server aggregates the received data and records it for each user. It receives data on the number of times a user has used their phone while walking as input and stores it in a database for each user. The stored data is then output as information on the number of times each user has used their phone while walking.

[0651] Step 7:

[0652] The server uses a generative AI model to send prompt messages to users and generate warning messages. It takes user data on the number of times they've used their phone while walking as input and generates a prompt message such as, "You've used your phone while walking XX times this month. For safety reasons, please stop before using your phone." This generated warning message is output.

[0653] Step 8:

[0654] The server awards bonus points to users who do not use their smartphones while walking. It retrieves user information for those who have used their smartphones while walking 0 times as input, and awards 10 bonus points per day to these users. The output is the result of this point awarding process.

[0655] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0656] This invention is a system that detects when a user is walking while using their smartphone, recognizes the user's emotions, and then issues a warning message. In addition to a basic walking / smartphone detection function using an accelerometer and GPS sensor, this system incorporates an emotion engine.

[0657] Detection of walking while using a smartphone and display of warning messages.

[0658] The device monitors the user's movement and changes in location using an accelerometer and a GPS sensor. Specifically, the accelerometer measures the user's movement, and the GPS sensor confirms changes in location. If the device determines that the user is operating their smartphone while moving at regular intervals, it considers this to be walking while using a smartphone. For example, if the user operates their smartphone while walking for one second, it will be detected.

[0659] Emotion recognition and message adjustment using an emotion engine.

[0660] If a user is walking while using their smartphone, the device activates its emotion engine to identify the user's current emotional state through facial recognition and voice analysis. For example, it uses the camera to analyze the user's facial expressions and estimates emotions from voice input. Based on this emotional information, the content and timing of warning messages are adjusted in real time.

[0661] For example, if a user is feeling stressed, the standard warning message will be changed to something like, "You seem tired today. For your safety, please take a short break before using your smartphone." This ensures that appropriate warnings are given, taking the user's feelings into consideration.

[0662] Counting and sending the number of times people walk while using their smartphones.

[0663] The device counts the number of times a user walks while using their smartphone and records this data internally. At the end of each month, this count data is sent to a server. For example, the device automatically sends the data on the number of times a user walks while using their smartphone to the server on the last day of the month.

[0664] Data aggregation by the server and a re-emphasis of warnings.

[0665] The server receives data sent from the terminal and stores it in a database. The received data is aggregated, and the number of times each user has used their smartphone while walking is calculated. Based on this, a generative AI sends another warning message to the user. For example, a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," is sent.

[0666] Bonus points awarded

[0667] Furthermore, the server identifies users who did not use their smartphones while walking and awards them bonus points as an incentive. For example, users who did not use their smartphones while walking receive 10 bonus points per day. This is achieved by extracting users with zero instances of using their smartphones while walking from the database and awarding points to those users.

[0668] Bonus point adjustment based on user emotions

[0669] The emotion engine also has a function that adjusts the bonus points a user receives according to their emotional state. For example, if a user is in a positive emotional state, they may be awarded more points than the standard amount.

[0670] As described above, the present invention is a system that detects a user's walking while using their smartphone in real time, provides appropriate warnings that take emotions into consideration, and offers incentives to promote safe behavior. For example, if a user walks while using their smartphone five times in a day, the emotion engine recognizes the user's state and displays an appropriate message, and sends this data to the server at the end of the month. The server aggregates the data and provides further warnings and bonus points.

[0671] The following describes the processing flow.

[0672] Step 1:

[0673] The device activates its accelerometer and GPS sensor to monitor the user's movement and location. The accelerometer measures the user's movement, and the GPS sensor confirms changes in position. Specifically, data is obtained using `AccelSensor().get_current_acceleration()` and `GPSSensor().get_current_position()`.

[0674] Step 2:

[0675] The device updates acceleration data and location data at regular intervals (for example, every second) to determine whether movement and changes in location meet certain conditions. Specifically, time.sleep(1) is used to acquire and check data every second.

[0676] Step 3:

[0677] If the device determines, based on the acquired data, that the user is using their smartphone while walking, it will display a warning message on the screen. For example, it might display a message such as, "You are currently using your smartphone while walking. Please stop and operate it for your safety," using display.show_message().

[0678] Step 4:

[0679] The device simultaneously activates an emotion engine and uses sensors such as cameras and microphones to detect the user's emotional state. For example, this may involve using a facial recognition camera and a voice recognition module to identify emotions from the user's facial expressions and voice.

[0680] Step 5:

[0681] The device adjusts the content and timing of warning messages in real time based on the user's emotions detected by the emotion engine. For example, if the user is feeling stressed, it will display a message such as, "You seem tired today. For your safety, please take a short break before using your smartphone."

[0682] Step 6:

[0683] The device counts the number of times the user is using their phone while walking as internal data and saves it to the device's storage area. For example, the count is incremented with `self.walk_count += 1` and the data is saved with `local_storage.save("walk_count", self.walk_count)`.

[0684] Step 7:

[0685] At the end of the month, the device sends the counted number of times a user has walked while using their phone to the server. Specifically, it uses `if datetime.datetime.now().day == datetime.date.today().replace(day=1).days_in_month: send_data_to_server()` to detect the end of the month and send the data.

[0686] Step 8:

[0687] The server receives data sent from the terminal and saves it to the database. The received data is saved using database.save("user_walk_count", user_id, walk_count).

[0688] Step 9:

[0689] The server aggregates the data and calculates the number of times each user walks while using their smartphone. For example, the data is extracted and aggregated using `walk_count = database.get("user_walk_count", user_id)`.

[0690] Step 10:

[0691] Based on the aggregated results, the server generates and sends a warning message to each user corresponding to the number of times they have used their smartphone while walking. For example, it might send a message such as, "You used your smartphone while walking 5 times this month. For your safety, please stop walking before using your smartphone."

[0692] Step 11:

[0693] The server identifies users who did not walk while using their smartphones and awards bonus points as an incentive to those identified. For example, it identifies target users using `users_no_walking = database.query("SELECT user_id FROM users WHERE walk_count = 0")` and awards points to those users.

[0694] Step 12:

[0695] The server uses an emotion engine to adjust bonus point allocation based on the user's emotions. Users with positive emotions may receive bonus points above the standard rate. For example, if a user expresses positive emotions, they may receive double the usual points.

[0696] (Example 2)

[0697] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0698] Conventional technologies lacked sufficient means to warn users of the dangers of walking while using a smartphone, and failed to provide warning messages that took into account the user's emotional state. Furthermore, the lack of a system to record the number of times users walked while using their smartphones and provide feedback to the user made it difficult to encourage improved behavior. In addition, there was no detailed system for providing incentives for refraining from walking while using a smartphone. As a result, it was difficult to continuously encourage safe behavior among users.

[0699] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0700] In this invention, the server includes means for detecting the user's movement using an acceleration sensor, means for acquiring and monitoring changes in location information using a location information acquisition device, means for displaying a warning message when it is determined that the user is operating a mobile device while moving, means for recognizing the user's emotions using facial recognition and voice analysis, means for adjusting the content of the warning message based on the recognized emotion information, means for counting and recording the number of times the user is using their smartphone while walking, means for transmitting the counted number of times the user is using their smartphone while walking to a recording device at the end of the month, means for the recording device to aggregate the received data and record it for each user, means for sending another warning message to the user based on the number of times the user is using their smartphone while walking, means for awarding reward points to users who do not use their smartphone while walking, and means for adjusting reward points based on the user's emotion information. This makes it possible to provide warning messages that take into account the user's emotional state, record and provide feedback on the number of times the user is using their smartphone while walking, and continuously encourage safe behavior by the user.

[0701] An "accelerometer" is a device that measures the user's speed and direction of movement.

[0702] A "location information acquisition device" is a device used to monitor and track a user's location.

[0703] A "mobile device" is an electronic device that a user can carry around and use for communication and information processing.

[0704] A "warning message" is a message displayed to alert the user.

[0705] "Facial recognition" is a technology that analyzes a user's facial expressions captured by a camera to identify their emotions.

[0706] "Voice analysis" is a technology that analyzes a user's voice collected through a microphone to identify their emotions.

[0707] "Emotional information" refers to data that indicates the user's current emotional state.

[0708] A "recording device" is a device used to store and save data.

[0709] "Reward points" are incentive points awarded to users for specific actions.

[0710] This invention is a system that detects when a user is operating a smartphone while walking, recognizes their emotions at that time, and issues an appropriate warning message. This system includes an accelerometer, a location information acquisition device, an emotion engine, and a server.

[0711] Hardware configuration

[0712] 1. An accelerometer (e.g., MPU-6050) is a device that measures the user's movement speed and direction.

[0713] 2. A location information acquisition device (e.g., U-blox NEO-6M) is a device for monitoring and tracking the user's location.

[0714] 3. A mobile device is an electronic device that a user can carry with them and performs communication and information processing, and includes a built-in camera and microphone.

[0715] 4. A server is a device for accumulating and storing data.

[0716] Software Configuration

[0717] 5. Facial recognition technology (e.g., OpenCV and dlib) is a technology that analyzes a user's facial expressions captured by a camera to identify their emotions.

[0718] 6. Speech analysis technology (e.g., Google Cloud Speech-to-Text) is a technology that analyzes a user's voice collected by a microphone to identify their emotions.

[0719] 7. The emotion engine is a software module that performs facial recognition and voice analysis to identify the user's emotional state.

[0720] System Functions

[0721] Detection of using a smartphone while walking

[0722] The device uses an accelerometer and a location information acquisition device to monitor the user's movement and changes in location. Next, the device analyzes the collected data at regular intervals and recognizes it as "walking while using a smartphone" if it determines that the user is operating their smartphone while walking.

[0723] Emotion recognition and adjustment of warning messages

[0724] If the device detects that the user is walking while using their smartphone, it activates its emotion engine. The device uses its camera and microphone to collect the user's facial expressions and voice, and analyzes them with the emotion engine. Based on the user's emotional state, the device generates an appropriate warning message and displays it on its display screen.

[0725] Specific example

[0726] For example, when a user is using their smartphone on a train platform, the device's accelerometer (MPU-6050) and location information acquisition device (U-blox NEO-6M) detect the user's movement. If the device detects that the user is walking while using their smartphone for one second, it displays a warning message such as, "Using your smartphone while walking is dangerous. Please stop before using it." Furthermore, if the device analyzes that the user is experiencing stress, it adjusts the message to something like, "You seem tired today. For your safety, please take a short break before using your smartphone."

[0727] Recording and sending the number of times I walk while using my smartphone.

[0728] Each time the device detects someone walking while using their smartphone, it records the number of occurrences in its internal database. On the last day of the month, the device sends the recorded number of times someone is walking while using their smartphone to a server.

[0729] Server-based data aggregation and resending of warning messages.

[0730] The server receives data on the number of times users have used their smartphones while walking from their devices and stores it in a database. The server then aggregates the number of times each user has used their smartphones while walking and uses a generation AI model to generate and send a warning message. For example, a possible message might be: "You have used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone."

[0731] Bonus points are awarded and adjustments are made based on emotions.

[0732] The server extracts users from the database who have zero instances of walking while using their smartphones and awards them reward points as an incentive. Furthermore, it adjusts the reward points based on the user's emotional state. For example, users with a positive emotional state are awarded points above the standard rate.

[0733] Examples of prompts for generative AI models

[0734] The following are possible inputs for a generative AI model.

[0735] Let's assume the system detects that the user is walking while using their smartphone. Using data from the accelerometer and location information acquisition device, the system uses an emotion engine to recognize the user's emotions (through facial expressions and voice analysis). If the user is experiencing stress, generate an appropriate warning message.

[0736] Thus, the system of the present invention can continuously promote safe behavior by providing real-time alerts and incentives that take into account the user's emotional state.

[0737] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0738] Step 1:

[0739] The device detects that the user is operating a smartphone.

[0740] Input: User's smartphone operation (touchscreen use, app launching, etc.)

[0741] Data processing: None

[0742] Output: Smartphone operation detection signal

[0743] Specific operation: A system service on the device monitors the smartphone's operation events.

[0744] Step 2:

[0745] The terminal monitors the user's movement and location information using an accelerometer and a location information acquisition device.

[0746] Input: Data from acceleration sensor (MPU-6050), data from location information acquisition device (U-blox NEO-6M)

[0747] Data processing: Real-time analysis of acceleration data and location information.

[0748] Output: Changes in movement and position information

[0749] Specific operation: The device collects acceleration data at regular intervals and obtains location information. The collected data is analyzed to determine whether movement has occurred and whether the location has changed.

[0750] Step 3:

[0751] The device determines whether the user is operating their smartphone while walking.

[0752] Input: Changes in movement and location information, detection signals from smartphone operation.

[0753] Data processing: Analysis of data over a specific period (e.g., 1 second).

[0754] Output: Result of detecting walking while using a smartphone (True / False)

[0755] Specific operation: To determine whether the user is walking and operating a smartphone, data from one second is integrated and analyzed.

[0756] Step 4:

[0757] The device displays a warning message if it detects someone walking while using their smartphone.

[0758] Input: Result of detecting walking while using a smartphone

[0759] Data processing: None

[0760] Output: Command to display warning message

[0761] Specific action: The device displays a warning message on its screen (e.g., "Using your phone while walking is dangerous. Please stop before using it.").

[0762] Step 5:

[0763] The device activates an emotion engine when it detects someone walking while using their smartphone, and recognizes the user's emotions.

[0764] Input: Camera video (facial expression data), microphone audio (audio data)

[0765] Data processing: Analysis of facial expression data and voice data (using OpenCV, dlib, and Google Cloud Speech-to-Text).

[0766] Output: Emotional information (e.g., stress, joy, etc.)

[0767] Specific operation: The device captures the user's face with its camera and collects audio with its microphone. This data is then analyzed by an emotion engine to recognize the user's emotions.

[0768] Step 6:

[0769] The device adjusts the content of the warning message based on the emotional information it recognizes.

[0770] Input: Sentiment information, existing warning messages

[0771] Data processing: Generating messages based on emotional information

[0772] Output: Adjusted warning message

[0773] Specific action: For example, if the user is feeling stressed, a message will be displayed saying, "You seem tired today. For your safety, please take a short break before using your smartphone."

[0774] Step 7:

[0775] The device counts the number of times a user walks while using their smartphone and records the data in an internal database.

[0776] Input: Result of detecting walking while using a smartphone

[0777] Data processing: Incrementing the count, recording to the database.

[0778] Output: Updated number of times people have walked while using their smartphones.

[0779] Specific action: Each time walking while using a smartphone is detected, the recorded count increases by one.

[0780] Step 8:

[0781] At the end of the month, the device sends data on the number of times the user has walked while using their smartphone to the server.

[0782] Input: Data on the number of times people walk while using their smartphones.

[0783] Data processing: Data format conversion, application of transmission protocols.

[0784] Output: Sending data to the server

[0785] Specific operation: On the last day of the month, the device converts the counted number of times a user has walked while using their phone into an appropriate format and sends it to the server.

[0786] Step 9:

[0787] The server aggregates the received data and records it for each user.

[0788] Input: Data on the number of times people walk while using their smartphones.

[0789] Data processing: Data aggregation and recording for each user.

[0790] Output: Aggregated data on walking while using a smartphone

[0791] Specific operation: The server aggregates the received data for each user and stores it in the database.

[0792] Step 10:

[0793] The server sends another warning message to the user based on the number of times they have used their phone while walking.

[0794] Input: Aggregated data on walking while using a smartphone

[0795] Data processing: Generating warning messages using a generative AI model.

[0796] Output: Sending a re-warning message

[0797] Specific operation: Based on the number of times each user walks while using their smartphone, the server uses a generative AI model to generate an appropriate warning message and sends it to the user.

[0798] Step 11:

[0799] The server will award reward points to users who do not use their smartphones while walking.

[0800] Input: Aggregated data on walking while using a smartphone

[0801] Data processing: Calculation and awarding of reward points

[0802] Output: User accounts to which reward points have been awarded

[0803] Specific operation: The server calculates reward points as an incentive for users who have zero instances of using their smartphones while walking during the month, and adds them to the user's account.

[0804] Step 12:

[0805] The server adjusts reward points based on the user's sentiment information.

[0806] Input: Sentiment information, reward point data

[0807] Data processing: Adjusting reward points based on emotional information

[0808] Output: Adjusted reward points

[0809] Specific action: Users in a positive emotional state will be awarded points above the standard level, and their reward points will be adjusted accordingly.

[0810] (Application Example 2)

[0811] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0812] Conventional pedestrian detection systems have faced challenges in effectively detecting "walking while using a smartphone" behavior and issuing appropriate warnings. In particular, they lacked the technology to send messages that took into account the user's emotional state, making them ineffective in improving user responses and behavior. Furthermore, they lacked means of providing incentives to promote safe behavior.

[0813] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0814] In this invention, the server includes means for detecting the user's movement using an acceleration sensor, means for acquiring and monitoring changes in location information using a GPS sensor, means for displaying a warning message when it is determined that the user is operating a mobile device while moving, means for counting and recording the number of times the user is using their smartphone while walking, means for sending the counted number of times the user is using their smartphone while walking to the server at the end of the month, means for the server to aggregate the received data and record it for each user, means for sending another warning message to the user based on the number of times the user is using their smartphone while walking, means for awarding bonus points to users who do not use their smartphone while walking, means for analyzing the user's emotional state from their facial expressions and voice and adjusting the warning message accordingly, and means for adjusting bonus points according to the user's emotional state. This makes it possible to provide appropriate warning messages tailored to the user's current emotional state and to provide incentives to curb smartphone use while walking.

[0815] An "accelerometer" is a device used to detect the acceleration of a user's movements and actions.

[0816] A "GPS sensor" is a device used to acquire location information on Earth and monitor changes in that location.

[0817] A "mobile device" is an electronic device that a user carries and uses, and specifically refers to a smartphone.

[0818] A "warning message" is a notification designed to remind users to stop using their smartphones while walking.

[0819] "Means of counting" refers to the devices or functions necessary to record the number of times someone is walking while using their smartphone.

[0820] A "server" is a computer system that collects and manages data and sends warning messages and bonus points to users.

[0821] "Means of aggregation" refers to a function that allows the server to organize the data it receives and compile statistics for each user.

[0822] "Bonus points" are points awarded to users as an incentive, and are granted when certain conditions are met.

[0823] "Emotional state" refers to the user's psychological state at that time, as analyzed from their facial expressions and voice.

[0824] "Means of analyzing facial expressions and voice" refer to devices and software that read emotions from a user's facial movements and voice.

[0825] "Means of adjustment" refers to functions that change the content of messages or the amount of points awarded based on specific conditions.

[0826] This invention is a system that detects when a user is walking while using their smartphone, recognizes the user's emotions, and then issues a warning message. In addition to a basic walking / smartphone detection function using an accelerometer and GPS sensor, this system incorporates an emotion engine.

[0827] Terminal-side embodiment

[0828] The device monitors the user's movement and changes in location using an accelerometer and GPS sensor. Specifically, the accelerometer measures the user's movement, and the GPS sensor confirms changes in location. If the device determines that the user is operating their smartphone while moving at regular intervals, it considers this to be walking while using a smartphone. For example, if a user is walking while operating their smartphone for one second, this will be detected. At this time, an emotion engine is activated that uses the camera and microphone to analyze facial expressions and voice in order to recognize the user's emotional state. The emotion engine analyzes the user's facial expression and voice data in real time to identify the user's current emotional state.

[0829] Display of warning message

[0830] When a user is detected walking while using their smartphone, the emotion engine analyzes the user's emotional state and generates a warning message based on that analysis. For example, if the user is feeling stressed, a message such as, "You seem tired today. For your safety, please take a short break before using your smartphone," will be displayed. For other emotional states, a standard warning message will be displayed.

[0831] Counting the number of times people walk while using their smartphones and transmitting data.

[0832] The device counts the number of times a user walks while using their smartphone and records this data internally. At the end of each month, this data is automatically sent to the server. Specifically, on the last day of the month, the device sends the data on the number of times a user walks while using their smartphone to the server, which receives it and stores it in its database.

[0833] The server aggregates the data and issues another warning.

[0834] The server aggregates the data sent from the terminals and calculates the number of times each user has used their smartphone while walking. Based on this, the server sends another warning message to the user. For example, a message like, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," is possible.

[0835] Bonus points awarded

[0836] The server has a function to identify users who do not use their smartphones while walking and to award them bonus points as an incentive. It extracts users from the database who have zero instances of using their smartphones while walking and awards them 10 bonus points per day. Furthermore, if a user's emotional state is positive, they may be awarded more points than the standard amount.

[0837] Hardware and software to be used

[0838] To implement this system, a mobile device with a built-in accelerometer and GPS sensor is required, as well as facial recognition software and voice analysis software to act as the emotion engine. Specifically, a general-purpose facial recognition library and voice analysis tool can be used for emotion recognition. In addition, a database management system and data aggregation software are required on the server.

[0839] Examples of specific cases and prompt statements

[0840] As a concrete example, consider a system where an autonomous vehicle detects pedestrians wearing smart glasses and issues a warning if they are using their smartphones while walking. The following is an example of a prompt to be input to the generating AI model:

[0841] Example of a prompt:

[0842] Please provide a code example for building a system that detects when a user is walking while using their smartphone and sends the most appropriate warning message based on their emotions. Please include the following sections:

[0843] 1. Code to acquire data from the accelerometer and GPS sensor.

[0844] 2. Code that recognizes the user's emotions and adjusts the warning message accordingly.

[0845] 3. Code to send a warning message to the user.

[0846] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0847] Step 1:

[0848] The device acquires data from the accelerometer and GPS sensor.

[0849] Input: Device's accelerometer, GPS sensor

[0850] Data processing: Collection of acceleration data and location data.

[0851] Output: User movement and location change information

[0852] Specific operation: The device's built-in accelerometer measures the user's movement data (accelerometer data), and the GPS sensor acquires the user's location information. This data is collected and used as basic data necessary for detecting walking while using a smartphone.

[0853] Step 2:

[0854] The device detects when the user is operating their smartphone while on the move.

[0855] Input: Acceleration data, location data

[0856] Data processing: Comparison of movement patterns and device operations at regular intervals.

[0857] Output: Detection results for walking while using a smartphone

[0858] Specific operation: The device determines whether the user is operating their smartphone while moving, based on acceleration data and location data collected at regular intervals. If it is determined that the user is using their smartphone while walking, that information is sent to the next step.

[0859] Step 3:

[0860] The device activates an emotion engine that analyzes the user's facial expressions and voice.

[0861] Input: User facial expression data (camera), audio data (microphone)

[0862] Data processing: facial expression recognition, voice analysis

[0863] Output: User's emotional state

[0864] Specific operation: The system uses the camera and microphone built into the device to acquire user facial expression and voice data. By analyzing this data, the system identifies the user's emotional state. The emotion engine uses, for example, a facial recognition library or a voice analysis tool.

[0865] Step 4:

[0866] The device generates and displays warning messages based on the user's emotional state.

[0867] Input: Emotional state, detection results of walking while using a smartphone

[0868] Data processing: Adjusting message content based on emotional state.

[0869] Output: Warning message

[0870] Specific operation: The device adjusts the content of the warning message based on the detected emotional state. For example, if the user is feeling stressed, it will generate a gentle message. The generated message will be displayed on the device's screen.

[0871] Step 5:

[0872] The device counts and records the number of times you use your smartphone while walking.

[0873] Input: Detection results of walking while using a smartphone

[0874] Data processing: Counting the number of times a person walks while using their smartphone, and recording it in the internal data.

[0875] Output: Count data

[0876] Specific operation: The device counts the number of times a user is using their phone while walking and records this data in its internal storage. The recorded data is later sent to a server.

[0877] Step 6:

[0878] At the end of each month, the device sends the counted data to the server.

[0879] Input: Data on the number of times people walk while using their smartphones.

[0880] Data processing: Preparing and sending data for transmission.

[0881] Output: Data sent to the server

[0882] Specific operation: At the end of each month, the device automatically sends data on the number of times the user is using their smartphone while walking to the server. This transmission process uses network communication functionality.

[0883] Step 7:

[0884] The server aggregates the data it receives and records it for each user.

[0885] Input: Data on the number of times people walk while using their smartphones.

[0886] Data processing: Data aggregation, creation of user-specific statistical data.

[0887] Output: Data on the number of times users walk while using their smartphones.

[0888] Specific operation: The server stores data on the number of times a user walks while using their smartphone, received from the terminal, into a database and generates statistical data for each user. Based on this, individual user reports are created.

[0889] Step 8:

[0890] The server will send the warning message to the user again.

[0891] Input: User-specific data on the number of times a user walks while using their smartphone.

[0892] Data processing: Generating warning messages

[0893] Output: Warning message

[0894] Specific operation: The server generates a warning message again based on each user's data on the number of times they use their smartphone while walking, and sends it to the user. The message is sent via smartphone or email.

[0895] Step 9:

[0896] The server will award bonus points to users who do not use their smartphones while walking.

[0897] Input: User-specific data on the number of times a user walks while using their smartphone.

[0898] Data processing: Calculation and awarding of bonus points

[0899] Output: Bonus Points

[0900] Specific operation: The server calculates bonus points for users who have zero instances of using their smartphones while walking, and records this in the database. As an incentive, the bonus points are added to the user's account.

[0901] Step 10:

[0902] The server adjusts bonus points based on the user's emotional state.

[0903] Input: Emotional state data, bonus points

[0904] Data processing: Adjustment of bonus points

[0905] Output: Adjusted bonus points

[0906] Specific operation: The server considers the user's emotional state data and adjusts the bonus points awarded to users in a positive emotional state to be above the standard level. The adjusted points are recorded in the database.

[0907] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0908] The data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0909] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0910] [Third Embodiment]

[0911] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0912] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0913] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0914] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0915] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0916] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0917] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0918] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0919] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0920] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0921] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0922] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0923] This invention provides a system that detects when a user is walking while using their smartphone, issues a warning, and further collects the number of times this has happened, thereby raising awareness and providing incentives to the user. This system uses an accelerometer and GPS sensor installed in the device to detect walking while using a smartphone.

[0924] The device uses an accelerometer to detect whether the user is operating their smartphone while walking, and a GPS sensor to check for changes in location. For example, if the user is operating their smartphone while moving at regular intervals (e.g., every second), the device will determine this as walking while using a smartphone. Once this determination is made, a warning message will be displayed on the screen. A specific example of such a message might be displayed: "You are currently walking while using your smartphone. For your safety, please stop and operate your phone."

[0925] This system also counts the number of times a user walks while using their smartphone and records it as internal data. At the end of each month, this count data is sent to a server. For example, the device automatically sends the data on the number of times a user walks while using their smartphone to the server on the last day of the month.

[0926] The server aggregates the received data and records the number of times each user has used their smartphone while walking. For example, the server stores each user's smartphone usage count in a database and checks the aggregated results at the end of the month. Based on this data, a generative AI sends another warning message to the user. For example, a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," might be sent.

[0927] Furthermore, the server identifies users who did not use their smartphones while walking and rewards them with PayPay points as an incentive. Specifically, users who did not use their smartphones while walking will receive 10 PayPay points per day. This is achieved by extracting users from the database who have zero instances of using their smartphones while walking and then awarding points to those users.

[0928] As described above, the present invention detects a user's walking while using their smartphone in real time and issues an immediate warning. Furthermore, it is a system that collects data on a server at the end of the month and provides incentives to promote safe behavior such as not walking while using a smartphone.

[0929] For example, if a user uses their smartphone while walking five times in a day, the device counts each instance, and at the end of the month, it sends the total count to the server. Based on this data, the server warns the user, "You used your smartphone while walking five times this month," and notifies users who did not use their smartphone while walking, "You have been awarded 10 PayPay points for the day."

[0930] This system is an effective means of ensuring user safety and raising awareness of the dangers of operating a smartphone while walking.

[0931] The following describes the processing flow.

[0932] Step 1:

[0933] The device activates its accelerometer and GPS sensor to monitor the user's movement and location. This involves measuring the user's movement with the accelerometer and checking for changes in location with the GPS sensor. Specifically, it uses AccelSensor().get_current_acceleration() and GPSSensor().get_current_position() to obtain current data.

[0934] Step 2:

[0935] The device acquires acceleration and location data again at regular intervals (e.g., every second) and determines if there are any changes. Specifically, time.sleep(1) is used to acquire and check data every second. This allows the system to determine if the user is continuously moving while operating the smartphone.

[0936] Step 3:

[0937] Based on the acquired data, the device will display a warning message on the screen if it determines that the user is using their phone while walking. For example, a message such as "You are currently using your phone while walking. Please stop and operate your phone for your safety." will be displayed using `display.show_message()`.

[0938] Step 4:

[0939] The device counts the number of times a user walks while using their phone as internal data and saves it to the device's storage area. For example, the count is incremented with `self.walk_count += 1` and saved with `local_storage.save("walk_count", self.walk_count)`.

[0940] Step 5:

[0941] At the end of the month, the device sends data on the number of times the user has walked while using their phone to the server. This includes a process that detects the end of the month and sends the data, such as `if datetime.datetime.now().day == datetime.date.today().replace(day=1).days_in_month: send_data_to_server()`.

[0942] Step 6:

[0943] The server receives data sent from the terminal and saves it to the database. Specifically, the data is saved using `database.save("user_walk_count", user_id, walk_count)`.

[0944] Step 7:

[0945] The server aggregates the data stored in the database and calculates the number of times each user walks while using their smartphone. For example, the aggregation is performed as follows: walk_count = database.get("user_walk_count", user_id).

[0946] Step 8:

[0947] Based on the aggregated results, the server generates and sends a warning message to each user corresponding to the number of times they have used their smartphone while walking. For example, it might send a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone."

[0948] Step 9:

[0949] The server extracts and identifies users who did not walk while using their smartphones. Specifically, it extracts these users from the database using `users_no_walking = database.query("SELECT user_id FROM users WHERE walk_count = 0")`.

[0950] Step 10:

[0951] The server awards bonus points to users who do not use their smartphones while walking and sends a notification message. For example, points are awarded using `for user_id in users_no_walking: grant_paypay_points(user_id, 10)` and a notification is sent stating, "10 PayPay points have been awarded for today."

[0952] (Example 1)

[0953] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0954] Using a smartphone while walking is a dangerous act for both the user and those around them, and can cause traffic accidents and falls. However, there is no system that can instantly detect when a user is operating a smartphone while walking and issue a warning. Furthermore, there is a lack of a system that collects data on users' smartphone-using behavior while walking and provides appropriate warnings and incentives based on that data. As a result, user awareness and improvement of behavior are insufficient.

[0955] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0956] In this invention, the server includes means for inputting a prompt sentence into a generating AI model based on received data and generating a warning message again, means for sending the generated warning message to the user based on the number of times the user has walked while using their smartphone, and means for awarding reward points to users who have not walked while using their smartphone. This enables users to recognize the dangers of walking while using their smartphone in real time and improve their behavior.

[0957] An "accelerometer" is a device used to detect the acceleration of an object and to measure the state of motion of that object.

[0958] A "GPS sensor" is a device that receives signals from satellites and determines the current location.

[0959] A "mobile device" is a portable electronic device that can be used while on the go, and includes smartphones, tablets, and other similar devices.

[0960] A "warning message" is a text or visual notification used to alert or warn a user.

[0961] "Counting" refers to the operation or process of counting the number of times a particular event has occurred.

[0962] A "server" is a computer system that provides various services and data processing to clients over a network.

[0963] A "database" is a software system that systematically stores data and manages it so that it can be easily searched and manipulated.

[0964] A "generative AI model" refers to an algorithm or program that uses artificial intelligence technology to automatically generate text and other content.

[0965] A "prompt statement" is an instruction or command given to a generative AI model, serving as a guideline for generating a specific output.

[0966] "Reward points" are incentives awarded to users who perform specific actions or meet certain conditions, and include those that can be used as monetary value or service benefits.

[0967] This invention is a system that detects the act of a user operating a smartphone while walking, commonly known as "walking while using a smartphone," and issues warnings and cautions in response. Furthermore, it records and aggregates the number of times a user walks while using a smartphone and provides incentives based on this information to promote improved user behavior.

[0968] The main components of the system are an accelerometer, GPS sensor, warning message display function, server, and generative AI model, all installed in the terminal.

[0969] Hardware and software to be used

[0970] 1. Accelerometer:

[0971] It is built into the device and detects the user's movement. Specifically, it is used to detect walking motion.

[0972] 2. GPS sensor:

[0973] It is built into the device and obtains the user's location information. By checking for changes in location information, it determines whether the user is moving.

[0974] 3. Terminal built-in software:

[0975] The system analyzes data from the accelerometer and GPS sensor to determine whether the user is operating their smartphone while walking. For example, if the user is moving and operating the screen every second, it is determined to be walking while using a smartphone.

[0976] 4. Warning message display function:

[0977] When walking while using a smartphone is detected, a warning message will be displayed on the screen. For example, a message such as "You are currently walking while using your smartphone. For your safety, please stop and operate your phone" will be displayed.

[0978] 5. Server:

[0979] The system receives and compiles data on the number of times users walk while using their smartphones, recording this data in a database for each user, and reviewing the results at the end of the month.

[0980] 6. Generative AI Models:

[0981] Based on the data collected by the server, prompts are entered to generate an appropriate warning message. The generated message is then sent to the user.

[0982] Example of a prompt message: "The user has used their smartphone while walking 50 times this month. Generate the following warning message: 'You have used your smartphone while walking 50 times this month. For your safety, please stop walking before using your smartphone.'"

[0983] 7. Incentive provision function:

[0984] The system extracts users from the database who have zero instances of walking while using their smartphones, and rewards these users with bonus points. For example, users who do not walk while using their smartphones will receive 10 bonus points per day.

[0985] Specific examples of system usage

[0986] When a user uses their smartphone while walking, the device's accelerometer and GPS sensor detect the movement and changes in location. The device recognizes this as walking while using a smartphone and displays a warning message on the screen: "You are currently using your smartphone while walking. For your safety, please stop and operate it."

[0987] Next, the system counts the number of times a user uses their phone while walking and records this data. At the end of the month, this data is sent to a server, which aggregates the data for each user. The server uses a generative AI model to generate a re-warning message and sends it to the user. A specific message such as, "User A used their phone while walking 5 times this month. For safety reasons, please stop walking before using your phone," is sent. In addition, users who do not use their phone while walking are rewarded with reward points.

[0988] In this way, the present invention is an effective method for ensuring user safety and raising awareness of the dangers of using a smartphone while walking, by detecting and warning against walking while using a smartphone in real time, and by providing incentives aimed at improving behavior.

[0989] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0990] Program processing flow

[0991] Step 1: Collect input data

[0992] The device collects data from its accelerometer and GPS sensor to detect whether the user is walking with their smartphone. The accelerometer detects movement, and the GPS sensor checks for changes in location.

[0993] Input: Acceleration data, GPS data

[0994] Output: Detected motion and position change data

[0995] Step 2: Detection of using a smartphone while walking

[0996] The device analyzes the collected sensor data to determine whether the user is operating their smartphone while moving at regular intervals. For example, if it detects movement and screen touches every second, it is determined to be walking while using a smartphone.

[0997] Input: Detected motion and position change data

[0998] Output: Result of detecting walking while using a smartphone

[0999] Specific action: The user walks 10 meters while touching the screen every second.

[1000] Step 3: Display of warning message

[1001] If the device detects that the user is using their phone while walking, it will immediately display a warning message on the screen. The message displayed will read, "You are currently using your phone while walking. For your safety, please stop and operate your phone."

[1002] Input: Result of detecting walking while using a smartphone

[1003] Output: Display of warning message

[1004] Specific action: The device displays the message, "You are currently using your phone while walking. For your safety, please stop and operate your phone."

[1005] Step 4: Count the number of times you use your phone while walking.

[1006] Each time the device detects that a user is walking while using their phone, it counts the number of times and records it in an internal database. This allows the device to accumulate data on how often a user has walked while using their phone.

[1007] Input: Result of detecting walking while using a smartphone

[1008] Output: Number of times people walked while using their smartphones were counted.

[1009] Specific operation: If detected 5 times in a day, the count is saved in internal memory.

[1010] Step 5: Send data on the number of times you walk while using your smartphone.

[1011] On the last day of the month, the device automatically sends accumulated data on the number of times users have walked while using their smartphone to the server.

[1012] Input: Data on the number of times people have walked while using their smartphones.

[1013] Output: Sending data to the server

[1014] Specific action: At 23:59 on the last day of the month, send data in the format "This month's number of times using a smartphone while walking is 5".

[1015] Step 6: Calculation

[1016] The server receives data sent from the terminals and records the number of times each user walks while using their smartphone in a database. The aggregated data is analyzed at the end of the month.

[1017] Input: Data on the number of times users have used their smartphones while walking.

[1018] Output: Aggregated database records

[1019] Specific action: The server records in the database that "User A used their smartphone while walking 5 times this month."

[1020] Step 7: Generate warning messages using a generative AI model.

[1021] The server inputs prompt messages into the AI ​​model, which then generates appropriate warning messages for each user.

[1022] Input: Aggregated data, prompt text

[1023] Output: Generated warning message

[1024] Specific prompt example: "The user has used their smartphone while walking 50 times this month. Please generate the following warning message: 'You have used your smartphone while walking 50 times this month. For your safety, please stop walking before using your smartphone.'"

[1025] Step 8: Sending a warning message

[1026] The server sends a warning message generated from the generated AI model to the user.

[1027] Input: Generated warning message

[1028] Output: Sending a message to the user

[1029] Specific action: A message is sent stating, "User A has used their smartphone while walking 5 times this month. For safety reasons, please stop walking before using your smartphone."

[1030] Step 9: Granting Incentives

[1031] The server extracts users from the database who have zero instances of using their smartphones while walking, and awards reward points to those users.

[1032] Input: Aggregated database list

[1033] Output: Reward points awarded

[1034] Specific action: A notification is sent stating, "User B has been awarded a total of 300 reward points."

[1035] The above outlines the specific processing steps of this system's program. This enables real-time detection, warning, and incentive provision for walking while using a smartphone, thereby ensuring user safety and promoting improvement of risky behaviors.

[1036] (Application Example 1)

[1037] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1038] The practice of using smartphones while walking, known as "walking while using a smartphone," increases the risk of traffic accidents and crime. However, currently, there is no system that not only displays warning messages but also collects data, analyzes user behavior, and issues appropriate warnings or provides incentives. Therefore, a more effective system is needed to ensure user safety and curb walking while using smartphones.

[1039] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1040] In this invention, the server includes means for sending prompt messages to the user and generating warning messages using a generation AI model, means for analyzing the user's tendency to walk while using a smartphone based on cumulative user data and generating corresponding warning content, and means for awarding bonus points to users who do not walk while using a smartphone. This makes it possible to significantly reduce the risks of walking while using a smartphone by analyzing user behavior in detail and providing appropriate warnings and incentives.

[1041] An "accelerometer" is a device that detects the acceleration of an object and is used in mobile devices to detect the user's movement.

[1042] A "GPS sensor" is a device that uses the Geographic Information System (GPS) to acquire location information and monitor changes in that location.

[1043] "Mobile devices" is a general term for electronic devices that can be carried and used, such as smartphones and tablets.

[1044] A "warning message" is a notification designed to alert a user to specific actions or situations.

[1045] "Means of recording" refers to systems and methods for storing data and information over long periods of time.

[1046] A "server" is a computer system used to store and process data on a network.

[1047] A "generative AI model" is an algorithm or model that uses machine learning and artificial intelligence technologies to automatically generate new information or messages.

[1048] A "prompt message" is a text message created by a generative AI model to encourage a specific action.

[1049] "Cumulative data" refers to the total amount of data accumulated over a certain period of time, as well as its records.

[1050] An "incentive" refers to a reward or perk offered to encourage a particular behavior.

[1051] "Means of analyzing trends" refers to methods and mechanisms for analyzing data and identifying specific patterns or trends.

[1052] "Bonus points" refer to additional points awarded to users who meet specific conditions.

[1053] This invention relates to a system that detects when a user is walking while using their smartphone, issues a warning, and further compiles the number of times this has happened, thereby providing awareness and incentives. The embodiments for carrying out this invention will be described in detail below.

[1054] composition

[1055] This system is primarily composed of the following hardware and software.

[1056] 1. Hardware

[1057] Mobile devices: Smartphones, tablets, etc.

[1058] Accelerometer sensor installed in mobile devices

[1059] GPS sensor installed in mobile devices

[1060] Servers that process and store data

[1061] 2. Software

[1062] Generative AI Models: Artificial intelligence algorithms for generating warning messages and prompt texts.

[1063] Sensor library for acquiring data from accelerometers and GPS sensors.

[1064] Python scripts and communication libraries (such as requests) for sending and analyzing data to and from the server.

[1065] operation

[1066] Detection of users walking while using their smartphones

[1067] 1. Mobile devices use accelerometers and GPS sensors to detect the user's movement.

[1068] 2. The accelerometer analyzes whether the user is walking.

[1069] 3. The GPS sensor monitors changes in location information.

[1070] If the device detects that a user is operating a mobile device while moving, it will display a warning message in real time. For example, the display might show a warning message such as, "You are currently using your phone while walking. Please stop and operate it for your safety."

[1071] Data recording and transmission

[1072] 1. The mobile device counts the number of times a person is using their smartphone while walking and records this as internal data.

[1073] 2. At the end of the month, the counted number of times a person was using their smartphone while walking is sent to the server.

[1074] Data aggregation and warning message generation

[1075] 1. The server aggregates the received data and records it for each user.

[1076] 2. Use a generative AI model to send a prompt message to the user and generate a warning message.

[1077] Specifically, the system analyzes user trends based on cumulative data of walking while using a smartphone and generates corresponding warning messages. An example of a prompt message is: "You have walked while using your smartphone XX times this month. For your safety, please stop walking before using your smartphone."

[1078] Granting of incentives

[1079] 1. The server will award bonus points to users who do not use their smartphones while walking. For example, users who do not use their smartphones while walking will receive 10 bonus points per day.

[1080] With the above configuration and operation, this invention can detect a user's walking while using a smartphone in real time, issue an immediate warning, and further provide incentives to promote safe behavior by aggregating the data at the end of the month.

[1081] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1082] Step 1:

[1083] Mobile devices detect user movement using accelerometers and GPS sensors. They acquire accelerometer and GPS data as input and analyze whether the user is walking. Based on this analysis, they obtain an output indicating that the user is moving.

[1084] Step 2:

[1085] The mobile device determines whether the user is operating their smartphone while moving. Based on acquired sensor data and screen operation status, it determines whether the user is walking while using their smartphone. Based on this determination, it outputs that the user is walking while using their smartphone.

[1086] Step 3:

[1087] The mobile device displays a warning message if it detects that the user is using their phone while walking. The input is the detection result for using a phone while walking, and a generation AI model is used to generate a prompt message. The output is a warning message that reads, "You are currently using your phone while walking. Please stop and operate it for your safety."

[1088] Step 4:

[1089] The mobile device counts the number of times a user is using their phone while walking and records this data internally. It takes the number of times walking while using a phone is detected as input and saves this information to internal memory. The stored data then outputs the number of times the user is using their phone while walking.

[1090] Step 5:

[1091] At the end of the month, the mobile device sends the counted number of times a user has walked while using their smartphone to the server. As input, it retrieves the count data stored internally and sends it to the server. As a result of this transmission, it receives output indicating that the count data has been saved on the server.

[1092] Step 6:

[1093] The server aggregates the received data and records it for each user. It receives data on the number of times a user has used their phone while walking as input and stores it in a database for each user. The stored data is then output as information on the number of times each user has used their phone while walking.

[1094] Step 7:

[1095] The server uses a generative AI model to send prompt messages to users and generate warning messages. It takes user data on the number of times they've used their phone while walking as input and generates a prompt message such as, "You've used your phone while walking XX times this month. For safety reasons, please stop before using your phone." This generated warning message is output.

[1096] Step 8:

[1097] The server awards bonus points to users who do not use their smartphones while walking. It retrieves user information for those who have used their smartphones while walking 0 times as input, and awards 10 bonus points per day to these users. The output is the result of this point awarding process.

[1098] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1099] This invention is a system that detects when a user is walking while using their smartphone, recognizes the user's emotions, and then issues a warning message. In addition to a basic walking / smartphone detection function using an accelerometer and GPS sensor, this system incorporates an emotion engine.

[1100] Detection of walking while using a smartphone and display of warning messages.

[1101] The device monitors the user's movement and changes in location using an accelerometer and a GPS sensor. Specifically, the accelerometer measures the user's movement, and the GPS sensor confirms changes in location. If the device determines that the user is operating their smartphone while moving at regular intervals, it considers this to be walking while using a smartphone. For example, if the user operates their smartphone while walking for one second, it will be detected.

[1102] Emotion recognition and message adjustment using an emotion engine.

[1103] If a user is walking while using their smartphone, the device activates its emotion engine to identify the user's current emotional state through facial recognition and voice analysis. For example, it uses the camera to analyze the user's facial expressions and estimates emotions from voice input. Based on this emotional information, the content and timing of warning messages are adjusted in real time.

[1104] For example, if a user is feeling stressed, the standard warning message will be changed to something like, "You seem tired today. For your safety, please take a short break before using your smartphone." This ensures that appropriate warnings are given, taking the user's feelings into consideration.

[1105] Counting and sending the number of times people walk while using their smartphones.

[1106] The device counts the number of times a user walks while using their smartphone and records this data internally. At the end of each month, this count data is sent to a server. For example, the device automatically sends the data on the number of times a user walks while using their smartphone to the server on the last day of the month.

[1107] Data aggregation by the server and a re-emphasis of warnings.

[1108] The server receives data sent from the terminal and stores it in a database. The received data is aggregated, and the number of times each user has used their smartphone while walking is calculated. Based on this, a generative AI sends another warning message to the user. For example, a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," is sent.

[1109] Bonus points awarded

[1110] Furthermore, the server identifies users who did not use their smartphones while walking and awards them bonus points as an incentive. For example, users who did not use their smartphones while walking receive 10 bonus points per day. This is achieved by extracting users with zero instances of using their smartphones while walking from the database and awarding points to those users.

[1111] Bonus point adjustment based on user emotions

[1112] The emotion engine also has a function that adjusts the bonus points a user receives according to their emotional state. For example, if a user is in a positive emotional state, they may be awarded more points than the standard amount.

[1113] As described above, the present invention is a system that detects a user's walking while using their smartphone in real time, provides appropriate warnings that take emotions into consideration, and offers incentives to promote safe behavior. For example, if a user walks while using their smartphone five times in a day, the emotion engine recognizes the user's state and displays an appropriate message, and sends this data to the server at the end of the month. The server aggregates the data and provides further warnings and bonus points.

[1114] The following describes the processing flow.

[1115] Step 1:

[1116] The device activates its accelerometer and GPS sensor to monitor the user's movement and location. The accelerometer measures the user's movement, and the GPS sensor confirms changes in position. Specifically, data is obtained using `AccelSensor().get_current_acceleration()` and `GPSSensor().get_current_position()`.

[1117] Step 2:

[1118] The device updates acceleration data and location data at regular intervals (for example, every second) to determine whether movement and changes in location meet certain conditions. Specifically, time.sleep(1) is used to acquire and check data every second.

[1119] Step 3:

[1120] If the device determines, based on the acquired data, that the user is using their smartphone while walking, it will display a warning message on the screen. For example, it might display a message such as, "You are currently using your smartphone while walking. Please stop and operate it for your safety," using display.show_message().

[1121] Step 4:

[1122] The device simultaneously activates an emotion engine and uses sensors such as cameras and microphones to detect the user's emotional state. For example, this may involve using a facial recognition camera and a voice recognition module to identify emotions from the user's facial expressions and voice.

[1123] Step 5:

[1124] The device adjusts the content and timing of warning messages in real time based on the user's emotions detected by the emotion engine. For example, if the user is feeling stressed, it will display a message such as, "You seem tired today. For your safety, please take a short break before using your smartphone."

[1125] Step 6:

[1126] The device counts the number of times the user is using their phone while walking as internal data and saves it to the device's storage area. For example, the count is incremented with `self.walk_count += 1` and the data is saved with `local_storage.save("walk_count", self.walk_count)`.

[1127] Step 7:

[1128] At the end of the month, the device sends the counted number of times a user has walked while using their phone to the server. Specifically, it uses `if datetime.datetime.now().day == datetime.date.today().replace(day=1).days_in_month: send_data_to_server()` to detect the end of the month and send the data.

[1129] Step 8:

[1130] The server receives data sent from the terminal and saves it to the database. The received data is saved using database.save("user_walk_count", user_id, walk_count).

[1131] Step 9:

[1132] The server aggregates the data and calculates the number of times each user walks while using their smartphone. For example, the data is extracted and aggregated using `walk_count = database.get("user_walk_count", user_id)`.

[1133] Step 10:

[1134] Based on the aggregated results, the server generates and sends a warning message to each user corresponding to the number of times they have used their smartphone while walking. For example, it might send a message such as, "You used your smartphone while walking 5 times this month. For your safety, please stop walking before using your smartphone."

[1135] Step 11:

[1136] The server identifies users who did not walk while using their smartphones and awards bonus points as an incentive to those identified. For example, it identifies target users using `users_no_walking = database.query("SELECT user_id FROM users WHERE walk_count = 0")` and awards points to those users.

[1137] Step 12:

[1138] The server uses an emotion engine to adjust bonus point allocation based on the user's emotions. Users with positive emotions may receive bonus points above the standard rate. For example, if a user expresses positive emotions, they may receive double the usual points.

[1139] (Example 2)

[1140] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1141] Conventional technologies lacked sufficient means to warn users of the dangers of walking while using a smartphone, and failed to provide warning messages that took into account the user's emotional state. Furthermore, the lack of a system to record the number of times users walked while using their smartphones and provide feedback to the user made it difficult to encourage improved behavior. In addition, there was no detailed system for providing incentives for refraining from walking while using a smartphone. As a result, it was difficult to continuously encourage safe behavior among users.

[1142] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1143] In this invention, the server includes means for detecting the user's movement using an acceleration sensor, means for acquiring and monitoring changes in location information using a location information acquisition device, means for displaying a warning message when it is determined that the user is operating a mobile device while moving, means for recognizing the user's emotions using facial recognition and voice analysis, means for adjusting the content of the warning message based on the recognized emotion information, means for counting and recording the number of times the user is using their smartphone while walking, means for transmitting the counted number of times the user is using their smartphone while walking to a recording device at the end of the month, means for the recording device to aggregate the received data and record it for each user, means for sending another warning message to the user based on the number of times the user is using their smartphone while walking, means for awarding reward points to users who do not use their smartphone while walking, and means for adjusting reward points based on the user's emotion information. This makes it possible to provide warning messages that take into account the user's emotional state, record and provide feedback on the number of times the user is using their smartphone while walking, and continuously encourage safe behavior by the user.

[1144] An "accelerometer" is a device that measures the user's speed and direction of movement.

[1145] A "location information acquisition device" is a device used to monitor and track a user's location.

[1146] A "mobile device" is an electronic device that a user can carry around and use for communication and information processing.

[1147] A "warning message" is a message displayed to alert the user.

[1148] "Facial recognition" is a technology that analyzes a user's facial expressions captured by a camera to identify their emotions.

[1149] "Voice analysis" is a technology that analyzes a user's voice collected through a microphone to identify their emotions.

[1150] "Emotional information" refers to data that indicates the user's current emotional state.

[1151] A "recording device" is a device used to store and save data.

[1152] "Reward points" are incentive points awarded to users for specific actions.

[1153] This invention is a system that detects when a user is operating a smartphone while walking, recognizes their emotions at that time, and issues an appropriate warning message. This system includes an accelerometer, a location information acquisition device, an emotion engine, and a server.

[1154] Hardware configuration

[1155] 1. An accelerometer (e.g., MPU-6050) is a device that measures the user's movement speed and direction.

[1156] 2. A location information acquisition device (e.g., U-blox NEO-6M) is a device for monitoring and tracking the user's location.

[1157] 3. A mobile device is an electronic device that a user can carry with them and performs communication and information processing, and includes a built-in camera and microphone.

[1158] 4. A server is a device for accumulating and storing data.

[1159] Software Configuration

[1160] 5. Facial recognition technology (e.g., OpenCV and dlib) is a technology that analyzes a user's facial expressions captured by a camera to identify their emotions.

[1161] 6. Speech analysis technology (e.g., Google Cloud Speech-to-Text) is a technology that analyzes a user's voice collected by a microphone to identify their emotions.

[1162] 7. The emotion engine is a software module that performs facial recognition and voice analysis to identify the user's emotional state.

[1163] System Functions

[1164] Detection of using a smartphone while walking

[1165] The device uses an accelerometer and a location information acquisition device to monitor the user's movement and changes in location. Next, the device analyzes the collected data at regular intervals and recognizes it as "walking while using a smartphone" if it determines that the user is operating their smartphone while walking.

[1166] Emotion recognition and adjustment of warning messages

[1167] If the device detects that the user is walking while using their smartphone, it activates its emotion engine. The device uses its camera and microphone to collect the user's facial expressions and voice, and analyzes them with the emotion engine. Based on the user's emotional state, the device generates an appropriate warning message and displays it on its display screen.

[1168] Specific example

[1169] For example, when a user is using their smartphone on a train platform, the device's accelerometer (MPU-6050) and location information acquisition device (U-blox NEO-6M) detect the user's movement. If the device detects that the user is walking while using their smartphone for one second, it displays a warning message such as, "Using your smartphone while walking is dangerous. Please stop before using it." Furthermore, if the device analyzes that the user is experiencing stress, it adjusts the message to something like, "You seem tired today. For your safety, please take a short break before using your smartphone."

[1170] Recording and sending the number of times I walk while using my smartphone.

[1171] Each time the device detects someone walking while using their smartphone, it records the number of occurrences in its internal database. On the last day of the month, the device sends the recorded number of times someone is walking while using their smartphone to a server.

[1172] Server-based data aggregation and resending of warning messages.

[1173] The server receives data on the number of times users have used their smartphones while walking from their devices and stores it in a database. The server then aggregates the number of times each user has used their smartphones while walking and uses a generation AI model to generate and send a warning message. For example, a possible message might be: "You have used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone."

[1174] Bonus points are awarded and adjustments are made based on emotions.

[1175] The server extracts users from the database who have zero instances of walking while using their smartphones and awards them reward points as an incentive. Furthermore, it adjusts the reward points based on the user's emotional state. For example, users with a positive emotional state are awarded points above the standard rate.

[1176] Examples of prompts for generative AI models

[1177] The following are possible inputs for a generative AI model.

[1178] Let's assume the system detects that the user is walking while using their smartphone. Using data from the accelerometer and location information acquisition device, the system uses an emotion engine to recognize the user's emotions (through facial expressions and voice analysis). If the user is experiencing stress, generate an appropriate warning message.

[1179] Thus, the system of the present invention can continuously promote safe behavior by providing real-time alerts and incentives that take into account the user's emotional state.

[1180] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1181] Step 1:

[1182] The device detects that the user is operating a smartphone.

[1183] Input: User's smartphone operation (touchscreen use, app launching, etc.)

[1184] Data processing: None

[1185] Output: Smartphone operation detection signal

[1186] Specific operation: A system service on the device monitors the smartphone's operation events.

[1187] Step 2:

[1188] The terminal monitors the user's movement and location information using an accelerometer and a location information acquisition device.

[1189] Input: Data from acceleration sensor (MPU-6050), data from location information acquisition device (U-blox NEO-6M)

[1190] Data processing: Real-time analysis of acceleration data and location information.

[1191] Output: Changes in movement and position information

[1192] Specific operation: The device collects acceleration data at regular intervals and obtains location information. The collected data is analyzed to determine whether movement has occurred and whether the location has changed.

[1193] Step 3:

[1194] The device determines whether the user is operating their smartphone while walking.

[1195] Input: Changes in movement and location information, detection signals from smartphone operation.

[1196] Data processing: Analysis of data over a specific period (e.g., 1 second).

[1197] Output: Result of detecting walking while using a smartphone (True / False)

[1198] Specific operation: To determine whether the user is walking and operating a smartphone, data from one second is integrated and analyzed.

[1199] Step 4:

[1200] The device displays a warning message if it detects someone walking while using their smartphone.

[1201] Input: Result of detecting walking while using a smartphone

[1202] Data processing: None

[1203] Output: Command to display warning message

[1204] Specific action: The device displays a warning message on its screen (e.g., "Using your phone while walking is dangerous. Please stop before using it.").

[1205] Step 5:

[1206] The device activates an emotion engine when it detects someone walking while using their smartphone, and recognizes the user's emotions.

[1207] Input: Camera video (facial expression data), microphone audio (audio data)

[1208] Data processing: Analysis of facial expression data and voice data (using OpenCV, dlib, and Google Cloud Speech-to-Text).

[1209] Output: Emotional information (e.g., stress, joy, etc.)

[1210] Specific operation: The device captures the user's face with its camera and collects audio with its microphone. This data is then analyzed by an emotion engine to recognize the user's emotions.

[1211] Step 6:

[1212] The device adjusts the content of the warning message based on the emotional information it recognizes.

[1213] Input: Sentiment information, existing warning messages

[1214] Data processing: Generating messages based on emotional information

[1215] Output: Adjusted warning message

[1216] Specific action: For example, if the user is feeling stressed, a message will be displayed saying, "You seem tired today. For your safety, please take a short break before using your smartphone."

[1217] Step 7:

[1218] The device counts the number of times a user walks while using their smartphone and records the data in an internal database.

[1219] Input: Result of detecting walking while using a smartphone

[1220] Data processing: Incrementing the count, recording to the database.

[1221] Output: Updated number of times people have walked while using their smartphones.

[1222] Specific action: Each time walking while using a smartphone is detected, the recorded count increases by one.

[1223] Step 8:

[1224] At the end of the month, the device sends data on the number of times the user has walked while using their smartphone to the server.

[1225] Input: Data on the number of times people walk while using their smartphones.

[1226] Data processing: Data format conversion, application of transmission protocols.

[1227] Output: Sending data to the server

[1228] Specific operation: On the last day of the month, the device converts the counted number of times a user has walked while using their phone into an appropriate format and sends it to the server.

[1229] Step 9:

[1230] The server aggregates the received data and records it for each user.

[1231] Input: Data on the number of times people walk while using their smartphones.

[1232] Data processing: Data aggregation and recording for each user.

[1233] Output: Aggregated data on walking while using a smartphone

[1234] Specific operation: The server aggregates the received data for each user and stores it in the database.

[1235] Step 10:

[1236] The server sends another warning message to the user based on the number of times they have used their phone while walking.

[1237] Input: Aggregated data on walking while using a smartphone

[1238] Data processing: Generating warning messages using a generative AI model.

[1239] Output: Sending a re-warning message

[1240] Specific operation: Based on the number of times each user walks while using their smartphone, the server uses a generative AI model to generate an appropriate warning message and sends it to the user.

[1241] Step 11:

[1242] The server will award reward points to users who do not use their smartphones while walking.

[1243] Input: Aggregated data on walking while using a smartphone

[1244] Data processing: Calculation and awarding of reward points

[1245] Output: User accounts to which reward points have been awarded

[1246] Specific operation: The server calculates reward points as an incentive for users who have zero instances of using their smartphones while walking during the month, and adds them to the user's account.

[1247] Step 12:

[1248] The server adjusts reward points based on the user's sentiment information.

[1249] Input: Sentiment information, reward point data

[1250] Data processing: Adjusting reward points based on emotional information

[1251] Output: Adjusted reward points

[1252] Specific action: Users in a positive emotional state will be awarded points above the standard level, and their reward points will be adjusted accordingly.

[1253] (Application Example 2)

[1254] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1255] Conventional pedestrian detection systems have faced challenges in effectively detecting "walking while using a smartphone" behavior and issuing appropriate warnings. In particular, they lacked the technology to send messages that took into account the user's emotional state, making them ineffective in improving user responses and behavior. Furthermore, they lacked means of providing incentives to promote safe behavior.

[1256] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1257] In this invention, the server includes means for detecting the user's movement using an acceleration sensor, means for acquiring and monitoring changes in location information using a GPS sensor, means for displaying a warning message when it is determined that the user is operating a mobile device while moving, means for counting and recording the number of times the user is using their smartphone while walking, means for sending the counted number of times the user is using their smartphone while walking to the server at the end of the month, means for the server to aggregate the received data and record it for each user, means for sending another warning message to the user based on the number of times the user is using their smartphone while walking, means for awarding bonus points to users who do not use their smartphone while walking, means for analyzing the user's emotional state from their facial expressions and voice and adjusting the warning message accordingly, and means for adjusting bonus points according to the user's emotional state. This makes it possible to provide appropriate warning messages tailored to the user's current emotional state and to provide incentives to curb smartphone use while walking.

[1258] An "accelerometer" is a device used to detect the acceleration of a user's movements and actions.

[1259] A "GPS sensor" is a device used to acquire location information on Earth and monitor changes in that location.

[1260] A "mobile device" is an electronic device that a user carries and uses, and specifically refers to a smartphone.

[1261] A "warning message" is a notification designed to remind users to stop using their smartphones while walking.

[1262] "Means of counting" refers to the devices or functions necessary to record the number of times someone is walking while using their smartphone.

[1263] A "server" is a computer system that collects and manages data and sends warning messages and bonus points to users.

[1264] "Means of aggregation" refers to a function that allows the server to organize the data it receives and compile statistics for each user.

[1265] "Bonus points" are points awarded to users as an incentive, and are granted when certain conditions are met.

[1266] "Emotional state" refers to the user's psychological state at that time, as analyzed from their facial expressions and voice.

[1267] "Means of analyzing facial expressions and voice" refer to devices and software that read emotions from a user's facial movements and voice.

[1268] "Means of adjustment" refers to functions that change the content of messages or the amount of points awarded based on specific conditions.

[1269] This invention is a system that detects when a user is walking while using their smartphone, recognizes the user's emotions, and then issues a warning message. In addition to a basic walking / smartphone detection function using an accelerometer and GPS sensor, this system incorporates an emotion engine.

[1270] Terminal-side embodiment

[1271] The device monitors the user's movement and changes in location using an accelerometer and GPS sensor. Specifically, the accelerometer measures the user's movement, and the GPS sensor confirms changes in location. If the device determines that the user is operating their smartphone while moving at regular intervals, it considers this to be walking while using a smartphone. For example, if a user is walking while operating their smartphone for one second, this will be detected. At this time, an emotion engine is activated that uses the camera and microphone to analyze facial expressions and voice in order to recognize the user's emotional state. The emotion engine analyzes the user's facial expression and voice data in real time to identify the user's current emotional state.

[1272] Display of warning message

[1273] When a user is detected walking while using their smartphone, the emotion engine analyzes the user's emotional state and generates a warning message based on that analysis. For example, if the user is feeling stressed, a message such as, "You seem tired today. For your safety, please take a short break before using your smartphone," will be displayed. For other emotional states, a standard warning message will be displayed.

[1274] Counting the number of times people walk while using their smartphones and transmitting data.

[1275] The device counts the number of times a user walks while using their smartphone and records this data internally. At the end of each month, this data is automatically sent to the server. Specifically, on the last day of the month, the device sends the data on the number of times a user walks while using their smartphone to the server, which receives it and stores it in its database.

[1276] The server aggregates the data and issues another warning.

[1277] The server aggregates the data sent from the terminals and calculates the number of times each user has used their smartphone while walking. Based on this, the server sends another warning message to the user. For example, a message like, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," is possible.

[1278] Bonus points awarded

[1279] The server has a function to identify users who do not use their smartphones while walking and to award them bonus points as an incentive. It extracts users from the database who have zero instances of using their smartphones while walking and awards them 10 bonus points per day. Furthermore, if a user's emotional state is positive, they may be awarded more points than the standard amount.

[1280] Hardware and software to be used

[1281] To implement this system, a mobile device with a built-in accelerometer and GPS sensor is required, as well as facial recognition software and voice analysis software to act as the emotion engine. Specifically, a general-purpose facial recognition library and voice analysis tool can be used for emotion recognition. In addition, a database management system and data aggregation software are required on the server.

[1282] Examples of specific cases and prompt statements

[1283] As a concrete example, consider a system where an autonomous vehicle detects pedestrians wearing smart glasses and issues a warning if they are using their smartphones while walking. The following is an example of a prompt to be input to the generating AI model:

[1284] Example of a prompt:

[1285] Please provide a code example for building a system that detects when a user is walking while using their smartphone and sends the most appropriate warning message based on their emotions. Please include the following sections:

[1286] 1. Code to acquire data from the accelerometer and GPS sensor.

[1287] 2. Code that recognizes the user's emotions and adjusts the warning message accordingly.

[1288] 3. Code to send a warning message to the user.

[1289] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1290] Step 1:

[1291] The device acquires data from the accelerometer and GPS sensor.

[1292] Input: Device's accelerometer, GPS sensor

[1293] Data processing: Collection of acceleration data and location data.

[1294] Output: User movement and location change information

[1295] Specific operation: The device's built-in accelerometer measures the user's movement data (accelerometer data), and the GPS sensor acquires the user's location information. This data is collected and used as basic data necessary for detecting walking while using a smartphone.

[1296] Step 2:

[1297] The device detects when the user is operating their smartphone while on the move.

[1298] Input: Acceleration data, location data

[1299] Data processing: Comparison of movement patterns and device operations at regular intervals.

[1300] Output: Detection results for walking while using a smartphone

[1301] Specific operation: The device determines whether the user is operating their smartphone while moving, based on acceleration data and location data collected at regular intervals. If it is determined that the user is using their smartphone while walking, that information is sent to the next step.

[1302] Step 3:

[1303] The device activates an emotion engine that analyzes the user's facial expressions and voice.

[1304] Input: User facial expression data (camera), audio data (microphone)

[1305] Data processing: facial expression recognition, voice analysis

[1306] Output: User's emotional state

[1307] Specific operation: The system uses the camera and microphone built into the device to acquire user facial expression and voice data. By analyzing this data, the system identifies the user's emotional state. The emotion engine uses, for example, a facial recognition library or a voice analysis tool.

[1308] Step 4:

[1309] The device generates and displays warning messages based on the user's emotional state.

[1310] Input: Emotional state, detection results of walking while using a smartphone

[1311] Data processing: Adjusting message content based on emotional state.

[1312] Output: Warning message

[1313] Specific operation: The device adjusts the content of the warning message based on the detected emotional state. For example, if the user is feeling stressed, it will generate a gentle message. The generated message will be displayed on the device's screen.

[1314] Step 5:

[1315] The device counts and records the number of times you use your smartphone while walking.

[1316] Input: Detection results of walking while using a smartphone

[1317] Data processing: Counting the number of times a person walks while using their smartphone, and recording it in the internal data.

[1318] Output: Count data

[1319] Specific operation: The device counts the number of times a user is using their phone while walking and records this data in its internal storage. The recorded data is later sent to a server.

[1320] Step 6:

[1321] At the end of each month, the device sends the counted data to the server.

[1322] Input: Data on the number of times people walk while using their smartphones.

[1323] Data processing: Preparing and sending data for transmission.

[1324] Output: Data sent to the server

[1325] Specific operation: At the end of each month, the device automatically sends data on the number of times the user is using their smartphone while walking to the server. This transmission process uses network communication functionality.

[1326] Step 7:

[1327] The server aggregates the data it receives and records it for each user.

[1328] Input: Data on the number of times people walk while using their smartphones.

[1329] Data processing: Data aggregation, creation of user-specific statistical data.

[1330] Output: Data on the number of times users walk while using their smartphones.

[1331] Specific operation: The server stores data on the number of times a user walks while using their smartphone, received from the terminal, into a database and generates statistical data for each user. Based on this, individual user reports are created.

[1332] Step 8:

[1333] The server will send the warning message to the user again.

[1334] Input: User-specific data on the number of times a user walks while using their smartphone.

[1335] Data processing: Generating warning messages

[1336] Output: Warning message

[1337] Specific operation: The server generates a warning message again based on each user's data on the number of times they use their smartphone while walking, and sends it to the user. The message is sent via smartphone or email.

[1338] Step 9:

[1339] The server will award bonus points to users who do not use their smartphones while walking.

[1340] Input: User-specific data on the number of times a user walks while using their smartphone.

[1341] Data processing: Calculation and awarding of bonus points

[1342] Output: Bonus Points

[1343] Specific operation: The server calculates bonus points for users who have zero instances of using their smartphones while walking, and records this in the database. As an incentive, the bonus points are added to the user's account.

[1344] Step 10:

[1345] The server adjusts bonus points based on the user's emotional state.

[1346] Input: Emotional state data, bonus points

[1347] Data processing: Adjustment of bonus points

[1348] Output: Adjusted bonus points

[1349] Specific operation: The server considers the user's emotional state data and adjusts the bonus points awarded to users in a positive emotional state to be above the standard level. The adjusted points are recorded in the database.

[1350] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1351] The data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1352] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1353] [Fourth Embodiment]

[1354] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1355] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1356] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1357] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1358] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1359] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1360] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1361] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1362] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1363] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1364] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1365] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1366] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1367] This invention provides a system that detects when a user is walking while using their smartphone, issues a warning, and further collects the number of times this has happened, thereby raising awareness and providing incentives to the user. This system uses an accelerometer and GPS sensor installed in the device to detect walking while using a smartphone.

[1368] The device uses an accelerometer to detect whether the user is operating their smartphone while walking, and a GPS sensor to check for changes in location. For example, if the user is operating their smartphone while moving at regular intervals (e.g., every second), the device will determine this as walking while using a smartphone. Once this determination is made, a warning message will be displayed on the screen. A specific example of such a message might be displayed: "You are currently walking while using your smartphone. For your safety, please stop and operate your phone."

[1369] This system also counts the number of times a user walks while using their smartphone and records it as internal data. At the end of each month, this count data is sent to a server. For example, the device automatically sends the data on the number of times a user walks while using their smartphone to the server on the last day of the month.

[1370] The server aggregates the received data and records the number of times each user has used their smartphone while walking. For example, the server stores each user's smartphone usage count in a database and checks the aggregated results at the end of the month. Based on this data, a generative AI sends another warning message to the user. For example, a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," might be sent.

[1371] Furthermore, the server identifies users who did not use their smartphones while walking and rewards them with PayPay points as an incentive. Specifically, users who did not use their smartphones while walking will receive 10 PayPay points per day. This is achieved by extracting users from the database who have zero instances of using their smartphones while walking and then awarding points to those users.

[1372] As described above, the present invention detects a user's walking while using their smartphone in real time and issues an immediate warning. Furthermore, it is a system that collects data on a server at the end of the month and provides incentives to promote safe behavior such as not walking while using a smartphone.

[1373] For example, if a user uses their smartphone while walking five times in a day, the device counts each instance, and at the end of the month, it sends the total count to the server. Based on this data, the server warns the user, "You used your smartphone while walking five times this month," and notifies users who did not use their smartphone while walking, "You have been awarded 10 PayPay points for the day."

[1374] This system is an effective means of ensuring user safety and raising awareness of the dangers of operating a smartphone while walking.

[1375] The following describes the processing flow.

[1376] Step 1:

[1377] The device activates its accelerometer and GPS sensor to monitor the user's movement and location. This involves measuring the user's movement with the accelerometer and checking for changes in location with the GPS sensor. Specifically, it uses AccelSensor().get_current_acceleration() and GPSSensor().get_current_position() to obtain current data.

[1378] Step 2:

[1379] The device acquires acceleration and location data again at regular intervals (e.g., every second) and determines if there are any changes. Specifically, time.sleep(1) is used to acquire and check data every second. This allows the system to determine if the user is continuously moving while operating the smartphone.

[1380] Step 3:

[1381] Based on the acquired data, the device will display a warning message on the screen if it determines that the user is using their phone while walking. For example, a message such as "You are currently using your phone while walking. Please stop and operate your phone for your safety." will be displayed using `display.show_message()`.

[1382] Step 4:

[1383] The device counts the number of times a user walks while using their phone as internal data and saves it to the device's storage area. For example, the count is incremented with `self.walk_count += 1` and saved with `local_storage.save("walk_count", self.walk_count)`.

[1384] Step 5:

[1385] At the end of the month, the device sends data on the number of times the user has walked while using their phone to the server. This includes a process that detects the end of the month and sends the data, such as `if datetime.datetime.now().day == datetime.date.today().replace(day=1).days_in_month: send_data_to_server()`.

[1386] Step 6:

[1387] The server receives data sent from the terminal and saves it to the database. Specifically, the data is saved using `database.save("user_walk_count", user_id, walk_count)`.

[1388] Step 7:

[1389] The server aggregates the data stored in the database and calculates the number of times each user walks while using their smartphone. For example, the aggregation is performed as follows: walk_count = database.get("user_walk_count", user_id).

[1390] Step 8:

[1391] Based on the aggregated results, the server generates and sends a warning message to each user corresponding to the number of times they have used their smartphone while walking. For example, it might send a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone."

[1392] Step 9:

[1393] The server extracts and identifies users who did not walk while using their smartphones. Specifically, it extracts these users from the database using `users_no_walking = database.query("SELECT user_id FROM users WHERE walk_count = 0")`.

[1394] Step 10:

[1395] The server awards bonus points to users who do not use their smartphones while walking and sends a notification message. For example, points are awarded using `for user_id in users_no_walking: grant_paypay_points(user_id, 10)` and a notification is sent stating, "10 PayPay points have been awarded for today."

[1396] (Example 1)

[1397] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1398] Using a smartphone while walking is a dangerous act for both the user and those around them, and can cause traffic accidents and falls. However, there is no system that can instantly detect when a user is operating a smartphone while walking and issue a warning. Furthermore, there is a lack of a system that collects data on users' smartphone-using behavior while walking and provides appropriate warnings and incentives based on that data. As a result, user awareness and improvement of behavior are insufficient.

[1399] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1400] In this invention, the server includes means for inputting a prompt sentence into a generating AI model based on received data and generating a warning message again, means for sending the generated warning message to the user based on the number of times the user has walked while using their smartphone, and means for awarding reward points to users who have not walked while using their smartphone. This enables users to recognize the dangers of walking while using their smartphone in real time and improve their behavior.

[1401] An "accelerometer" is a device used to detect the acceleration of an object and to measure the state of motion of that object.

[1402] A "GPS sensor" is a device that receives signals from satellites and determines the current location.

[1403] A "mobile device" is a portable electronic device that can be used while on the go, and includes smartphones, tablets, and other similar devices.

[1404] A "warning message" is a text or visual notification used to alert or warn a user.

[1405] "Counting" refers to the operation or process of counting the number of times a particular event has occurred.

[1406] A "server" is a computer system that provides various services and data processing to clients over a network.

[1407] A "database" is a software system that systematically stores data and manages it so that it can be easily searched and manipulated.

[1408] A "generative AI model" refers to an algorithm or program that uses artificial intelligence technology to automatically generate text and other content.

[1409] A "prompt statement" is an instruction or command given to a generative AI model, serving as a guideline for generating a specific output.

[1410] "Reward points" are incentives awarded to users who perform specific actions or meet certain conditions, and include those that can be used as monetary value or service benefits.

[1411] This invention is a system that detects the act of a user operating a smartphone while walking, commonly known as "walking while using a smartphone," and issues warnings and cautions in response. Furthermore, it records and aggregates the number of times a user walks while using a smartphone and provides incentives based on this information to promote improved user behavior.

[1412] The main components of the system are an accelerometer, GPS sensor, warning message display function, server, and generative AI model, all installed in the terminal.

[1413] Hardware and software to be used

[1414] 1. Accelerometer:

[1415] It is built into the device and detects the user's movement. Specifically, it is used to detect walking motion.

[1416] 2. GPS sensor:

[1417] It is built into the device and obtains the user's location information. By checking for changes in location information, it determines whether the user is moving.

[1418] 3. Terminal built-in software:

[1419] The system analyzes data from the accelerometer and GPS sensor to determine whether the user is operating their smartphone while walking. For example, if the user is moving and operating the screen every second, it is determined to be walking while using a smartphone.

[1420] 4. Warning message display function:

[1421] When walking while using a smartphone is detected, a warning message will be displayed on the screen. For example, a message such as "You are currently walking while using your smartphone. For your safety, please stop and operate your phone" will be displayed.

[1422] 5. Server:

[1423] The system receives and compiles data on the number of times users walk while using their smartphones, recording this data in a database for each user, and reviewing the results at the end of the month.

[1424] 6. Generative AI Models:

[1425] Based on the data collected by the server, prompts are entered to generate an appropriate warning message. The generated message is then sent to the user.

[1426] Example of a prompt message: "The user has used their smartphone while walking 50 times this month. Generate the following warning message: 'You have used your smartphone while walking 50 times this month. For your safety, please stop walking before using your smartphone.'"

[1427] 7. Incentive provision function:

[1428] The system extracts users from the database who have zero instances of walking while using their smartphones, and rewards these users with bonus points. For example, users who do not walk while using their smartphones will receive 10 bonus points per day.

[1429] Specific examples of system usage

[1430] When a user uses their smartphone while walking, the device's accelerometer and GPS sensor detect the movement and changes in location. The device recognizes this as walking while using a smartphone and displays a warning message on the screen: "You are currently using your smartphone while walking. For your safety, please stop and operate it."

[1431] Next, the system counts the number of times a user uses their phone while walking and records this data. At the end of the month, this data is sent to a server, which aggregates the data for each user. The server uses a generative AI model to generate a re-warning message and sends it to the user. A specific message such as, "User A used their phone while walking 5 times this month. For safety reasons, please stop walking before using your phone," is sent. In addition, users who do not use their phone while walking are rewarded with reward points.

[1432] In this way, the present invention is an effective method for ensuring user safety and raising awareness of the dangers of using a smartphone while walking, by detecting and warning against walking while using a smartphone in real time, and by providing incentives aimed at improving behavior.

[1433] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1434] Program processing flow

[1435] Step 1: Collect input data

[1436] The device collects data from its accelerometer and GPS sensor to detect whether the user is walking with their smartphone. The accelerometer detects movement, and the GPS sensor checks for changes in location.

[1437] Input: Acceleration data, GPS data

[1438] Output: Detected motion and position change data

[1439] Step 2: Detection of using a smartphone while walking

[1440] The device analyzes the collected sensor data to determine whether the user is operating their smartphone while moving at regular intervals. For example, if it detects movement and screen touches every second, it is determined to be walking while using a smartphone.

[1441] Input: Detected motion and position change data

[1442] Output: Result of detecting walking while using a smartphone

[1443] Specific action: The user walks 10 meters while touching the screen every second.

[1444] Step 3: Display of warning message

[1445] If the device detects that the user is using their phone while walking, it will immediately display a warning message on the screen. The message displayed will read, "You are currently using your phone while walking. For your safety, please stop and operate your phone."

[1446] Input: Result of detecting walking while using a smartphone

[1447] Output: Display of warning message

[1448] Specific action: The device displays the message, "You are currently using your phone while walking. For your safety, please stop and operate your phone."

[1449] Step 4: Count the number of times you use your phone while walking.

[1450] Each time the device detects that a user is walking while using their phone, it counts the number of times and records it in an internal database. This allows the device to accumulate data on how often a user has walked while using their phone.

[1451] Input: Result of detecting walking while using a smartphone

[1452] Output: Number of times people walked while using their smartphones were counted.

[1453] Specific operation: If detected 5 times in a day, the count is saved in internal memory.

[1454] Step 5: Send data on the number of times you walk while using your smartphone.

[1455] On the last day of the month, the device automatically sends accumulated data on the number of times users have walked while using their smartphone to the server.

[1456] Input: Data on the number of times people have walked while using their smartphones.

[1457] Output: Sending data to the server

[1458] Specific action: At 23:59 on the last day of the month, send data in the format "This month's number of times using a smartphone while walking is 5".

[1459] Step 6: Calculation

[1460] The server receives data sent from the terminals and records the number of times each user walks while using their smartphone in a database. The aggregated data is analyzed at the end of the month.

[1461] Input: Data on the number of times users have used their smartphones while walking.

[1462] Output: Aggregated database records

[1463] Specific action: The server records in the database that "User A used their smartphone while walking 5 times this month."

[1464] Step 7: Generate warning messages using a generative AI model.

[1465] The server inputs prompt messages into the AI ​​model, which then generates appropriate warning messages for each user.

[1466] Input: Aggregated data, prompt text

[1467] Output: Generated warning message

[1468] Specific prompt example: "The user has used their smartphone while walking 50 times this month. Please generate the following warning message: 'You have used your smartphone while walking 50 times this month. For your safety, please stop walking before using your smartphone.'"

[1469] Step 8: Sending a warning message

[1470] The server sends a warning message generated from the generated AI model to the user.

[1471] Input: Generated warning message

[1472] Output: Sending a message to the user

[1473] Specific action: A message is sent stating, "User A has used their smartphone while walking 5 times this month. For safety reasons, please stop walking before using your smartphone."

[1474] Step 9: Granting Incentives

[1475] The server extracts users from the database who have zero instances of using their smartphones while walking, and awards reward points to those users.

[1476] Input: Aggregated database list

[1477] Output: Reward points awarded

[1478] Specific action: A notification is sent stating, "User B has been awarded a total of 300 reward points."

[1479] The above outlines the specific processing steps of this system's program. This enables real-time detection, warning, and incentive provision for walking while using a smartphone, thereby ensuring user safety and promoting improvement of risky behaviors.

[1480] (Application Example 1)

[1481] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1482] The practice of using smartphones while walking, known as "walking while using a smartphone," increases the risk of traffic accidents and crime. However, currently, there is no system that not only displays warning messages but also collects data, analyzes user behavior, and issues appropriate warnings or provides incentives. Therefore, a more effective system is needed to ensure user safety and curb walking while using smartphones.

[1483] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1484] In this invention, the server includes means for sending prompt messages to the user and generating warning messages using a generation AI model, means for analyzing the user's tendency to walk while using a smartphone based on cumulative user data and generating corresponding warning content, and means for awarding bonus points to users who do not walk while using a smartphone. This makes it possible to significantly reduce the risks of walking while using a smartphone by analyzing user behavior in detail and providing appropriate warnings and incentives.

[1485] An "accelerometer" is a device that detects the acceleration of an object and is used in mobile devices to detect the user's movement.

[1486] A "GPS sensor" is a device that uses the Geographic Information System (GPS) to acquire location information and monitor changes in that location.

[1487] "Mobile devices" is a general term for electronic devices that can be carried and used, such as smartphones and tablets.

[1488] A "warning message" is a notification designed to alert a user to specific actions or situations.

[1489] "Means of recording" refers to systems and methods for storing data and information over long periods of time.

[1490] A "server" is a computer system used to store and process data on a network.

[1491] A "generative AI model" is an algorithm or model that uses machine learning and artificial intelligence technologies to automatically generate new information or messages.

[1492] A "prompt message" is a text message created by a generative AI model to encourage a specific action.

[1493] "Cumulative data" refers to the total amount of data accumulated over a certain period of time, as well as its records.

[1494] An "incentive" refers to a reward or perk offered to encourage a particular behavior.

[1495] "Means of analyzing trends" refers to methods and mechanisms for analyzing data and identifying specific patterns or trends.

[1496] "Bonus points" refer to additional points awarded to users who meet specific conditions.

[1497] This invention relates to a system that detects when a user is walking while using their smartphone, issues a warning, and further compiles the number of times this has happened, thereby providing awareness and incentives. The embodiments for carrying out this invention will be described in detail below.

[1498] composition

[1499] This system is primarily composed of the following hardware and software.

[1500] 1. Hardware

[1501] Mobile devices: Smartphones, tablets, etc.

[1502] Accelerometer sensor installed in mobile devices

[1503] GPS sensor installed in mobile devices

[1504] Servers that process and store data

[1505] 2. Software

[1506] Generative AI Models: Artificial intelligence algorithms for generating warning messages and prompt texts.

[1507] Sensor library for acquiring data from accelerometers and GPS sensors.

[1508] Python scripts and communication libraries (such as requests) for sending and analyzing data to and from the server.

[1509] operation

[1510] Detection of users walking while using their smartphones

[1511] 1. Mobile devices use accelerometers and GPS sensors to detect the user's movement.

[1512] 2. The accelerometer analyzes whether the user is walking.

[1513] 3. The GPS sensor monitors changes in location information.

[1514] If the device detects that a user is operating a mobile device while moving, it will display a warning message in real time. For example, the display might show a warning message such as, "You are currently using your phone while walking. Please stop and operate it for your safety."

[1515] Data recording and transmission

[1516] 1. The mobile device counts the number of times a person is using their smartphone while walking and records this as internal data.

[1517] 2. At the end of the month, the counted number of times a person was using their smartphone while walking is sent to the server.

[1518] Data aggregation and warning message generation

[1519] 1. The server aggregates the received data and records it for each user.

[1520] 2. Use a generative AI model to send a prompt message to the user and generate a warning message.

[1521] Specifically, the system analyzes user trends based on cumulative data of walking while using a smartphone and generates corresponding warning messages. An example of a prompt message is: "You have walked while using your smartphone XX times this month. For your safety, please stop walking before using your smartphone."

[1522] Granting of incentives

[1523] 1. The server will award bonus points to users who do not use their smartphones while walking. For example, users who do not use their smartphones while walking will receive 10 bonus points per day.

[1524] With the above configuration and operation, this invention can detect a user's walking while using a smartphone in real time, issue an immediate warning, and further provide incentives to promote safe behavior by aggregating the data at the end of the month.

[1525] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1526] Step 1:

[1527] Mobile devices detect user movement using accelerometers and GPS sensors. They acquire accelerometer and GPS data as input and analyze whether the user is walking. Based on this analysis, they obtain an output indicating that the user is moving.

[1528] Step 2:

[1529] The mobile device determines whether the user is operating their smartphone while moving. Based on acquired sensor data and screen operation status, it determines whether the user is walking while using their smartphone. Based on this determination, it outputs that the user is walking while using their smartphone.

[1530] Step 3:

[1531] The mobile device displays a warning message if it detects that the user is using their phone while walking. The input is the detection result for using a phone while walking, and a generation AI model is used to generate a prompt message. The output is a warning message that reads, "You are currently using your phone while walking. Please stop and operate it for your safety."

[1532] Step 4:

[1533] The mobile device counts the number of times a user is using their phone while walking and records this data internally. It takes the number of times walking while using a phone is detected as input and saves this information to internal memory. The stored data then outputs the number of times the user is using their phone while walking.

[1534] Step 5:

[1535] At the end of the month, the mobile device sends the counted number of times a user has walked while using their smartphone to the server. As input, it retrieves the count data stored internally and sends it to the server. As a result of this transmission, it receives output indicating that the count data has been saved on the server.

[1536] Step 6:

[1537] The server aggregates the received data and records it for each user. It receives data on the number of times a user has used their phone while walking as input and stores it in a database for each user. The stored data is then output as information on the number of times each user has used their phone while walking.

[1538] Step 7:

[1539] The server uses a generative AI model to send prompt messages to users and generate warning messages. It takes user data on the number of times they've used their phone while walking as input and generates a prompt message such as, "You've used your phone while walking XX times this month. For safety reasons, please stop before using your phone." This generated warning message is output.

[1540] Step 8:

[1541] The server awards bonus points to users who do not use their smartphones while walking. It retrieves user information for those who have used their smartphones while walking 0 times as input, and awards 10 bonus points per day to these users. The output is the result of this point awarding process.

[1542] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1543] This invention is a system that detects when a user is walking while using their smartphone, recognizes the user's emotions, and then issues a warning message. In addition to a basic walking / smartphone detection function using an accelerometer and GPS sensor, this system incorporates an emotion engine.

[1544] Detection of walking while using a smartphone and display of warning messages.

[1545] The device monitors the user's movement and changes in location using an accelerometer and a GPS sensor. Specifically, the accelerometer measures the user's movement, and the GPS sensor confirms changes in location. If the device determines that the user is operating their smartphone while moving at regular intervals, it considers this to be walking while using a smartphone. For example, if the user operates their smartphone while walking for one second, it will be detected.

[1546] Emotion recognition and message adjustment using an emotion engine.

[1547] If a user is walking while using their smartphone, the device activates its emotion engine to identify the user's current emotional state through facial recognition and voice analysis. For example, it uses the camera to analyze the user's facial expressions and estimates emotions from voice input. Based on this emotional information, the content and timing of warning messages are adjusted in real time.

[1548] For example, if a user is feeling stressed, the standard warning message will be changed to something like, "You seem tired today. For your safety, please take a short break before using your smartphone." This ensures that appropriate warnings are given, taking the user's feelings into consideration.

[1549] Counting and sending the number of times people walk while using their smartphones.

[1550] The device counts the number of times a user walks while using their smartphone and records this data internally. At the end of each month, this count data is sent to a server. For example, the device automatically sends the data on the number of times a user walks while using their smartphone to the server on the last day of the month.

[1551] Data aggregation by the server and a re-emphasis of warnings.

[1552] The server receives data sent from the terminal and stores it in a database. The received data is aggregated, and the number of times each user has used their smartphone while walking is calculated. Based on this, a generative AI sends another warning message to the user. For example, a message such as, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," is sent.

[1553] Bonus points awarded

[1554] Furthermore, the server identifies users who did not use their smartphones while walking and awards them bonus points as an incentive. For example, users who did not use their smartphones while walking receive 10 bonus points per day. This is achieved by extracting users with zero instances of using their smartphones while walking from the database and awarding points to those users.

[1555] Bonus point adjustment based on user emotions

[1556] The emotion engine also has a function that adjusts the bonus points a user receives according to their emotional state. For example, if a user is in a positive emotional state, they may be awarded more points than the standard amount.

[1557] As described above, the present invention is a system that detects a user's walking while using their smartphone in real time, provides appropriate warnings that take emotions into consideration, and offers incentives to promote safe behavior. For example, if a user walks while using their smartphone five times in a day, the emotion engine recognizes the user's state and displays an appropriate message, and sends this data to the server at the end of the month. The server aggregates the data and provides further warnings and bonus points.

[1558] The following describes the processing flow.

[1559] Step 1:

[1560] The device activates its accelerometer and GPS sensor to monitor the user's movement and location. The accelerometer measures the user's movement, and the GPS sensor confirms changes in position. Specifically, data is obtained using `AccelSensor().get_current_acceleration()` and `GPSSensor().get_current_position()`.

[1561] Step 2:

[1562] The device updates acceleration data and location data at regular intervals (for example, every second) to determine whether movement and changes in location meet certain conditions. Specifically, time.sleep(1) is used to acquire and check data every second.

[1563] Step 3:

[1564] If the device determines, based on the acquired data, that the user is using their smartphone while walking, it will display a warning message on the screen. For example, it might display a message such as, "You are currently using your smartphone while walking. Please stop and operate it for your safety," using display.show_message().

[1565] Step 4:

[1566] The device simultaneously activates an emotion engine and uses sensors such as cameras and microphones to detect the user's emotional state. For example, this may involve using a facial recognition camera and a voice recognition module to identify emotions from the user's facial expressions and voice.

[1567] Step 5:

[1568] The device adjusts the content and timing of warning messages in real time based on the user's emotions detected by the emotion engine. For example, if the user is feeling stressed, it will display a message such as, "You seem tired today. For your safety, please take a short break before using your smartphone."

[1569] Step 6:

[1570] The device counts the number of times the user is using their phone while walking as internal data and saves it to the device's storage area. For example, the count is incremented with `self.walk_count += 1` and the data is saved with `local_storage.save("walk_count", self.walk_count)`.

[1571] Step 7:

[1572] At the end of the month, the device sends the counted number of times a user has walked while using their phone to the server. Specifically, it uses `if datetime.datetime.now().day == datetime.date.today().replace(day=1).days_in_month: send_data_to_server()` to detect the end of the month and send the data.

[1573] Step 8:

[1574] The server receives data sent from the terminal and saves it to the database. The received data is saved using database.save("user_walk_count", user_id, walk_count).

[1575] Step 9:

[1576] The server aggregates the data and calculates the number of times each user walks while using their smartphone. For example, the data is extracted and aggregated using `walk_count = database.get("user_walk_count", user_id)`.

[1577] Step 10:

[1578] Based on the aggregated results, the server generates and sends a warning message to each user corresponding to the number of times they have used their smartphone while walking. For example, it might send a message such as, "You used your smartphone while walking 5 times this month. For your safety, please stop walking before using your smartphone."

[1579] Step 11:

[1580] The server identifies users who did not walk while using their smartphones and awards bonus points as an incentive to those identified. For example, it identifies target users using `users_no_walking = database.query("SELECT user_id FROM users WHERE walk_count = 0")` and awards points to those users.

[1581] Step 12:

[1582] The server uses an emotion engine to adjust bonus point allocation based on the user's emotions. Users with positive emotions may receive bonus points above the standard rate. For example, if a user expresses positive emotions, they may receive double the usual points.

[1583] (Example 2)

[1584] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1585] Conventional technologies lacked sufficient means to warn users of the dangers of walking while using a smartphone, and failed to provide warning messages that took into account the user's emotional state. Furthermore, the lack of a system to record the number of times users walked while using their smartphones and provide feedback to the user made it difficult to encourage improved behavior. In addition, there was no detailed system for providing incentives for refraining from walking while using a smartphone. As a result, it was difficult to continuously encourage safe behavior among users.

[1586] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1587] In this invention, the server includes means for detecting the user's movement using an acceleration sensor, means for acquiring and monitoring changes in location information using a location information acquisition device, means for displaying a warning message when it is determined that the user is operating a mobile device while moving, means for recognizing the user's emotions using facial recognition and voice analysis, means for adjusting the content of the warning message based on the recognized emotion information, means for counting and recording the number of times the user is using their smartphone while walking, means for transmitting the counted number of times the user is using their smartphone while walking to a recording device at the end of the month, means for the recording device to aggregate the received data and record it for each user, means for sending another warning message to the user based on the number of times the user is using their smartphone while walking, means for awarding reward points to users who do not use their smartphone while walking, and means for adjusting reward points based on the user's emotion information. This makes it possible to provide warning messages that take into account the user's emotional state, record and provide feedback on the number of times the user is using their smartphone while walking, and continuously encourage safe behavior by the user.

[1588] An "accelerometer" is a device that measures the user's speed and direction of movement.

[1589] A "location information acquisition device" is a device used to monitor and track a user's location.

[1590] A "mobile device" is an electronic device that a user can carry around and use for communication and information processing.

[1591] A "warning message" is a message displayed to alert the user.

[1592] "Facial recognition" is a technology that analyzes a user's facial expressions captured by a camera to identify their emotions.

[1593] "Voice analysis" is a technology that analyzes a user's voice collected through a microphone to identify their emotions.

[1594] "Emotional information" refers to data that indicates the user's current emotional state.

[1595] A "recording device" is a device used to store and save data.

[1596] "Reward points" are incentive points awarded to users for specific actions.

[1597] This invention is a system that detects when a user is operating a smartphone while walking, recognizes their emotions at that time, and issues an appropriate warning message. This system includes an accelerometer, a location information acquisition device, an emotion engine, and a server.

[1598] Hardware configuration

[1599] 1. An accelerometer (e.g., MPU-6050) is a device that measures the user's movement speed and direction.

[1600] 2. A location information acquisition device (e.g., U-blox NEO-6M) is a device for monitoring and tracking the user's location.

[1601] 3. A mobile device is an electronic device that a user can carry with them and performs communication and information processing, and includes a built-in camera and microphone.

[1602] 4. A server is a device for accumulating and storing data.

[1603] Software Configuration

[1604] 5. Facial recognition technology (e.g., OpenCV and dlib) is a technology that analyzes a user's facial expressions captured by a camera to identify their emotions.

[1605] 6. Speech analysis technology (e.g., Google Cloud Speech-to-Text) is a technology that analyzes a user's voice collected by a microphone to identify their emotions.

[1606] 7. The emotion engine is a software module that performs facial recognition and voice analysis to identify the user's emotional state.

[1607] System Functions

[1608] Detection of using a smartphone while walking

[1609] The device uses an accelerometer and a location information acquisition device to monitor the user's movement and changes in location. Next, the device analyzes the collected data at regular intervals and recognizes it as "walking while using a smartphone" if it determines that the user is operating their smartphone while walking.

[1610] Emotion recognition and adjustment of warning messages

[1611] If the device detects that the user is walking while using their smartphone, it activates its emotion engine. The device uses its camera and microphone to collect the user's facial expressions and voice, and analyzes them with the emotion engine. Based on the user's emotional state, the device generates an appropriate warning message and displays it on its display screen.

[1612] Specific example

[1613] For example, when a user is using their smartphone on a train platform, the device's accelerometer (MPU-6050) and location information acquisition device (U-blox NEO-6M) detect the user's movement. If the device detects that the user is walking while using their smartphone for one second, it displays a warning message such as, "Using your smartphone while walking is dangerous. Please stop before using it." Furthermore, if the device analyzes that the user is experiencing stress, it adjusts the message to something like, "You seem tired today. For your safety, please take a short break before using your smartphone."

[1614] Recording and sending the number of times I walk while using my smartphone.

[1615] Each time the device detects someone walking while using their smartphone, it records the number of occurrences in its internal database. On the last day of the month, the device sends the recorded number of times someone is walking while using their smartphone to a server.

[1616] Server-based data aggregation and resending of warning messages.

[1617] The server receives data on the number of times users have used their smartphones while walking from their devices and stores it in a database. The server then aggregates the number of times each user has used their smartphones while walking and uses a generation AI model to generate and send a warning message. For example, a possible message might be: "You have used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone."

[1618] Bonus points are awarded and adjustments are made based on emotions.

[1619] The server extracts users from the database who have zero instances of walking while using their smartphones and awards them reward points as an incentive. Furthermore, it adjusts the reward points based on the user's emotional state. For example, users with a positive emotional state are awarded points above the standard rate.

[1620] Examples of prompts for generative AI models

[1621] The following are possible inputs for a generative AI model.

[1622] Let's assume the system detects that the user is walking while using their smartphone. Using data from the accelerometer and location information acquisition device, the system uses an emotion engine to recognize the user's emotions (through facial expressions and voice analysis). If the user is experiencing stress, generate an appropriate warning message.

[1623] Thus, the system of the present invention can continuously promote safe behavior by providing real-time alerts and incentives that take into account the user's emotional state.

[1624] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1625] Step 1:

[1626] The device detects that the user is operating a smartphone.

[1627] Input: User's smartphone operation (touchscreen use, app launching, etc.)

[1628] Data processing: None

[1629] Output: Smartphone operation detection signal

[1630] Specific operation: A system service on the device monitors the smartphone's operation events.

[1631] Step 2:

[1632] The terminal monitors the user's movement and location information using an accelerometer and a location information acquisition device.

[1633] Input: Data from acceleration sensor (MPU-6050), data from location information acquisition device (U-blox NEO-6M)

[1634] Data processing: Real-time analysis of acceleration data and location information.

[1635] Output: Changes in movement and position information

[1636] Specific operation: The device collects acceleration data at regular intervals and obtains location information. The collected data is analyzed to determine whether movement has occurred and whether the location has changed.

[1637] Step 3:

[1638] The device determines whether the user is operating their smartphone while walking.

[1639] Input: Changes in movement and location information, detection signals from smartphone operation.

[1640] Data processing: Analysis of data over a specific period (e.g., 1 second).

[1641] Output: Result of detecting walking while using a smartphone (True / False)

[1642] Specific operation: To determine whether the user is walking and operating a smartphone, data from one second is integrated and analyzed.

[1643] Step 4:

[1644] The device displays a warning message if it detects someone walking while using their smartphone.

[1645] Input: Result of detecting walking while using a smartphone

[1646] Data processing: None

[1647] Output: Command to display warning message

[1648] Specific action: The device displays a warning message on its screen (e.g., "Using your phone while walking is dangerous. Please stop before using it.").

[1649] Step 5:

[1650] The device activates an emotion engine when it detects someone walking while using their smartphone, and recognizes the user's emotions.

[1651] Input: Camera video (facial expression data), microphone audio (audio data)

[1652] Data processing: Analysis of facial expression data and voice data (using OpenCV, dlib, and Google Cloud Speech-to-Text).

[1653] Output: Emotional information (e.g., stress, joy, etc.)

[1654] Specific operation: The device captures the user's face with its camera and collects audio with its microphone. This data is then analyzed by an emotion engine to recognize the user's emotions.

[1655] Step 6:

[1656] The device adjusts the content of the warning message based on the emotional information it recognizes.

[1657] Input: Sentiment information, existing warning messages

[1658] Data processing: Generating messages based on emotional information

[1659] Output: Adjusted warning message

[1660] Specific action: For example, if the user is feeling stressed, a message will be displayed saying, "You seem tired today. For your safety, please take a short break before using your smartphone."

[1661] Step 7:

[1662] The device counts the number of times a user walks while using their smartphone and records the data in an internal database.

[1663] Input: Result of detecting walking while using a smartphone

[1664] Data processing: Incrementing the count, recording to the database.

[1665] Output: Updated number of times people have walked while using their smartphones.

[1666] Specific action: Each time walking while using a smartphone is detected, the recorded count increases by one.

[1667] Step 8:

[1668] At the end of the month, the device sends data on the number of times the user has walked while using their smartphone to the server.

[1669] Input: Data on the number of times people walk while using their smartphones.

[1670] Data processing: Data format conversion, application of transmission protocols.

[1671] Output: Sending data to the server

[1672] Specific operation: On the last day of the month, the device converts the counted number of times a user has walked while using their phone into an appropriate format and sends it to the server.

[1673] Step 9:

[1674] The server aggregates the received data and records it for each user.

[1675] Input: Data on the number of times people walk while using their smartphones.

[1676] Data processing: Data aggregation and recording for each user.

[1677] Output: Aggregated data on walking while using a smartphone

[1678] Specific operation: The server aggregates the received data for each user and stores it in the database.

[1679] Step 10:

[1680] The server sends another warning message to the user based on the number of times they have used their phone while walking.

[1681] Input: Aggregated data on walking while using a smartphone

[1682] Data processing: Generating warning messages using a generative AI model.

[1683] Output: Sending a re-warning message

[1684] Specific operation: Based on the number of times each user walks while using their smartphone, the server uses a generative AI model to generate an appropriate warning message and sends it to the user.

[1685] Step 11:

[1686] The server will award reward points to users who do not use their smartphones while walking.

[1687] Input: Aggregated data on walking while using a smartphone

[1688] Data processing: Calculation and awarding of reward points

[1689] Output: User accounts to which reward points have been awarded

[1690] Specific operation: The server calculates reward points as an incentive for users who have zero instances of using their smartphones while walking during the month, and adds them to the user's account.

[1691] Step 12:

[1692] The server adjusts reward points based on the user's sentiment information.

[1693] Input: Sentiment information, reward point data

[1694] Data processing: Adjusting reward points based on emotional information

[1695] Output: Adjusted reward points

[1696] Specific action: Users in a positive emotional state will be awarded points above the standard level, and their reward points will be adjusted accordingly.

[1697] (Application Example 2)

[1698] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1699] Conventional pedestrian detection systems have faced challenges in effectively detecting "walking while using a smartphone" behavior and issuing appropriate warnings. In particular, they lacked the technology to send messages that took into account the user's emotional state, making them ineffective in improving user responses and behavior. Furthermore, they lacked means of providing incentives to promote safe behavior.

[1700] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1701] In this invention, the server includes means for detecting the user's movement using an acceleration sensor, means for acquiring and monitoring changes in location information using a GPS sensor, means for displaying a warning message when it is determined that the user is operating a mobile device while moving, means for counting and recording the number of times the user is using their smartphone while walking, means for sending the counted number of times the user is using their smartphone while walking to the server at the end of the month, means for the server to aggregate the received data and record it for each user, means for sending another warning message to the user based on the number of times the user is using their smartphone while walking, means for awarding bonus points to users who do not use their smartphone while walking, means for analyzing the user's emotional state from their facial expressions and voice and adjusting the warning message accordingly, and means for adjusting bonus points according to the user's emotional state. This makes it possible to provide appropriate warning messages tailored to the user's current emotional state and to provide incentives to curb smartphone use while walking.

[1702] An "accelerometer" is a device used to detect the acceleration of a user's movements and actions.

[1703] A "GPS sensor" is a device used to acquire location information on Earth and monitor changes in that location.

[1704] A "mobile device" is an electronic device that a user carries and uses, and specifically refers to a smartphone.

[1705] A "warning message" is a notification designed to remind users to stop using their smartphones while walking.

[1706] "Means of counting" refers to the devices or functions necessary to record the number of times someone is walking while using their smartphone.

[1707] A "server" is a computer system that collects and manages data and sends warning messages and bonus points to users.

[1708] "Means of aggregation" refers to a function that allows the server to organize the data it receives and compile statistics for each user.

[1709] "Bonus points" are points awarded to users as an incentive, and are granted when certain conditions are met.

[1710] "Emotional state" refers to the user's psychological state at that time, as analyzed from their facial expressions and voice.

[1711] "Means of analyzing facial expressions and voice" refer to devices and software that read emotions from a user's facial movements and voice.

[1712] "Means of adjustment" refers to functions that change the content of messages or the amount of points awarded based on specific conditions.

[1713] This invention is a system that detects when a user is walking while using their smartphone, recognizes the user's emotions, and then issues a warning message. In addition to a basic walking / smartphone detection function using an accelerometer and GPS sensor, this system incorporates an emotion engine.

[1714] Terminal-side embodiment

[1715] The device monitors the user's movement and changes in location using an accelerometer and GPS sensor. Specifically, the accelerometer measures the user's movement, and the GPS sensor confirms changes in location. If the device determines that the user is operating their smartphone while moving at regular intervals, it considers this to be walking while using a smartphone. For example, if a user is walking while operating their smartphone for one second, this will be detected. At this time, an emotion engine is activated that uses the camera and microphone to analyze facial expressions and voice in order to recognize the user's emotional state. The emotion engine analyzes the user's facial expression and voice data in real time to identify the user's current emotional state.

[1716] Display of warning message

[1717] When a user is detected walking while using their smartphone, the emotion engine analyzes the user's emotional state and generates a warning message based on that analysis. For example, if the user is feeling stressed, a message such as, "You seem tired today. For your safety, please take a short break before using your smartphone," will be displayed. For other emotional states, a standard warning message will be displayed.

[1718] Counting the number of times people walk while using their smartphones and transmitting data.

[1719] The device counts the number of times a user walks while using their smartphone and records this data internally. At the end of each month, this data is automatically sent to the server. Specifically, on the last day of the month, the device sends the data on the number of times a user walks while using their smartphone to the server, which receives it and stores it in its database.

[1720] The server aggregates the data and issues another warning.

[1721] The server aggregates the data sent from the terminals and calculates the number of times each user has used their smartphone while walking. Based on this, the server sends another warning message to the user. For example, a message like, "You used your smartphone while walking XX times this month. For your safety, please stop walking before using your smartphone," is possible.

[1722] Bonus points awarded

[1723] The server has a function to identify users who do not use their smartphones while walking and to award them bonus points as an incentive. It extracts users from the database who have zero instances of using their smartphones while walking and awards them 10 bonus points per day. Furthermore, if a user's emotional state is positive, they may be awarded more points than the standard amount.

[1724] Hardware and software to be used

[1725] To implement this system, a mobile device with a built-in accelerometer and GPS sensor is required, as well as facial recognition software and voice analysis software to act as the emotion engine. Specifically, a general-purpose facial recognition library and voice analysis tool can be used for emotion recognition. In addition, a database management system and data aggregation software are required on the server.

[1726] Examples of specific cases and prompt statements

[1727] As a concrete example, consider a system where an autonomous vehicle detects pedestrians wearing smart glasses and issues a warning if they are using their smartphones while walking. The following is an example of a prompt to be input to the generating AI model:

[1728] Example of a prompt:

[1729] Please provide a code example for building a system that detects when a user is walking while using their smartphone and sends the most appropriate warning message based on their emotions. Please include the following sections:

[1730] 1. Code to acquire data from the accelerometer and GPS sensor.

[1731] 2. Code that recognizes the user's emotions and adjusts the warning message accordingly.

[1732] 3. Code to send a warning message to the user.

[1733] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1734] Step 1:

[1735] The device acquires data from the accelerometer and GPS sensor.

[1736] Input: Device's accelerometer, GPS sensor

[1737] Data processing: Collection of acceleration data and location data.

[1738] Output: User movement and location change information

[1739] Specific operation: The device's built-in accelerometer measures the user's movement data (accelerometer data), and the GPS sensor acquires the user's location information. This data is collected and used as basic data necessary for detecting walking while using a smartphone.

[1740] Step 2:

[1741] The device detects when the user is operating their smartphone while on the move.

[1742] Input: Acceleration data, location data

[1743] Data processing: Comparison of movement patterns and device operations at regular intervals.

[1744] Output: Detection results for walking while using a smartphone

[1745] Specific operation: The device determines whether the user is operating their smartphone while moving, based on acceleration data and location data collected at regular intervals. If it is determined that the user is using their smartphone while walking, that information is sent to the next step.

[1746] Step 3:

[1747] The device activates an emotion engine that analyzes the user's facial expressions and voice.

[1748] Input: User facial expression data (camera), audio data (microphone)

[1749] Data processing: facial expression recognition, voice analysis

[1750] Output: User's emotional state

[1751] Specific operation: The system uses the camera and microphone built into the device to acquire user facial expression and voice data. By analyzing this data, the system identifies the user's emotional state. The emotion engine uses, for example, a facial recognition library or a voice analysis tool.

[1752] Step 4:

[1753] The device generates and displays warning messages based on the user's emotional state.

[1754] Input: Emotional state, detection results of walking while using a smartphone

[1755] Data processing: Adjusting message content based on emotional state.

[1756] Output: Warning message

[1757] Specific operation: The device adjusts the content of the warning message based on the detected emotional state. For example, if the user is feeling stressed, it will generate a gentle message. The generated message will be displayed on the device's screen.

[1758] Step 5:

[1759] The device counts and records the number of times you use your smartphone while walking.

[1760] Input: Detection results of walking while using a smartphone

[1761] Data processing: Counting the number of times a person walks while using their smartphone, and recording it in the internal data.

[1762] Output: Count data

[1763] Specific operation: The device counts the number of times a user is using their phone while walking and records this data in its internal storage. The recorded data is later sent to a server.

[1764] Step 6:

[1765] At the end of each month, the device sends the counted data to the server.

[1766] Input: Data on the number of times people walk while using their smartphones.

[1767] Data processing: Preparing and sending data for transmission.

[1768] Output: Data sent to the server

[1769] Specific operation: At the end of each month, the device automatically sends data on the number of times the user is using their smartphone while walking to the server. This transmission process uses network communication functionality.

[1770] Step 7:

[1771] The server aggregates the data it receives and records it for each user.

[1772] Input: Data on the number of times people walk while using their smartphones.

[1773] Data processing: Data aggregation, creation of user-specific statistical data.

[1774] Output: Data on the number of times users walk while using their smartphones.

[1775] Specific operation: The server stores data on the number of times a user walks while using their smartphone, received from the terminal, into a database and generates statistical data for each user. Based on this, individual user reports are created.

[1776] Step 8:

[1777] The server will send the warning message to the user again.

[1778] Input: User-specific data on the number of times a user walks while using their smartphone.

[1779] Data processing: Generating warning messages

[1780] Output: Warning message

[1781] Specific operation: The server generates a warning message again based on each user's data on the number of times they use their smartphone while walking, and sends it to the user. The message is sent via smartphone or email.

[1782] Step 9:

[1783] The server will award bonus points to users who do not use their smartphones while walking.

[1784] Input: User-specific data on the number of times a user walks while using their smartphone.

[1785] Data processing: Calculation and awarding of bonus points

[1786] Output: Bonus Points

[1787] Specific operation: The server calculates bonus points for users who have zero instances of using their smartphones while walking, and records this in the database. As an incentive, the bonus points are added to the user's account.

[1788] Step 10:

[1789] The server adjusts bonus points based on the user's emotional state.

[1790] Input: Emotional state data, bonus points

[1791] Data processing: Adjustment of bonus points

[1792] Output: Adjusted bonus points

[1793] Specific operation: The server considers the user's emotional state data and adjusts the bonus points awarded to users in a positive emotional state to be above the standard level. The adjusted points are recorded in the database.

[1794] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1795] The data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1796] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1797] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1798] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1799] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1800] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1801] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1802] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1803] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1804] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1805] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1806] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1807] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the networ...

Claims

1. A means for detecting user movement using an accelerometer, A means for acquiring and monitoring changes in location information using a GPS sensor, A means of displaying a warning message when it is determined that the user is operating a mobile device while moving, A method for counting and recording the number of times someone walks while using their smartphone, A method for sending the number of times people are walking while using their smartphones, counted at the end of the month, to a server, A means of aggregating the data received by the server and recording it for each user, A means of sending a warning message to the user again based on the number of times they have used their smartphone while walking, A system that includes a mechanism for awarding bonus points to users who do not use their smartphones while walking.

2. The system according to claim 1, comprising means for monitoring changes in acceleration and location information at regular intervals in order to detect when a user is walking while using their smartphone.

3. The system according to claim 1, which includes means for visually displaying a warning message on a display when displaying a warning message.

Images