System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing an image; means for performing character recognition on the captured image; means for formatting the character recognized data; and means for importing the formatted data into a calendar application.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, family schedules are handwritten on a paper calendar, while individual schedules are managed using a calendar app, resulting in dual management. This method fragments information, making efficient management difficult. There is also a need to maintain the convenience of paper calendars. Furthermore, the time and cost involved in digitizing handwritten content is also an issue. There is a need to solve these issues and realize efficient and smart schedule management. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means. First, an image acquisition means is provided, allowing a handwritten calendar to be easily photographed using a smartphone or other device. Next, a character recognition means is used to convert the handwritten character information from the acquired image into digital data. This data is then converted into a format that can be understood by a calendar app using a formatting means. Finally, a means is used to import the formatted data into the calendar app, allowing schedules to be easily digitized and shared.

[0006] In addition, a means for analyzing the acquired image, including color-coded information, and identifying each individual's schedule is added, making schedule management even more efficient. Furthermore, a means for saving formatted data and sending it to the user's terminal is provided, making it easy to access and share the data. These means of the present invention enable efficient schedule management while maintaining the convenience of a handwritten calendar.

[0007] "Means for acquiring images" refers to equipment or technology for taking images of handwritten calendars and other items and acquiring the data in a digital format.

[0008] "Character recognition means" is a technology that reads handwritten character information from an image and converts it into digital text data.

[0009] A "formatting method" is a technique or process that converts recognized digital text data into a format that is easy for a calendar application to understand (e.g., iCalendar format).

[0010] "Means for importing into a calendar app" refers to the functionality or process for bringing formatted data into a calendar app.

[0011] "Color-coding information" is information for distinguishing between individual schedules, which are expressed using different colors on a handwritten calendar.

[0012] "Means for analysis" refers to the techniques and processes for analyzing images and data in detail and extracting or identifying the necessary information.

[0013] "Data storage" refers to the technology or process used to securely hold formatted data in a database or storage.

[0014] "Means for transmitting to the user's terminal" refers to a communication technology for transmitting data from the server to the user's device (such as a smartphone). [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention provides a system for digitizing handwritten calendars and managing them in a unified manner using a calendar application. Specific embodiments of the system are described below.

[0037] Server Processing

[0038] 1. Receiving image uploads

[0039] When a handwritten calendar image is uploaded from a user's terminal, the server receives it and stores the received image data in a temporary directory.

[0040] 2. Image Analysis

[0041] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[0042] 3. Data Formatting

[0043] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI. Using scripts or application logic, the extracted information is converted into a format that the calendar app can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify who owns a particular event.

[0044] 4. Data storage and transmission

[0045] The server stores the formatted data in a database and sends a message to the user's device to notify them that the data is ready. When the user makes a request to retrieve new calendar data, the server sends the data to the device.

[0046] Terminal handling

[0047] 1. Image capture

[0048] The user takes a photo of the handwritten calendar using the smartphone camera, and checks the image to make sure the entire calendar is clearly visible.

[0049] 2. Upload an image

[0050] When the user presses the image upload button, the device uploads the captured image data to the server. The device sends the image data to the server using an HTTP request.

[0051] 3. Receiving and importing data

[0052] The device that receives the notification from the server launches the calendar app with the user's confirmation. The calendar app communicates with the server and sends a request to obtain new formatted data. Upon receiving the formatted calendar data from the server, the calendar app imports it and adds new events to the existing calendar. The calendar app uses color-coding information to distinguish and display each event for each individual.

[0053] User operations

[0054] 1. Calendar photoshoot

[0055] The user takes a photo of a handwritten calendar using their smartphone, and the events are color-coded to indicate who owns them.

[0056] 2. Upload confirmation

[0057] When uploading a captured image to a server, the user performs a confirmation operation.

[0058] 3. Data confirmation

[0059] The user can check their calendar app to ensure the new events have been imported correctly, and if necessary, they can make additional edits or adjustments.

[0060] Specific examples

[0061] Example 1: Image upload and analysis

[0062] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server then uses generative AI to analyze the text in the image and convert each person's schedule into digital text. The text is then formatted and saved in a calendar app format, and the user's device is notified.

[0063] Example 2: Data import

[0064] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new event data from the server and automatically imports it into their calendar. The user can then check each person's events, which are correctly reflected in the calendar, and edit them if necessary.

[0065] The system of the present invention allows for centralized digital management while maintaining the convenience of a handwritten calendar, making it easier to share and coordinate schedules and allowing all family members to quickly understand the latest information.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The user uses the smartphone camera to take a picture of a handwritten calendar hanging on the wall, and checks the image to make sure the entire calendar is clearly visible.

[0069] Step 2:

[0070] The device prepares to upload the captured image data to the server through the application interface. When the user presses the upload button, the device sends the image data to the server via an HTTP request.

[0071] Step 3:

[0072] The server receives the image upload request sent from the user's terminal, saves the image data in a temporary directory, and sends a response confirming receipt to the terminal.

[0073] Step 4:

[0074] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[0075] Step 5:

[0076] The server extracts the event date, time, content, and color-coded information from the parsed text data, using scripts or application logic to convert the event data into a format that the calendar app can understand (e.g., iCalendar format).

[0077] Step 6:

[0078] The server stores the formatted data in a database, a process that ensures that the scheduled data is easily accessible at a later time and is accurately maintained.

[0079] Step 7:

[0080] The server sends a message to the user's device to notify them that the formatted data is ready, possibly via push notification or email.

[0081] Step 8:

[0082] The device receives a notification from the server informing the user that new calendar data is available. The user confirms the notification and launches the calendar app.

[0083] Step 9:

[0084] The device's calendar app sends a request to the server to get the new format data, and the server sends the saved format data to the device.

[0085] Step 10:

[0086] The calendar application on the device imports the received format data and adds new events to the existing calendar. The calendar application uses color-coding information to distinguish and display each event for each individual.

[0087] Step 11:

[0088] The user can verify that the new event has been imported correctly within their calendar app, and can make additional edits or adjustments if needed.

[0089] Through these processing steps, the handwritten calendar information is digitized and can be shared with the whole family via a smartphone calendar app.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] Systems that digitize and centrally manage handwritten calendars face the problem of cumbersome and time-consuming manual data entry. It is also difficult to manage individual appointments by color-coding them, making sharing and coordination difficult. Furthermore, there is a need to improve the accuracy of handwritten character recognition and accurately convert them into digital data.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes a means for receiving and temporarily storing image data from the user's device, a means for inputting the image data into a generative AI model and analyzing handwritten characters, and a means for converting the generated character recognition data into a format that can be understood by a calendar application. This allows handwritten calendars to be digitized and centrally managed, and schedules to be color-coded and shared.

[0095] "Means for acquiring images" refers to devices or software that allow a user to take a photo of a handwritten calendar with a smartphone or camera and acquire the image data.

[0096] The "means for character recognition of acquired images" refers to software or algorithms for analyzing handwritten characters from acquired image data and extracting them as digital text data.

[0097] A "means for formatting character-recognized data" is software or script that converts text data obtained by character recognition into a specific data format that can be recognized by a calendar application (e.g., iCalendar format).

[0098] "Means for saving formatted data in a database and transmitting it to a user's terminal" refers to communication means or software that saves the converted data in a database and transmits it to a user's terminal as needed.

[0099] The "means for receiving image data from the user's terminal and temporarily storing it" refers to server-side storage or software that receives image data sent from the user's terminal and temporarily stores it.

[0100] The "means of inputting image data into a generative AI model and analyzing handwritten characters" refers to a method of supplying received and stored image data to a generative AI model and converting handwritten characters into text data.

[0101] "Means for converting the generated character recognition data into a format understandable by a calendar application" refers to software or scripts that convert the text data output by the generative AI model into a format that can be imported and understood by a calendar application (e.g., iCalendar format).

[0102] The "means for sending a notification to the user's terminal that the data is ready" refers to a communication function or software for notifying the user's terminal that the formatted data is ready.

[0103] The present invention provides a system for digitizing handwritten calendars and managing them in a unified manner using a calendar application. Specific embodiments of the system are described below.

[0104] System Overview

[0105] The system starts when a user takes a photo of a handwritten calendar with their smartphone and uploads it to a server. The server analyzes the uploaded image using a generative AI model and converts it into digital data. The converted data is then formatted into a format that the calendar app can understand and saved in a database. Finally, the formatted data is sent to the user's device and imported into the calendar app.

[0106] Hardware and Software

[0107] Smartphone: Users take a photo of the handwritten calendar using the camera on their smartphone. iPhones, Android devices, etc. can be used.

[0108] Server: A server is used to receive, analyze, format, store, and transmit image data. Specifically, a web framework such as Flask or Django is used.

[0109] Generative AI models: used to analyze handwritten characters in images, for example, using pre-trained models using PyTorch or TensorFlow.

[0110] OCR technology: Google Cloud Vision API and Tesseract are used to analyze handwritten characters in images.

[0111] Database: Use a database such as MySQL to store the analyzed data.

[0112] Calendar app: Use an application such as Google Calendar to import digital data.

[0113] Processing Flow

[0114] A user uses the system in the following steps:

[0115] 1. Image capture

[0116] A user takes a photo of a handwritten calendar using the camera on their smartphone. When taking the photo, they color-code the events so that it is clear who owns each one. For example, they could take a photo of a handwritten family calendar and mark each event with a different color.

[0117] 2. Upload an image

[0118] When a user presses the image upload button in the app, the smartphone uploads the captured image data to the server. This is done by sending the data to the server using an HTTP POST request. For example, the image data is POSTed to the / upload_image endpoint of the Flask server.

[0119] 3. Image Analysis

[0120] The server reads the saved image data and passes it to the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data. For example, it can call the Google Cloud Vision API to perform character recognition.

[0121] 4. Data Formatting

[0122] The server extracts date, time, content, and color-coded information from the text data output by the generation AI. Using a script (e.g., a Python script), it converts the extracted information into a format understandable by a calendar app, such as iCalendar format. It adds the color-coded information as metadata. For example, data such as "{"date": "2023-10-01", "events": ["Meeting", "Meeting"]}" is converted to the format "BEGIN:VEVENT\nDTSTART:20231001T090000Z\nSUMMARY:Meeting\nEND:VEVENT\n".

[0123] 5. Data storage and transmission

[0124] The server saves the formatted data in a database (e.g., MySQL) and notifies the user's device that the data is ready. Specifically, this is done using WebSocket or push notification. For example, the server inserts iCalendar format data into the calendar_events table in a MySQL database and sends a message to the user's device via WebSocket saying "Calendar data has been updated."

[0125] 6. Receiving and Importing Data

[0126] The device that receives the notification from the server displays the notification to the user and launches the calendar app. The calendar app communicates with the server to send a request for new data and receives the formatted data. The received data is then imported into the calendar app, where events are displayed in color.

[0127] Specific examples

[0128] Example 1: Image upload and analysis

[0129] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server then uses a generative AI model to analyze the text in the image and convert each person's schedule into digital text. The text is then formatted and saved in a calendar app format, and the user's device is notified.

[0130] Example 2: Data import

[0131] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new event data from the server and automatically imports it into their calendar. The user can then check each person's events, which are correctly reflected in the calendar, and edit them if necessary.

[0132] Prompt Sentence Examples

[0133] Please explain the system that takes a photo of a family's handwritten calendar with a smartphone, digitizes it on the cloud, and imports it into a calendar app. For example, what steps are taken to process the data, and how does the user operate it?

[0134] This invention digitizes handwritten calendars and centralizes their management, making it easier to share and coordinate schedules, and allowing all family members to quickly grasp the latest information.

[0135] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0136] Step 1:

[0137] Image capture

[0138] A user launches the camera app on their smartphone and takes a photo of a handwritten calendar. When taking the photo, they color-code the events to identify who owns them. For example, they can take a photo of a family calendar and mark each event with a different color.

[0139] Input: Handwritten calendar

[0140] Output: Captured image data (JPEG or PNG format)

[0141] Step 2:

[0142] Image upload

[0143] When a user presses the image upload button in the app, the device sends the captured image data to the server using an HTTP POST request. For example, the image data is POSTed to http: / / example.com / upload_image.

[0144] Input: Captured image data

[0145] Output: HTTP POST request to the server

[0146] Step 3:

[0147] Receiving and saving images on the server

[0148] The server receives the image data sent from the user's device and stores it in a temporary directory. For example, it processes the request using a web framework such as Flask or Django and stores the image file as / tmp / uploaded_images / image1.jpg.

[0149] Input: HTTP POST request (image data)

[0150] Output: Temporarily saved image file

[0151] Step 4:

[0152] Image analysis

[0153] The server reads the temporarily saved image data and inputs it into the generative AI model. The generative AI model analyzes the handwritten characters in the image using OCR technology (e.g., Google Cloud Vision API or Tesseract) and converts them into text data. For example, input / tmp / uploaded_images / image1.jpg into the Google Cloud Vision API and obtain JSON-formatted text data as the character recognition results.

[0154] Input: Saved image file

[0155] Output: Parsed text data (JSON format)

[0156] Step 5:

[0157] Data Formatting

[0158] The server extracts date, time, content, and color-coding information from the text data output by the generative AI model. It then uses a script (e.g., a Python script) to convert the extracted information into a format that calendar apps can understand, such as iCalendar, and adds the color-coding information as metadata.

[0159] Input: Parsed text data (JSON format)

[0160] Output: Formatted calendar data (iCalendar format)

[0161] Step 6:

[0162] Data Retention and Notification

[0163] The server saves the formatted data in a database (e.g., MySQL) and sends a notification to the user's device that the data is ready. For example, it inserts the iCalendar format data into the calendar_events table in the MySQL database and sends a message to the user's device via WebSocket saying "Calendar data has been updated."

[0164] Input: Formatted calendar data (iCalendar format)

[0165] Output: Calendar data stored in the database, and notification messages

[0166] Step 7:

[0167] Receiving and Importing Data

[0168] When the device receives the notification from the server, it displays the notification to the user and launches the calendar app. The calendar app then communicates with the server and sends a request to retrieve new data. The formatted calendar data is then received and automatically imported into the calendar. For example, the calendar app may use a REST API to retrieve iCalendar formatted data, import it into the calendar, and display it.

[0169] Input: Notification message, formatted calendar data

[0170] Output: New events imported into the Calendar app

[0171] This process converts handwritten calendars into digital data and efficiently manages them centrally. Users can easily digitize their handwritten calendars and manage them in a calendar app.

[0172] (Application example 1)

[0173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0174] Existing factory work schedule management methods often require a lot of time and effort in the process of digitizing handwritten schedules, making it difficult to streamline the process. Furthermore, it is difficult to integrate and manage the schedules of multiple individuals, who need to be color-coded or otherwise differentiated, making centralized schedule management difficult. The objective of this invention is to efficiently and accurately digitize handwritten work schedules and realize centralized schedule management.

[0175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0176] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into a calendar application, and means for importing the formatted data into a work schedule system. This makes it possible to efficiently digitize handwritten work schedules and centrally manage them, including each individual's schedule.

[0177] The "means for acquiring images" has a function that allows a user to take a photograph of a handwritten calendar or work schedule and send it to the server in digital image format.

[0178] "Means for character recognition" refers to a technology that extracts character information from an acquired image and converts it into digital text format, and uses OCR technology.

[0179] The "formatting means" has a function for converting the character-recognized data into a specific format, such as the iCalendar format.

[0180] The "means for importing into a calendar application" has a function for importing formatted data into a calendar application and displaying it as a schedule item.

[0181] The "means for importing into the work schedule system" refers to a function that imports formatted data into the factory's work schedule system and registers and manages it as schedule information.

[0182] The "means for analyzing and identifying the schedule of each individual" has a function for distinguishing schedules of different individuals by taking into account color-coding information in the acquired image.

[0183] The "means for storing and transmitting to the user's terminal" has the function of storing formatted data in a database and transmitting the data to the user's terminal as required.

[0184] The present invention relates to a system for digitizing handwritten calendars and work schedules and managing them in a unified manner. Specific embodiments for carrying out the present invention will be described below.

[0185] Server Action:

[0186] When a handwritten calendar image is uploaded from the user's device, the server first stores it in a temporary directory. The server then analyzes the stored image data using OCR technology. Specifically, it uses software such as Tesseract OCR or Google Cloud Vision API. The analyzed data is then formatted into a specific format (e.g., iCalendar format) and stored in a database. The server then imports the formatted data into the work schedule system and sends it to the user's device as needed.

[0187] Terminal handling:

[0188] The device first takes a photo of the handwritten calendar through user operation and then sends the image to the server by pressing the upload button. At this time, the image data is sent using an HTTP request. When the device receives a notification from the server that the new schedule data is ready, it retrieves it and imports it into the calendar app. The device also has the function to import the data received from the server into the work schedule system.

[0189] User Action:

[0190] The user takes a photo of a handwritten calendar using their smartphone. When taking the photo, the appointments are color-coded to indicate who owns them. The image is uploaded to the server, and upon receiving a notification from the server, the user checks the calendar app to confirm that the new schedule has been imported correctly. The user also verifies that the new schedule has been integrated into the factory's work scheduling system.

[0191] Examples:

[0192] At the beginning of the month, a user takes a photo of a manually written factory work schedule and uploads the image to the server. The server uses OCR technology to analyze the text in the image and converts the schedules for each worker and machine into digital text. The server then formats the text into iCalendar format or a format that can be understood by the factory's scheduling system. The formatted data is stored in a database, and an import notification is sent to the user's device. The user receives the notification, launches their calendar app, and checks the new schedule data. They also verify that the new schedule has been correctly integrated into the factory's work scheduling system.

[0193] Example prompt sentence:

[0194] Write code for an application that parses handwritten calendar images to digitize schedule events for import into a factory scheduling system.

[0195] In this way, the invention is a system that efficiently digitizes handwritten calendars and schedules and centralizes factory schedule management, thereby improving work efficiency and reducing errors.

[0196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0197] Step 1:

[0198] The user takes a photo of a handwritten calendar with the smartphone camera and generates an image file. The input is the handwritten calendar, and the output is a digital image file. The user checks that the image is clearly captured.

[0199] Step 2:

[0200] A user uses an application on their device to upload images taken with a camera to a server. The input is a digital image file, and the output is an HTTP request to send that file to the server. The transmission begins when the user presses the upload button.

[0201] Step 3:

[0202] The server receives the uploaded image and stores it in a temporary storage directory. The input is the image data sent via the HTTP request, and the output is the image file in the server's temporary storage directory. This step is where the file is saved.

[0203] Step 4:

[0204] The server analyzes the stored image data using OCR technology. The input is the image file stored on the server, and the output is the extracted text data. Specifically, the server performs image analysis using Google Cloud Vision API and Tesseract OCR.

[0205] Step 5:

[0206] The server formats the text data analyzed by OCR technology into a format that can be understood by a calendar application or work schedule system. The input is the text data obtained by OCR analysis, and the output is formatted data (e.g., iCalendar format). The server uses scripts or application logic to format the data.

[0207] Step 6:

[0208] The server saves the formatted data in a database and notifies the user's terminal that the data is ready. The input is the formatted schedule data, and the output is the record saved in the database and a notification message to the terminal.

[0209] Step 7:

[0210] The device receives the notification from the server, and after obtaining the user's confirmation, launches the calendar app and instructs it to retrieve new schedule data. The input is the notification message from the server, and the output is a request to retrieve new schedule data.

[0211] Step 8:

[0212] The calendar app communicates with the server, gets the formatted schedule data, and imports it into the calendar. The input is the formatted data sent from the server, and the output is the schedule events imported into the calendar app. Through this series of processes, the new schedule is correctly reflected in the calendar.

[0213] Step 9:

[0214] The user checks the calendar app and verifies that the new schedule has been imported correctly. Similarly, the user verifies that the new schedule has been integrated into the factory's work scheduling system. The input is the imported schedule data, and the output is the user's confirmation operation.

[0215] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0216] The present invention combines a system that digitizes handwritten calendars and centrally manages them using a calendar application with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention will be described below.

[0217] Server Processing

[0218] 1. Receiving image uploads

[0219] When a handwritten calendar image is uploaded from a user's terminal, the server receives it and stores the received image data in a temporary directory.

[0220] 2. Image Analysis

[0221] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[0222] 3. Data Formatting

[0223] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI. It uses scripts and application logic to convert the event data into a format that the calendar app can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify who owns a specific event.

[0224] 4. Activating the Emotional Engine

[0225] Before importing data into the calendar app, the server activates an emotion engine to recognize the user's emotions. This emotion engine analyzes the user's past behavioral data and text input to recognize the user's emotional state at that time.

[0226] 5. Emotional Data Integration

[0227] The server then incorporates the recognized emotion data into the calendar data. For example, if the user feels nervous or anxious about an important appointment, the server stores this information as metadata. The server also arranges for the emotion data to be displayed in the calendar app.

[0228] 6. Data storage and transmission

[0229] The server stores the formatted data and emotion data in a database and sends a message to the user's device to notify them that the data is ready. When the user makes a request to obtain new calendar data, the server sends the data to the device.

[0230] Terminal handling

[0231] 1. Image capture

[0232] The user takes a photo of the handwritten calendar using the smartphone camera, and checks the image to make sure the entire calendar is clearly visible.

[0233] 2. Upload an image

[0234] When the user presses the image upload button, the device uploads the captured image data to the server. The device sends the image data to the server using an HTTP request.

[0235] 3. Receiving and importing data

[0236] The device that receives the notification from the server launches the calendar app with the user's confirmation. The calendar app communicates with the server and sends a request to obtain new format data and emotion data. Upon receiving the data from the server, the calendar app imports it and adds a new event to the existing calendar. The calendar app uses color-coding information and emotion data to distinguish and display each event for each individual.

[0237] User operations

[0238] 1. Calendar photoshoot

[0239] The user takes a photo of a handwritten calendar using their smartphone, and the events are color-coded to indicate who owns them.

[0240] 2. Upload confirmation

[0241] When uploading a captured image to a server, the user performs a confirmation operation.

[0242] 3. Data confirmation

[0243] The user can check their calendar app to ensure the new appointment and emotion data has been imported correctly. If necessary, the user can make additional edits or adjustments.

[0244] Specific examples

[0245] Example 1: Image upload and analysis

[0246] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server uses generative AI to analyze the text in the image and convert each person's schedule into digital text. It then uses an emotion engine to analyze the user's emotional state and integrates the emotional data into the calendar data. The data is then formatted and saved in a calendar app format, and notified to the user's device.

[0247] Example 2: Data import

[0248] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new events and emotion data from the server and automatically imports them into their calendar. The user can then check the events and emotions of each person correctly reflected in the calendar and edit them if necessary.

[0249] The system of the present invention not only digitizes handwritten calendar information and allows it to be shared with the whole family via a smartphone calendar app, but also reflects the user's emotional state, making it easier to share and coordinate schedules and allowing all family members to quickly grasp the latest information and emotional state.

[0250] The processing flow will be explained below.

[0251] Step 1:

[0252] The user takes a photo of the handwritten calendar using the smartphone camera, checks that the entire calendar is clearly visible, and prepares to upload it to the server via the app.

[0253] Step 2:

[0254] The user presses the upload button on the app to send the captured image data to the server. The device then sends the image data to the server using an HTTP request.

[0255] Step 3:

[0256] The server receives the image upload request sent from the user's terminal, saves the image data in a temporary storage directory, and sends a response confirming receipt to the terminal.

[0257] Step 4:

[0258] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[0259] Step 5:

[0260] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI, and uses scripts and logic to convert the extracted information into a format that the calendar app can understand (e.g., iCalendar format).

[0261] Step 6:

[0262] The server starts an emotion engine and analyzes the user's past behavioral data and schedule information. The emotion engine recognizes the user's emotional state and acquires that data.

[0263] Step 7:

[0264] The server integrates the user's emotional data into the schedule data. For example, if the user feels nervous or anxious about an important appointment, the server stores this information as metadata and stores it together with the calendar information.

[0265] Step 8:

[0266] The server stores the formatted data and emotion data in a database. It also sends a message to the user's device to notify them that the data is ready. Possible notification methods include push notifications and emails.

[0267] Step 9:

[0268] The device receives a notification from the server and notifies the user that new calendar data and emotion data are available. The user confirms the notification and launches the calendar app.

[0269] Step 10:

[0270] The device's calendar app sends a request to the server to get new formatting and emotion data, which the server then sends to the device.

[0271] Step 11:

[0272] The calendar application of the device imports the received format data and emotion data and adds a new event to the existing calendar. The calendar application uses color-coding information and emotion data to distinguish and display each event for each individual.

[0273] Step 12:

[0274] Users can then review their new events and emotion data in their calendar app to ensure it has been imported correctly. If necessary, users can make additional edits or adjustments.

[0275] Through these processing steps, the handwritten calendar information is digitized and can be shared with the whole family via a smartphone calendar app, and the user's emotional state is also reflected, making it easier to share and adjust schedules, and allowing all family members to quickly grasp the latest information and emotional state.

[0276] Example 2

[0277] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0278] Current digital calendar management systems lack the ability to digitize handwritten calendar information, making it difficult to manage schedules that take the user's emotional state into account. They also lack the ability to accurately identify handwritten information and reflect specific color coding or emotional data in the calendar. As a result, sharing and adjusting schedules becomes cumbersome and stressful for users.

[0279] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0280] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into a calendar application, means for activating an emotion analysis engine for recognizing the user's emotions, and means for integrating the emotion data into calendar data, thereby enabling accurate digitization of handwritten calendar information and schedule management that reflects the user's emotional state.

[0281] "Means for acquiring images" refers to equipment or software that allows a user to photograph or scan a handwritten calendar image and provide it to a server as digital data.

[0282] "Means for character recognition of acquired images" refers to the function of using OCR technology to extract character information from acquired handwritten calendar images and convert it into text data.

[0283] "Means for formatting the recognized data" refers to the script or application logic used to convert the text data obtained using OCR technology into a format that can be understood by a calendar application (e.g., iCalendar format).

[0284] "Means for importing formatted data into a calendar application" refers to functionality that allows formatted data to be read into a calendar application so that it can be displayed as an appointment on a digital calendar.

[0285] "Means for launching an emotion analysis engine to recognize a user's emotions" refers to a function for analyzing a user's past behavioral data and text inputs to run an engine to recognize the user's emotional state.

[0286] "Means for integrating emotional data into calendar data" refers to a function that integrates the recognized emotional data of a user with existing calendar data and reflects it in a calendar application as emotional information.

[0287] "Color coding information" refers to color information used to distinguish different events or people in a handwritten calendar.

[0288] "Calendar application" refers to software that allows a user to digitally manage their schedules.

[0289] The present invention combines a system that digitizes handwritten calendars and centrally manages them using a calendar application with an emotion analysis engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[0290] Server Processing

[0291] 1. Receiving image uploads

[0292] The server receives handwritten calendar image data from the user's device and stores it in a temporary directory. At this time, it assigns a unique ID to the file name for management purposes.

[0293] 2. Image Analysis

[0294] The server retrieves the calendar image from the temporary storage directory and inputs it into the generative AI model. The prompt text used is, "Please analyze the handwritten characters in this image and convert them into text data." The generative AI model then uses OCR technology to convert the handwritten characters in the image into text format and returns the results to the server.

[0295] 3. Data Formatting

[0296] The server parses the text data returned by the generative AI model to extract the date, time, content, and color-coding information for each event. This process is performed using Python scripts or custom application logic. The extracted data is then converted into a format that calendar applications can understand, such as iCalendar, and the color-coding information is added as metadata.

[0297] 4. Activating the Emotional Engine

[0298] The server starts the emotion engine before importing the data. The emotion engine analyzes the user's past behavioral data and text input to recognize their current emotional state. This process uses emotion analysis algorithms and natural language processing techniques.

[0299] 5. Emotional Data Integration

[0300] The server integrates the emotional data recognized by the emotion engine into the calendar data, saving the emotional information (e.g., "The user is feeling anxious about a meeting where traffic congestion is expected") as metadata, and formats the information so that it can be displayed in the calendar application.

[0301] 6. Data storage and transmission

[0302] The server stores the formatted data and emotion data in a database. It then notifies the user's device that the data is ready. This notification can be done via push notification or email. When the user makes a request, the server sends the data for import to the device.

[0303] Terminal handling

[0304] 1. Image capture

[0305] The user takes a photo of the handwritten calendar using the smartphone's camera. The user checks that the image is clear and shows the entire calendar. The user can also color-code events as needed.

[0306] 2. Upload an image

[0307] When a user presses the image upload button, the device sends the captured image to the server using an HTTP request (such as the POST method). This request includes the image file as well as metadata such as user information and the date and time the image was taken.

[0308] 3. Receiving and importing data

[0309] When the device receives a notification from the server that the data is ready, it launches the calendar application with the user's confirmation. The calendar application sends a request to the server to obtain new data and receives the data returned from the server. The received data is automatically imported into the calendar, and the schedule and emotion data are displayed.

[0310] User operations

[0311] 1. Calendar photoshoot

[0312] The user takes a photo of the handwritten calendar using the smartphone camera. When taking the photo, each event is color-coded to distinguish whose event it is.

[0313] 2. Upload confirmation

[0314] When uploading a captured image to a server, the user displays a confirmation screen to confirm the contents of the image to be sent. This confirmation operation uses check boxes and confirmation buttons.

[0315] 3. Data confirmation

[0316] Users can check in their calendar application whether the new events and emotion data have been imported correctly. If there are any errors in the imported content, users can manually correct or add them.

[0317] As described above, the system of the present invention digitizes handwritten calendar information and further integrates the user's emotional data, enabling more comprehensive schedule management.

[0318] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0319] Step 1:

[0320] The user takes a photo of the handwritten calendar using the smartphone camera. At this time, the events are color-coded to indicate who owns them. The captured image data is saved in the device.

[0321] Input: Handwritten calendar

[0322] Output: Captured image data

[0323] Step 2:

[0324] When a user presses the image upload button, the device sends the saved image data to the server as an HTTP request (POST method). The request includes the image file, user information, and the date and time the image was taken.

[0325] Input: Captured image data, user information, date and time of capture

[0326] Output: HTTP request sent to the server

[0327] Step 3:

[0328] The server stores the handwritten calendar image data received from the user's device in a temporary directory, and manages it by assigning a unique ID to the file name.

[0329] Input: Image data sent as an HTTP request

[0330] Output: Image data saved on the server (in the temporary directory)

[0331] Step 4:

[0332] The server retrieves the calendar image from the temporary storage directory and inputs it into the generative AI model. The prompt text used is, "Please analyze the handwritten characters in this image and convert them into text data." The generative AI model then uses OCR technology to convert the handwritten characters in the image into text format.

[0333] Input: Calendar image data retrieved from temporary directory

[0334] Output: Text data analyzed from handwritten characters

[0335] Step 5:

[0336] The server parses the text data returned by the generative AI model to extract the date, time, content, and color-coding information for each event. This analysis is done using Python scripts or custom application logic. The extracted data is then converted into a format that calendar applications can understand, such as iCalendar, and the color-coding information is added as metadata.

[0337] Input: Text data returned by the generative AI model

[0338] Output: Formatted data including iCalendar data and color coding information

[0339] Step 6:

[0340] The server starts the emotion engine before importing the data. The emotion engine analyzes the user's past behavioral data and text input to recognize their current emotional state. Natural language processing and emotion recognition algorithms are used for emotion analysis.

[0341] Input: User's past behavior data, text input

[0342] Output: Recognized user emotion data

[0343] Step 7:

[0344] The server integrates the emotional data recognized by the emotion engine into the calendar data. For example, it adds emotional information such as "the user is feeling anxious about an important meeting" as metadata. The server then formats the emotional data so that it can be displayed in the calendar application.

[0345] Input: Recognized emotion data, iCalendar format data

[0346] Output: Formatted calendar data with emotion data integrated

[0347] Step 8:

[0348] The server stores the formatted data and emotion data in a database. It then notifies the user's device that the data is ready. When the user makes a request, the server sends the data for import to the device.

[0349] Input: Formatted calendar data with emotion data integrated

[0350] Output: Data stored in the database, notifications sent to users

[0351] Step 9:

[0352] When the device receives a notification from the server that the data is ready, it launches the calendar application with the user's confirmation. The calendar application sends a request to the server to obtain new data and receives the data returned from the server. The received data is automatically imported into the calendar, and the schedule and emotion data are displayed.

[0353] Input: Notification from the server that data is ready

[0354] Output: Calendar data and emotion data imported into a calendar application

[0355] Step 10:

[0356] Users can check in their calendar application whether the new events and emotion data have been imported correctly. If there are any errors in the imported content, users can manually correct or add them.

[0357] Input: Calendar data and emotion data imported into a calendar application

[0358] Output: Confirmed or modified calendar data

[0359] (Application example 2)

[0360] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0361] Conventional handwritten shift schedules and schedule management systems were difficult to digitize, making it time-consuming to share and update information across the organization. Furthermore, they were unable to reflect employees' emotional states, making it difficult to optimize the work environment. For this reason, there was a demand for a system that could easily digitize handwritten shift schedules and reflect employees' emotional states, thereby streamlining store operations and contributing to employee motivation management.

[0362] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0363] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into the system, means for recognizing an emotional state based on the formatted data, and means for integrating and storing the emotional data in a database, thereby making it possible to digitize handwritten shift schedules and integrate the emotional data.

[0364] The "means for acquiring images" refers to a means by which a user photographs a handwritten shift table or schedule and provides the image to the server as a digital image.

[0365] "Means for character recognition in acquired images" refers to a means for analyzing handwritten characters in digital images using OCR technology and converting them into text data.

[0366] "Means for formatting character-recognized data" refers to means for converting text data obtained by OCR into a format that can be understood by the system.

[0367] A "means for importing formatted data into a system" is a means for incorporating formatted data into a particular system or application.

[0368] The "means for recognizing an emotional state based on formatted data" is a means for analyzing data and past information to estimate a user's emotional state.

[0369] The "means for integrating emotional data and storing it in a database" is a means for integrating recognized emotional data with formatted data, and storing the data in an organized manner in a database.

[0370] The present invention includes the following configuration and processing means to provide a system that digitizes handwritten shift tables and schedules and further reflects the emotional state of employees.

[0371] System Configuration

[0372] The system mainly consists of a server, a terminal (e.g., a smartphone), and a user interface. Specific usage methods for each component are explained below.

[0373] Server Processing

[0374] 1. Receiving image uploads

[0375] When an image of a handwritten shift schedule is uploaded from a user's device, the server receives it. The received image data is temporarily stored in a directory. The server uses a cloud storage service (e.g., Amazon S3) to store the image data.

[0376] 2. Image Analysis

[0377] The server inputs the stored image data into a generative AI model using OCR technology, for example, by using the Google Cloud Vision API to analyze handwritten characters in the image and convert them into text data.

[0378] 3. Data Formatting

[0379] The server extracts the shift schedule's date, time, content, and color-coded information from the text data output by the generation AI and converts it into a format the system can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify specific employees.

[0380] 4. Activating the Emotional Engine

[0381] Before importing the shift schedule data, the server starts an emotion engine to recognize the emotions of employees. This emotion engine analyzes employees' past work history and comments to recognize their emotional state at that time, using, for example, IBM Watson's emotion analysis API.

[0382] 5. Emotional Data Integration

[0383] The server then incorporates the recognized emotion data into the shift schedule data. For example, if an employee is feeling stressed or tired, the emotion data is saved as metadata and used for future shift management.

[0384] 6. Data storage and transmission

[0385] The server stores the formatted data and emotion data in a database (e.g., MySQL) and sends a message to the user's device to notify them that the data is ready. You can use a notification service such as Firebase Cloud Messaging (FCM).

[0386] Terminal handling

[0387] 1. Image capture

[0388] The user takes a photo of the handwritten shift schedule using the smartphone camera, and checks the image to make sure the entire shift schedule is clearly visible.

[0389] 2. Upload an image

[0390] When the user presses the image upload button, the device uploads the captured image data to the server. The image data is sent to the server using an HTTP request.

[0391] 3. Receiving and importing data

[0392] Upon receiving the notification from the server, the device launches the system app with the user's confirmation. The app communicates with the server and sends a request to obtain new format data and emotion data. Upon receiving the data from the server, the app imports it and adds a new shift schedule to the existing system. The app uses color-coding information and emotion data to distinguish and display the status of each employee.

[0393] Specific examples

[0394] The store manager, who is the user, takes a photo of a handwritten shift schedule with their smartphone and uploads the image to the server. The server uses OCR technology to analyze the handwritten characters in the image and convert the shift information into text data. Next, it uses an emotion engine to analyze past work history and comments to recognize employee emotions. This emotion data is integrated into the shift data and saved in a database. The device receives a notification from the server and launches the shift management app to check the updated data.

[0395] Prompt Sentence Examples

[0396] "Analyze employees' work logs from the past six months to understand their emotional state, especially their stress and fatigue levels."

[0397] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0398] Step 1:

[0399] Taking and reviewing images

[0400] The user takes a photo of a handwritten shift schedule using the smartphone camera. This image is checked to see if the entire shift schedule is clearly captured. The input is the physical handwritten shift schedule, and the output is a digital image file.

[0401] Step 2:

[0402] Uploading an image

[0403] When a user presses the image upload button, the device uploads the captured image data to the server. The image data is sent using an HTTP request, with the input being a digital image file and the output being image data stored on the server.

[0404] Step 3:

[0405] Image analysis

[0406] The server analyzes the stored image data using OCR technology. Specifically, it uses an OCR engine such as Google Cloud Vision API to analyze the handwritten characters in the image and convert them into text data. The input is the uploaded image data, and the output is the analyzed text data.

[0407] Step 4:

[0408] Data Formatting

[0409] The server extracts the shift schedule date, time, content, and color-coded information from the text data output by the generative AI model and converts it into a format that the system can understand (e.g., iCalendar format). The input is the text data generated by OCR, and the output is the formatted shift schedule data.

[0410] Step 5:

[0411] Emotion engine activation and analysis

[0412] Before importing the formatted shift data, the server launches an emotion engine using IBM Watson's emotion analysis API to analyze employees' past work history and comments. The input is the employee's past data and formatted shift data, and the output is recognized emotion data.

[0413] Step 6:

[0414] Emotional Data Integration

[0415] The server integrates the recognized emotion data into the shift schedule data. Based on the analysis results, specific emotions (e.g., stress or fatigue) are added to the shift schedule data as metadata. The input is formatted shift schedule data and emotion data, and the output is shift schedule data integrated with emotion data.

[0416] Step 7:

[0417] Data Retention and Notification

[0418] The server saves the consolidated shift data in a database (e.g., MySQL) and sends notifications to the user's device via Firebase Cloud Messaging (FCM), etc. The input is the consolidated shift data and user device information, and the output is the data saved in the database and the sent notifications.

[0419] Step 8:

[0420] Receiving and displaying data

[0421] The user's device receives the notification from the server, launches the shift management app, and retrieves the new shift data. The retrieved data is displayed in the app, and the shift schedule reflecting each employee's emotional state is confirmed. The input is the integrated shift data sent from the server, and the output is the new shift schedule displayed in the shift management app on the device.

[0422] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0423] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0424] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0425] [Second embodiment]

[0426] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0427] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0428] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0429] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0430] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0431] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0432] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0433] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0434] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0435] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0436] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0437] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0438] The present invention provides a system for digitizing handwritten calendars and managing them in a unified manner using a calendar application. Specific embodiments of the system are described below.

[0439] Server Processing

[0440] 1. Receiving image uploads

[0441] When a handwritten calendar image is uploaded from a user's terminal, the server receives it and stores the received image data in a temporary directory.

[0442] 2. Image Analysis

[0443] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[0444] 3. Data Formatting

[0445] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI. Using scripts or application logic, the extracted information is converted into a format that the calendar app can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify who owns a particular event.

[0446] 4. Data storage and transmission

[0447] The server stores the formatted data in a database and sends a message to the user's device to notify them that the data is ready. When the user makes a request to retrieve new calendar data, the server sends the data to the device.

[0448] Terminal handling

[0449] 1. Image capture

[0450] The user takes a photo of the handwritten calendar using the smartphone camera, and checks the image to make sure the entire calendar is clearly visible.

[0451] 2. Upload an image

[0452] When the user presses the image upload button, the device uploads the captured image data to the server. The device sends the image data to the server using an HTTP request.

[0453] 3. Receiving and importing data

[0454] The device that receives the notification from the server launches the calendar app with the user's confirmation. The calendar app communicates with the server and sends a request to obtain new formatted data. Upon receiving the formatted calendar data from the server, the calendar app imports it and adds new events to the existing calendar. The calendar app uses color-coding information to distinguish and display each event for each individual.

[0455] User operations

[0456] 1. Calendar photoshoot

[0457] The user takes a photo of a handwritten calendar using their smartphone, and the events are color-coded to indicate who owns them.

[0458] 2. Upload confirmation

[0459] When uploading a captured image to a server, the user performs a confirmation operation.

[0460] 3. Data confirmation

[0461] The user can check their calendar app to ensure the new events have been imported correctly, and if necessary, they can make additional edits or adjustments.

[0462] Specific examples

[0463] Example 1: Image upload and analysis

[0464] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server then uses generative AI to analyze the text in the image and convert each person's schedule into digital text. The text is then formatted and saved in a calendar app format, and the user's device is notified.

[0465] Example 2: Data import

[0466] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new event data from the server and automatically imports it into their calendar. The user can then check each person's events, which are correctly reflected in the calendar, and edit them if necessary.

[0467] The system of the present invention allows for centralized digital management while maintaining the convenience of a handwritten calendar, making it easier to share and coordinate schedules and allowing all family members to quickly understand the latest information.

[0468] The processing flow will be explained below.

[0469] Step 1:

[0470] The user uses the smartphone camera to take a picture of a handwritten calendar hanging on the wall, and checks the image to make sure the entire calendar is clearly visible.

[0471] Step 2:

[0472] The device prepares to upload the captured image data to the server through the application interface. When the user presses the upload button, the device sends the image data to the server via an HTTP request.

[0473] Step 3:

[0474] The server receives the image upload request sent from the user's terminal, saves the image data in a temporary directory, and sends a response confirming receipt to the terminal.

[0475] Step 4:

[0476] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[0477] Step 5:

[0478] The server extracts the event date, time, content, and color-coded information from the parsed text data, using scripts or application logic to convert the event data into a format that the calendar app can understand (e.g., iCalendar format).

[0479] Step 6:

[0480] The server stores the formatted data in a database, a process that ensures that the scheduled data is easily accessible at a later time and is accurately maintained.

[0481] Step 7:

[0482] The server sends a message to the user's device to notify them that the formatted data is ready, possibly via push notification or email.

[0483] Step 8:

[0484] The device receives a notification from the server informing the user that new calendar data is available. The user confirms the notification and launches the calendar app.

[0485] Step 9:

[0486] The device's calendar app sends a request to the server to get the new format data, and the server sends the saved format data to the device.

[0487] Step 10:

[0488] The calendar application on the device imports the received format data and adds new events to the existing calendar. The calendar application uses color-coding information to distinguish and display each event for each individual.

[0489] Step 11:

[0490] The user can verify that the new event has been imported correctly within their calendar app, and can make additional edits or adjustments if needed.

[0491] Through these processing steps, the handwritten calendar information is digitized and can be shared with the whole family via a smartphone calendar app.

[0492] Example 1

[0493] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0494] Systems that digitize and centrally manage handwritten calendars face the problem of cumbersome and time-consuming manual data entry. It is also difficult to manage individual appointments by color-coding them, making sharing and coordination difficult. Furthermore, there is a need to improve the accuracy of handwritten character recognition and accurately convert them into digital data.

[0495] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0496] In this invention, the server includes a means for receiving and temporarily storing image data from the user's device, a means for inputting the image data into a generative AI model and analyzing handwritten characters, and a means for converting the generated character recognition data into a format that can be understood by a calendar application. This allows handwritten calendars to be digitized and centrally managed, and schedules to be color-coded and shared.

[0497] "Means for acquiring images" refers to devices or software that allow a user to take a photo of a handwritten calendar with a smartphone or camera and acquire the image data.

[0498] The "means for character recognition of acquired images" refers to software or algorithms for analyzing handwritten characters from acquired image data and extracting them as digital text data.

[0499] A "means for formatting character-recognized data" is software or script that converts text data obtained by character recognition into a specific data format that can be recognized by a calendar application (e.g., iCalendar format).

[0500] "Means for saving formatted data in a database and transmitting it to a user's terminal" refers to communication means or software that saves the converted data in a database and transmits it to a user's terminal as needed.

[0501] The "means for receiving image data from the user's terminal and temporarily storing it" refers to server-side storage or software that receives image data sent from the user's terminal and temporarily stores it.

[0502] The "means of inputting image data into a generative AI model and analyzing handwritten characters" refers to a method of supplying received and stored image data to a generative AI model and converting handwritten characters into text data.

[0503] "Means for converting the generated character recognition data into a format understandable by a calendar application" refers to software or scripts that convert the text data output by the generative AI model into a format that can be imported and understood by a calendar application (e.g., iCalendar format).

[0504] The "means for sending a notification to the user's terminal that the data is ready" refers to a communication function or software for notifying the user's terminal that the formatted data is ready.

[0505] The present invention provides a system for digitizing handwritten calendars and managing them in a unified manner using a calendar application. Specific embodiments of the system are described below.

[0506] System Overview

[0507] The system starts when a user takes a photo of a handwritten calendar with their smartphone and uploads it to a server. The server analyzes the uploaded image using a generative AI model and converts it into digital data. The converted data is then formatted into a format that the calendar app can understand and saved in a database. Finally, the formatted data is sent to the user's device and imported into the calendar app.

[0508] Hardware and Software

[0509] Smartphone: Users take a photo of the handwritten calendar using the camera on their smartphone. iPhones, Android devices, etc. can be used.

[0510] Server: A server is used to receive, analyze, format, store, and transmit image data. Specifically, a web framework such as Flask or Django is used.

[0511] Generative AI models: used to analyze handwritten characters in images, for example, using pre-trained models using PyTorch or TensorFlow.

[0512] OCR technology: Google Cloud Vision API and Tesseract are used to analyze handwritten characters in images.

[0513] Database: Use a database such as MySQL to store the analyzed data.

[0514] Calendar app: Use an application such as Google Calendar to import digital data.

[0515] Processing Flow

[0516] A user uses the system in the following steps:

[0517] 1. Image capture

[0518] A user takes a photo of a handwritten calendar using the camera on their smartphone. When taking the photo, they color-code the events so that it is clear who owns each one. For example, they could take a photo of a handwritten family calendar and mark each event with a different color.

[0519] 2. Upload an image

[0520] When a user presses the image upload button in the app, the smartphone uploads the captured image data to the server. This is done by sending the data to the server using an HTTP POST request. For example, the image data is POSTed to the / upload_image endpoint of the Flask server.

[0521] 3. Image Analysis

[0522] The server reads the saved image data and passes it to the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data. For example, it can call the Google Cloud Vision API to perform character recognition.

[0523] 4. Data Formatting

[0524] The server extracts date, time, content, and color-coded information from the text data output by the generation AI. Using a script (e.g., a Python script), it converts the extracted information into a format understandable by a calendar app, such as iCalendar format. It adds the color-coded information as metadata. For example, data such as "{"date": "2023-10-01", "events": ["Meeting", "Meeting"]}" is converted to the format "BEGIN:VEVENT\nDTSTART:20231001T090000Z\nSUMMARY:Meeting\nEND:VEVENT\n".

[0525] 5. Data storage and transmission

[0526] The server saves the formatted data in a database (e.g., MySQL) and notifies the user's device that the data is ready. Specifically, this is done using WebSocket or push notification. For example, the server inserts iCalendar format data into the calendar_events table in a MySQL database and sends a message to the user's device via WebSocket saying "Calendar data has been updated."

[0527] 6. Receiving and Importing Data

[0528] The device that receives the notification from the server displays the notification to the user and launches the calendar app. The calendar app communicates with the server to send a request for new data and receives the formatted data. The received data is then imported into the calendar app, where events are displayed in color.

[0529] Specific examples

[0530] Example 1: Image upload and analysis

[0531] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server then uses a generative AI model to analyze the text in the image and convert each person's schedule into digital text. The text is then formatted and saved in a calendar app format, and the user's device is notified.

[0532] Example 2: Data import

[0533] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new event data from the server and automatically imports it into their calendar. The user can then check each person's events, which are correctly reflected in the calendar, and edit them if necessary.

[0534] Prompt Sentence Examples

[0535] Please explain the system that takes a photo of a family's handwritten calendar with a smartphone, digitizes it on the cloud, and imports it into a calendar app. For example, what steps are taken to process the data, and how does the user operate it?

[0536] This invention digitizes handwritten calendars and centralizes their management, making it easier to share and coordinate schedules, and allowing all family members to quickly grasp the latest information.

[0537] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0538] Step 1:

[0539] Image capture

[0540] A user launches the camera app on their smartphone and takes a photo of a handwritten calendar. When taking the photo, they color-code the events to identify who owns them. For example, they can take a photo of a family calendar and mark each event with a different color.

[0541] Input: Handwritten calendar

[0542] Output: Captured image data (JPEG or PNG format)

[0543] Step 2:

[0544] Image upload

[0545] When a user presses the image upload button in the app, the device sends the captured image data to the server using an HTTP POST request. For example, the image data is POSTed to http: / / example.com / upload_image.

[0546] Input: Captured image data

[0547] Output: HTTP POST request to the server

[0548] Step 3:

[0549] Receiving and saving images on the server

[0550] The server receives the image data sent from the user's device and stores it in a temporary directory. For example, it processes the request using a web framework such as Flask or Django and stores the image file as / tmp / uploaded_images / image1.jpg.

[0551] Input: HTTP POST request (image data)

[0552] Output: Temporarily saved image file

[0553] Step 4:

[0554] Image analysis

[0555] The server reads the temporarily saved image data and inputs it into the generative AI model. The generative AI model analyzes the handwritten characters in the image using OCR technology (e.g., Google Cloud Vision API or Tesseract) and converts them into text data. For example, input / tmp / uploaded_images / image1.jpg into the Google Cloud Vision API and obtain JSON-formatted text data as the character recognition results.

[0556] Input: Saved image file

[0557] Output: Parsed text data (JSON format)

[0558] Step 5:

[0559] Data Formatting

[0560] The server extracts date, time, content, and color-coding information from the text data output by the generative AI model. It then uses a script (e.g., a Python script) to convert the extracted information into a format that calendar apps can understand, such as iCalendar, and adds the color-coding information as metadata.

[0561] Input: Parsed text data (JSON format)

[0562] Output: Formatted calendar data (iCalendar format)

[0563] Step 6:

[0564] Data Retention and Notification

[0565] The server saves the formatted data in a database (e.g., MySQL) and sends a notification to the user's device that the data is ready. For example, it inserts the iCalendar format data into the calendar_events table in the MySQL database and sends a message to the user's device via WebSocket saying "Calendar data has been updated."

[0566] Input: Formatted calendar data (iCalendar format)

[0567] Output: Calendar data stored in the database, and notification messages

[0568] Step 7:

[0569] Receiving and Importing Data

[0570] When the device receives the notification from the server, it displays the notification to the user and launches the calendar app. The calendar app then communicates with the server and sends a request to retrieve new data. The formatted calendar data is then received and automatically imported into the calendar. For example, the calendar app may use a REST API to retrieve iCalendar formatted data, import it into the calendar, and display it.

[0571] Input: Notification message, formatted calendar data

[0572] Output: New events imported into the Calendar app

[0573] This process converts handwritten calendars into digital data and efficiently manages them centrally. Users can easily digitize their handwritten calendars and manage them in a calendar app.

[0574] (Application example 1)

[0575] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0576] Existing factory work schedule management methods often require a lot of time and effort in the process of digitizing handwritten schedules, making it difficult to streamline the process. Furthermore, it is difficult to integrate and manage the schedules of multiple individuals, who need to be color-coded or otherwise differentiated, making centralized schedule management difficult. The objective of this invention is to efficiently and accurately digitize handwritten work schedules and realize centralized schedule management.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0578] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into a calendar application, and means for importing the formatted data into a work schedule system. This makes it possible to efficiently digitize handwritten work schedules and centrally manage them, including each individual's schedule.

[0579] The "means for acquiring images" has a function that allows a user to take a photograph of a handwritten calendar or work schedule and send it to the server in digital image format.

[0580] "Means for character recognition" refers to a technology that extracts character information from an acquired image and converts it into digital text format, and uses OCR technology.

[0581] The "formatting means" has a function for converting the character-recognized data into a specific format, such as the iCalendar format.

[0582] The "means for importing into a calendar application" has a function for importing formatted data into a calendar application and displaying it as a schedule item.

[0583] The "means for importing into the work schedule system" refers to a function that imports formatted data into the factory's work schedule system and registers and manages it as schedule information.

[0584] The "means for analyzing and identifying the schedule of each individual" has a function for distinguishing schedules of different individuals by taking into account color-coding information in the acquired image.

[0585] The "means for storing and transmitting to the user's terminal" has the function of storing formatted data in a database and transmitting the data to the user's terminal as required.

[0586] The present invention relates to a system for digitizing handwritten calendars and work schedules and managing them in a unified manner. Specific embodiments for carrying out the present invention will be described below.

[0587] Server Action:

[0588] When a handwritten calendar image is uploaded from the user's device, the server first stores it in a temporary directory. The server then analyzes the stored image data using OCR technology. Specifically, it uses software such as Tesseract OCR or Google Cloud Vision API. The analyzed data is then formatted into a specific format (e.g., iCalendar format) and stored in a database. The server then imports the formatted data into the work schedule system and sends it to the user's device as needed.

[0589] Terminal handling:

[0590] The device first takes a photo of the handwritten calendar through user operation and then sends the image to the server by pressing the upload button. At this time, the image data is sent using an HTTP request. When the device receives a notification from the server that the new schedule data is ready, it retrieves it and imports it into the calendar app. The device also has the function to import the data received from the server into the work schedule system.

[0591] User Action:

[0592] The user takes a photo of a handwritten calendar using their smartphone. When taking the photo, the appointments are color-coded to indicate who owns them. The image is uploaded to the server, and upon receiving a notification from the server, the user checks the calendar app to confirm that the new schedule has been imported correctly. The user also verifies that the new schedule has been integrated into the factory's work scheduling system.

[0593] Examples:

[0594] At the beginning of the month, a user takes a photo of a manually written factory work schedule and uploads the image to the server. The server uses OCR technology to analyze the text in the image and converts the schedules for each worker and machine into digital text. The server then formats the text into iCalendar format or a format that can be understood by the factory's scheduling system. The formatted data is stored in a database, and an import notification is sent to the user's device. The user receives the notification, launches their calendar app, and checks the new schedule data. They also verify that the new schedule has been correctly integrated into the factory's work scheduling system.

[0595] Example prompt sentence:

[0596] Write code for an application that parses handwritten calendar images to digitize schedule events for import into a factory scheduling system.

[0597] In this way, the invention is a system that efficiently digitizes handwritten calendars and schedules and centralizes factory schedule management, thereby improving work efficiency and reducing errors.

[0598] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0599] Step 1:

[0600] The user takes a photo of a handwritten calendar with the smartphone camera and generates an image file. The input is the handwritten calendar, and the output is a digital image file. The user checks that the image is clearly captured.

[0601] Step 2:

[0602] A user uses an application on their device to upload images taken with a camera to a server. The input is a digital image file, and the output is an HTTP request to send that file to the server. The transmission begins when the user presses the upload button.

[0603] Step 3:

[0604] The server receives the uploaded image and stores it in a temporary storage directory. The input is the image data sent via the HTTP request, and the output is the image file in the server's temporary storage directory. This step is where the file is saved.

[0605] Step 4:

[0606] The server analyzes the stored image data using OCR technology. The input is the image file stored on the server, and the output is the extracted text data. Specifically, the server performs image analysis using Google Cloud Vision API and Tesseract OCR.

[0607] Step 5:

[0608] The server formats the text data analyzed by OCR technology into a format that can be understood by a calendar application or work schedule system. The input is the text data obtained by OCR analysis, and the output is formatted data (e.g., iCalendar format). The server uses scripts or application logic to format the data.

[0609] Step 6:

[0610] The server saves the formatted data in a database and notifies the user's terminal that the data is ready. The input is the formatted schedule data, and the output is the record saved in the database and a notification message to the terminal.

[0611] Step 7:

[0612] The device receives the notification from the server, and after obtaining the user's confirmation, launches the calendar app and instructs it to retrieve new schedule data. The input is the notification message from the server, and the output is a request to retrieve new schedule data.

[0613] Step 8:

[0614] The calendar app communicates with the server, gets the formatted schedule data, and imports it into the calendar. The input is the formatted data sent from the server, and the output is the schedule events imported into the calendar app. Through this series of processes, the new schedule is correctly reflected in the calendar.

[0615] Step 9:

[0616] The user checks the calendar app and verifies that the new schedule has been imported correctly. Similarly, the user verifies that the new schedule has been integrated into the factory's work scheduling system. The input is the imported schedule data, and the output is the user's confirmation operation.

[0617] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0618] The present invention combines a system that digitizes handwritten calendars and centrally manages them using a calendar application with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention will be described below.

[0619] Server Processing

[0620] 1. Receiving image uploads

[0621] When a handwritten calendar image is uploaded from a user's terminal, the server receives it and stores the received image data in a temporary directory.

[0622] 2. Image Analysis

[0623] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[0624] 3. Data Formatting

[0625] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI. It uses scripts and application logic to convert the event data into a format that the calendar app can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify who owns a specific event.

[0626] 4. Activating the Emotional Engine

[0627] Before importing data into the calendar app, the server activates an emotion engine to recognize the user's emotions. This emotion engine analyzes the user's past behavioral data and text input to recognize the user's emotional state at that time.

[0628] 5. Emotional Data Integration

[0629] The server then incorporates the recognized emotion data into the calendar data. For example, if the user feels nervous or anxious about an important appointment, the server stores this information as metadata. The server also arranges for the emotion data to be displayed in the calendar app.

[0630] 6. Data storage and transmission

[0631] The server stores the formatted data and emotion data in a database and sends a message to the user's device to notify them that the data is ready. When the user makes a request to obtain new calendar data, the server sends the data to the device.

[0632] Terminal handling

[0633] 1. Image capture

[0634] The user takes a photo of the handwritten calendar using the smartphone camera, and checks the image to make sure the entire calendar is clearly visible.

[0635] 2. Upload an image

[0636] When the user presses the image upload button, the device uploads the captured image data to the server. The device sends the image data to the server using an HTTP request.

[0637] 3. Receiving and importing data

[0638] The device that receives the notification from the server launches the calendar app with the user's confirmation. The calendar app communicates with the server and sends a request to obtain new format data and emotion data. Upon receiving the data from the server, the calendar app imports it and adds a new event to the existing calendar. The calendar app uses color-coding information and emotion data to distinguish and display each event for each individual.

[0639] User operations

[0640] 1. Calendar photoshoot

[0641] The user takes a photo of a handwritten calendar using their smartphone, and the events are color-coded to indicate who owns them.

[0642] 2. Upload confirmation

[0643] When uploading a captured image to a server, the user performs a confirmation operation.

[0644] 3. Data confirmation

[0645] The user can check their calendar app to ensure the new appointment and emotion data has been imported correctly. If necessary, the user can make additional edits or adjustments.

[0646] Specific examples

[0647] Example 1: Image upload and analysis

[0648] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server uses generative AI to analyze the text in the image and convert each person's schedule into digital text. It then uses an emotion engine to analyze the user's emotional state and integrates the emotional data into the calendar data. The data is then formatted and saved in a calendar app format, and notified to the user's device.

[0649] Example 2: Data import

[0650] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new events and emotion data from the server and automatically imports them into their calendar. The user can then check the events and emotions of each person correctly reflected in the calendar and edit them if necessary.

[0651] The system of the present invention not only digitizes handwritten calendar information and allows it to be shared with the whole family via a smartphone calendar app, but also reflects the user's emotional state, making it easier to share and coordinate schedules and allowing all family members to quickly grasp the latest information and emotional state.

[0652] The processing flow will be explained below.

[0653] Step 1:

[0654] The user takes a photo of the handwritten calendar using the smartphone camera, checks that the entire calendar is clearly visible, and prepares to upload it to the server via the app.

[0655] Step 2:

[0656] The user presses the upload button on the app to send the captured image data to the server. The device then sends the image data to the server using an HTTP request.

[0657] Step 3:

[0658] The server receives the image upload request sent from the user's terminal, saves the image data in a temporary storage directory, and sends a response confirming receipt to the terminal.

[0659] Step 4:

[0660] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[0661] Step 5:

[0662] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI, and uses scripts and logic to convert the extracted information into a format that the calendar app can understand (e.g., iCalendar format).

[0663] Step 6:

[0664] The server starts an emotion engine and analyzes the user's past behavioral data and schedule information. The emotion engine recognizes the user's emotional state and acquires that data.

[0665] Step 7:

[0666] The server integrates the user's emotional data into the schedule data. For example, if the user feels nervous or anxious about an important appointment, the server stores this information as metadata and stores it together with the calendar information.

[0667] Step 8:

[0668] The server stores the formatted data and emotion data in a database. It also sends a message to the user's device to notify them that the data is ready. Possible notification methods include push notifications and emails.

[0669] Step 9:

[0670] The device receives a notification from the server and notifies the user that new calendar data and emotion data are available. The user confirms the notification and launches the calendar app.

[0671] Step 10:

[0672] The device's calendar app sends a request to the server to get new formatting and emotion data, which the server then sends to the device.

[0673] Step 11:

[0674] The calendar application of the device imports the received format data and emotion data and adds a new event to the existing calendar. The calendar application uses color-coding information and emotion data to distinguish and display each event for each individual.

[0675] Step 12:

[0676] Users can then review their new events and emotion data in their calendar app to ensure it has been imported correctly. If necessary, users can make additional edits or adjustments.

[0677] Through these processing steps, the handwritten calendar information is digitized and can be shared with the whole family via a smartphone calendar app, and the user's emotional state is also reflected, making it easier to share and adjust schedules, and allowing all family members to quickly grasp the latest information and emotional state.

[0678] Example 2

[0679] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] Current digital calendar management systems lack the ability to digitize handwritten calendar information, making it difficult to manage schedules that take the user's emotional state into account. They also lack the ability to accurately identify handwritten information and reflect specific color coding or emotional data in the calendar. As a result, sharing and adjusting schedules becomes cumbersome and stressful for users.

[0681] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0682] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into a calendar application, means for activating an emotion analysis engine for recognizing the user's emotions, and means for integrating the emotion data into calendar data, thereby enabling accurate digitization of handwritten calendar information and schedule management that reflects the user's emotional state.

[0683] "Means for acquiring images" refers to equipment or software that allows a user to photograph or scan a handwritten calendar image and provide it to a server as digital data.

[0684] "Means for character recognition of acquired images" refers to the function of using OCR technology to extract character information from acquired handwritten calendar images and convert it into text data.

[0685] "Means for formatting the recognized data" refers to the script or application logic used to convert the text data obtained using OCR technology into a format that can be understood by a calendar application (e.g., iCalendar format).

[0686] "Means for importing formatted data into a calendar application" refers to functionality that allows formatted data to be read into a calendar application so that it can be displayed as an appointment on a digital calendar.

[0687] "Means for launching an emotion analysis engine to recognize a user's emotions" refers to a function for analyzing a user's past behavioral data and text inputs to run an engine to recognize the user's emotional state.

[0688] "Means for integrating emotional data into calendar data" refers to a function that integrates the recognized emotional data of a user with existing calendar data and reflects it in a calendar application as emotional information.

[0689] "Color coding information" refers to color information used to distinguish different events or people in a handwritten calendar.

[0690] "Calendar application" refers to software that allows a user to digitally manage their schedules.

[0691] The present invention combines a system that digitizes handwritten calendars and centrally manages them using a calendar application with an emotion analysis engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[0692] Server Processing

[0693] 1. Receiving image uploads

[0694] The server receives handwritten calendar image data from the user's device and stores it in a temporary directory. At this time, it assigns a unique ID to the file name for management purposes.

[0695] 2. Image Analysis

[0696] The server retrieves the calendar image from the temporary storage directory and inputs it into the generative AI model. The prompt text used is, "Please analyze the handwritten characters in this image and convert them into text data." The generative AI model then uses OCR technology to convert the handwritten characters in the image into text format and returns the results to the server.

[0697] 3. Data Formatting

[0698] The server parses the text data returned by the generative AI model to extract the date, time, content, and color-coding information for each event. This process is performed using Python scripts or custom application logic. The extracted data is then converted into a format that calendar applications can understand, such as iCalendar, and the color-coding information is added as metadata.

[0699] 4. Activating the Emotional Engine

[0700] The server starts the emotion engine before importing the data. The emotion engine analyzes the user's past behavioral data and text input to recognize their current emotional state. This process uses emotion analysis algorithms and natural language processing techniques.

[0701] 5. Emotional Data Integration

[0702] The server integrates the emotional data recognized by the emotion engine into the calendar data, saving the emotional information (e.g., "The user is feeling anxious about a meeting where traffic congestion is expected") as metadata, and formats the information so that it can be displayed in the calendar application.

[0703] 6. Data storage and transmission

[0704] The server stores the formatted data and emotion data in a database. It then notifies the user's device that the data is ready. This notification can be done via push notification or email. When the user makes a request, the server sends the data for import to the device.

[0705] Terminal handling

[0706] 1. Image capture

[0707] The user takes a photo of the handwritten calendar using the smartphone's camera. The user checks that the image is clear and shows the entire calendar. The user can also color-code events as needed.

[0708] 2. Upload an image

[0709] When a user presses the image upload button, the device sends the captured image to the server using an HTTP request (such as the POST method). This request includes the image file as well as metadata such as user information and the date and time the image was taken.

[0710] 3. Receiving and importing data

[0711] When the device receives a notification from the server that the data is ready, it launches the calendar application with the user's confirmation. The calendar application sends a request to the server to obtain new data and receives the data returned from the server. The received data is automatically imported into the calendar, and the schedule and emotion data are displayed.

[0712] User operations

[0713] 1. Calendar photoshoot

[0714] The user takes a photo of the handwritten calendar using the smartphone camera. When taking the photo, each event is color-coded to distinguish whose event it is.

[0715] 2. Upload confirmation

[0716] When uploading a captured image to a server, the user displays a confirmation screen to confirm the contents of the image to be sent. This confirmation operation uses check boxes and confirmation buttons.

[0717] 3. Data confirmation

[0718] Users can check in their calendar application whether the new events and emotion data have been imported correctly. If there are any errors in the imported content, users can manually correct or add them.

[0719] As described above, the system of the present invention digitizes handwritten calendar information and further integrates the user's emotional data, enabling more comprehensive schedule management.

[0720] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0721] Step 1:

[0722] The user takes a photo of the handwritten calendar using the smartphone camera. At this time, the events are color-coded to indicate who owns them. The captured image data is saved in the device.

[0723] Input: Handwritten calendar

[0724] Output: Captured image data

[0725] Step 2:

[0726] When a user presses the image upload button, the device sends the saved image data to the server as an HTTP request (POST method). The request includes the image file, user information, and the date and time the image was taken.

[0727] Input: Captured image data, user information, date and time of capture

[0728] Output: HTTP request sent to the server

[0729] Step 3:

[0730] The server stores the handwritten calendar image data received from the user's device in a temporary directory, and manages it by assigning a unique ID to the file name.

[0731] Input: Image data sent as an HTTP request

[0732] Output: Image data saved on the server (in the temporary directory)

[0733] Step 4:

[0734] The server retrieves the calendar image from the temporary storage directory and inputs it into the generative AI model. The prompt text used is, "Please analyze the handwritten characters in this image and convert them into text data." The generative AI model then uses OCR technology to convert the handwritten characters in the image into text format.

[0735] Input: Calendar image data retrieved from temporary directory

[0736] Output: Text data analyzed from handwritten characters

[0737] Step 5:

[0738] The server parses the text data returned by the generative AI model to extract the date, time, content, and color-coding information for each event. This analysis is done using Python scripts or custom application logic. The extracted data is then converted into a format that calendar applications can understand, such as iCalendar, and the color-coding information is added as metadata.

[0739] Input: Text data returned by the generative AI model

[0740] Output: Formatted data including iCalendar data and color coding information

[0741] Step 6:

[0742] The server starts the emotion engine before importing the data. The emotion engine analyzes the user's past behavioral data and text input to recognize their current emotional state. Natural language processing and emotion recognition algorithms are used for emotion analysis.

[0743] Input: User's past behavior data, text input

[0744] Output: Recognized user emotion data

[0745] Step 7:

[0746] The server integrates the emotional data recognized by the emotion engine into the calendar data. For example, it adds emotional information such as "the user is feeling anxious about an important meeting" as metadata. The server then formats the emotional data so that it can be displayed in the calendar application.

[0747] Input: Recognized emotion data, iCalendar format data

[0748] Output: Formatted calendar data with emotion data integrated

[0749] Step 8:

[0750] The server stores the formatted data and emotion data in a database. It then notifies the user's device that the data is ready. When the user makes a request, the server sends the data for import to the device.

[0751] Input: Formatted calendar data with emotion data integrated

[0752] Output: Data stored in the database, notifications sent to users

[0753] Step 9:

[0754] When the device receives a notification from the server that the data is ready, it launches the calendar application with the user's confirmation. The calendar application sends a request to the server to obtain new data and receives the data returned from the server. The received data is automatically imported into the calendar, and the schedule and emotion data are displayed.

[0755] Input: Notification from the server that data is ready

[0756] Output: Calendar data and emotion data imported into a calendar application

[0757] Step 10:

[0758] Users can check in their calendar application whether the new events and emotion data have been imported correctly. If there are any errors in the imported content, users can manually correct or add them.

[0759] Input: Calendar data and emotion data imported into a calendar application

[0760] Output: Confirmed or modified calendar data

[0761] (Application example 2)

[0762] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0763] Conventional handwritten shift schedules and schedule management systems were difficult to digitize, making it time-consuming to share and update information across the organization. Furthermore, they were unable to reflect employees' emotional states, making it difficult to optimize the work environment. For this reason, there was a demand for a system that could easily digitize handwritten shift schedules and reflect employees' emotional states, thereby streamlining store operations and contributing to employee motivation management.

[0764] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0765] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into the system, means for recognizing an emotional state based on the formatted data, and means for integrating and storing the emotional data in a database, thereby making it possible to digitize handwritten shift schedules and integrate the emotional data.

[0766] The "means for acquiring images" refers to a means by which a user photographs a handwritten shift table or schedule and provides the image to the server as a digital image.

[0767] "Means for character recognition in acquired images" refers to a means for analyzing handwritten characters in digital images using OCR technology and converting them into text data.

[0768] "Means for formatting character-recognized data" refers to means for converting text data obtained by OCR into a format that can be understood by the system.

[0769] A "means for importing formatted data into a system" is a means for incorporating formatted data into a particular system or application.

[0770] The "means for recognizing an emotional state based on formatted data" is a means for analyzing data and past information to estimate a user's emotional state.

[0771] The "means for integrating emotional data and storing it in a database" is a means for integrating recognized emotional data with formatted data, and storing the data in an organized manner in a database.

[0772] The present invention includes the following configuration and processing means to provide a system that digitizes handwritten shift tables and schedules and further reflects the emotional state of employees.

[0773] System Configuration

[0774] The system mainly consists of a server, a terminal (e.g., a smartphone), and a user interface. Specific usage methods for each component are explained below.

[0775] Server Processing

[0776] 1. Receiving image uploads

[0777] When an image of a handwritten shift schedule is uploaded from a user's device, the server receives it. The received image data is temporarily stored in a directory. The server uses a cloud storage service (e.g., Amazon S3) to store the image data.

[0778] 2. Image Analysis

[0779] The server inputs the stored image data into a generative AI model using OCR technology, for example, by using the Google Cloud Vision API to analyze handwritten characters in the image and convert them into text data.

[0780] 3. Data Formatting

[0781] The server extracts the shift schedule's date, time, content, and color-coded information from the text data output by the generation AI and converts it into a format the system can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify specific employees.

[0782] 4. Activating the Emotional Engine

[0783] Before importing the shift schedule data, the server starts an emotion engine to recognize the emotions of employees. This emotion engine analyzes employees' past work history and comments to recognize their emotional state at that time, using, for example, IBM Watson's emotion analysis API.

[0784] 5. Emotional Data Integration

[0785] The server then incorporates the recognized emotion data into the shift schedule data. For example, if an employee is feeling stressed or tired, the emotion data is saved as metadata and used for future shift management.

[0786] 6. Data storage and transmission

[0787] The server stores the formatted data and emotion data in a database (e.g., MySQL) and sends a message to the user's device to notify them that the data is ready. You can use a notification service such as Firebase Cloud Messaging (FCM).

[0788] Terminal handling

[0789] 1. Image capture

[0790] The user takes a photo of the handwritten shift schedule using the smartphone camera, and checks the image to make sure the entire shift schedule is clearly visible.

[0791] 2. Upload an image

[0792] When the user presses the image upload button, the device uploads the captured image data to the server. The image data is sent to the server using an HTTP request.

[0793] 3. Receiving and importing data

[0794] Upon receiving the notification from the server, the device launches the system app with the user's confirmation. The app communicates with the server and sends a request to obtain new format data and emotion data. Upon receiving the data from the server, the app imports it and adds a new shift schedule to the existing system. The app uses color-coding information and emotion data to distinguish and display the status of each employee.

[0795] Specific examples

[0796] The store manager, who is the user, takes a photo of a handwritten shift schedule with their smartphone and uploads the image to the server. The server uses OCR technology to analyze the handwritten characters in the image and convert the shift information into text data. Next, it uses an emotion engine to analyze past work history and comments to recognize employee emotions. This emotion data is integrated into the shift data and saved in a database. The device receives a notification from the server and launches the shift management app to check the updated data.

[0797] Prompt Sentence Examples

[0798] "Analyze employees' work logs from the past six months to understand their emotional state, especially their stress and fatigue levels."

[0799] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0800] Step 1:

[0801] Taking and reviewing images

[0802] The user takes a photo of a handwritten shift schedule using the smartphone camera. This image is checked to see if the entire shift schedule is clearly captured. The input is the physical handwritten shift schedule, and the output is a digital image file.

[0803] Step 2:

[0804] Uploading an image

[0805] When a user presses the image upload button, the device uploads the captured image data to the server. The image data is sent using an HTTP request, with the input being a digital image file and the output being image data stored on the server.

[0806] Step 3:

[0807] Image analysis

[0808] The server analyzes the stored image data using OCR technology. Specifically, it uses an OCR engine such as Google Cloud Vision API to analyze the handwritten characters in the image and convert them into text data. The input is the uploaded image data, and the output is the analyzed text data.

[0809] Step 4:

[0810] Data Formatting

[0811] The server extracts the shift schedule date, time, content, and color-coded information from the text data output by the generative AI model and converts it into a format that the system can understand (e.g., iCalendar format). The input is the text data generated by OCR, and the output is the formatted shift schedule data.

[0812] Step 5:

[0813] Emotion engine activation and analysis

[0814] Before importing the formatted shift data, the server launches an emotion engine using IBM Watson's emotion analysis API to analyze employees' past work history and comments. The input is the employee's past data and formatted shift data, and the output is recognized emotion data.

[0815] Step 6:

[0816] Emotional Data Integration

[0817] The server integrates the recognized emotion data into the shift schedule data. Based on the analysis results, specific emotions (e.g., stress or fatigue) are added to the shift schedule data as metadata. The input is formatted shift schedule data and emotion data, and the output is shift schedule data integrated with emotion data.

[0818] Step 7:

[0819] Data Retention and Notification

[0820] The server saves the consolidated shift data in a database (e.g., MySQL) and sends notifications to the user's device via Firebase Cloud Messaging (FCM), etc. The input is the consolidated shift data and user device information, and the output is the data saved in the database and the sent notifications.

[0821] Step 8:

[0822] Receiving and displaying data

[0823] The user's device receives the notification from the server, launches the shift management app, and retrieves the new shift data. The retrieved data is displayed in the app, and the shift schedule reflecting each employee's emotional state is confirmed. The input is the integrated shift data sent from the server, and the output is the new shift schedule displayed in the shift management app on the device.

[0824] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0825] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0826] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0827] [Third embodiment]

[0828] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0829] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0830] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0831] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0832] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0833] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0834] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0835] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0836] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0837] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0838] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0839] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0840] The present invention provides a system for digitizing handwritten calendars and managing them in a unified manner using a calendar application. Specific embodiments of the system are described below.

[0841] Server Processing

[0842] 1. Receiving image uploads

[0843] When a handwritten calendar image is uploaded from a user's terminal, the server receives it and stores the received image data in a temporary directory.

[0844] 2. Image Analysis

[0845] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[0846] 3. Data Formatting

[0847] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI. Using scripts or application logic, the extracted information is converted into a format that the calendar app can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify who owns a particular event.

[0848] 4. Data storage and transmission

[0849] The server stores the formatted data in a database and sends a message to the user's device to notify them that the data is ready. When the user makes a request to retrieve new calendar data, the server sends the data to the device.

[0850] Terminal handling

[0851] 1. Image capture

[0852] The user takes a photo of the handwritten calendar using the smartphone camera, and checks the image to make sure the entire calendar is clearly visible.

[0853] 2. Upload an image

[0854] When the user presses the image upload button, the device uploads the captured image data to the server. The device sends the image data to the server using an HTTP request.

[0855] 3. Receiving and importing data

[0856] The device that receives the notification from the server launches the calendar app with the user's confirmation. The calendar app communicates with the server and sends a request to obtain new formatted data. Upon receiving the formatted calendar data from the server, the calendar app imports it and adds new events to the existing calendar. The calendar app uses color-coding information to distinguish and display each event for each individual.

[0857] User operations

[0858] 1. Calendar photoshoot

[0859] The user takes a photo of a handwritten calendar using their smartphone, and the events are color-coded to indicate who owns them.

[0860] 2. Upload confirmation

[0861] When uploading a captured image to a server, the user performs a confirmation operation.

[0862] 3. Data confirmation

[0863] The user can check their calendar app to ensure the new events have been imported correctly, and if necessary, they can make additional edits or adjustments.

[0864] Specific examples

[0865] Example 1: Image upload and analysis

[0866] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server then uses generative AI to analyze the text in the image and convert each person's schedule into digital text. The text is then formatted and saved in a calendar app format, and the user's device is notified.

[0867] Example 2: Data import

[0868] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new event data from the server and automatically imports it into their calendar. The user can then check each person's events, which are correctly reflected in the calendar, and edit them if necessary.

[0869] The system of the present invention allows for centralized digital management while maintaining the convenience of a handwritten calendar, making it easier to share and coordinate schedules and allowing all family members to quickly understand the latest information.

[0870] The processing flow will be explained below.

[0871] Step 1:

[0872] The user uses the smartphone camera to take a picture of a handwritten calendar hanging on the wall, and checks the image to make sure the entire calendar is clearly visible.

[0873] Step 2:

[0874] The device prepares to upload the captured image data to the server through the application interface. When the user presses the upload button, the device sends the image data to the server via an HTTP request.

[0875] Step 3:

[0876] The server receives the image upload request sent from the user's terminal, saves the image data in a temporary directory, and sends a response confirming receipt to the terminal.

[0877] Step 4:

[0878] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[0879] Step 5:

[0880] The server extracts the event date, time, content, and color-coded information from the parsed text data, using scripts or application logic to convert the event data into a format that the calendar app can understand (e.g., iCalendar format).

[0881] Step 6:

[0882] The server stores the formatted data in a database, a process that ensures that the scheduled data is easily accessible at a later time and is accurately maintained.

[0883] Step 7:

[0884] The server sends a message to the user's device to notify them that the formatted data is ready, possibly via push notification or email.

[0885] Step 8:

[0886] The device receives a notification from the server informing the user that new calendar data is available. The user confirms the notification and launches the calendar app.

[0887] Step 9:

[0888] The device's calendar app sends a request to the server to get the new format data, and the server sends the saved format data to the device.

[0889] Step 10:

[0890] The calendar application on the device imports the received format data and adds new events to the existing calendar. The calendar application uses color-coding information to distinguish and display each event for each individual.

[0891] Step 11:

[0892] The user can verify that the new event has been imported correctly within their calendar app, and can make additional edits or adjustments if needed.

[0893] Through these processing steps, the handwritten calendar information is digitized and can be shared with the whole family via a smartphone calendar app.

[0894] Example 1

[0895] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0896] Systems that digitize and centrally manage handwritten calendars face the problem of cumbersome and time-consuming manual data entry. It is also difficult to manage individual appointments by color-coding them, making sharing and coordination difficult. Furthermore, there is a need to improve the accuracy of handwritten character recognition and accurately convert them into digital data.

[0897] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0898] In this invention, the server includes a means for receiving and temporarily storing image data from the user's device, a means for inputting the image data into a generative AI model and analyzing handwritten characters, and a means for converting the generated character recognition data into a format that can be understood by a calendar application. This allows handwritten calendars to be digitized and centrally managed, and schedules to be color-coded and shared.

[0899] "Means for acquiring images" refers to devices or software that allow a user to take a photo of a handwritten calendar with a smartphone or camera and acquire the image data.

[0900] The "means for character recognition of acquired images" refers to software or algorithms for analyzing handwritten characters from acquired image data and extracting them as digital text data.

[0901] A "means for formatting character-recognized data" is software or script that converts text data obtained by character recognition into a specific data format that can be recognized by a calendar application (e.g., iCalendar format).

[0902] "Means for saving formatted data in a database and transmitting it to a user's terminal" refers to communication means or software that saves the converted data in a database and transmits it to a user's terminal as needed.

[0903] The "means for receiving image data from the user's terminal and temporarily storing it" refers to server-side storage or software that receives image data sent from the user's terminal and temporarily stores it.

[0904] The "means of inputting image data into a generative AI model and analyzing handwritten characters" refers to a method of supplying received and stored image data to a generative AI model and converting handwritten characters into text data.

[0905] "Means for converting the generated character recognition data into a format understandable by a calendar application" refers to software or scripts that convert the text data output by the generative AI model into a format that can be imported and understood by a calendar application (e.g., iCalendar format).

[0906] The "means for sending a notification to the user's terminal that the data is ready" refers to a communication function or software for notifying the user's terminal that the formatted data is ready.

[0907] The present invention provides a system for digitizing handwritten calendars and managing them in a unified manner using a calendar application. Specific embodiments of the system are described below.

[0908] System Overview

[0909] The system starts when a user takes a photo of a handwritten calendar with their smartphone and uploads it to a server. The server analyzes the uploaded image using a generative AI model and converts it into digital data. The converted data is then formatted into a format that the calendar app can understand and saved in a database. Finally, the formatted data is sent to the user's device and imported into the calendar app.

[0910] Hardware and Software

[0911] Smartphone: Users take a photo of the handwritten calendar using the camera on their smartphone. iPhones, Android devices, etc. can be used.

[0912] Server: A server is used to receive, analyze, format, store, and transmit image data. Specifically, a web framework such as Flask or Django is used.

[0913] Generative AI models: used to analyze handwritten characters in images, for example, using pre-trained models using PyTorch or TensorFlow.

[0914] OCR technology: Google Cloud Vision API and Tesseract are used to analyze handwritten characters in images.

[0915] Database: Use a database such as MySQL to store the analyzed data.

[0916] Calendar app: Use an application such as Google Calendar to import digital data.

[0917] Processing Flow

[0918] A user uses the system in the following steps:

[0919] 1. Image capture

[0920] A user takes a photo of a handwritten calendar using the camera on their smartphone. When taking the photo, they color-code the events so that it is clear who owns each one. For example, they could take a photo of a handwritten family calendar and mark each event with a different color.

[0921] 2. Upload an image

[0922] When a user presses the image upload button in the app, the smartphone uploads the captured image data to the server. This is done by sending the data to the server using an HTTP POST request. For example, the image data is POSTed to the / upload_image endpoint of the Flask server.

[0923] 3. Image Analysis

[0924] The server reads the saved image data and passes it to the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data. For example, it can call the Google Cloud Vision API to perform character recognition.

[0925] 4. Data Formatting

[0926] The server extracts date, time, content, and color-coded information from the text data output by the generation AI. Using a script (e.g., a Python script), it converts the extracted information into a format understandable by a calendar app, such as iCalendar format. It adds the color-coded information as metadata. For example, data such as "{"date": "2023-10-01", "events": ["Meeting", "Meeting"]}" is converted to the format "BEGIN:VEVENT\nDTSTART:20231001T090000Z\nSUMMARY:Meeting\nEND:VEVENT\n".

[0927] 5. Data storage and transmission

[0928] The server saves the formatted data in a database (e.g., MySQL) and notifies the user's device that the data is ready. Specifically, this is done using WebSocket or push notification. For example, the server inserts iCalendar format data into the calendar_events table in a MySQL database and sends a message to the user's device via WebSocket saying "Calendar data has been updated."

[0929] 6. Receiving and Importing Data

[0930] The device that receives the notification from the server displays the notification to the user and launches the calendar app. The calendar app communicates with the server to send a request for new data and receives the formatted data. The received data is then imported into the calendar app, where events are displayed in color.

[0931] Specific examples

[0932] Example 1: Image upload and analysis

[0933] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server then uses a generative AI model to analyze the text in the image and convert each person's schedule into digital text. The text is then formatted and saved in a calendar app format, and the user's device is notified.

[0934] Example 2: Data import

[0935] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new event data from the server and automatically imports it into their calendar. The user can then check each person's events, which are correctly reflected in the calendar, and edit them if necessary.

[0936] Prompt Sentence Examples

[0937] Please explain the system that takes a photo of a family's handwritten calendar with a smartphone, digitizes it on the cloud, and imports it into a calendar app. For example, what steps are taken to process the data, and how does the user operate it?

[0938] This invention digitizes handwritten calendars and centralizes their management, making it easier to share and coordinate schedules, and allowing all family members to quickly grasp the latest information.

[0939] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0940] Step 1:

[0941] Image capture

[0942] A user launches the camera app on their smartphone and takes a photo of a handwritten calendar. When taking the photo, they color-code the events to identify who owns them. For example, they can take a photo of a family calendar and mark each event with a different color.

[0943] Input: Handwritten calendar

[0944] Output: Captured image data (JPEG or PNG format)

[0945] Step 2:

[0946] Image upload

[0947] When a user presses the image upload button in the app, the device sends the captured image data to the server using an HTTP POST request. For example, the image data is POSTed to http: / / example.com / upload_image.

[0948] Input: Captured image data

[0949] Output: HTTP POST request to the server

[0950] Step 3:

[0951] Receiving and saving images on the server

[0952] The server receives the image data sent from the user's device and stores it in a temporary directory. For example, it processes the request using a web framework such as Flask or Django and stores the image file as / tmp / uploaded_images / image1.jpg.

[0953] Input: HTTP POST request (image data)

[0954] Output: Temporarily saved image file

[0955] Step 4:

[0956] Image analysis

[0957] The server reads the temporarily saved image data and inputs it into the generative AI model. The generative AI model analyzes the handwritten characters in the image using OCR technology (e.g., Google Cloud Vision API or Tesseract) and converts them into text data. For example, input / tmp / uploaded_images / image1.jpg into the Google Cloud Vision API and obtain JSON-formatted text data as the character recognition results.

[0958] Input: Saved image file

[0959] Output: Parsed text data (JSON format)

[0960] Step 5:

[0961] Data Formatting

[0962] The server extracts date, time, content, and color-coding information from the text data output by the generative AI model. It then uses a script (e.g., a Python script) to convert the extracted information into a format that calendar apps can understand, such as iCalendar, and adds the color-coding information as metadata.

[0963] Input: Parsed text data (JSON format)

[0964] Output: Formatted calendar data (iCalendar format)

[0965] Step 6:

[0966] Data Retention and Notification

[0967] The server saves the formatted data in a database (e.g., MySQL) and sends a notification to the user's device that the data is ready. For example, it inserts the iCalendar format data into the calendar_events table in the MySQL database and sends a message to the user's device via WebSocket saying "Calendar data has been updated."

[0968] Input: Formatted calendar data (iCalendar format)

[0969] Output: Calendar data stored in the database, and notification messages

[0970] Step 7:

[0971] Receiving and Importing Data

[0972] When the device receives the notification from the server, it displays the notification to the user and launches the calendar app. The calendar app then communicates with the server and sends a request to retrieve new data. The formatted calendar data is then received and automatically imported into the calendar. For example, the calendar app may use a REST API to retrieve iCalendar formatted data, import it into the calendar, and display it.

[0973] Input: Notification message, formatted calendar data

[0974] Output: New events imported into the Calendar app

[0975] This process converts handwritten calendars into digital data and efficiently manages them centrally. Users can easily digitize their handwritten calendars and manage them in a calendar app.

[0976] (Application example 1)

[0977] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0978] Existing factory work schedule management methods often require a lot of time and effort in the process of digitizing handwritten schedules, making it difficult to streamline the process. Furthermore, it is difficult to integrate and manage the schedules of multiple individuals, who need to be color-coded or otherwise differentiated, making centralized schedule management difficult. The objective of this invention is to efficiently and accurately digitize handwritten work schedules and realize centralized schedule management.

[0979] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0980] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into a calendar application, and means for importing the formatted data into a work schedule system. This makes it possible to efficiently digitize handwritten work schedules and centrally manage them, including each individual's schedule.

[0981] The "means for acquiring images" has a function that allows a user to take a photograph of a handwritten calendar or work schedule and send it to the server in digital image format.

[0982] "Means for character recognition" refers to a technology that extracts character information from an acquired image and converts it into digital text format, and uses OCR technology.

[0983] The "formatting means" has a function for converting the character-recognized data into a specific format, such as the iCalendar format.

[0984] The "means for importing into a calendar application" has a function for importing formatted data into a calendar application and displaying it as a schedule item.

[0985] The "means for importing into the work schedule system" refers to a function that imports formatted data into the factory's work schedule system and registers and manages it as schedule information.

[0986] The "means for analyzing and identifying the schedule of each individual" has a function for distinguishing schedules of different individuals by taking into account color-coding information in the acquired image.

[0987] The "means for storing and transmitting to the user's terminal" has the function of storing formatted data in a database and transmitting the data to the user's terminal as required.

[0988] The present invention relates to a system for digitizing handwritten calendars and work schedules and managing them in a unified manner. Specific embodiments for carrying out the present invention will be described below.

[0989] Server Action:

[0990] When a handwritten calendar image is uploaded from the user's device, the server first stores it in a temporary directory. The server then analyzes the stored image data using OCR technology. Specifically, it uses software such as Tesseract OCR or Google Cloud Vision API. The analyzed data is then formatted into a specific format (e.g., iCalendar format) and stored in a database. The server then imports the formatted data into the work schedule system and sends it to the user's device as needed.

[0991] Terminal handling:

[0992] The device first takes a photo of the handwritten calendar through user operation and then sends the image to the server by pressing the upload button. At this time, the image data is sent using an HTTP request. When the device receives a notification from the server that the new schedule data is ready, it retrieves it and imports it into the calendar app. The device also has the function to import the data received from the server into the work schedule system.

[0993] User Action:

[0994] The user takes a photo of a handwritten calendar using their smartphone. When taking the photo, the appointments are color-coded to indicate who owns them. The image is uploaded to the server, and upon receiving a notification from the server, the user checks the calendar app to confirm that the new schedule has been imported correctly. The user also verifies that the new schedule has been integrated into the factory's work scheduling system.

[0995] Examples:

[0996] At the beginning of the month, a user takes a photo of a manually written factory work schedule and uploads the image to the server. The server uses OCR technology to analyze the text in the image and converts the schedules for each worker and machine into digital text. The server then formats the text into iCalendar format or a format that can be understood by the factory's scheduling system. The formatted data is stored in a database, and an import notification is sent to the user's device. The user receives the notification, launches their calendar app, and checks the new schedule data. They also verify that the new schedule has been correctly integrated into the factory's work scheduling system.

[0997] Example prompt sentence:

[0998] Write code for an application that parses handwritten calendar images to digitize schedule events for import into a factory scheduling system.

[0999] In this way, the invention is a system that efficiently digitizes handwritten calendars and schedules and centralizes factory schedule management, thereby improving work efficiency and reducing errors.

[1000] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1001] Step 1:

[1002] The user takes a photo of a handwritten calendar with the smartphone camera and generates an image file. The input is the handwritten calendar, and the output is a digital image file. The user checks that the image is clearly captured.

[1003] Step 2:

[1004] A user uses an application on their device to upload images taken with a camera to a server. The input is a digital image file, and the output is an HTTP request to send that file to the server. The transmission begins when the user presses the upload button.

[1005] Step 3:

[1006] The server receives the uploaded image and stores it in a temporary storage directory. The input is the image data sent via the HTTP request, and the output is the image file in the server's temporary storage directory. This step is where the file is saved.

[1007] Step 4:

[1008] The server analyzes the stored image data using OCR technology. The input is the image file stored on the server, and the output is the extracted text data. Specifically, the server performs image analysis using Google Cloud Vision API and Tesseract OCR.

[1009] Step 5:

[1010] The server formats the text data analyzed by OCR technology into a format that can be understood by a calendar application or work schedule system. The input is the text data obtained by OCR analysis, and the output is formatted data (e.g., iCalendar format). The server uses scripts or application logic to format the data.

[1011] Step 6:

[1012] The server saves the formatted data in a database and notifies the user's terminal that the data is ready. The input is the formatted schedule data, and the output is the record saved in the database and a notification message to the terminal.

[1013] Step 7:

[1014] The device receives the notification from the server, and after obtaining the user's confirmation, launches the calendar app and instructs it to retrieve new schedule data. The input is the notification message from the server, and the output is a request to retrieve new schedule data.

[1015] Step 8:

[1016] The calendar app communicates with the server, gets the formatted schedule data, and imports it into the calendar. The input is the formatted data sent from the server, and the output is the schedule events imported into the calendar app. Through this series of processes, the new schedule is correctly reflected in the calendar.

[1017] Step 9:

[1018] The user checks the calendar app and verifies that the new schedule has been imported correctly. Similarly, the user verifies that the new schedule has been integrated into the factory's work scheduling system. The input is the imported schedule data, and the output is the user's confirmation operation.

[1019] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1020] The present invention combines a system that digitizes handwritten calendars and centrally manages them using a calendar application with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention will be described below.

[1021] Server Processing

[1022] 1. Receiving image uploads

[1023] When a handwritten calendar image is uploaded from a user's terminal, the server receives it and stores the received image data in a temporary directory.

[1024] 2. Image Analysis

[1025] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[1026] 3. Data Formatting

[1027] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI. It uses scripts and application logic to convert the event data into a format that the calendar app can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify who owns a specific event.

[1028] 4. Activating the Emotional Engine

[1029] Before importing data into the calendar app, the server activates an emotion engine to recognize the user's emotions. This emotion engine analyzes the user's past behavioral data and text input to recognize the user's emotional state at that time.

[1030] 5. Emotional Data Integration

[1031] The server then incorporates the recognized emotion data into the calendar data. For example, if the user feels nervous or anxious about an important appointment, the server stores this information as metadata. The server also arranges for the emotion data to be displayed in the calendar app.

[1032] 6. Data storage and transmission

[1033] The server stores the formatted data and emotion data in a database and sends a message to the user's device to notify them that the data is ready. When the user makes a request to obtain new calendar data, the server sends the data to the device.

[1034] Terminal handling

[1035] 1. Image capture

[1036] The user takes a photo of the handwritten calendar using the smartphone camera, and checks the image to make sure the entire calendar is clearly visible.

[1037] 2. Upload an image

[1038] When the user presses the image upload button, the device uploads the captured image data to the server. The device sends the image data to the server using an HTTP request.

[1039] 3. Receiving and importing data

[1040] The device that receives the notification from the server launches the calendar app with the user's confirmation. The calendar app communicates with the server and sends a request to obtain new format data and emotion data. Upon receiving the data from the server, the calendar app imports it and adds a new event to the existing calendar. The calendar app uses color-coding information and emotion data to distinguish and display each event for each individual.

[1041] User operations

[1042] 1. Calendar photoshoot

[1043] The user takes a photo of a handwritten calendar using their smartphone, and the events are color-coded to indicate who owns them.

[1044] 2. Upload confirmation

[1045] When uploading a captured image to a server, the user performs a confirmation operation.

[1046] 3. Data confirmation

[1047] The user can check their calendar app to ensure the new appointment and emotion data has been imported correctly. If necessary, the user can make additional edits or adjustments.

[1048] Specific examples

[1049] Example 1: Image upload and analysis

[1050] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server uses generative AI to analyze the text in the image and convert each person's schedule into digital text. It then uses an emotion engine to analyze the user's emotional state and integrates the emotional data into the calendar data. The data is then formatted and saved in a calendar app format, and notified to the user's device.

[1051] Example 2: Data import

[1052] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new events and emotion data from the server and automatically imports them into their calendar. The user can then check the events and emotions of each person correctly reflected in the calendar and edit them if necessary.

[1053] The system of the present invention not only digitizes handwritten calendar information and allows it to be shared with the whole family via a smartphone calendar app, but also reflects the user's emotional state, making it easier to share and coordinate schedules and allowing all family members to quickly grasp the latest information and emotional state.

[1054] The processing flow will be explained below.

[1055] Step 1:

[1056] The user takes a photo of the handwritten calendar using the smartphone camera, checks that the entire calendar is clearly visible, and prepares to upload it to the server via the app.

[1057] Step 2:

[1058] The user presses the upload button on the app to send the captured image data to the server. The device then sends the image data to the server using an HTTP request.

[1059] Step 3:

[1060] The server receives the image upload request sent from the user's terminal, saves the image data in a temporary storage directory, and sends a response confirming receipt to the terminal.

[1061] Step 4:

[1062] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[1063] Step 5:

[1064] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI, and uses scripts and logic to convert the extracted information into a format that the calendar app can understand (e.g., iCalendar format).

[1065] Step 6:

[1066] The server starts an emotion engine and analyzes the user's past behavioral data and schedule information. The emotion engine recognizes the user's emotional state and acquires that data.

[1067] Step 7:

[1068] The server integrates the user's emotional data into the schedule data. For example, if the user feels nervous or anxious about an important appointment, the server stores this information as metadata and stores it together with the calendar information.

[1069] Step 8:

[1070] The server stores the formatted data and emotion data in a database. It also sends a message to the user's device to notify them that the data is ready. Possible notification methods include push notifications and emails.

[1071] Step 9:

[1072] The device receives a notification from the server and notifies the user that new calendar data and emotion data are available. The user confirms the notification and launches the calendar app.

[1073] Step 10:

[1074] The device's calendar app sends a request to the server to get new formatting and emotion data, which the server then sends to the device.

[1075] Step 11:

[1076] The calendar application of the device imports the received format data and emotion data and adds a new event to the existing calendar. The calendar application uses color-coding information and emotion data to distinguish and display each event for each individual.

[1077] Step 12:

[1078] Users can then review their new events and emotion data in their calendar app to ensure it has been imported correctly. If necessary, users can make additional edits or adjustments.

[1079] Through these processing steps, the handwritten calendar information is digitized and can be shared with the whole family via a smartphone calendar app, and the user's emotional state is also reflected, making it easier to share and adjust schedules, and allowing all family members to quickly grasp the latest information and emotional state.

[1080] Example 2

[1081] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1082] Current digital calendar management systems lack the ability to digitize handwritten calendar information, making it difficult to manage schedules that take the user's emotional state into account. They also lack the ability to accurately identify handwritten information and reflect specific color coding or emotional data in the calendar. As a result, sharing and adjusting schedules becomes cumbersome and stressful for users.

[1083] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1084] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into a calendar application, means for activating an emotion analysis engine for recognizing the user's emotions, and means for integrating the emotion data into calendar data, thereby enabling accurate digitization of handwritten calendar information and schedule management that reflects the user's emotional state.

[1085] "Means for acquiring images" refers to equipment or software that allows a user to photograph or scan a handwritten calendar image and provide it to a server as digital data.

[1086] "Means for character recognition of acquired images" refers to the function of using OCR technology to extract character information from acquired handwritten calendar images and convert it into text data.

[1087] "Means for formatting the recognized data" refers to the script or application logic used to convert the text data obtained using OCR technology into a format that can be understood by a calendar application (e.g., iCalendar format).

[1088] "Means for importing formatted data into a calendar application" refers to functionality that allows formatted data to be read into a calendar application so that it can be displayed as an appointment on a digital calendar.

[1089] "Means for launching an emotion analysis engine to recognize a user's emotions" refers to a function for analyzing a user's past behavioral data and text inputs to run an engine to recognize the user's emotional state.

[1090] "Means for integrating emotional data into calendar data" refers to a function that integrates the recognized emotional data of a user with existing calendar data and reflects it in a calendar application as emotional information.

[1091] "Color coding information" refers to color information used to distinguish different events or people in a handwritten calendar.

[1092] "Calendar application" refers to software that allows a user to digitally manage their schedules.

[1093] The present invention combines a system that digitizes handwritten calendars and centrally manages them using a calendar application with an emotion analysis engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[1094] Server Processing

[1095] 1. Receiving image uploads

[1096] The server receives handwritten calendar image data from the user's device and stores it in a temporary directory. At this time, it assigns a unique ID to the file name for management purposes.

[1097] 2. Image Analysis

[1098] The server retrieves the calendar image from the temporary storage directory and inputs it into the generative AI model. The prompt text used is, "Please analyze the handwritten characters in this image and convert them into text data." The generative AI model then uses OCR technology to convert the handwritten characters in the image into text format and returns the results to the server.

[1099] 3. Data Formatting

[1100] The server parses the text data returned by the generative AI model to extract the date, time, content, and color-coding information for each event. This process is performed using Python scripts or custom application logic. The extracted data is then converted into a format that calendar applications can understand, such as iCalendar, and the color-coding information is added as metadata.

[1101] 4. Activating the Emotional Engine

[1102] The server starts the emotion engine before importing the data. The emotion engine analyzes the user's past behavioral data and text input to recognize their current emotional state. This process uses emotion analysis algorithms and natural language processing techniques.

[1103] 5. Emotional Data Integration

[1104] The server integrates the emotional data recognized by the emotion engine into the calendar data, saving the emotional information (e.g., "The user is feeling anxious about a meeting where traffic congestion is expected") as metadata, and formats the information so that it can be displayed in the calendar application.

[1105] 6. Data storage and transmission

[1106] The server stores the formatted data and emotion data in a database. It then notifies the user's device that the data is ready. This notification can be done via push notification or email. When the user makes a request, the server sends the data for import to the device.

[1107] Terminal handling

[1108] 1. Image capture

[1109] The user takes a photo of the handwritten calendar using the smartphone's camera. The user checks that the image is clear and shows the entire calendar. The user can also color-code events as needed.

[1110] 2. Upload an image

[1111] When a user presses the image upload button, the device sends the captured image to the server using an HTTP request (such as the POST method). This request includes the image file as well as metadata such as user information and the date and time the image was taken.

[1112] 3. Receiving and importing data

[1113] When the device receives a notification from the server that the data is ready, it launches the calendar application with the user's confirmation. The calendar application sends a request to the server to obtain new data and receives the data returned from the server. The received data is automatically imported into the calendar, and the schedule and emotion data are displayed.

[1114] User operations

[1115] 1. Calendar photoshoot

[1116] The user takes a photo of the handwritten calendar using the smartphone camera. When taking the photo, each event is color-coded to distinguish whose event it is.

[1117] 2. Upload confirmation

[1118] When uploading a captured image to a server, the user displays a confirmation screen to confirm the contents of the image to be sent. This confirmation operation uses check boxes and confirmation buttons.

[1119] 3. Data confirmation

[1120] Users can check in their calendar application whether the new events and emotion data have been imported correctly. If there are any errors in the imported content, users can manually correct or add them.

[1121] As described above, the system of the present invention digitizes handwritten calendar information and further integrates the user's emotional data, enabling more comprehensive schedule management.

[1122] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1123] Step 1:

[1124] The user takes a photo of the handwritten calendar using the smartphone camera. At this time, the events are color-coded to indicate who owns them. The captured image data is saved in the device.

[1125] Input: Handwritten calendar

[1126] Output: Captured image data

[1127] Step 2:

[1128] When a user presses the image upload button, the device sends the saved image data to the server as an HTTP request (POST method). The request includes the image file, user information, and the date and time the image was taken.

[1129] Input: Captured image data, user information, date and time of capture

[1130] Output: HTTP request sent to the server

[1131] Step 3:

[1132] The server stores the handwritten calendar image data received from the user's device in a temporary directory, and manages it by assigning a unique ID to the file name.

[1133] Input: Image data sent as an HTTP request

[1134] Output: Image data saved on the server (in the temporary directory)

[1135] Step 4:

[1136] The server retrieves the calendar image from the temporary storage directory and inputs it into the generative AI model. The prompt text used is, "Please analyze the handwritten characters in this image and convert them into text data." The generative AI model then uses OCR technology to convert the handwritten characters in the image into text format.

[1137] Input: Calendar image data retrieved from temporary directory

[1138] Output: Text data analyzed from handwritten characters

[1139] Step 5:

[1140] The server parses the text data returned by the generative AI model to extract the date, time, content, and color-coding information for each event. This analysis is done using Python scripts or custom application logic. The extracted data is then converted into a format that calendar applications can understand, such as iCalendar, and the color-coding information is added as metadata.

[1141] Input: Text data returned by the generative AI model

[1142] Output: Formatted data including iCalendar data and color coding information

[1143] Step 6:

[1144] The server starts the emotion engine before importing the data. The emotion engine analyzes the user's past behavioral data and text input to recognize their current emotional state. Natural language processing and emotion recognition algorithms are used for emotion analysis.

[1145] Input: User's past behavior data, text input

[1146] Output: Recognized user emotion data

[1147] Step 7:

[1148] The server integrates the emotional data recognized by the emotion engine into the calendar data. For example, it adds emotional information such as "the user is feeling anxious about an important meeting" as metadata. The server then formats the emotional data so that it can be displayed in the calendar application.

[1149] Input: Recognized emotion data, iCalendar format data

[1150] Output: Formatted calendar data with emotion data integrated

[1151] Step 8:

[1152] The server stores the formatted data and emotion data in a database. It then notifies the user's device that the data is ready. When the user makes a request, the server sends the data for import to the device.

[1153] Input: Formatted calendar data with emotion data integrated

[1154] Output: Data stored in the database, notifications sent to users

[1155] Step 9:

[1156] When the device receives a notification from the server that the data is ready, it launches the calendar application with the user's confirmation. The calendar application sends a request to the server to obtain new data and receives the data returned from the server. The received data is automatically imported into the calendar, and the schedule and emotion data are displayed.

[1157] Input: Notification from the server that data is ready

[1158] Output: Calendar data and emotion data imported into a calendar application

[1159] Step 10:

[1160] Users can check in their calendar application whether the new events and emotion data have been imported correctly. If there are any errors in the imported content, users can manually correct or add them.

[1161] Input: Calendar data and emotion data imported into a calendar application

[1162] Output: Confirmed or modified calendar data

[1163] (Application example 2)

[1164] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1165] Conventional handwritten shift schedules and schedule management systems were difficult to digitize, making it time-consuming to share and update information across the organization. Furthermore, they were unable to reflect employees' emotional states, making it difficult to optimize the work environment. For this reason, there was a demand for a system that could easily digitize handwritten shift schedules and reflect employees' emotional states, thereby streamlining store operations and contributing to employee motivation management.

[1166] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1167] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into the system, means for recognizing an emotional state based on the formatted data, and means for integrating and storing the emotional data in a database, thereby making it possible to digitize handwritten shift schedules and integrate the emotional data.

[1168] The "means for acquiring images" refers to a means by which a user photographs a handwritten shift table or schedule and provides the image to the server as a digital image.

[1169] "Means for character recognition in acquired images" refers to a means for analyzing handwritten characters in digital images using OCR technology and converting them into text data.

[1170] "Means for formatting character-recognized data" refers to means for converting text data obtained by OCR into a format that can be understood by the system.

[1171] A "means for importing formatted data into a system" is a means for incorporating formatted data into a particular system or application.

[1172] The "means for recognizing an emotional state based on formatted data" is a means for analyzing data and past information to estimate a user's emotional state.

[1173] The "means for integrating emotional data and storing it in a database" is a means for integrating recognized emotional data with formatted data, and storing the data in an organized manner in a database.

[1174] The present invention includes the following configuration and processing means to provide a system that digitizes handwritten shift tables and schedules and further reflects the emotional state of employees.

[1175] System Configuration

[1176] The system mainly consists of a server, a terminal (e.g., a smartphone), and a user interface. Specific usage methods for each component are explained below.

[1177] Server Processing

[1178] 1. Receiving image uploads

[1179] When an image of a handwritten shift schedule is uploaded from a user's device, the server receives it. The received image data is temporarily stored in a directory. The server uses a cloud storage service (e.g., Amazon S3) to store the image data.

[1180] 2. Image Analysis

[1181] The server inputs the stored image data into a generative AI model using OCR technology, for example, by using the Google Cloud Vision API to analyze handwritten characters in the image and convert them into text data.

[1182] 3. Data Formatting

[1183] The server extracts the shift schedule's date, time, content, and color-coded information from the text data output by the generation AI and converts it into a format the system can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify specific employees.

[1184] 4. Activating the Emotional Engine

[1185] Before importing the shift schedule data, the server starts an emotion engine to recognize the emotions of employees. This emotion engine analyzes employees' past work history and comments to recognize their emotional state at that time, using, for example, IBM Watson's emotion analysis API.

[1186] 5. Emotional Data Integration

[1187] The server then incorporates the recognized emotion data into the shift schedule data. For example, if an employee is feeling stressed or tired, the emotion data is saved as metadata and used for future shift management.

[1188] 6. Data storage and transmission

[1189] The server stores the formatted data and emotion data in a database (e.g., MySQL) and sends a message to the user's device to notify them that the data is ready. You can use a notification service such as Firebase Cloud Messaging (FCM).

[1190] Terminal handling

[1191] 1. Image capture

[1192] The user takes a photo of the handwritten shift schedule using the smartphone camera, and checks the image to make sure the entire shift schedule is clearly visible.

[1193] 2. Upload an image

[1194] When the user presses the image upload button, the device uploads the captured image data to the server. The image data is sent to the server using an HTTP request.

[1195] 3. Receiving and importing data

[1196] Upon receiving the notification from the server, the device launches the system app with the user's confirmation. The app communicates with the server and sends a request to obtain new format data and emotion data. Upon receiving the data from the server, the app imports it and adds a new shift schedule to the existing system. The app uses color-coding information and emotion data to distinguish and display the status of each employee.

[1197] Specific examples

[1198] The store manager, who is the user, takes a photo of a handwritten shift schedule with their smartphone and uploads the image to the server. The server uses OCR technology to analyze the handwritten characters in the image and convert the shift information into text data. Next, it uses an emotion engine to analyze past work history and comments to recognize employee emotions. This emotion data is integrated into the shift data and saved in a database. The device receives a notification from the server and launches the shift management app to check the updated data.

[1199] Prompt Sentence Examples

[1200] "Analyze employees' work logs from the past six months to understand their emotional state, especially their stress and fatigue levels."

[1201] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1202] Step 1:

[1203] Taking and reviewing images

[1204] The user takes a photo of a handwritten shift schedule using the smartphone camera. This image is checked to see if the entire shift schedule is clearly captured. The input is the physical handwritten shift schedule, and the output is a digital image file.

[1205] Step 2:

[1206] Uploading an image

[1207] When a user presses the image upload button, the device uploads the captured image data to the server. The image data is sent using an HTTP request, with the input being a digital image file and the output being image data stored on the server.

[1208] Step 3:

[1209] Image analysis

[1210] The server analyzes the stored image data using OCR technology. Specifically, it uses an OCR engine such as Google Cloud Vision API to analyze the handwritten characters in the image and convert them into text data. The input is the uploaded image data, and the output is the analyzed text data.

[1211] Step 4:

[1212] Data Formatting

[1213] The server extracts the shift schedule date, time, content, and color-coded information from the text data output by the generative AI model and converts it into a format that the system can understand (e.g., iCalendar format). The input is the text data generated by OCR, and the output is the formatted shift schedule data.

[1214] Step 5:

[1215] Emotion engine activation and analysis

[1216] Before importing the formatted shift data, the server launches an emotion engine using IBM Watson's emotion analysis API to analyze employees' past work history and comments. The input is the employee's past data and formatted shift data, and the output is recognized emotion data.

[1217] Step 6:

[1218] Emotional Data Integration

[1219] The server integrates the recognized emotion data into the shift schedule data. Based on the analysis results, specific emotions (e.g., stress or fatigue) are added to the shift schedule data as metadata. The input is formatted shift schedule data and emotion data, and the output is shift schedule data integrated with emotion data.

[1220] Step 7:

[1221] Data Retention and Notification

[1222] The server saves the consolidated shift data in a database (e.g., MySQL) and sends notifications to the user's device via Firebase Cloud Messaging (FCM), etc. The input is the consolidated shift data and user device information, and the output is the data saved in the database and the sent notifications.

[1223] Step 8:

[1224] Receiving and displaying data

[1225] The user's device receives the notification from the server, launches the shift management app, and retrieves the new shift data. The retrieved data is displayed in the app, and the shift schedule reflecting each employee's emotional state is confirmed. The input is the integrated shift data sent from the server, and the output is the new shift schedule displayed in the shift management app on the device.

[1226] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1227] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1228] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1229] [Fourth embodiment]

[1230] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1231] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1232] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1233] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1234] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1235] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1236] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1237] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1238] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1239] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1240] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1241] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1242] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1243] The present invention provides a system for digitizing handwritten calendars and managing them in a unified manner using a calendar application. Specific embodiments of the system are described below.

[1244] Server Processing

[1245] 1. Receiving image uploads

[1246] When a handwritten calendar image is uploaded from a user's terminal, the server receives it and stores the received image data in a temporary directory.

[1247] 2. Image Analysis

[1248] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[1249] 3. Data Formatting

[1250] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI. Using scripts or application logic, the extracted information is converted into a format that the calendar app can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify who owns a particular event.

[1251] 4. Data storage and transmission

[1252] The server stores the formatted data in a database and sends a message to the user's device to notify them that the data is ready. When the user makes a request to retrieve new calendar data, the server sends the data to the device.

[1253] Terminal handling

[1254] 1. Image capture

[1255] The user takes a photo of the handwritten calendar using the smartphone camera, and checks the image to make sure the entire calendar is clearly visible.

[1256] 2. Upload an image

[1257] When the user presses the image upload button, the device uploads the captured image data to the server. The device sends the image data to the server using an HTTP request.

[1258] 3. Receiving and importing data

[1259] The device that receives the notification from the server launches the calendar app with the user's confirmation. The calendar app communicates with the server and sends a request to obtain new formatted data. Upon receiving the formatted calendar data from the server, the calendar app imports it and adds new events to the existing calendar. The calendar app uses color-coding information to distinguish and display each event for each individual.

[1260] User operations

[1261] 1. Calendar photoshoot

[1262] The user takes a photo of a handwritten calendar using their smartphone, and the events are color-coded to indicate who owns them.

[1263] 2. Upload confirmation

[1264] When uploading a captured image to a server, the user performs a confirmation operation.

[1265] 3. Data confirmation

[1266] The user can check their calendar app to ensure the new events have been imported correctly, and if necessary, they can make additional edits or adjustments.

[1267] Specific examples

[1268] Example 1: Image upload and analysis

[1269] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server then uses generative AI to analyze the text in the image and convert each person's schedule into digital text. The text is then formatted and saved in a calendar app format, and the user's device is notified.

[1270] Example 2: Data import

[1271] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new event data from the server and automatically imports it into their calendar. The user can then check each person's events, which are correctly reflected in the calendar, and edit them if necessary.

[1272] The system of the present invention allows for centralized digital management while maintaining the convenience of a handwritten calendar, making it easier to share and coordinate schedules and allowing all family members to quickly understand the latest information.

[1273] The processing flow will be explained below.

[1274] Step 1:

[1275] The user uses the smartphone camera to take a picture of a handwritten calendar hanging on the wall, and checks the image to make sure the entire calendar is clearly visible.

[1276] Step 2:

[1277] The device prepares to upload the captured image data to the server through the application interface. When the user presses the upload button, the device sends the image data to the server via an HTTP request.

[1278] Step 3:

[1279] The server receives the image upload request sent from the user's terminal, saves the image data in a temporary directory, and sends a response confirming receipt to the terminal.

[1280] Step 4:

[1281] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[1282] Step 5:

[1283] The server extracts the event date, time, content, and color-coded information from the parsed text data, using scripts or application logic to convert the event data into a format that the calendar app can understand (e.g., iCalendar format).

[1284] Step 6:

[1285] The server stores the formatted data in a database, a process that ensures that the scheduled data is easily accessible at a later time and is accurately maintained.

[1286] Step 7:

[1287] The server sends a message to the user's device to notify them that the formatted data is ready, possibly via push notification or email.

[1288] Step 8:

[1289] The device receives a notification from the server informing the user that new calendar data is available. The user confirms the notification and launches the calendar app.

[1290] Step 9:

[1291] The device's calendar app sends a request to the server to get the new format data, and the server sends the saved format data to the device.

[1292] Step 10:

[1293] The calendar application on the device imports the received format data and adds new events to the existing calendar. The calendar application uses color-coding information to distinguish and display each event for each individual.

[1294] Step 11:

[1295] The user can verify that the new event has been imported correctly within their calendar app, and can make additional edits or adjustments if needed.

[1296] Through these processing steps, the handwritten calendar information is digitized and can be shared with the whole family via a smartphone calendar app.

[1297] Example 1

[1298] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1299] Systems that digitize and centrally manage handwritten calendars face the problem of cumbersome and time-consuming manual data entry. It is also difficult to manage individual appointments by color-coding them, making sharing and coordination difficult. Furthermore, there is a need to improve the accuracy of handwritten character recognition and accurately convert them into digital data.

[1300] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1301] In this invention, the server includes a means for receiving and temporarily storing image data from the user's device, a means for inputting the image data into a generative AI model and analyzing handwritten characters, and a means for converting the generated character recognition data into a format that can be understood by a calendar application. This allows handwritten calendars to be digitized and centrally managed, and schedules to be color-coded and shared.

[1302] "Means for acquiring images" refers to devices or software that allow a user to take a photo of a handwritten calendar with a smartphone or camera and acquire the image data.

[1303] The "means for character recognition of acquired images" refers to software or algorithms for analyzing handwritten characters from acquired image data and extracting them as digital text data.

[1304] A "means for formatting character-recognized data" is software or script that converts text data obtained by character recognition into a specific data format that can be recognized by a calendar application (e.g., iCalendar format).

[1305] "Means for saving formatted data in a database and transmitting it to a user's terminal" refers to communication means or software that saves the converted data in a database and transmits it to a user's terminal as needed.

[1306] The "means for receiving image data from the user's terminal and temporarily storing it" refers to server-side storage or software that receives image data sent from the user's terminal and temporarily stores it.

[1307] The "means of inputting image data into a generative AI model and analyzing handwritten characters" refers to a method of supplying received and stored image data to a generative AI model and converting handwritten characters into text data.

[1308] "Means for converting the generated character recognition data into a format understandable by a calendar application" refers to software or scripts that convert the text data output by the generative AI model into a format that can be imported and understood by a calendar application (e.g., iCalendar format).

[1309] The "means for sending a notification to the user's terminal that the data is ready" refers to a communication function or software for notifying the user's terminal that the formatted data is ready.

[1310] The present invention provides a system for digitizing handwritten calendars and managing them in a unified manner using a calendar application. Specific embodiments of the system are described below.

[1311] System Overview

[1312] The system starts when a user takes a photo of a handwritten calendar with their smartphone and uploads it to a server. The server analyzes the uploaded image using a generative AI model and converts it into digital data. The converted data is then formatted into a format that the calendar app can understand and saved in a database. Finally, the formatted data is sent to the user's device and imported into the calendar app.

[1313] Hardware and Software

[1314] Smartphone: Users take a photo of the handwritten calendar using the camera on their smartphone. iPhones, Android devices, etc. can be used.

[1315] Server: A server is used to receive, analyze, format, store, and transmit image data. Specifically, a web framework such as Flask or Django is used.

[1316] Generative AI models: used to analyze handwritten characters in images, for example, using pre-trained models using PyTorch or TensorFlow.

[1317] OCR technology: Google Cloud Vision API and Tesseract are used to analyze handwritten characters in images.

[1318] Database: Use a database such as MySQL to store the analyzed data.

[1319] Calendar app: Use an application such as Google Calendar to import digital data.

[1320] Processing Flow

[1321] A user uses the system in the following steps:

[1322] 1. Image capture

[1323] A user takes a photo of a handwritten calendar using the camera on their smartphone. When taking the photo, they color-code the events so that it is clear who owns each one. For example, they could take a photo of a handwritten family calendar and mark each event with a different color.

[1324] 2. Upload an image

[1325] When a user presses the image upload button in the app, the smartphone uploads the captured image data to the server. This is done by sending the data to the server using an HTTP POST request. For example, the image data is POSTed to the / upload_image endpoint of the Flask server.

[1326] 3. Image Analysis

[1327] The server reads the saved image data and passes it to the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data. For example, it can call the Google Cloud Vision API to perform character recognition.

[1328] 4. Data Formatting

[1329] The server extracts date, time, content, and color-coded information from the text data output by the generation AI. Using a script (e.g., a Python script), it converts the extracted information into a format understandable by a calendar app, such as iCalendar format. It adds the color-coded information as metadata. For example, data such as "{"date": "2023-10-01", "events": ["Meeting", "Meeting"]}" is converted to the format "BEGIN:VEVENT\nDTSTART:20231001T090000Z\nSUMMARY:Meeting\nEND:VEVENT\n".

[1330] 5. Data storage and transmission

[1331] The server saves the formatted data in a database (e.g., MySQL) and notifies the user's device that the data is ready. Specifically, this is done using WebSocket or push notification. For example, the server inserts iCalendar format data into the calendar_events table in a MySQL database and sends a message to the user's device via WebSocket saying "Calendar data has been updated."

[1332] 6. Receiving and Importing Data

[1333] The device that receives the notification from the server displays the notification to the user and launches the calendar app. The calendar app communicates with the server to send a request for new data and receives the formatted data. The received data is then imported into the calendar app, where events are displayed in color.

[1334] Specific examples

[1335] Example 1: Image upload and analysis

[1336] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server then uses a generative AI model to analyze the text in the image and convert each person's schedule into digital text. The text is then formatted and saved in a calendar app format, and the user's device is notified.

[1337] Example 2: Data import

[1338] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new event data from the server and automatically imports it into their calendar. The user can then check each person's events, which are correctly reflected in the calendar, and edit them if necessary.

[1339] Prompt Sentence Examples

[1340] Please explain the system that takes a photo of a family's handwritten calendar with a smartphone, digitizes it on the cloud, and imports it into a calendar app. For example, what steps are taken to process the data, and how does the user operate it?

[1341] This invention digitizes handwritten calendars and centralizes their management, making it easier to share and coordinate schedules, and allowing all family members to quickly grasp the latest information.

[1342] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1343] Step 1:

[1344] Image capture

[1345] A user launches the camera app on their smartphone and takes a photo of a handwritten calendar. When taking the photo, they color-code the events to identify who owns them. For example, they can take a photo of a family calendar and mark each event with a different color.

[1346] Input: Handwritten calendar

[1347] Output: Captured image data (JPEG or PNG format)

[1348] Step 2:

[1349] Image upload

[1350] When a user presses the image upload button in the app, the device sends the captured image data to the server using an HTTP POST request. For example, the image data is POSTed to http: / / example.com / upload_image.

[1351] Input: Captured image data

[1352] Output: HTTP POST request to the server

[1353] Step 3:

[1354] Receiving and saving images on the server

[1355] The server receives the image data sent from the user's device and stores it in a temporary directory. For example, it processes the request using a web framework such as Flask or Django and stores the image file as / tmp / uploaded_images / image1.jpg.

[1356] Input: HTTP POST request (image data)

[1357] Output: Temporarily saved image file

[1358] Step 4:

[1359] Image analysis

[1360] The server reads the temporarily saved image data and inputs it into the generative AI model. The generative AI model analyzes the handwritten characters in the image using OCR technology (e.g., Google Cloud Vision API or Tesseract) and converts them into text data. For example, input / tmp / uploaded_images / image1.jpg into the Google Cloud Vision API and obtain JSON-formatted text data as the character recognition results.

[1361] Input: Saved image file

[1362] Output: Parsed text data (JSON format)

[1363] Step 5:

[1364] Data Formatting

[1365] The server extracts date, time, content, and color-coding information from the text data output by the generative AI model. It then uses a script (e.g., a Python script) to convert the extracted information into a format that calendar apps can understand, such as iCalendar, and adds the color-coding information as metadata.

[1366] Input: Parsed text data (JSON format)

[1367] Output: Formatted calendar data (iCalendar format)

[1368] Step 6:

[1369] Data Retention and Notification

[1370] The server saves the formatted data in a database (e.g., MySQL) and sends a notification to the user's device that the data is ready. For example, it inserts the iCalendar format data into the calendar_events table in the MySQL database and sends a message to the user's device via WebSocket saying "Calendar data has been updated."

[1371] Input: Formatted calendar data (iCalendar format)

[1372] Output: Calendar data stored in the database, and notification messages

[1373] Step 7:

[1374] Receiving and Importing Data

[1375] When the device receives the notification from the server, it displays the notification to the user and launches the calendar app. The calendar app then communicates with the server and sends a request to retrieve new data. The formatted calendar data is then received and automatically imported into the calendar. For example, the calendar app may use a REST API to retrieve iCalendar formatted data, import it into the calendar, and display it.

[1376] Input: Notification message, formatted calendar data

[1377] Output: New events imported into the Calendar app

[1378] This process converts handwritten calendars into digital data and efficiently manages them centrally. Users can easily digitize their handwritten calendars and manage them in a calendar app.

[1379] (Application example 1)

[1380] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1381] Existing factory work schedule management methods often require a lot of time and effort in the process of digitizing handwritten schedules, making it difficult to streamline the process. Furthermore, it is difficult to integrate and manage the schedules of multiple individuals, who need to be color-coded or otherwise differentiated, making centralized schedule management difficult. The objective of this invention is to efficiently and accurately digitize handwritten work schedules and realize centralized schedule management.

[1382] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1383] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into a calendar application, and means for importing the formatted data into a work schedule system. This makes it possible to efficiently digitize handwritten work schedules and centrally manage them, including each individual's schedule.

[1384] The "means for acquiring images" has a function that allows a user to take a photograph of a handwritten calendar or work schedule and send it to the server in digital image format.

[1385] "Means for character recognition" refers to a technology that extracts character information from an acquired image and converts it into digital text format, and uses OCR technology.

[1386] The "formatting means" has a function for converting the character-recognized data into a specific format, such as the iCalendar format.

[1387] The "means for importing into a calendar application" has a function for importing formatted data into a calendar application and displaying it as a schedule item.

[1388] The "means for importing into the work schedule system" refers to a function that imports formatted data into the factory's work schedule system and registers and manages it as schedule information.

[1389] The "means for analyzing and identifying the schedule of each individual" has a function for distinguishing schedules of different individuals by taking into account color-coding information in the acquired image.

[1390] The "means for storing and transmitting to the user's terminal" has the function of storing formatted data in a database and transmitting the data to the user's terminal as required.

[1391] The present invention relates to a system for digitizing handwritten calendars and work schedules and managing them in a unified manner. Specific embodiments for carrying out the present invention will be described below.

[1392] Server Action:

[1393] When a handwritten calendar image is uploaded from the user's device, the server first stores it in a temporary directory. The server then analyzes the stored image data using OCR technology. Specifically, it uses software such as Tesseract OCR or Google Cloud Vision API. The analyzed data is then formatted into a specific format (e.g., iCalendar format) and stored in a database. The server then imports the formatted data into the work schedule system and sends it to the user's device as needed.

[1394] Terminal handling:

[1395] The device first takes a photo of the handwritten calendar through user operation and then sends the image to the server by pressing the upload button. At this time, the image data is sent using an HTTP request. When the device receives a notification from the server that the new schedule data is ready, it retrieves it and imports it into the calendar app. The device also has the function to import the data received from the server into the work schedule system.

[1396] User Action:

[1397] The user takes a photo of a handwritten calendar using their smartphone. When taking the photo, the appointments are color-coded to indicate who owns them. The image is uploaded to the server, and upon receiving a notification from the server, the user checks the calendar app to confirm that the new schedule has been imported correctly. The user also verifies that the new schedule has been integrated into the factory's work scheduling system.

[1398] Examples:

[1399] At the beginning of the month, a user takes a photo of a manually written factory work schedule and uploads the image to the server. The server uses OCR technology to analyze the text in the image and converts the schedules for each worker and machine into digital text. The server then formats the text into iCalendar format or a format that can be understood by the factory's scheduling system. The formatted data is stored in a database, and an import notification is sent to the user's device. The user receives the notification, launches their calendar app, and checks the new schedule data. They also verify that the new schedule has been correctly integrated into the factory's work scheduling system.

[1400] Example prompt sentence:

[1401] Write code for an application that parses handwritten calendar images to digitize schedule events for import into a factory scheduling system.

[1402] In this way, the invention is a system that efficiently digitizes handwritten calendars and schedules and centralizes factory schedule management, thereby improving work efficiency and reducing errors.

[1403] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1404] Step 1:

[1405] The user takes a photo of a handwritten calendar with the smartphone camera and generates an image file. The input is the handwritten calendar, and the output is a digital image file. The user checks that the image is clearly captured.

[1406] Step 2:

[1407] A user uses an application on their device to upload images taken with a camera to a server. The input is a digital image file, and the output is an HTTP request to send that file to the server. The transmission begins when the user presses the upload button.

[1408] Step 3:

[1409] The server receives the uploaded image and stores it in a temporary storage directory. The input is the image data sent via the HTTP request, and the output is the image file in the server's temporary storage directory. This step is where the file is saved.

[1410] Step 4:

[1411] The server analyzes the stored image data using OCR technology. The input is the image file stored on the server, and the output is the extracted text data. Specifically, the server performs image analysis using Google Cloud Vision API and Tesseract OCR.

[1412] Step 5:

[1413] The server formats the text data analyzed by OCR technology into a format that can be understood by a calendar application or work schedule system. The input is the text data obtained by OCR analysis, and the output is formatted data (e.g., iCalendar format). The server uses scripts or application logic to format the data.

[1414] Step 6:

[1415] The server saves the formatted data in a database and notifies the user's terminal that the data is ready. The input is the formatted schedule data, and the output is the record saved in the database and a notification message to the terminal.

[1416] Step 7:

[1417] The device receives the notification from the server, and after obtaining the user's confirmation, launches the calendar app and instructs it to retrieve new schedule data. The input is the notification message from the server, and the output is a request to retrieve new schedule data.

[1418] Step 8:

[1419] The calendar app communicates with the server, gets the formatted schedule data, and imports it into the calendar. The input is the formatted data sent from the server, and the output is the schedule events imported into the calendar app. Through this series of processes, the new schedule is correctly reflected in the calendar.

[1420] Step 9:

[1421] The user checks the calendar app and verifies that the new schedule has been imported correctly. Similarly, the user verifies that the new schedule has been integrated into the factory's work scheduling system. The input is the imported schedule data, and the output is the user's confirmation operation.

[1422] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1423] The present invention combines a system that digitizes handwritten calendars and centrally manages them using a calendar application with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention will be described below.

[1424] Server Processing

[1425] 1. Receiving image uploads

[1426] When a handwritten calendar image is uploaded from a user's terminal, the server receives it and stores the received image data in a temporary directory.

[1427] 2. Image Analysis

[1428] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[1429] 3. Data Formatting

[1430] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI. It uses scripts and application logic to convert the event data into a format that the calendar app can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify who owns a specific event.

[1431] 4. Activating the Emotional Engine

[1432] Before importing data into the calendar app, the server activates an emotion engine to recognize the user's emotions. This emotion engine analyzes the user's past behavioral data and text input to recognize the user's emotional state at that time.

[1433] 5. Emotional Data Integration

[1434] The server then incorporates the recognized emotion data into the calendar data. For example, if the user feels nervous or anxious about an important appointment, the server stores this information as metadata. The server also arranges for the emotion data to be displayed in the calendar app.

[1435] 6. Data storage and transmission

[1436] The server stores the formatted data and emotion data in a database and sends a message to the user's device to notify them that the data is ready. When the user makes a request to obtain new calendar data, the server sends the data to the device.

[1437] Terminal handling

[1438] 1. Image capture

[1439] The user takes a photo of the handwritten calendar using the smartphone camera, and checks the image to make sure the entire calendar is clearly visible.

[1440] 2. Upload an image

[1441] When the user presses the image upload button, the device uploads the captured image data to the server. The device sends the image data to the server using an HTTP request.

[1442] 3. Receiving and importing data

[1443] The device that receives the notification from the server launches the calendar app with the user's confirmation. The calendar app communicates with the server and sends a request to obtain new format data and emotion data. Upon receiving the data from the server, the calendar app imports it and adds a new event to the existing calendar. The calendar app uses color-coding information and emotion data to distinguish and display each event for each individual.

[1444] User operations

[1445] 1. Calendar photoshoot

[1446] The user takes a photo of a handwritten calendar using their smartphone, and the events are color-coded to indicate who owns them.

[1447] 2. Upload confirmation

[1448] When uploading a captured image to a server, the user performs a confirmation operation.

[1449] 3. Data confirmation

[1450] The user can check their calendar app to ensure the new appointment and emotion data has been imported correctly. If necessary, the user can make additional edits or adjustments.

[1451] Specific examples

[1452] Example 1: Image upload and analysis

[1453] At the beginning of each month, users take a photo of their family's handwritten calendar and upload it to the server. The server uses generative AI to analyze the text in the image and convert each person's schedule into digital text. It then uses an emotion engine to analyze the user's emotional state and integrates the emotional data into the calendar data. The data is then formatted and saved in a calendar app format, and notified to the user's device.

[1454] Example 2: Data import

[1455] When a user receives a notification from the server, they launch the calendar app on their smartphone to check for updates. The calendar app retrieves the new events and emotion data from the server and automatically imports them into their calendar. The user can then check the events and emotions of each person correctly reflected in the calendar and edit them if necessary.

[1456] The system of the present invention not only digitizes handwritten calendar information and allows it to be shared with the whole family via a smartphone calendar app, but also reflects the user's emotional state, making it easier to share and coordinate schedules and allowing all family members to quickly grasp the latest information and emotional state.

[1457] The processing flow will be explained below.

[1458] Step 1:

[1459] The user takes a photo of the handwritten calendar using the smartphone camera, checks that the entire calendar is clearly visible, and prepares to upload it to the server via the app.

[1460] Step 2:

[1461] The user presses the upload button on the app to send the captured image data to the server. The device then sends the image data to the server using an HTTP request.

[1462] Step 3:

[1463] The server receives the image upload request sent from the user's terminal, saves the image data in a temporary storage directory, and sends a response confirming receipt to the terminal.

[1464] Step 4:

[1465] The server inputs the stored image data into the generative AI model, which then uses OCR technology to analyze the handwritten characters in the image and convert them into text data.

[1466] Step 5:

[1467] The server extracts the event date, time, content, and color-coded information from the text data output by the generation AI, and uses scripts and logic to convert the extracted information into a format that the calendar app can understand (e.g., iCalendar format).

[1468] Step 6:

[1469] The server starts an emotion engine and analyzes the user's past behavioral data and schedule information. The emotion engine recognizes the user's emotional state and acquires that data.

[1470] Step 7:

[1471] The server integrates the user's emotional data into the schedule data. For example, if the user feels nervous or anxious about an important appointment, the server stores this information as metadata and stores it together with the calendar information.

[1472] Step 8:

[1473] The server stores the formatted data and emotion data in a database. It also sends a message to the user's device to notify them that the data is ready. Possible notification methods include push notifications and emails.

[1474] Step 9:

[1475] The device receives a notification from the server and notifies the user that new calendar data and emotion data are available. The user confirms the notification and launches the calendar app.

[1476] Step 10:

[1477] The device's calendar app sends a request to the server to get new formatting and emotion data, which the server then sends to the device.

[1478] Step 11:

[1479] The calendar application of the device imports the received format data and emotion data and adds a new event to the existing calendar. The calendar application uses color-coding information and emotion data to distinguish and display each event for each individual.

[1480] Step 12:

[1481] Users can then review their new events and emotion data in their calendar app to ensure it has been imported correctly. If necessary, users can make additional edits or adjustments.

[1482] Through these processing steps, the handwritten calendar information is digitized and can be shared with the whole family via a smartphone calendar app, and the user's emotional state is also reflected, making it easier to share and adjust schedules, and allowing all family members to quickly grasp the latest information and emotional state.

[1483] Example 2

[1484] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1485] Current digital calendar management systems lack the ability to digitize handwritten calendar information, making it difficult to manage schedules that take the user's emotional state into account. They also lack the ability to accurately identify handwritten information and reflect specific color coding or emotional data in the calendar. As a result, sharing and adjusting schedules becomes cumbersome and stressful for users.

[1486] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1487] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into a calendar application, means for activating an emotion analysis engine for recognizing the user's emotions, and means for integrating the emotion data into calendar data, thereby enabling accurate digitization of handwritten calendar information and schedule management that reflects the user's emotional state.

[1488] "Means for acquiring images" refers to equipment or software that allows a user to photograph or scan a handwritten calendar image and provide it to a server as digital data.

[1489] "Means for character recognition of acquired images" refers to the function of using OCR technology to extract character information from acquired handwritten calendar images and convert it into text data.

[1490] "Means for formatting the recognized data" refers to the script or application logic used to convert the text data obtained using OCR technology into a format that can be understood by a calendar application (e.g., iCalendar format).

[1491] "Means for importing formatted data into a calendar application" refers to functionality that allows formatted data to be read into a calendar application so that it can be displayed as an appointment on a digital calendar.

[1492] "Means for launching an emotion analysis engine to recognize a user's emotions" refers to a function for analyzing a user's past behavioral data and text inputs to run an engine to recognize the user's emotional state.

[1493] "Means for integrating emotional data into calendar data" refers to a function that integrates the recognized emotional data of a user with existing calendar data and reflects it in a calendar application as emotional information.

[1494] "Color coding information" refers to color information used to distinguish different events or people in a handwritten calendar.

[1495] "Calendar application" refers to software that allows a user to digitally manage their schedules.

[1496] The present invention combines a system that digitizes handwritten calendars and centrally manages them using a calendar application with an emotion analysis engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[1497] Server Processing

[1498] 1. Receiving image uploads

[1499] The server receives handwritten calendar image data from the user's device and stores it in a temporary directory. At this time, it assigns a unique ID to the file name for management purposes.

[1500] 2. Image Analysis

[1501] The server retrieves the calendar image from the temporary storage directory and inputs it into the generative AI model. The prompt text used is, "Please analyze the handwritten characters in this image and convert them into text data." The generative AI model then uses OCR technology to convert the handwritten characters in the image into text format and returns the results to the server.

[1502] 3. Data Formatting

[1503] The server parses the text data returned by the generative AI model to extract the date, time, content, and color-coding information for each event. This process is performed using Python scripts or custom application logic. The extracted data is then converted into a format that calendar applications can understand, such as iCalendar, and the color-coding information is added as metadata.

[1504] 4. Activating the Emotional Engine

[1505] The server starts the emotion engine before importing the data. The emotion engine analyzes the user's past behavioral data and text input to recognize their current emotional state. This process uses emotion analysis algorithms and natural language processing techniques.

[1506] 5. Emotional Data Integration

[1507] The server integrates the emotional data recognized by the emotion engine into the calendar data, saving the emotional information (e.g., "The user is feeling anxious about a meeting where traffic congestion is expected") as metadata, and formats the information so that it can be displayed in the calendar application.

[1508] 6. Data storage and transmission

[1509] The server stores the formatted data and emotion data in a database. It then notifies the user's device that the data is ready. This notification can be done via push notification or email. When the user makes a request, the server sends the data for import to the device.

[1510] Terminal handling

[1511] 1. Image capture

[1512] The user takes a photo of the handwritten calendar using the smartphone's camera. The user checks that the image is clear and shows the entire calendar. The user can also color-code events as needed.

[1513] 2. Upload an image

[1514] When a user presses the image upload button, the device sends the captured image to the server using an HTTP request (such as the POST method). This request includes the image file as well as metadata such as user information and the date and time the image was taken.

[1515] 3. Receiving and importing data

[1516] When the device receives a notification from the server that the data is ready, it launches the calendar application with the user's confirmation. The calendar application sends a request to the server to obtain new data and receives the data returned from the server. The received data is automatically imported into the calendar, and the schedule and emotion data are displayed.

[1517] User operations

[1518] 1. Calendar photoshoot

[1519] The user takes a photo of the handwritten calendar using the smartphone camera. When taking the photo, each event is color-coded to distinguish whose event it is.

[1520] 2. Upload confirmation

[1521] When uploading a captured image to a server, the user displays a confirmation screen to confirm the contents of the image to be sent. This confirmation operation uses check boxes and confirmation buttons.

[1522] 3. Data confirmation

[1523] Users can check in their calendar application whether the new events and emotion data have been imported correctly. If there are any errors in the imported content, users can manually correct or add them.

[1524] As described above, the system of the present invention digitizes handwritten calendar information and further integrates the user's emotional data, enabling more comprehensive schedule management.

[1525] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1526] Step 1:

[1527] The user takes a photo of the handwritten calendar using the smartphone camera. At this time, the events are color-coded to indicate who owns them. The captured image data is saved in the device.

[1528] Input: Handwritten calendar

[1529] Output: Captured image data

[1530] Step 2:

[1531] When a user presses the image upload button, the device sends the saved image data to the server as an HTTP request (POST method). The request includes the image file, user information, and the date and time the image was taken.

[1532] Input: Captured image data, user information, date and time of capture

[1533] Output: HTTP request sent to the server

[1534] Step 3:

[1535] The server stores the handwritten calendar image data received from the user's device in a temporary directory, and manages it by assigning a unique ID to the file name.

[1536] Input: Image data sent as an HTTP request

[1537] Output: Image data saved on the server (in the temporary directory)

[1538] Step 4:

[1539] The server retrieves the calendar image from the temporary storage directory and inputs it into the generative AI model. The prompt text used is, "Please analyze the handwritten characters in this image and convert them into text data." The generative AI model then uses OCR technology to convert the handwritten characters in the image into text format.

[1540] Input: Calendar image data retrieved from temporary directory

[1541] Output: Text data analyzed from handwritten characters

[1542] Step 5:

[1543] The server parses the text data returned by the generative AI model to extract the date, time, content, and color-coding information for each event. This analysis is done using Python scripts or custom application logic. The extracted data is then converted into a format that calendar applications can understand, such as iCalendar, and the color-coding information is added as metadata.

[1544] Input: Text data returned by the generative AI model

[1545] Output: Formatted data including iCalendar data and color coding information

[1546] Step 6:

[1547] The server starts the emotion engine before importing the data. The emotion engine analyzes the user's past behavioral data and text input to recognize their current emotional state. Natural language processing and emotion recognition algorithms are used for emotion analysis.

[1548] Input: User's past behavior data, text input

[1549] Output: Recognized user emotion data

[1550] Step 7:

[1551] The server integrates the emotional data recognized by the emotion engine into the calendar data. For example, it adds emotional information such as "the user is feeling anxious about an important meeting" as metadata. The server then formats the emotional data so that it can be displayed in the calendar application.

[1552] Input: Recognized emotion data, iCalendar format data

[1553] Output: Formatted calendar data with emotion data integrated

[1554] Step 8:

[1555] The server stores the formatted data and emotion data in a database. It then notifies the user's device that the data is ready. When the user makes a request, the server sends the data for import to the device.

[1556] Input: Formatted calendar data with emotion data integrated

[1557] Output: Data stored in the database, notifications sent to users

[1558] Step 9:

[1559] When the device receives a notification from the server that the data is ready, it launches the calendar application with the user's confirmation. The calendar application sends a request to the server to obtain new data and receives the data returned from the server. The received data is automatically imported into the calendar, and the schedule and emotion data are displayed.

[1560] Input: Notification from the server that data is ready

[1561] Output: Calendar data and emotion data imported into a calendar application

[1562] Step 10:

[1563] Users can check in their calendar application whether the new events and emotion data have been imported correctly. If there are any errors in the imported content, users can manually correct or add them.

[1564] Input: Calendar data and emotion data imported into a calendar application

[1565] Output: Confirmed or modified calendar data

[1566] (Application example 2)

[1567] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1568] Conventional handwritten shift schedules and schedule management systems were difficult to digitize, making it time-consuming to share and update information across the organization. Furthermore, they were unable to reflect employees' emotional states, making it difficult to optimize the work environment. For this reason, there was a demand for a system that could easily digitize handwritten shift schedules and reflect employees' emotional states, thereby streamlining store operations and contributing to employee motivation management.

[1569] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1570] In this invention, the server includes means for acquiring images, means for character recognition of the acquired images, means for formatting the character-recognized data, means for importing the formatted data into the system, means for recognizing an emotional state based on the formatted data, and means for integrating and storing the emotional data in a database, thereby making it possible to digitize handwritten shift schedules and integrate the emotional data.

[1571] The "means for acquiring images" refers to a means by which a user photographs a handwritten shift table or schedule and provides the image to the server as a digital image.

[1572] "Means for character recognition in acquired images" refers to a means for analyzing handwritten characters in digital images using OCR technology and converting them into text data.

[1573] "Means for formatting character-recognized data" refers to means for converting text data obtained by OCR into a format that can be understood by the system.

[1574] A "means for importing formatted data into a system" is a means for incorporating formatted data into a particular system or application.

[1575] The "means for recognizing an emotional state based on formatted data" is a means for analyzing data and past information to estimate a user's emotional state.

[1576] The "means for integrating emotional data and storing it in a database" is a means for integrating recognized emotional data with formatted data, and storing the data in an organized manner in a database.

[1577] The present invention includes the following configuration and processing means to provide a system that digitizes handwritten shift tables and schedules and further reflects the emotional state of employees.

[1578] System Configuration

[1579] The system mainly consists of a server, a terminal (e.g., a smartphone), and a user interface. Specific usage methods for each component are explained below.

[1580] Server Processing

[1581] 1. Receiving image uploads

[1582] When an image of a handwritten shift schedule is uploaded from a user's device, the server receives it. The received image data is temporarily stored in a directory. The server uses a cloud storage service (e.g., Amazon S3) to store the image data.

[1583] 2. Image Analysis

[1584] The server inputs the stored image data into a generative AI model using OCR technology, for example, by using the Google Cloud Vision API to analyze handwritten characters in the image and convert them into text data.

[1585] 3. Data Formatting

[1586] The server extracts the shift schedule's date, time, content, and color-coded information from the text data output by the generation AI and converts it into a format the system can understand (e.g., iCalendar format). The color-coded information is added as metadata to identify specific employees.

[1587] 4. Activating the Emotional Engine

[1588] Before importing the shift schedule data, the server starts an emotion engine to recognize the emotions of employees. This emotion engine analyzes employees' past work history and comments to recognize their emotional state at that time, using, for example, IBM Watson's emotion analysis API.

[1589] 5. Emotional Data Integration

[1590] The server then incorporates the recognized emotion data into the shift schedule data. For example, if an employee is feeling stressed or tired, the emotion data is saved as metadata and used for future shift management.

[1591] 6. Data storage and transmission

[1592] The server stores the formatted data and emotion data in a database (e.g., MySQL) and sends a message to the user's device to notify them that the data is ready. You can use a notification service such as Firebase Cloud Messaging (FCM).

[1593] Terminal handling

[1594] 1. Image capture

[1595] The user takes a photo of the handwritten shift schedule using the smartphone camera, and checks the image to make sure the entire shift schedule is clearly visible.

[1596] 2. Upload an image

[1597] When the user presses the image upload button, the device uploads the captured image data to the server. The image data is sent to the server using an HTTP request.

[1598] 3. Receiving and importing data

[1599] Upon receiving the notification from the server, the device launches the system app with the user's confirmation. The app communicates with the server and sends a request to obtain new format data and emotion data. Upon receiving the data from the server, the app imports it and adds a new shift schedule to the existing system. The app uses color-coding information and emotion data to distinguish and display the status of each employee.

[1600] Specific examples

[1601] The store manager, who is the user, takes a photo of a handwritten shift schedule with their smartphone and uploads the image to the server. The server uses OCR technology to analyze the handwritten characters in the image and convert the shift information into text data. Next, it uses an emotion engine to analyze past work history and comments to recognize employee emotions. This emotion data is integrated into the shift data and saved in a database. The device receives a notification from the server and launches the shift management app to check the updated data.

[1602] Prompt Sentence Examples

[1603] "Analyze employees' work logs from the past six months to understand their emotional state, especially their stress and fatigue levels."

[1604] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1605] Step 1:

[1606] Taking and reviewing images

[1607] The user takes a photo of a handwritten shift schedule using the smartphone camera. This image is checked to see if the entire shift schedule is clearly captured. The input is the physical handwritten shift schedule, and the output is a digital image file.

[1608] Step 2:

[1609] Uploading an image

[1610] When a user presses the image upload button, the device uploads the captured image data to the server. The image data is sent using an HTTP request, with the input being a digital image file and the output being image data stored on the server.

[1611] Step 3:

[1612] Image analysis

[1613] The server analyzes the stored image data using OCR technology. Specifically, it uses an OCR engine such as Google Cloud Vision API to analyze the handwritten characters in the image and convert them into text data. The input is the uploaded image data, and the output is the analyzed text data.

[1614] Step 4:

[1615] Data Formatting

[1616] The server extracts the shift schedule date, time, content, and color-coded information from the text data output by the generative AI model and converts it into a format that the system can understand (e.g., iCalendar format). The input is the text data generated by OCR, and the output is the formatted shift schedule data.

[1617] Step 5:

[1618] Emotion engine activation and analysis

[1619] Before importing the formatted shift data, the server launches an emotion engine using IBM Watson's emotion analysis API to analyze employees' past work history and comments. The input is the employee's past data and formatted shift data, and the output is recognized emotion data.

[1620] Step 6:

[1621] Emotional Data Integration

[1622] The server integrates the recognized emotion data into the shift schedule data. Based on the analysis results, specific emotions (e.g., stress or fatigue) are added to the shift schedule data as metadata. The input is formatted shift schedule data and emotion data, and the output is shift schedule data integrated with emotion data.

[1623] Step 7:

[1624] Data Retention and Notification

[1625] The server saves the consolidated shift data in a database (e.g., MySQL) and sends notifications to the user's device via Firebase Cloud Messaging (FCM), etc. The input is the consolidated shift data and user device information, and the output is the data saved in the database and the sent notifications.

[1626] Step 8:

[1627] Receiving and displaying data

[1628] The user's device receives the notification from the server, launches the shift management app, and retrieves the new shift data. The retrieved data is displayed in the app, and the shift schedule reflecting each employee's emotional state is confirmed. The input is the integrated shift data sent from the server, and the output is the new shift schedule displayed in the shift management app on the device.

[1629] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1630] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1631] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1632] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1633] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1634] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1635] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1636] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1637] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1638] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1639] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1640] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1641] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1642] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1643] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1644] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1645] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1646] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1647] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1648] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1649] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1650] The following is further disclosed regarding the above embodiment.

[1651] (Claim 1)

[1652] a means for acquiring an image;

[1653] A means for character recognition of the acquired image;

[1654] A means for formatting the character recognized data;

[1655] A way to import the formatted data into a calendar app;

[1656] A system including:

[1657] (Claim 2)

[1658] Further comprising a means for analyzing the acquired image including color-coded information and identifying each individual's schedule.

[1659] 10. The system of claim 1.

[1660] (Claim 3)

[1661] further comprising means for storing and transmitting the formatted data to a user terminal;

[1662] 10. The system of claim 1.

[1663] "Example 1"

[1664] (Claim 1)

[1665] a means for acquiring an image;

[1666] A means for character recognition of the acquired image;

[1667] means for formatting the character recognized data;

[1668] means for storing the formatted data in a database and transmitting the data to a user's terminal;

[1669] means for receiving and temporarily storing image data from a user's terminal;

[1670] A means of inputting image data into a generative AI model and analyzing handwritten characters;

[1671] A means of converting the generated character recognition data into a format that the calendar application can understand;

[1672] means for sending a notification to the user's terminal that the data is ready;

[1673] A system including:

[1674] (Claim 2)

[1675] Further comprising a means for analyzing the acquired image including color-coded information and identifying each individual's schedule.

[1676] 10. The system of claim 1.

[1677] (Claim 3)

[1678] further comprising means for storing and transmitting the formatted data to a user terminal;

[1679] 10. The system of claim 1.

[1680] "Application Example 1"

[1681] (Claim 1)

[1682] a means for acquiring an image;

[1683] A means for character recognition of the acquired image;

[1684] A means for formatting the character recognized data;

[1685] A way to import the formatted data into a calendar app;

[1686] means for importing the formatted data into a work scheduling system;

[1687] A system including:

[1688] (Claim 2)

[1689] Further comprising a means for analyzing the acquired image including color-coded information and identifying each individual's schedule.

[1690] 10. The system of claim 1.

[1691] (Claim 3)

[1692] further comprising means for storing and transmitting the formatted data to a user terminal;

[1693] 10. The system of claim 1.

[1694] "Example 2: Combining Emotion Engines"

[1695] (Claim 1)

[1696] a means for acquiring an image;

[1697] A means for character recognition of the acquired image;

[1698] A means for formatting the character recognized data;

[1699] means for importing the formatted data into a calendar application;

[1700] means for activating a sentiment analysis engine for recognizing a user's sentiment;

[1701] a means for integrating emotion data with calendar data;

[1702] A system including:

[1703] (Claim 2)

[1704] Further comprising a means for analyzing the acquired image including color-coded information and identifying each individual's schedule.

[1705] 10. The system of claim 1.

[1706] (Claim 3)

[1707] further comprising means for storing and transmitting the formatted data to a user terminal;

[1708] 10. The system of claim 1.

[1709] "Application example 2 when combining emotion engines"

[1710] (Claim 1)

[1711] a means for acquiring an image;

[1712] A means for character recognition of the acquired image;

[1713] A means for formatting the character recognized data;

[1714] a means for importing the formatted data into the system;

[1715] means for recognizing an emotional state based on the formatted data;

[1716] a means for integrating and storing the emotion data in a database;

[1717] A system including:

[1718] (Claim 2)

[1719] Further comprising a means for analyzing the acquired image including color-coded information and identifying each individual's schedule.

[1720] 10. The system of claim 1.

[1721] (Claim 3)

[1722] further comprising means for storing and transmitting the formatted data to a user terminal;

[1723] 10. The system of claim 1. [Explanation of symbols]

[1724] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring an image; A means for character recognition of the acquired image; A means for formatting the character recognized data; A way to import the formatted data into a calendar app; A system including:

2. Further comprising a means for analyzing the acquired image including color-coded information and identifying each individual's schedule. The system of claim 1 .

3. further comprising means for storing and transmitting the formatted data to a user terminal; The system of claim 1 .

Images