System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving an image of an educational scene photographed by a user, a means for performing image analysis on the received image to extract text information, a means for automatically generating a class notebook on the basis of the extracted text information, and a means for returning the generated class notebook to a user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In educational settings and online training, efficiently taking notes on lecture content is a difficult task for many students and participants. Traditional methods require manual note-taking, which often takes time and distracts students from the lesson. Furthermore, manually transcribing images and blackboard content into text is time-consuming and cumbersome, which can reduce learning efficiency. [Means for solving the problem]

[0005] This invention provides a system that automatically generates class notes simply by allowing users to take images of educational scenes with their smartphones. This system includes a means for receiving the images taken by the user and performing image analysis to extract text information. It also includes a means for automatically generating class notes based on the extracted text information and returning the generated class notes to the user's device. This system allows users to create class notes without hassle and efficiently organize their learning content. Furthermore, the notes are linked to the user ID and stored in a database, allowing for easy later access.

[0006] "User" refers to any individual or entity that uses the System to capture and upload images of educational situations.

[0007] "Images of educational scenes" refers to still images taken of educational scenes such as classes and training sessions.

[0008] "Image analysis" refers to a series of processes performed to extract useful information from received image data.

[0009] "Text information" refers to character and sentence data extracted through image analysis.

[0010] "Class notes" refer to learning materials that have been formalized by generative AI based on extracted text information.

[0011] "Generative AI" refers to artificial intelligence that automatically converts extracted text information into study materials or note format.

[0012] "Database" refers to an information management system for storing and managing notes and user information generated within the system.

[0013] "User terminal" refers to a mobile device such as a smartphone or tablet that a user uses to use the system.

[0014] "Uploading" refers to the act of sending captured images or data from a user terminal to a server.

[0015] "Server" refers to a computer system that centrally controls the entire system and processes data. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that automatically generates lesson notes by allowing a user to take images of a teaching scene using a device such as a smartphone or tablet and upload the images to a server. A specific embodiment of this system will be described here.

[0038] Client side (terminal)

[0039] Taking and saving photos

[0040] A user takes a picture of a class or training session using a smartphone camera, and the device temporarily stores the photograph in the smartphone's internal memory.

[0041] Prepare to upload

[0042] The device moves the captured photo to the designated temporary storage for uploading, and then displays an "Upload" button to the user, offering them the option to send the photo to the server.

[0043] Upload Process

[0044] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0045] View notes

[0046] When the terminal receives the class notes returned from the server, it provides the user with an interface that displays the notes. Through this interface, the user can view the generated notes and use them for study or review.

[0047] Server side

[0048] Receiving and saving images

[0049] The server receives the photo data sent from the device and temporarily stores it in the server's storage.

[0050] Image analysis and OCR processing

[0051] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used in this process, specifically by pre-processing the images (converting them to grayscale and removing noise) and then applying character recognition algorithms.

[0052] Note generation

[0053] The server passes the extracted text information to a generation AI, which then automatically generates class notes. The generation AI then formats the received text data into notes, adding headings and bullet points.

[0054] Saving and returning to the database

[0055] The generated class notes are linked to the user ID and stored in a database. An HTTP response is also created and sent to return the generated notes to the device.

[0056] Specific examples

[0057] For example, imagine a user taking a picture of a formula or explanation written on a blackboard during a math class with their smartphone. When the user takes a photo and presses the "upload" button, the photo is sent to a server. The server analyzes the received photo and uses OCR technology to extract the formula and text. Next, the generative AI creates notes based on the extracted text data. Specifically, it formats the formula, adds paragraphs, and emphasizes important points.

[0058] The generated math notes are stored in a database and then sent back to the device. The user can view the notes on their smartphone app and use them for review or summarizing after class.

[0059] In this way, this system efficiently analyzes images of educational scenes taken by the user and automatically generates and provides lesson notes, thereby significantly improving learning efficiency.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[0063] Step 2:

[0064] The device moves the saved photos to the specified temporary upload storage. The device displays an "Upload" button, giving the user the option to send the photos to the server.

[0065] Step 3:

[0066] The user presses the "Upload" button. The device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0067] Step 4:

[0068] The server receives the HTTP request sent from the device, analyzes the photo data in the request, and temporarily stores it.

[0069] Step 5:

[0070] The server applies image analysis algorithms to the stored photos, including pre-processing the images (grayscale conversion and noise reduction).

[0071] Step 6:

[0072] The server uses OCR technology to extract text from the image, which involves detecting text regions and running character recognition algorithms.

[0073] Step 7:

[0074] The server sends the extracted text data to the generation AI, which analyzes the text data and automatically generates class notes. Specific operations include formatting the text, adding headings, and generating bullet points.

[0075] Step 8:

[0076] The server associates the generated class notes with the user ID and stores them in a database. The server then returns the generated notes to the device as an HTTP response.

[0077] Step 9:

[0078] The device analyzes the HTTP response received from the server, extracts the generated note data, and displays the note on the user interface.

[0079] Example 1

[0080] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0081] Traditional methods of taking class notes require students to take notes by hand during lectures, which requires a lot of time and effort. Furthermore, there is a risk of overlooking information or making typos, which can lead to reduced learning effectiveness. Furthermore, there are few efficient ways to digitize lecture content, making it difficult to search and share later.

[0082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0083] In this invention, the server includes means for receiving images of educational scenes taken by a user, means for temporarily saving the images in a storage device of the terminal, means for uploading the saved images to the server, means for performing image analysis on the received images to extract text information, means for inputting the extracted text information into a generative AI model to automatically generate class notes, means for returning the generated class notes to the user terminal, and means for displaying the generated class notes. This makes it possible to automatically digitize lecture content and efficiently generate accurate class notes.

[0084] "User" refers to a person who uses this system to take images of educational scenes and generate notes.

[0085] "Terminal" refers to a device, such as a smartphone or tablet, that a user uses to take images and upload them to a server.

[0086] "Server" refers to a computer system that receives and analyzes image data sent from a terminal and has the function of generating notes and returning them to the user.

[0087] "Image analysis" refers to a series of processes in which a terminal or server processes image data received and extracts useful information from the image.

[0088] "Text information" refers to information such as character data and mathematical formulas extracted from an image through image analysis.

[0089] "Generative AI model" refers to the artificial intelligence algorithm used to automatically generate notes based on text information.

[0090] A "prompt sentence" refers to an input sentence that instructs the generative AI model on what kind of note to generate.

[0091] "Class notes" refers to documents that organize the contents of lectures and are automatically generated by a generative AI model based on images of educational scenes taken by users.

[0092] "Storage device" refers to hardware for temporarily or long-term storage of image data and generated note data.

[0093] "Upload" refers to the process by which a device sends stored image data to a server.

[0094] A "user identifier" is information used to uniquely identify a user and is used to associate the user with the notes created in the database.

[0095] An "end-user terminal" refers to a terminal that is directly operated by a user and is responsible for taking images and displaying notes.

[0096] This invention is a system that automatically generates lesson notes by allowing a user to take images of a teaching scene using a device such as a smartphone or tablet and upload the images to a server. A specific embodiment of this system will be described here.

[0097] Client side (terminal)

[0098] 1. Taking and saving photos

[0099] Users use their smartphone camera to take pictures of classes and training sessions, and the photos are first temporarily saved in the device's internal memory.

[0100] 2. Prepare for upload

[0101] The device moves the temporarily saved photo data to temporary storage for uploading, and then the application displays an "Upload" button to the user, which the user can press to prepare the photo data for transmission to the server.

[0102] 3. Upload Process

[0103] When a user presses the "Upload" button, the device creates an HTTP request, attaches the saved photo, and sends it to the server. This process uses the smartphone's communication function and specifies the destination endpoint URL.

[0104] 4. View notes

[0105] The terminal receives the class notes returned from the server and provides an interface that allows users to view the notes. Users can use this interface to view the generated notes and use them for study or review.

[0106] Server side

[0107] 1. Receiving and saving images

[0108] The server receives the photo data sent from the device and temporarily stores it in the server's storage. The format and size of the received data are verified.

[0109] 2. Image analysis and OCR processing

[0110] The server analyzes the stored photo data and uses OCR technology to extract text information from the images. Specifically, the images are pre-processed (by converting them to grayscale and removing noise), and then a character recognition algorithm is applied.

[0111] 3. Creating notes

[0112] The server inputs the text information extracted by OCR into a generative AI model, which then automatically generates class notes. The generative AI model then formats the extracted text data into notes, adding headings and bullet points.

[0113] 4. Saving to database and returning

[0114] The generated class notes are linked to the user ID and stored in a database. An HTTP response is also created and sent to return the generated notes to the device.

[0115] Specific examples

[0116] For example, consider the case where a user takes a photo of a math class using their smartphone. During class, the user takes a photo of the formulas and explanations on the blackboard with their smartphone and presses the "upload" button, sending the photo to a server. The server analyzes the received photo and uses OCR technology to extract the formulas and characters. Next, a generative AI model creates class notes based on the extracted text data. Specifically, it formats formulas, adds paragraphs, and emphasizes important points.

[0117] The generated math notes are stored in a database and then sent back to the device, where users can view them using a smartphone application and use them for review or summarizing after class.

[0118] Examples of prompt statements

[0119] An example of a prompt sentence to input to the generative AI model is, "Please extract text data from the photo below and create class notes." By inputting this prompt sentence and the text data extracted by OCR into the AI ​​model, specific class notes are generated.

[0120] In this way, this system efficiently analyzes images of educational scenes taken by the user and automatically generates and provides lesson notes, thereby significantly improving learning efficiency.

[0121] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0122] Step 1:

[0123] A user takes a picture of a class or training session using the smartphone camera. The captured image is the input, and the device temporarily stores this image in its internal memory. Specifically, the smartphone's camera app accepts the operation of the capture button and saves the captured image data in a temporary directory within the camera app.

[0124] Step 2:

[0125] The device moves images stored in internal memory to temporary storage for uploading. The input to this process is image data stored in internal memory, and the output is image data moved to temporary storage for uploading. Specifically, the file is moved within the application, and an "Upload" button is displayed on the screen.

[0126] Step 3:

[0127] When a user presses the "Upload" button in the application, the device creates an HTTP request and sends the image data to the server. The input to this process is the image data in temporary upload storage, and the output is the HTTP request sent to the server. Specifically, the application constructs an HTTP request in the background and sends the request with the image data attached in multipart format to the specified server endpoint.

[0128] Step 4:

[0129] The server receives image data sent from the device and temporarily stores it in the server's storage. The input to this process is the HTTP request, and the output is image data stored in the server's temporary storage. Specifically, the server analyzes the request and stores the received image data in a specific directory.

[0130] Step 5:

[0131] The server performs image analysis and OCR processing on the stored image data to extract text information. The input to this process is the stored image data, and the output is the extracted text information. Specifically, the OCR software preprocesses the image (such as grayscale conversion and noise removal) and applies character recognition algorithms to generate the text information.

[0132] Step 6:

[0133] The server inputs the extracted text information into a generative AI model to automatically generate class notes. The input for this process is the text information extracted by OCR, and the output is the generated class notes. Specifically, the AI ​​model (such as the GPT series) receives the prompt and text information and generates formatted class notes.

[0134] Step 7:

[0135] The server associates the generated class notes with the user ID, stores them in a database, and returns them to the user's device. The input to this process is the generated class notes and the user ID, and the output is storage in the database and an HTTP response to the device. Specifically, the server stores the notes in the database, creates an HTTP response, and sends it to the device.

[0136] Step 8:

[0137] The device receives the class notes returned from the server and provides an interface for the user to view them. The input to this process is the HTTP response returned from the server, and the output is the class notes displayed in the user interface. Specifically, the application analyzes the received data and renders the note contents in the display area within the app.

[0138] (Application example 1)

[0139] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0140] Autonomous vehicles face the challenge of efficiently analyzing the large amounts of video and sensor data collected during operation, and providing the navigation information necessary for operation and a rapid response method when a problem occurs. This will reduce the risk of accidents and improve passenger safety and vehicle efficiency.

[0141] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0142] In this invention, the server includes means for receiving images of the environment taken by a user, means for performing image processing on the received images to extract information, and means for automatically generating navigation information based on the extracted information. This makes it possible to quickly extract important information from the taken images and provide necessary navigation information and response methods to autonomously driven vehicles in real time.

[0143] "User" refers to the operator or manager of the autonomous vehicle.

[0144] "Environmental images" are video data of the surroundings collected by cameras mounted on autonomous vehicles.

[0145] "Image processing" is a technique for extracting necessary information from video data captured by a camera, and specifically includes grayscale conversion and noise removal.

[0146] "Information" refers to text data and important point information extracted through image processing.

[0147] "Navigation information" is data that includes route information useful for operating an autonomous vehicle and instructions on how to respond in the event of an accident.

[0148] A "terminal" is a device for using navigation information, specifically an in-vehicle computer or a smartphone.

[0149] "Identification information" is information for identifying an autonomous vehicle or a user.

[0150] "Data storage device" refers to a database or storage device installed within a server.

[0151] A "mobile terminal" is an information device that a user can carry around, and specifically includes a smartphone or tablet.

[0152] A "server" is a computer system that receives images, processes them, and generates the necessary information.

[0153] This invention is a system that efficiently processes video data captured by an autonomous vehicle while it is in operation, and generates and provides navigation information in real time. First, the system program will be explained.

[0154] Overview of the program

[0155] This system involves a series of steps: an in-vehicle camera collects images of the surrounding environment and sends them to a server, which processes the images, extracts the information, generates navigation information, and sends it back to the terminal.

[0156] Program processing overview

[0157] 1. Data collection and transmission

[0158] The device captures images of the environment using a camera mounted on the autonomous vehicle and temporarily stores them in its internal memory. At this point, the device compresses the image data as needed and moves it to temporary storage.

[0159] 2. Image analysis and information extraction on the server

[0160] The server receives the image data sent from the device and performs image processing. In this process, preprocessing (grayscale conversion, noise removal, etc.) is performed using OpenCV, and text information is extracted using Tesseract OCR. The extracted information is just raw data, so it needs to be processed in the next step.

[0161] 3. Generating navigation information

[0162] The server inputs the extracted text information into a generative AI model using Tensorflow (registered trademark) to generate navigation information, which includes important information and troubleshooting tips.

[0163] 4. Data storage and return

[0164] The generated navigation information is stored in a data storage device along with the identification information, and then sent back to the vehicle's onboard computer as an HTTP response, providing the necessary information for driving in real time.

[0165] Hardware and software used

[0166] Hardware:

[0167] On-board camera: a device that takes images of the environment

[0168] On-board computer: A device that manages the temporary storage and transmission of data.

[0169] Storage device: Data storage device

[0170] software:

[0171] OpenCV: Image processing library

[0172] Tesseract OCR: Character Recognition Engine

[0173] TensorFlow: A framework for running generative AI models

[0174] HTTP client: Software that manages the sending and receiving of data

[0175] Specific examples

[0176] For example, when an autonomous vehicle approaches an intersection, a camera collects information on signs and traffic lights. When the user presses the "upload" button, the image data is sent to a server. The server processes the image and generates navigation information such as "Follow the signs near the intersection to confirm your direction of travel." This navigation information is sent back to the user's device in real time and displayed on the driver's seat display.

[0177] Prompt Sentence Examples

[0178] Below are some example prompts to input to the generative AI model:

[0179] (prompt statement)

[0180] Please create a navigation notebook based on this video data, including important points, optimal route information, and recovery procedures in case of trouble.

[0181] Video data:

[0182] 1. Video file name | / path / to / data / video1.mp4

[0183] 2. Video file name | / path / to / data / video2.mp4

[0184] The above is an embodiment of this system.

[0185] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0186] Step 1:

[0187] The terminal captures images of the environment using a camera mounted on the autonomous vehicle and temporarily stores them in its internal memory. It receives raw image data captured by the camera as input and obtains image data stored in temporary storage as output. Specific operations include compressing the captured images and transferring them to temporary storage.

[0188] Step 2:

[0189] When the user presses the "Upload" button, the device sends the saved image data to the server as an HTTP request. The image data in temporary storage is used as input, and the data sent to the server is obtained as output. Specifically, the device performs a process to format the image data into an HTTP request.

[0190] Step 3:

[0191] The server receives image data sent from the terminal and saves it in a storage device. It receives image data sent from the terminal as input and obtains the saved data as output. Specifically, the server safely stores the data it receives in storage.

[0192] Step 4:

[0193] The server performs preprocessing on the stored image data using OpenCV. It receives the stored image data as input and obtains preprocessed data as output. Specifically, this includes processes such as grayscale conversion and noise removal.

[0194] Step 5:

[0195] The preprocessed image data is input into Tesseract OCR to extract text information. The preprocessed image data is used as input, and extracted text information is obtained as output. Specifically, the text in the image is analyzed and extracted as character data.

[0196] Step 6:

[0197] The extracted text information is input into a generative AI model using TensorFlow to generate navigation information. The extracted text information is used as input, and generated navigation information is obtained as output. Specific operations include the generative AI model analyzing the input data and generating the necessary information.

[0198] Step 7:

[0199] The generated navigation information is stored in a data storage device together with the identification information. The generated navigation information and the identification information are used as input, and information stored in a database is obtained as output. Specific operations include formatting the information into a database format.

[0200] Step 8:

[0201] The server returns the saved navigation information to the terminal as an HTTP response. It uses the navigation information saved in the database as input and obtains the data sent to the terminal as output. Specific operations include formatting the navigation information as an HTTP response and sending it to the terminal.

[0202] Step 9:

[0203] The terminal displays the received navigation information to the user. It uses the navigation information received from the server as input and obtains the information displayed on the user display as output. Specific operations include visually displaying the navigation information.

[0204] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0205] This invention is a system that allows users to take images of educational situations using devices such as smartphones or tablets, upload the image data to a server, automatically generate class notes, and further recognize the user's emotions and reflect the results in the notes, thereby improving the quality and efficiency of users' learning.

[0206] Client side (terminal)

[0207] Taking and saving photos

[0208] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[0209] Prepare to upload

[0210] The device moves the saved photos to the designated temporary upload storage, and then displays an "Upload" button to the user, giving them the option to send the photos to the server.

[0211] Upload Process

[0212] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0213] View notes

[0214] When the device receives the class notes returned from the server, it provides the user with an interface to display the notes. Through this interface, the user can view the generated notes and use them for study or review. In addition, the user's emotional information analyzed by the emotion engine is also reflected in the notes.

[0215] Server side

[0216] Receiving and saving images

[0217] The server receives the photo data sent from the device and temporarily stores it in the server's storage.

[0218] Image analysis and OCR processing

[0219] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used in this process, specifically by pre-processing the images (converting them to grayscale and removing noise) and then applying character recognition algorithms.

[0220] Emotion recognition by emotion engine

[0221] The server further executes an emotion engine that recognizes the user's emotion from image and text information, and collects the user's emotion data to identify the user's emotional state, such as relaxed, focused, or distressed.

[0222] Note generation

[0223] The server automatically generates class notes using a generation AI based on the extracted text information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes.

[0224] Saving and returning to the database

[0225] The generated class notes and the user's emotion data are linked to the user ID and stored in a database. Furthermore, an HTTP response is created and sent to return the generated notes to the device.

[0226] Specific examples

[0227] For example, imagine a user is taking a chemistry class and takes a photo of a chemical formula or explanation written on the blackboard with their smartphone. When the user takes a photo and presses the "upload" button, the photo is sent to the server. The server analyzes the received photo and uses OCR technology to extract the chemical formula and text. Next, the generative AI creates notes based on the extracted text data. Specifically, it formats the chemical formula, adds paragraphs, and emphasizes important points.

[0228] Furthermore, the emotion engine recognizes the user's emotions. For example, if the user feels "difficult to understand," the engine reflects this information in the notes. This information is recorded in a specific place in the notes, allowing the user to identify which part of the text they found difficult. This information is also stored in a database for future reference.

[0229] In this way, this system efficiently analyzes images of educational scenes taken by the user, automatically generates lesson notes, and uses an emotion engine to reflect the user's emotional information in the notes, thereby significantly improving learning efficiency.

[0230] The processing flow will be explained below.

[0231] Step 1:

[0232] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[0233] Step 2:

[0234] The device moves the saved photos to the specified temporary upload storage. The device displays an "Upload" button, giving the user the option to send the photos to the server.

[0235] Step 3:

[0236] The user presses the "Upload" button. The device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0237] Step 4:

[0238] The server receives the HTTP request sent from the device, analyzes the photo data in the request, and temporarily stores it.

[0239] Step 5:

[0240] The server applies image analysis algorithms to the stored photos, including pre-processing the images (grayscale conversion and noise reduction).

[0241] Step 6:

[0242] The server uses OCR technology to extract text from the image, which involves detecting text regions and running character recognition algorithms.

[0243] Step 7:

[0244] The server sends the extracted text data to the emotion engine, which evaluates the user's emotional state (e.g., relaxed, focused, or distressed) based on the analysis of the text and images. Specific operations include performing text analysis and facial recognition technology.

[0245] Step 8:

[0246] The server then sends the text data to a generation AI, which automatically generates class notes. The generation AI formats the received text data into notes, adding headings and bullet points. Emotion data from the emotion engine is also reflected in the notes.

[0247] Step 9:

[0248] The server associates the generated class notes with the user's emotion data and stores them in a database. The server then returns the generated notes to the device as an HTTP response.

[0249] Step 10:

[0250] The device analyzes the HTTP response received from the server and extracts the generated note data and emotional information. The device then displays the note on the user interface, along with emotional information such as which parts the user found difficult to understand.

[0251] Example 2

[0252] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0253] In modern education, detailed notes are necessary to effectively review the content of lessons, but traditional methods are time-consuming and burdensome for students. Furthermore, there is a lack of means to grasp students' understanding and emotions in real time and reflect them in their notes, making it difficult to optimize learning outcomes. Therefore, there is a need for the development of a system that can efficiently generate lesson notes and also reflect students' emotional information.

[0254] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0255] In this invention, the server includes means for receiving images of educational scenes taken by a user, means for performing image analysis on the received images and extracting text information, means for automatically generating class notes based on the extracted text information, means for returning the generated class notes to the user terminal, and means for recognizing the user's emotions and reflecting the emotional information in the class notes. This enables the automatic generation of class notes and the reflection of emotional information, allowing students to efficiently review the content of lessons and improving learning effectiveness.

[0256] "User" refers to an individual who uses this system to capture images of educational situations and generate lesson notes.

[0257] "Educational setting" refers to a situation in which learning or education is taking place, such as in classes, lectures, or training.

[0258] "Images" refers to photographs or scanned data of educational scenes taken by users.

[0259] "Image analysis" refers to the process of extracting text information from received image data.

[0260] "Text information" refers to character data or text data extracted by image analysis.

[0261] "Class notes" refers to documents that record learning content and are automatically generated based on extracted text information.

[0262] "Means for receiving" refers to the function of the server to receive and store image data sent from the user terminal.

[0263] "Means for extracting" refers to the function of extracting text information from an image through image analysis.

[0264] "Means of automatic generation" refers to algorithms or programs that create class notes based on extracted text information.

[0265] "Means for returning" refers to the function of sending the generated lesson notes to the user's terminal and displaying them.

[0266] "Means for recognizing emotions" refers to a function that analyzes the user's emotional state and collects that information.

[0267] "Means for reflecting emotional information" refers to the function of incorporating the recognized emotions of the user into class notes.

[0268] "Temporary storage means" refers to a function for temporarily storing images taken by a user terminal and later transmitting them to a server.

[0269] "Means for uploading" refers to a function for sending image data from a user terminal to a server.

[0270] "Means of saving to a database" refers to the function for associating generated class notes with user IDs and storing them for a long period of time.

[0271] This invention is a system that allows users to take images of educational situations using devices such as smartphones or tablets, upload the image data to a server, automatically generate class notes, recognize the user's emotions, and reflect the results in the notes. This system can improve the quality and efficiency of users' learning.

[0272] Hardware and Software Configuration

[0273] Device: The user captures the classroom scene using a smartphone or tablet (iOS, ANDROID (registered trademark)). The device includes a camera, memory, and network module.

[0274] Server: A server with high-performance storage and processing power. Key software includes image analysis algorithms, OCR technology, emotion recognition engines, and generative AI models.

[0275] Image analysis algorithms: Libraries such as Python and OpenCV are used to extract text information from images.

[0276] OCR technology: OCR software such as Tesseract is used to extract text.

[0277] Emotion recognition engine: Emotion analysis uses IBM Watson (registered trademark) and Microsoft (registered trademark) Azure (registered trademark) Emotion API.

[0278] Generative AI models: Generative AI models such as GPT-3 (registered trademark) are used to format text data into notes.

[0279] Overview of system processing

[0280] Client side (terminal)

[0281] Taking and saving photos: A user takes a photo of a class or training session using their smartphone camera. The device temporarily stores the photo in its internal memory. For example, a user may use their smartphone camera to take a photo of a chemical formula drawn on a blackboard during class.

[0282] Prepare for upload: The device moves the saved photos to the specified temporary upload storage, and displays an "Upload" button to the user, giving them the option to send the photos to the server. The moved photos are saved in " / storage / emulated / 0 / AppUploads / ".

[0283] Upload process: When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo. For example, an HTTP POST request is sent to "https: / / example.com / upload".

[0284] Displaying notes: The device receives the class note data returned from the server and provides the user with an interface for displaying the notes. Through this interface, the user can view the generated notes and use them for study or review. In addition, the user's emotional information analyzed by the emotion engine is also reflected in the notes.

[0285] Server side

[0286] Image reception and storage: The server receives the photo data sent from the device and temporarily stores it in the server's storage. The received image data is stored in " / var / www / uploads / ".

[0287] Image analysis and OCR processing: Image analysis algorithms are run on the server to extract text information from images. Specifically, the Python OCR library "Tesseract" is used to convert images into text data.

[0288] Emotion recognition using emotion engine: Implement an emotion engine that recognizes user emotions from images and text information. Call emotion recognition APIs (e.g., IBM Watson, Microsoft Azure Emotion API) to analyze emotions from the user's facial expressions and context.

[0289] Note generation: Class notes are automatically generated by a generation AI based on the extracted text information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the user's emotional information. For example, the AI ​​model can be given a prompt such as, "Please highlight the chemical formula and indicate any difficult parts to understand."

[0290] Saving to database and returning: The generated class notes and the user's emotion data are linked to the user ID and stored in a database. The generated notes are then returned to the device as an HTTP response. For example, the note data and user ID are saved in a MySQL (registered trademark) database, and JSON data is returned as a response.

[0291] Specific examples

[0292] For example, suppose a user takes a photo of a chemical formula or explanation written on a blackboard during a chemistry class using their smartphone. After taking the photo and pressing the "upload" button, the photo is sent to a server. The server analyzes the received photo and extracts the chemical formula and text using OCR technology. Next, a generative AI creates class notes based on the extracted text data. Specifically, it formats the chemical formula, adds paragraphs, and emphasizes important points. Furthermore, an emotion engine recognizes the user's emotions and, for example, if the user finds the text difficult to understand, reflects that information in the notes. This information is recorded in specific places in the notes, allowing the user to identify which parts were difficult. This information is also stored in a database for future reference. This system efficiently analyzes images of educational situations taken by users, automatically generates class notes, and uses the emotion engine to reflect the user's emotional information in the notes, significantly improving learning efficiency.

[0293] Example prompts to input to the generative AI model

[0294] "Extract text information from this photo and format it into a note. In particular, highlight the chemical formula and important points. Also, reflect the user's sentiment that it is difficult to understand."

[0295] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0296] Step 1: Take and save a photo

[0297] User: Uses a camera app on a smartphone or tablet to capture educational scenes.

[0298] Terminal: The captured photo is temporarily saved in the internal memory. The input is an image taken by the user, and the output is an image file saved in the internal memory. Specifically, the user takes a photo of a chemical formula drawn on a blackboard with the smartphone camera, and the image file is saved in the " / storage / emulated / 0 / Pictures / " directory.

[0299] Step 2: Prepare to upload

[0300] On device: Moves saved photos to the specified temporary upload storage. The input is an image file stored in the internal memory, and the output is an image file moved to the upload storage.

[0301] On the device: The user is presented with an "Upload" button, giving them the option to send the photo to the server. Specifically, the image file is moved to the " / storage / emulated / 0 / AppUploads / " directory, and then the "Upload" button is displayed in the UI.

[0302] Step 3: Upload Process

[0303] User: Presses the "Upload" button.

[0304] Terminal: Create an HTTP request, attach the saved photo, and send it to the specified endpoint on the server. The input is the image file saved in the upload storage, and the output is an HTTP request to the server. Specifically, an HTTP POST request is sent to "https: / / example.com / upload" with the photo data attached in multipart format.

[0305] Step 4: Receive and save images

[0306] Server: Receives photo data sent from the device. The received data is temporarily stored in the server's storage. The input is the image data sent from the device, and the output is an image file stored in the server's storage. Specifically, the image file is saved in the " / var / www / uploads / " directory.

[0307] Step 5: Image analysis and OCR processing

[0308] Server: Executes image analysis algorithms and extracts text information from images. After preprocessing the image (grayscale conversion and noise removal), character recognition is performed using OCR technology. The input is the received image file, and the output is the extracted text data. Specifically, the image is converted to text data using the Python OCR library "Tesseract."

[0309] Step 6: Emotion Recognition with the Emotion Engine

[0310] Server: Runs the emotion engine that recognizes the user's emotions from image and text information. The emotion engine collects the user's emotional data and identifies their emotional state, such as relaxed, focused, or difficult. The input is image and text data, and the output is the recognized emotion data. Specifically, it calls an emotion recognition API (e.g., IBM Watson, Microsoft Azure Emotion API) to analyze emotions from the user's facial expressions and context.

[0311] Step 7: Generate notes

[0312] Server: Class notes are automatically generated using a generative AI based on the extracted text information. The generative AI formats the text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes. The input is text data and emotion data, and the output is the generated class notes. Specifically, it uses a generative AI model such as GPT-3 to generate notes based on the input prompt.

[0313] Step 8: Saving and returning to the database

[0314] Server: The generated class notes and the user's emotion data are linked to the user ID and stored in a database. In addition, an HTTP response is created and sent to return the generated notes to the device. The input is the generated class notes and the user ID, and the output is the information stored in the database and the HTTP response for return. Specifically, the server saves the note data and the corresponding user ID in a MySQL database, and returns JSON data as a response to the device.

[0315] Step 9: Viewing Notes

[0316] Device: Receives class notes returned from the server and provides the user with an interface to display the notes. The input is the HTTP response from the server, and the output is the displayed note information. Specifically, the device analyzes the note data received in JSON format and formats it in an easy-to-read format within the app before displaying it.

[0317] (Application example 2)

[0318] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0319] In modern factories, product quality control and improved work efficiency are extremely important issues. However, traditional manual inspection records and worker emotional management are cumbersome and inefficient. Furthermore, because workers' emotional states are not reflected in inspection records, they often do not lead to improvements in the work environment. For this reason, there was a need for a system that automatically analyzes product images, generates accurate inspection records, and also reflects workers' emotional states.

[0320] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving images, means for performing image analysis and extracting text information, means for automatically generating notes, and means for recognizing the user's emotions and reflecting the information in the notes. This enables efficient quality control of products in factories, and further enables improvements to the work environment and early detection of problems by reflecting the emotional information of workers.

[0321] The "means for receiving images" refers to a function or process by which a user sends an image taken by the user to the server and the server receives the image.

[0322] "Means for performing image analysis and extracting text information" refers to a function or process that uses digital image processing technology on a received image to identify text contained in the image and extract it as characters.

[0323] "Means for automatically generating notes" refers to functions and processes for automatically generating structured notes based on extracted text information.

[0324] "Means for recognizing the user's emotions and reflecting the information in the notebook" refers to the functions and processes for analyzing the user's emotional state from images and surrounding data and reflecting the results in the notebook. For example, it could identify areas where the user is having difficulty and record that information in the notebook.

[0325] "Inside the factory" refers to the work area where a series of operations such as product manufacturing, assembly, inspection, and quality control are carried out.

[0326] "Quality control" refers to activities to check and ensure that the manufacturing process and finished products comply with prescribed standards and specifications.

[0327] "Working efficiently" means carrying out work in a way that maximizes effectiveness while minimizing the use of time and resources.

[0328] "Reflecting the emotional information of workers" means understanding the emotional state of workers and providing feedback to improve the work environment and methods based on that information.

[0329] "Improving the working environment" refers to activities that improve the physical and psychological working environment so that workers can work safely and efficiently.

[0330] "Early problem detection" means detecting product defects or problems with work procedures in advance and responding quickly.

[0331] The system for carrying out the present invention is for supporting quality control within a factory. Specific embodiments and processes thereof will be described below.

[0332] Client side (terminal)

[0333] 1. Taking and saving images

[0334] A robot in the factory takes pictures of the products, and the device temporarily stores the pictures in its internal memory.

[0335] 2. Prepare for upload

[0336] The device moves the saved image to a predetermined temporary storage location and then displays an "Upload" button to the user, providing them with the option to send the image to the server.

[0337] 3. Upload Process

[0338] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved image.

[0339] Server side

[0340] 1. Receiving and saving images

[0341] The server receives the image data sent from the device and temporarily stores the received data in the server's storage.

[0342] 2. Image analysis and OCR processing

[0343] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used here, specifically image preprocessing (grayscale conversion and noise removal) and text recognition algorithms are applied. Specific software includes OpenCV and Tesseract OCR.

[0344] 3. Emotion Recognition by Emotion Engine

[0345] The server also runs an emotion engine that recognizes user emotions from images and text information. The emotion engine collects the worker's emotional data and identifies their emotional states, such as relaxed, focused, or difficult. The emotion recognition model used here is trained with Keras.

[0346] 4. Creating Notes

[0347] The server automatically generates quality inspection notes using a generation AI based on the extracted text and emotion information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes.

[0348] 5. Saving to database and returning

[0349] The generated quality inspection note and emotion information are linked to the user ID and stored in a database. Furthermore, an HTTP response is created and sent to return the generated note to the device.

[0350] Specific examples

[0351] For example, a robot in a factory takes a picture of a product and automatically generates a quality inspection note based on that image. Text information (such as product dimensions and surface condition) is extracted from the captured image using OCR technology, and an emotion recognition model is used to recognize the worker's emotional state (such as "difficulty"). The generation AI uses this information to format the quality inspection note, creating a note that highlights the points where the worker found it difficult.

[0352] Prompt Sentence Examples

[0353] For example, the following prompt sentence could be input to a generative AI model:

[0354] prompt:

[0355] Please format the following quality inspection record as an inspection note and reflect the worker's emotional information.

[0356] Product condition: Surface scratches

[0357] Test result: Fail

[0358] Quality information: Further inspection required

[0359] Worker Emotions: Difficulty

[0360] This automatically generates quality inspection notes that are easy for users to understand, and provides information that is useful for improving the work environment and enhancing quality.

[0361] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0362] Step 1:

[0363] A robot takes images of products in a factory. Specifically, it acquires image data of the products using a camera mounted on the robot. The input is the physical appearance of the product, and the output is image data (JPEG or PNG format).

[0364] Step 2:

[0365] The captured image data is temporarily saved in the device's internal memory. Specifically, the image file is saved in the device's storage and its file path is recorded. The input is the image data captured in step 1, and the output is the file path of the saved image file.

[0366] Step 3:

[0367] The user presses the "Upload" button to send image data to the server. Specifically, the device creates an HTTP request, attaches the saved image file, and sends it to the specified endpoint on the server. The input is the file path of the saved image file, and the output is an HTTP request to the server.

[0368] Step 4:

[0369] The server receives image data sent from the terminal and temporarily stores the image data in the server's storage. The input is an HTTP request, and the output is an image file stored on the server.

[0370] Step 5:

[0371] The server runs an image analysis algorithm to extract text information from the image. Specifically, the image is converted to grayscale, noise is removed, and then Tesseract OCR is used to recognize text in the image. The input is the saved image file, and the output is the extracted text information.

[0372] Step 6:

[0373] The server uses an emotion recognition engine to analyze the user's emotional state. Specifically, it detects the worker's face in the image and labels the emotion using an emotion recognition model trained with Keras. The input is the saved image file, and the output is the recognized emotional state information.

[0374] Step 7:

[0375] The server uses the generative AI model to generate a quality inspection note based on the extracted text information and analyzed emotion information. Specifically, a prompt sentence is input to the generative AI model, and note-format text is output. The input is text information and emotion information, and the output is the generated quality inspection note.

[0376] Step 8:

[0377] The server associates the generated quality inspection note with the user ID and stores it in the database. Specifically, it associates the note text with the user ID, registers it in the database, and also stores the necessary metadata. The input is the generated quality inspection note and the user ID, and the output is the note record stored in the database.

[0378] Step 9:

[0379] The server returns the generated note to the terminal as an HTTP response. Specifically, it creates an HTTP response and sends it with the text of the generated note included as a payload. The input is the generated quality inspection note, and the output is the HTTP response.

[0380] Step 10:

[0381] The quality inspection note received by the terminal is displayed through the user interface. Specifically, the note text is displayed on the screen so that the user can check the contents. The input is the note text received as an HTTP response, and the output is the inspection note displayed on the terminal screen.

[0382] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0383] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0384] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0385] [Second embodiment]

[0386] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0387] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0388] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0389] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0390] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0391] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0392] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0393] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0394] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0395] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0396] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0397] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0398] This invention is a system that automatically generates lesson notes by allowing a user to take images of a teaching scene using a device such as a smartphone or tablet and upload the images to a server. A specific embodiment of this system will be described here.

[0399] Client side (terminal)

[0400] Taking and saving photos

[0401] A user takes a picture of a class or training session using a smartphone camera, and the device temporarily stores the photograph in the smartphone's internal memory.

[0402] Prepare to upload

[0403] The device moves the captured photo to the designated temporary storage for uploading, and then displays an "Upload" button to the user, offering them the option to send the photo to the server.

[0404] Upload Process

[0405] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0406] View notes

[0407] When the terminal receives the class notes returned from the server, it provides the user with an interface that displays the notes. Through this interface, the user can view the generated notes and use them for study or review.

[0408] Server side

[0409] Receiving and saving images

[0410] The server receives the photo data sent from the device and temporarily stores it in the server's storage.

[0411] Image analysis and OCR processing

[0412] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used in this process, specifically by pre-processing the images (converting them to grayscale and removing noise) and then applying character recognition algorithms.

[0413] Note generation

[0414] The server passes the extracted text information to a generation AI, which then automatically generates class notes. The generation AI then formats the received text data into notes, adding headings and bullet points.

[0415] Saving and returning to the database

[0416] The generated class notes are linked to the user ID and stored in a database. An HTTP response is also created and sent to return the generated notes to the device.

[0417] Specific examples

[0418] For example, imagine a user taking a picture of a formula or explanation written on a blackboard during a math class with their smartphone. When the user takes a photo and presses the "upload" button, the photo is sent to a server. The server analyzes the received photo and uses OCR technology to extract the formula and text. Next, the generative AI creates notes based on the extracted text data. Specifically, it formats the formula, adds paragraphs, and emphasizes important points.

[0419] The generated math notes are stored in a database and then sent back to the device. The user can view the notes on their smartphone app and use them for review or summarizing after class.

[0420] In this way, this system efficiently analyzes images of educational scenes taken by the user and automatically generates and provides lesson notes, thereby significantly improving learning efficiency.

[0421] The processing flow will be explained below.

[0422] Step 1:

[0423] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[0424] Step 2:

[0425] The device moves the saved photos to the specified temporary upload storage. The device displays an "Upload" button, giving the user the option to send the photos to the server.

[0426] Step 3:

[0427] The user presses the "Upload" button. The device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0428] Step 4:

[0429] The server receives the HTTP request sent from the device, analyzes the photo data in the request, and temporarily stores it.

[0430] Step 5:

[0431] The server applies image analysis algorithms to the stored photos, including pre-processing the images (grayscale conversion and noise reduction).

[0432] Step 6:

[0433] The server uses OCR technology to extract text from the image, which involves detecting text regions and running character recognition algorithms.

[0434] Step 7:

[0435] The server sends the extracted text data to the generation AI, which analyzes the text data and automatically generates class notes. Specific operations include formatting the text, adding headings, and generating bullet points.

[0436] Step 8:

[0437] The server associates the generated class notes with the user ID and stores them in a database. The server then returns the generated notes to the device as an HTTP response.

[0438] Step 9:

[0439] The device analyzes the HTTP response received from the server, extracts the generated note data, and displays the note on the user interface.

[0440] Example 1

[0441] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0442] Traditional methods of taking class notes require students to take notes by hand during lectures, which requires a lot of time and effort. Furthermore, there is a risk of overlooking information or making typos, which can lead to reduced learning effectiveness. Furthermore, there are few efficient ways to digitize lecture content, making it difficult to search and share later.

[0443] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0444] In this invention, the server includes means for receiving images of educational scenes taken by a user, means for temporarily saving the images in a storage device of the terminal, means for uploading the saved images to the server, means for performing image analysis on the received images to extract text information, means for inputting the extracted text information into a generative AI model to automatically generate class notes, means for returning the generated class notes to the user terminal, and means for displaying the generated class notes. This makes it possible to automatically digitize lecture content and efficiently generate accurate class notes.

[0445] "User" refers to a person who uses this system to take images of educational scenes and generate notes.

[0446] "Terminal" refers to a device, such as a smartphone or tablet, that a user uses to take images and upload them to a server.

[0447] "Server" refers to a computer system that receives and analyzes image data sent from a terminal and has the function of generating notes and returning them to the user.

[0448] "Image analysis" refers to a series of processes in which a terminal or server processes image data received and extracts useful information from the image.

[0449] "Text information" refers to information such as character data and mathematical formulas extracted from an image through image analysis.

[0450] "Generative AI model" refers to the artificial intelligence algorithm used to automatically generate notes based on text information.

[0451] A "prompt sentence" refers to an input sentence that instructs the generative AI model on what kind of note to generate.

[0452] "Class notes" refers to documents that organize the contents of lectures and are automatically generated by a generative AI model based on images of educational scenes taken by users.

[0453] "Storage device" refers to hardware for temporarily or long-term storage of image data and generated note data.

[0454] "Upload" refers to the process by which a device sends stored image data to a server.

[0455] A "user identifier" is information used to uniquely identify a user and is used to associate the user with the notes created in the database.

[0456] An "end-user terminal" refers to a terminal that is directly operated by a user and is responsible for taking images and displaying notes.

[0457] This invention is a system that automatically generates lesson notes by allowing a user to take images of a teaching scene using a device such as a smartphone or tablet and upload the images to a server. A specific embodiment of this system will be described here.

[0458] Client side (terminal)

[0459] 1. Taking and saving photos

[0460] Users use their smartphone camera to take pictures of classes and training sessions, and the photos are first temporarily saved in the device's internal memory.

[0461] 2. Prepare for upload

[0462] The device moves the temporarily saved photo data to temporary storage for uploading, and then the application displays an "Upload" button to the user, which the user can press to prepare the photo data for transmission to the server.

[0463] 3. Upload Process

[0464] When a user presses the "Upload" button, the device creates an HTTP request, attaches the saved photo, and sends it to the server. This process uses the smartphone's communication function and specifies the destination endpoint URL.

[0465] 4. View notes

[0466] The terminal receives the class notes returned from the server and provides an interface that allows users to view the notes. Users can use this interface to view the generated notes and use them for study or review.

[0467] Server side

[0468] 1. Receiving and saving images

[0469] The server receives the photo data sent from the device and temporarily stores it in the server's storage. The format and size of the received data are verified.

[0470] 2. Image analysis and OCR processing

[0471] The server analyzes the stored photo data and uses OCR technology to extract text information from the images. Specifically, the images are pre-processed (by converting them to grayscale and removing noise), and then a character recognition algorithm is applied.

[0472] 3. Creating notes

[0473] The server inputs the text information extracted by OCR into a generative AI model, which then automatically generates class notes. The generative AI model then formats the extracted text data into notes, adding headings and bullet points.

[0474] 4. Saving to database and returning

[0475] The generated class notes are linked to the user ID and stored in a database. An HTTP response is also created and sent to return the generated notes to the device.

[0476] Specific examples

[0477] For example, consider the case where a user takes a photo of a math class using their smartphone. During class, the user takes a photo of the formulas and explanations on the blackboard with their smartphone and presses the "upload" button, sending the photo to a server. The server analyzes the received photo and uses OCR technology to extract the formulas and characters. Next, a generative AI model creates class notes based on the extracted text data. Specifically, it formats formulas, adds paragraphs, and emphasizes important points.

[0478] The generated math notes are stored in a database and then sent back to the device, where users can view them using a smartphone application and use them for review or summarizing after class.

[0479] Examples of prompt statements

[0480] An example of a prompt sentence to input to the generative AI model is, "Please extract text data from the photo below and create class notes." By inputting this prompt sentence and the text data extracted by OCR into the AI ​​model, specific class notes are generated.

[0481] In this way, this system efficiently analyzes images of educational scenes taken by the user and automatically generates and provides lesson notes, thereby significantly improving learning efficiency.

[0482] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0483] Step 1:

[0484] A user takes a picture of a class or training session using the smartphone camera. The captured image is the input, and the device temporarily stores this image in its internal memory. Specifically, the smartphone's camera app accepts the operation of the capture button and saves the captured image data in a temporary directory within the camera app.

[0485] Step 2:

[0486] The device moves images stored in internal memory to temporary storage for uploading. The input to this process is image data stored in internal memory, and the output is image data moved to temporary storage for uploading. Specifically, the file is moved within the application, and an "Upload" button is displayed on the screen.

[0487] Step 3:

[0488] When a user presses the "Upload" button in the application, the device creates an HTTP request and sends the image data to the server. The input to this process is the image data in temporary upload storage, and the output is the HTTP request sent to the server. Specifically, the application constructs an HTTP request in the background and sends the request with the image data attached in multipart format to the specified server endpoint.

[0489] Step 4:

[0490] The server receives image data sent from the device and temporarily stores it in the server's storage. The input to this process is the HTTP request, and the output is image data stored in the server's temporary storage. Specifically, the server analyzes the request and stores the received image data in a specific directory.

[0491] Step 5:

[0492] The server performs image analysis and OCR processing on the stored image data to extract text information. The input to this process is the stored image data, and the output is the extracted text information. Specifically, the OCR software preprocesses the image (such as grayscale conversion and noise removal) and applies character recognition algorithms to generate the text information.

[0493] Step 6:

[0494] The server inputs the extracted text information into a generative AI model to automatically generate class notes. The input for this process is the text information extracted by OCR, and the output is the generated class notes. Specifically, the AI ​​model (such as the GPT series) receives the prompt and text information and generates formatted class notes.

[0495] Step 7:

[0496] The server associates the generated class notes with the user ID, stores them in a database, and returns them to the user's device. The input to this process is the generated class notes and the user ID, and the output is storage in the database and an HTTP response to the device. Specifically, the server stores the notes in the database, creates an HTTP response, and sends it to the device.

[0497] Step 8:

[0498] The device receives the class notes returned from the server and provides an interface for the user to view them. The input to this process is the HTTP response returned from the server, and the output is the class notes displayed in the user interface. Specifically, the application analyzes the received data and renders the note contents in the display area within the app.

[0499] (Application example 1)

[0500] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0501] Autonomous vehicles face the challenge of efficiently analyzing the large amounts of video and sensor data collected during operation, and providing the navigation information necessary for operation and a rapid response method when a problem occurs. This will reduce the risk of accidents and improve passenger safety and vehicle efficiency.

[0502] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0503] In this invention, the server includes means for receiving images of the environment taken by a user, means for performing image processing on the received images to extract information, and means for automatically generating navigation information based on the extracted information. This makes it possible to quickly extract important information from the taken images and provide necessary navigation information and response methods to autonomously driven vehicles in real time.

[0504] "User" refers to the operator or manager of the autonomous vehicle.

[0505] "Environmental images" are video data of the surroundings collected by cameras mounted on autonomous vehicles.

[0506] "Image processing" is a technique for extracting necessary information from video data captured by a camera, and specifically includes grayscale conversion and noise removal.

[0507] "Information" refers to text data and important point information extracted through image processing.

[0508] "Navigation information" is data that includes route information useful for operating an autonomous vehicle and instructions on how to respond in the event of an accident.

[0509] A "terminal" is a device for using navigation information, specifically an in-vehicle computer or a smartphone.

[0510] "Identification information" is information for identifying an autonomous vehicle or a user.

[0511] "Data storage device" refers to a database or storage device installed within a server.

[0512] A "mobile terminal" is an information device that a user can carry around, and specifically includes a smartphone or tablet.

[0513] A "server" is a computer system that receives images, processes them, and generates the necessary information.

[0514] This invention is a system that efficiently processes video data captured by an autonomous vehicle while it is in operation, and generates and provides navigation information in real time. First, the system program will be explained.

[0515] Overview of the program

[0516] This system involves a series of steps: an in-vehicle camera collects images of the surrounding environment and sends them to a server, which processes the images, extracts the information, generates navigation information, and sends it back to the terminal.

[0517] Program processing overview

[0518] 1. Data collection and transmission

[0519] The device captures images of the environment using a camera mounted on the autonomous vehicle and temporarily stores them in its internal memory. At this point, the device compresses the image data as needed and moves it to temporary storage.

[0520] 2. Image analysis and information extraction on the server

[0521] The server receives the image data sent from the device and performs image processing. In this process, preprocessing (grayscale conversion, noise removal, etc.) is performed using OpenCV, and text information is extracted using Tesseract OCR. The extracted information is just raw data, so it needs to be processed in the next step.

[0522] 3. Generating navigation information

[0523] The server inputs the extracted text information into a generative AI model using TensorFlow to generate navigation information, which includes important information on points of interest and how to respond when trouble occurs.

[0524] 4. Data storage and return

[0525] The generated navigation information is stored in a data storage device along with the identification information, and then sent back to the vehicle's onboard computer as an HTTP response, providing the necessary information for driving in real time.

[0526] Hardware and software used

[0527] Hardware:

[0528] On-board camera: a device that takes images of the environment

[0529] On-board computer: A device that manages the temporary storage and transmission of data.

[0530] Storage device: Data storage device

[0531] software:

[0532] OpenCV: Image processing library

[0533] Tesseract OCR: Character Recognition Engine

[0534] TensorFlow: A framework for running generative AI models

[0535] HTTP client: Software that manages the sending and receiving of data

[0536] Specific examples

[0537] For example, when an autonomous vehicle approaches an intersection, a camera collects information on signs and traffic lights. When the user presses the "upload" button, the image data is sent to a server. The server processes the image and generates navigation information such as "Follow the signs near the intersection to confirm your direction of travel." This navigation information is sent back to the user's device in real time and displayed on the driver's seat display.

[0538] Prompt Sentence Examples

[0539] Below are some example prompts to input to the generative AI model:

[0540] (prompt statement)

[0541] Please create a navigation notebook based on this video data, including important points, optimal route information, and recovery procedures in case of trouble.

[0542] Video data:

[0543] 1. Video file name | / path / to / data / video1.mp4

[0544] 2. Video file name | / path / to / data / video2.mp4

[0545] The above is an embodiment of this system.

[0546] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0547] Step 1:

[0548] The terminal captures images of the environment using a camera mounted on the autonomous vehicle and temporarily stores them in its internal memory. It receives raw image data captured by the camera as input and obtains image data stored in temporary storage as output. Specific operations include compressing the captured images and transferring them to temporary storage.

[0549] Step 2:

[0550] When the user presses the "Upload" button, the device sends the saved image data to the server as an HTTP request. The image data in temporary storage is used as input, and the data sent to the server is obtained as output. Specifically, the device performs a process to format the image data into an HTTP request.

[0551] Step 3:

[0552] The server receives image data sent from the terminal and saves it in a storage device. It receives image data sent from the terminal as input and obtains the saved data as output. Specifically, the server safely stores the data it receives in storage.

[0553] Step 4:

[0554] The server performs preprocessing on the stored image data using OpenCV. It receives the stored image data as input and obtains preprocessed data as output. Specifically, this includes processes such as grayscale conversion and noise removal.

[0555] Step 5:

[0556] The preprocessed image data is input into Tesseract OCR to extract text information. The preprocessed image data is used as input, and extracted text information is obtained as output. Specifically, the text in the image is analyzed and extracted as character data.

[0557] Step 6:

[0558] The extracted text information is input into a generative AI model using TensorFlow to generate navigation information. The extracted text information is used as input, and generated navigation information is obtained as output. Specific operations include the generative AI model analyzing the input data and generating the necessary information.

[0559] Step 7:

[0560] The generated navigation information is stored in a data storage device together with the identification information. The generated navigation information and the identification information are used as input, and information stored in a database is obtained as output. Specific operations include formatting the information into a database format.

[0561] Step 8:

[0562] The server returns the saved navigation information to the terminal as an HTTP response. It uses the navigation information saved in the database as input and obtains the data sent to the terminal as output. Specific operations include formatting the navigation information as an HTTP response and sending it to the terminal.

[0563] Step 9:

[0564] The terminal displays the received navigation information to the user. It uses the navigation information received from the server as input and obtains the information displayed on the user display as output. Specific operations include visually displaying the navigation information.

[0565] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0566] This invention is a system that allows users to take images of educational situations using devices such as smartphones or tablets, upload the image data to a server, automatically generate class notes, and further recognize the user's emotions and reflect the results in the notes, thereby improving the quality and efficiency of users' learning.

[0567] Client side (terminal)

[0568] Taking and saving photos

[0569] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[0570] Prepare to upload

[0571] The device moves the saved photos to the designated temporary upload storage, and then displays an "Upload" button to the user, giving them the option to send the photos to the server.

[0572] Upload Process

[0573] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0574] View notes

[0575] When the device receives the class notes returned from the server, it provides the user with an interface to display the notes. Through this interface, the user can view the generated notes and use them for study or review. In addition, the user's emotional information analyzed by the emotion engine is also reflected in the notes.

[0576] Server side

[0577] Receiving and saving images

[0578] The server receives the photo data sent from the device and temporarily stores it in the server's storage.

[0579] Image analysis and OCR processing

[0580] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used in this process, specifically by pre-processing the images (converting them to grayscale and removing noise) and then applying character recognition algorithms.

[0581] Emotion recognition by emotion engine

[0582] The server further executes an emotion engine that recognizes the user's emotion from image and text information, and collects the user's emotion data to identify the user's emotional state, such as relaxed, focused, or distressed.

[0583] Note generation

[0584] The server automatically generates class notes using a generation AI based on the extracted text information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes.

[0585] Saving and returning to the database

[0586] The generated class notes and the user's emotion data are linked to the user ID and stored in a database. Furthermore, an HTTP response is created and sent to return the generated notes to the device.

[0587] Specific examples

[0588] For example, imagine a user is taking a chemistry class and takes a photo of a chemical formula or explanation written on the blackboard with their smartphone. When the user takes a photo and presses the "upload" button, the photo is sent to the server. The server analyzes the received photo and uses OCR technology to extract the chemical formula and text. Next, the generative AI creates notes based on the extracted text data. Specifically, it formats the chemical formula, adds paragraphs, and emphasizes important points.

[0589] Furthermore, the emotion engine recognizes the user's emotions. For example, if the user feels "difficult to understand," the engine reflects this information in the notes. This information is recorded in a specific place in the notes, allowing the user to identify which part of the text they found difficult. This information is also stored in a database for future reference.

[0590] In this way, this system efficiently analyzes images of educational scenes taken by the user, automatically generates lesson notes, and uses an emotion engine to reflect the user's emotional information in the notes, thereby significantly improving learning efficiency.

[0591] The processing flow will be explained below.

[0592] Step 1:

[0593] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[0594] Step 2:

[0595] The device moves the saved photos to the specified temporary upload storage. The device displays an "Upload" button, giving the user the option to send the photos to the server.

[0596] Step 3:

[0597] The user presses the "Upload" button. The device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0598] Step 4:

[0599] The server receives the HTTP request sent from the device, analyzes the photo data in the request, and temporarily stores it.

[0600] Step 5:

[0601] The server applies image analysis algorithms to the stored photos, including pre-processing the images (grayscale conversion and noise reduction).

[0602] Step 6:

[0603] The server uses OCR technology to extract text from the image, which involves detecting text regions and running character recognition algorithms.

[0604] Step 7:

[0605] The server sends the extracted text data to the emotion engine, which evaluates the user's emotional state (e.g., relaxed, focused, or distressed) based on the analysis of the text and images. Specific operations include performing text analysis and facial recognition technology.

[0606] Step 8:

[0607] The server then sends the text data to a generation AI, which automatically generates class notes. The generation AI formats the received text data into notes, adding headings and bullet points. Emotion data from the emotion engine is also reflected in the notes.

[0608] Step 9:

[0609] The server associates the generated class notes with the user's emotion data and stores them in a database. The server then returns the generated notes to the device as an HTTP response.

[0610] Step 10:

[0611] The device analyzes the HTTP response received from the server and extracts the generated note data and emotional information. The device then displays the note on the user interface, along with emotional information such as which parts the user found difficult to understand.

[0612] Example 2

[0613] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] In modern education, detailed notes are necessary to effectively review the content of lessons, but traditional methods are time-consuming and burdensome for students. Furthermore, there is a lack of means to grasp students' understanding and emotions in real time and reflect them in their notes, making it difficult to optimize learning outcomes. Therefore, there is a need for the development of a system that can efficiently generate lesson notes and also reflect students' emotional information.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0616] In this invention, the server includes means for receiving images of educational scenes taken by a user, means for performing image analysis on the received images and extracting text information, means for automatically generating class notes based on the extracted text information, means for returning the generated class notes to the user terminal, and means for recognizing the user's emotions and reflecting the emotional information in the class notes. This enables the automatic generation of class notes and the reflection of emotional information, allowing students to efficiently review the content of lessons and improving learning effectiveness.

[0617] "User" refers to an individual who uses this system to capture images of educational situations and generate lesson notes.

[0618] "Educational setting" refers to a situation in which learning or education is taking place, such as in classes, lectures, or training.

[0619] "Images" refers to photographs or scanned data of educational scenes taken by users.

[0620] "Image analysis" refers to the process of extracting text information from received image data.

[0621] "Text information" refers to character data or text data extracted by image analysis.

[0622] "Class notes" refers to documents that record learning content and are automatically generated based on extracted text information.

[0623] "Means for receiving" refers to the function of the server to receive and store image data sent from the user terminal.

[0624] "Means for extracting" refers to the function of extracting text information from an image through image analysis.

[0625] "Means of automatic generation" refers to algorithms or programs that create class notes based on extracted text information.

[0626] "Means for returning" refers to the function of sending the generated lesson notes to the user's terminal and displaying them.

[0627] "Means for recognizing emotions" refers to a function that analyzes the user's emotional state and collects that information.

[0628] "Means for reflecting emotional information" refers to the function of incorporating the recognized emotions of the user into class notes.

[0629] "Temporary storage means" refers to a function for temporarily storing images taken by a user terminal and later transmitting them to a server.

[0630] "Means for uploading" refers to a function for sending image data from a user terminal to a server.

[0631] "Means of saving to a database" refers to the function for associating generated class notes with user IDs and storing them for a long period of time.

[0632] This invention is a system that allows users to take images of educational situations using devices such as smartphones or tablets, upload the image data to a server, automatically generate class notes, recognize the user's emotions, and reflect the results in the notes. This system can improve the quality and efficiency of users' learning.

[0633] Hardware and Software Configuration

[0634] Device: The user captures the classroom scene using a smartphone or tablet (iOS, Android). The device includes a camera, memory, and network module.

[0635] Server: A server with high-performance storage and processing power. Key software includes image analysis algorithms, OCR technology, emotion recognition engines, and generative AI models.

[0636] Image analysis algorithms: Libraries such as Python and OpenCV are used to extract text information from images.

[0637] OCR technology: OCR software such as Tesseract is used to extract text.

[0638] Emotion recognition engine: Sentiment analysis uses IBM Watson and Microsoft Azure's Emotion API.

[0639] Generative AI models: Generative AI models such as GPT-3 are used to format text data into notes.

[0640] Overview of system processing

[0641] Client side (terminal)

[0642] Taking and saving photos: A user takes a photo of a class or training session using their smartphone camera. The device temporarily stores the photo in its internal memory. For example, a user may use their smartphone camera to take a photo of a chemical formula drawn on a blackboard during class.

[0643] Prepare for upload: The device moves the saved photos to the specified temporary upload storage, and displays an "Upload" button to the user, giving them the option to send the photos to the server. The moved photos are saved in " / storage / emulated / 0 / AppUploads / ".

[0644] Upload process: When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo. For example, an HTTP POST request is sent to "https: / / example.com / upload".

[0645] Displaying notes: The device receives the class note data returned from the server and provides the user with an interface for displaying the notes. Through this interface, the user can view the generated notes and use them for study or review. In addition, the user's emotional information analyzed by the emotion engine is also reflected in the notes.

[0646] Server side

[0647] Image reception and storage: The server receives the photo data sent from the device and temporarily stores it in the server's storage. The received image data is stored in " / var / www / uploads / ".

[0648] Image analysis and OCR processing: Image analysis algorithms are run on the server to extract text information from images. Specifically, the Python OCR library "Tesseract" is used to convert images into text data.

[0649] Emotion recognition using emotion engine: Implement an emotion engine that recognizes user emotions from images and text information. Call emotion recognition APIs (e.g., IBM Watson, Microsoft Azure Emotion API) to analyze emotions from the user's facial expressions and context.

[0650] Note generation: Class notes are automatically generated by a generation AI based on the extracted text information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the user's emotional information. For example, the AI ​​model can be given a prompt such as, "Please highlight the chemical formula and indicate any difficult parts to understand."

[0651] Saving to database and returning: The generated class notes and the user's emotion data are linked to the user ID and stored in a database. The generated notes are then returned to the device as an HTTP response. For example, the note data and user ID are saved in a MySQL database, and JSON data is returned as a response.

[0652] Specific examples

[0653] For example, suppose a user takes a photo of a chemical formula or explanation written on a blackboard during a chemistry class using their smartphone. After taking the photo and pressing the "upload" button, the photo is sent to a server. The server analyzes the received photo and extracts the chemical formula and text using OCR technology. Next, a generative AI creates class notes based on the extracted text data. Specifically, it formats the chemical formula, adds paragraphs, and emphasizes important points. Furthermore, an emotion engine recognizes the user's emotions and, for example, if the user finds the text difficult to understand, reflects that information in the notes. This information is recorded in specific places in the notes, allowing the user to identify which parts were difficult. This information is also stored in a database for future reference. This system efficiently analyzes images of educational situations taken by users, automatically generates class notes, and uses the emotion engine to reflect the user's emotional information in the notes, significantly improving learning efficiency.

[0654] Example prompts to input to the generative AI model

[0655] "Extract text information from this photo and format it into a note. In particular, highlight the chemical formula and important points. Also, reflect the user's sentiment that it is difficult to understand."

[0656] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0657] Step 1: Take and save a photo

[0658] User: Uses a camera app on a smartphone or tablet to capture educational scenes.

[0659] Terminal: The captured photo is temporarily saved in the internal memory. The input is an image taken by the user, and the output is an image file saved in the internal memory. Specifically, the user takes a photo of a chemical formula drawn on a blackboard with the smartphone camera, and the image file is saved in the " / storage / emulated / 0 / Pictures / " directory.

[0660] Step 2: Prepare to upload

[0661] On device: Moves saved photos to the specified temporary upload storage. The input is an image file stored in the internal memory, and the output is an image file moved to the upload storage.

[0662] On the device: The user is presented with an "Upload" button, giving them the option to send the photo to the server. Specifically, the image file is moved to the " / storage / emulated / 0 / AppUploads / " directory, and then the "Upload" button is displayed in the UI.

[0663] Step 3: Upload Process

[0664] User: Presses the "Upload" button.

[0665] Terminal: Create an HTTP request, attach the saved photo, and send it to the specified endpoint on the server. The input is the image file saved in the upload storage, and the output is an HTTP request to the server. Specifically, an HTTP POST request is sent to "https: / / example.com / upload" with the photo data attached in multipart format.

[0666] Step 4: Receive and save images

[0667] Server: Receives photo data sent from the device. The received data is temporarily stored in the server's storage. The input is the image data sent from the device, and the output is an image file stored in the server's storage. Specifically, the image file is saved in the " / var / www / uploads / " directory.

[0668] Step 5: Image analysis and OCR processing

[0669] Server: Executes image analysis algorithms and extracts text information from images. After preprocessing the image (grayscale conversion and noise removal), character recognition is performed using OCR technology. The input is the received image file, and the output is the extracted text data. Specifically, the image is converted to text data using the Python OCR library "Tesseract."

[0670] Step 6: Emotion Recognition with the Emotion Engine

[0671] Server: Runs the emotion engine that recognizes the user's emotions from image and text information. The emotion engine collects the user's emotional data and identifies their emotional state, such as relaxed, focused, or difficult. The input is image and text data, and the output is the recognized emotion data. Specifically, it calls an emotion recognition API (e.g., IBM Watson, Microsoft Azure Emotion API) to analyze emotions from the user's facial expressions and context.

[0672] Step 7: Generate notes

[0673] Server: Class notes are automatically generated using a generative AI based on the extracted text information. The generative AI formats the text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes. The input is text data and emotion data, and the output is the generated class notes. Specifically, it uses a generative AI model such as GPT-3 to generate notes based on the input prompt.

[0674] Step 8: Saving and returning to the database

[0675] Server: The generated class notes and the user's emotion data are linked to the user ID and stored in a database. In addition, an HTTP response is created and sent to return the generated notes to the device. The input is the generated class notes and the user ID, and the output is the information stored in the database and the HTTP response for return. Specifically, the server saves the note data and the corresponding user ID in a MySQL database, and returns JSON data as a response to the device.

[0676] Step 9: Viewing Notes

[0677] Device: Receives class notes returned from the server and provides the user with an interface to display the notes. The input is the HTTP response from the server, and the output is the displayed note information. Specifically, the device analyzes the note data received in JSON format and formats it in an easy-to-read format within the app before displaying it.

[0678] (Application example 2)

[0679] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] In modern factories, product quality control and improved work efficiency are extremely important issues. However, traditional manual inspection records and worker emotional management are cumbersome and inefficient. Furthermore, because workers' emotional states are not reflected in inspection records, they often do not lead to improvements in the work environment. For this reason, there was a need for a system that automatically analyzes product images, generates accurate inspection records, and also reflects workers' emotional states.

[0681] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving images, means for performing image analysis and extracting text information, means for automatically generating notes, and means for recognizing the user's emotions and reflecting the information in the notes. This enables efficient quality control of products in factories, and further enables improvements to the work environment and early detection of problems by reflecting the emotional information of workers.

[0682] The "means for receiving images" refers to a function or process by which a user sends an image taken by the user to the server and the server receives the image.

[0683] "Means for performing image analysis and extracting text information" refers to a function or process that uses digital image processing technology on a received image to identify text contained in the image and extract it as characters.

[0684] "Means for automatically generating notes" refers to functions and processes for automatically generating structured notes based on extracted text information.

[0685] "Means for recognizing the user's emotions and reflecting the information in the notebook" refers to the functions and processes for analyzing the user's emotional state from images and surrounding data and reflecting the results in the notebook. For example, it could identify areas where the user is having difficulty and record that information in the notebook.

[0686] "Inside the factory" refers to the work area where a series of operations such as product manufacturing, assembly, inspection, and quality control are carried out.

[0687] "Quality control" refers to activities to check and ensure that the manufacturing process and finished products comply with prescribed standards and specifications.

[0688] "Working efficiently" means carrying out work in a way that maximizes effectiveness while minimizing the use of time and resources.

[0689] "Reflecting the emotional information of workers" means understanding the emotional state of workers and providing feedback to improve the work environment and methods based on that information.

[0690] "Improving the working environment" refers to activities that improve the physical and psychological working environment so that workers can work safely and efficiently.

[0691] "Early problem detection" means detecting product defects or problems with work procedures in advance and responding quickly.

[0692] The system for carrying out the present invention is for supporting quality control within a factory. Specific embodiments and processes thereof will be described below.

[0693] Client side (terminal)

[0694] 1. Taking and saving images

[0695] A robot in the factory takes pictures of the products, and the device temporarily stores the pictures in its internal memory.

[0696] 2. Prepare for upload

[0697] The device moves the saved image to a predetermined temporary storage location and then displays an "Upload" button to the user, providing them with the option to send the image to the server.

[0698] 3. Upload Process

[0699] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved image.

[0700] Server side

[0701] 1. Receiving and saving images

[0702] The server receives the image data sent from the device and temporarily stores the received data in the server's storage.

[0703] 2. Image analysis and OCR processing

[0704] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used here, specifically image preprocessing (grayscale conversion and noise removal) and text recognition algorithms are applied. Specific software includes OpenCV and Tesseract OCR.

[0705] 3. Emotion Recognition by Emotion Engine

[0706] The server also runs an emotion engine that recognizes user emotions from images and text information. The emotion engine collects the worker's emotional data and identifies their emotional states, such as relaxed, focused, or difficult. The emotion recognition model used here is trained with Keras.

[0707] 4. Creating Notes

[0708] The server automatically generates quality inspection notes using a generation AI based on the extracted text and emotion information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes.

[0709] 5. Saving to database and returning

[0710] The generated quality inspection note and emotion information are linked to the user ID and stored in a database. Furthermore, an HTTP response is created and sent to return the generated note to the device.

[0711] Specific examples

[0712] For example, a robot in a factory takes a picture of a product and automatically generates a quality inspection note based on that image. Text information (such as product dimensions and surface condition) is extracted from the captured image using OCR technology, and an emotion recognition model is used to recognize the worker's emotional state (such as "difficulty"). The generation AI uses this information to format the quality inspection note, creating a note that highlights the points where the worker found it difficult.

[0713] Prompt Sentence Examples

[0714] For example, the following prompt sentence could be input to a generative AI model:

[0715] prompt:

[0716] Please format the following quality inspection record as an inspection note and reflect the worker's emotional information.

[0717] Product condition: Surface scratches

[0718] Test result: Fail

[0719] Quality information: Further inspection required

[0720] Worker Emotions: Difficulty

[0721] This automatically generates quality inspection notes that are easy for users to understand, and provides information that is useful for improving the work environment and enhancing quality.

[0722] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0723] Step 1:

[0724] A robot takes images of products in a factory. Specifically, it acquires image data of the products using a camera mounted on the robot. The input is the physical appearance of the product, and the output is image data (JPEG or PNG format).

[0725] Step 2:

[0726] The captured image data is temporarily saved in the device's internal memory. Specifically, the image file is saved in the device's storage and its file path is recorded. The input is the image data captured in step 1, and the output is the file path of the saved image file.

[0727] Step 3:

[0728] The user presses the "Upload" button to send image data to the server. Specifically, the device creates an HTTP request, attaches the saved image file, and sends it to the specified endpoint on the server. The input is the file path of the saved image file, and the output is an HTTP request to the server.

[0729] Step 4:

[0730] The server receives image data sent from the terminal and temporarily stores the image data in the server's storage. The input is an HTTP request, and the output is an image file stored on the server.

[0731] Step 5:

[0732] The server runs an image analysis algorithm to extract text information from the image. Specifically, the image is converted to grayscale, noise is removed, and then Tesseract OCR is used to recognize text in the image. The input is the saved image file, and the output is the extracted text information.

[0733] Step 6:

[0734] The server uses an emotion recognition engine to analyze the user's emotional state. Specifically, it detects the worker's face in the image and labels the emotion using an emotion recognition model trained with Keras. The input is the saved image file, and the output is the recognized emotional state information.

[0735] Step 7:

[0736] The server uses the generative AI model to generate a quality inspection note based on the extracted text information and analyzed emotion information. Specifically, a prompt sentence is input to the generative AI model, and note-format text is output. The input is text information and emotion information, and the output is the generated quality inspection note.

[0737] Step 8:

[0738] The server associates the generated quality inspection note with the user ID and stores it in the database. Specifically, it associates the note text with the user ID, registers it in the database, and also stores the necessary metadata. The input is the generated quality inspection note and the user ID, and the output is the note record stored in the database.

[0739] Step 9:

[0740] The server returns the generated note to the terminal as an HTTP response. Specifically, it creates an HTTP response and sends it with the text of the generated note included as a payload. The input is the generated quality inspection note, and the output is the HTTP response.

[0741] Step 10:

[0742] The quality inspection note received by the terminal is displayed through the user interface. Specifically, the note text is displayed on the screen so that the user can check the contents. The input is the note text received as an HTTP response, and the output is the inspection note displayed on the terminal screen.

[0743] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0744] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0745] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0746] [Third embodiment]

[0747] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0748] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0749] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0750] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0751] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0752] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0753] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0754] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0755] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0756] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0757] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0758] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0759] This invention is a system that automatically generates lesson notes by allowing a user to take images of a teaching scene using a device such as a smartphone or tablet and upload the images to a server. A specific embodiment of this system will be described here.

[0760] Client side (terminal)

[0761] Taking and saving photos

[0762] A user takes a picture of a class or training session using a smartphone camera, and the device temporarily stores the photograph in the smartphone's internal memory.

[0763] Prepare to upload

[0764] The device moves the captured photo to the designated temporary storage for uploading, and then displays an "Upload" button to the user, offering them the option to send the photo to the server.

[0765] Upload Process

[0766] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0767] View notes

[0768] When the terminal receives the class notes returned from the server, it provides the user with an interface that displays the notes. Through this interface, the user can view the generated notes and use them for study or review.

[0769] Server side

[0770] Receiving and saving images

[0771] The server receives the photo data sent from the device and temporarily stores it in the server's storage.

[0772] Image analysis and OCR processing

[0773] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used in this process, specifically by pre-processing the images (converting them to grayscale and removing noise) and then applying character recognition algorithms.

[0774] Note generation

[0775] The server passes the extracted text information to a generation AI, which then automatically generates class notes. The generation AI then formats the received text data into notes, adding headings and bullet points.

[0776] Saving and returning to the database

[0777] The generated class notes are linked to the user ID and stored in a database. An HTTP response is also created and sent to return the generated notes to the device.

[0778] Specific examples

[0779] For example, imagine a user taking a picture of a formula or explanation written on a blackboard during a math class with their smartphone. When the user takes a photo and presses the "upload" button, the photo is sent to a server. The server analyzes the received photo and uses OCR technology to extract the formula and text. Next, the generative AI creates notes based on the extracted text data. Specifically, it formats the formula, adds paragraphs, and emphasizes important points.

[0780] The generated math notes are stored in a database and then sent back to the device. The user can view the notes on their smartphone app and use them for review or summarizing after class.

[0781] In this way, this system efficiently analyzes images of educational scenes taken by the user and automatically generates and provides lesson notes, thereby significantly improving learning efficiency.

[0782] The processing flow will be explained below.

[0783] Step 1:

[0784] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[0785] Step 2:

[0786] The device moves the saved photos to the specified temporary upload storage. The device displays an "Upload" button, giving the user the option to send the photos to the server.

[0787] Step 3:

[0788] The user presses the "Upload" button. The device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0789] Step 4:

[0790] The server receives the HTTP request sent from the device, analyzes the photo data in the request, and temporarily stores it.

[0791] Step 5:

[0792] The server applies image analysis algorithms to the stored photos, including pre-processing the images (grayscale conversion and noise reduction).

[0793] Step 6:

[0794] The server uses OCR technology to extract text from the image, which involves detecting text regions and running character recognition algorithms.

[0795] Step 7:

[0796] The server sends the extracted text data to the generation AI, which analyzes the text data and automatically generates class notes. Specific operations include formatting the text, adding headings, and generating bullet points.

[0797] Step 8:

[0798] The server associates the generated class notes with the user ID and stores them in a database. The server then returns the generated notes to the device as an HTTP response.

[0799] Step 9:

[0800] The device analyzes the HTTP response received from the server, extracts the generated note data, and displays the note on the user interface.

[0801] Example 1

[0802] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0803] Traditional methods of taking class notes require students to take notes by hand during lectures, which requires a lot of time and effort. Furthermore, there is a risk of overlooking information or making typos, which can lead to reduced learning effectiveness. Furthermore, there are few efficient ways to digitize lecture content, making it difficult to search and share later.

[0804] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0805] In this invention, the server includes means for receiving images of educational scenes taken by a user, means for temporarily saving the images in a storage device of the terminal, means for uploading the saved images to the server, means for performing image analysis on the received images to extract text information, means for inputting the extracted text information into a generative AI model to automatically generate class notes, means for returning the generated class notes to the user terminal, and means for displaying the generated class notes. This makes it possible to automatically digitize lecture content and efficiently generate accurate class notes.

[0806] "User" refers to a person who uses this system to take images of educational scenes and generate notes.

[0807] "Terminal" refers to a device, such as a smartphone or tablet, that a user uses to take images and upload them to a server.

[0808] "Server" refers to a computer system that receives and analyzes image data sent from a terminal and has the function of generating notes and returning them to the user.

[0809] "Image analysis" refers to a series of processes in which a terminal or server processes image data received and extracts useful information from the image.

[0810] "Text information" refers to information such as character data and mathematical formulas extracted from an image through image analysis.

[0811] "Generative AI model" refers to the artificial intelligence algorithm used to automatically generate notes based on text information.

[0812] A "prompt sentence" refers to an input sentence that instructs the generative AI model on what kind of note to generate.

[0813] "Class notes" refers to documents that organize the contents of lectures and are automatically generated by a generative AI model based on images of educational scenes taken by users.

[0814] "Storage device" refers to hardware for temporarily or long-term storage of image data and generated note data.

[0815] "Upload" refers to the process by which a device sends stored image data to a server.

[0816] A "user identifier" is information used to uniquely identify a user and is used to associate the user with the notes created in the database.

[0817] An "end-user terminal" refers to a terminal that is directly operated by a user and is responsible for taking images and displaying notes.

[0818] This invention is a system that automatically generates lesson notes by allowing a user to take images of a teaching scene using a device such as a smartphone or tablet and upload the images to a server. A specific embodiment of this system will be described here.

[0819] Client side (terminal)

[0820] 1. Taking and saving photos

[0821] Users use their smartphone camera to take pictures of classes and training sessions, and the photos are first temporarily saved in the device's internal memory.

[0822] 2. Prepare for upload

[0823] The device moves the temporarily saved photo data to temporary storage for uploading, and then the application displays an "Upload" button to the user, which the user can press to prepare the photo data for transmission to the server.

[0824] 3. Upload Process

[0825] When a user presses the "Upload" button, the device creates an HTTP request, attaches the saved photo, and sends it to the server. This process uses the smartphone's communication function and specifies the destination endpoint URL.

[0826] 4. View notes

[0827] The terminal receives the class notes returned from the server and provides an interface that allows users to view the notes. Users can use this interface to view the generated notes and use them for study or review.

[0828] Server side

[0829] 1. Receiving and saving images

[0830] The server receives the photo data sent from the device and temporarily stores it in the server's storage. The format and size of the received data are verified.

[0831] 2. Image analysis and OCR processing

[0832] The server analyzes the stored photo data and uses OCR technology to extract text information from the images. Specifically, the images are pre-processed (by converting them to grayscale and removing noise), and then a character recognition algorithm is applied.

[0833] 3. Creating notes

[0834] The server inputs the text information extracted by OCR into a generative AI model, which then automatically generates class notes. The generative AI model then formats the extracted text data into notes, adding headings and bullet points.

[0835] 4. Saving to database and returning

[0836] The generated class notes are linked to the user ID and stored in a database. An HTTP response is also created and sent to return the generated notes to the device.

[0837] Specific examples

[0838] For example, consider the case where a user takes a photo of a math class using their smartphone. During class, the user takes a photo of the formulas and explanations on the blackboard with their smartphone and presses the "upload" button, sending the photo to a server. The server analyzes the received photo and uses OCR technology to extract the formulas and characters. Next, a generative AI model creates class notes based on the extracted text data. Specifically, it formats formulas, adds paragraphs, and emphasizes important points.

[0839] The generated math notes are stored in a database and then sent back to the device, where users can view them using a smartphone application and use them for review or summarizing after class.

[0840] Examples of prompt statements

[0841] An example of a prompt sentence to input to the generative AI model is, "Please extract text data from the photo below and create class notes." By inputting this prompt sentence and the text data extracted by OCR into the AI ​​model, specific class notes are generated.

[0842] In this way, this system efficiently analyzes images of educational scenes taken by the user and automatically generates and provides lesson notes, thereby significantly improving learning efficiency.

[0843] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0844] Step 1:

[0845] A user takes a picture of a class or training session using the smartphone camera. The captured image is the input, and the device temporarily stores this image in its internal memory. Specifically, the smartphone's camera app accepts the operation of the capture button and saves the captured image data in a temporary directory within the camera app.

[0846] Step 2:

[0847] The device moves images stored in internal memory to temporary storage for uploading. The input to this process is image data stored in internal memory, and the output is image data moved to temporary storage for uploading. Specifically, the file is moved within the application, and an "Upload" button is displayed on the screen.

[0848] Step 3:

[0849] When a user presses the "Upload" button in the application, the device creates an HTTP request and sends the image data to the server. The input to this process is the image data in temporary upload storage, and the output is the HTTP request sent to the server. Specifically, the application constructs an HTTP request in the background and sends the request with the image data attached in multipart format to the specified server endpoint.

[0850] Step 4:

[0851] The server receives image data sent from the device and temporarily stores it in the server's storage. The input to this process is the HTTP request, and the output is image data stored in the server's temporary storage. Specifically, the server analyzes the request and stores the received image data in a specific directory.

[0852] Step 5:

[0853] The server performs image analysis and OCR processing on the stored image data to extract text information. The input to this process is the stored image data, and the output is the extracted text information. Specifically, the OCR software preprocesses the image (such as grayscale conversion and noise removal) and applies character recognition algorithms to generate the text information.

[0854] Step 6:

[0855] The server inputs the extracted text information into a generative AI model to automatically generate class notes. The input for this process is the text information extracted by OCR, and the output is the generated class notes. Specifically, the AI ​​model (such as the GPT series) receives the prompt and text information and generates formatted class notes.

[0856] Step 7:

[0857] The server associates the generated class notes with the user ID, stores them in a database, and returns them to the user's device. The input to this process is the generated class notes and the user ID, and the output is storage in the database and an HTTP response to the device. Specifically, the server stores the notes in the database, creates an HTTP response, and sends it to the device.

[0858] Step 8:

[0859] The device receives the class notes returned from the server and provides an interface for the user to view them. The input to this process is the HTTP response returned from the server, and the output is the class notes displayed in the user interface. Specifically, the application analyzes the received data and renders the note contents in the display area within the app.

[0860] (Application example 1)

[0861] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0862] Autonomous vehicles face the challenge of efficiently analyzing the large amounts of video and sensor data collected during operation, and providing the navigation information necessary for operation and a rapid response method when a problem occurs. This will reduce the risk of accidents and improve passenger safety and vehicle efficiency.

[0863] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0864] In this invention, the server includes means for receiving images of the environment taken by a user, means for performing image processing on the received images to extract information, and means for automatically generating navigation information based on the extracted information. This makes it possible to quickly extract important information from the taken images and provide necessary navigation information and response methods to autonomously driven vehicles in real time.

[0865] "User" refers to the operator or manager of the autonomous vehicle.

[0866] "Environmental images" are video data of the surroundings collected by cameras mounted on autonomous vehicles.

[0867] "Image processing" is a technique for extracting necessary information from video data captured by a camera, and specifically includes grayscale conversion and noise removal.

[0868] "Information" refers to text data and important point information extracted through image processing.

[0869] "Navigation information" is data that includes route information useful for operating an autonomous vehicle and instructions on how to respond in the event of an accident.

[0870] A "terminal" is a device for using navigation information, specifically an in-vehicle computer or a smartphone.

[0871] "Identification information" is information for identifying an autonomous vehicle or a user.

[0872] "Data storage device" refers to a database or storage device installed within a server.

[0873] A "mobile terminal" is an information device that a user can carry around, and specifically includes a smartphone or tablet.

[0874] A "server" is a computer system that receives images, processes them, and generates the necessary information.

[0875] This invention is a system that efficiently processes video data captured by an autonomous vehicle while it is in operation, and generates and provides navigation information in real time. First, the system program will be explained.

[0876] Overview of the program

[0877] This system involves a series of steps: an in-vehicle camera collects images of the surrounding environment and sends them to a server, which processes the images, extracts the information, generates navigation information, and sends it back to the terminal.

[0878] Program processing overview

[0879] 1. Data collection and transmission

[0880] The device captures images of the environment using a camera mounted on the autonomous vehicle and temporarily stores them in its internal memory. At this point, the device compresses the image data as needed and moves it to temporary storage.

[0881] 2. Image analysis and information extraction on the server

[0882] The server receives the image data sent from the device and performs image processing. In this process, preprocessing (grayscale conversion, noise removal, etc.) is performed using OpenCV, and text information is extracted using Tesseract OCR. The extracted information is just raw data, so it needs to be processed in the next step.

[0883] 3. Generating navigation information

[0884] The server inputs the extracted text information into a generative AI model using TensorFlow to generate navigation information, which includes important information on points of interest and how to respond when trouble occurs.

[0885] 4. Data storage and return

[0886] The generated navigation information is stored in a data storage device along with the identification information, and then sent back to the vehicle's onboard computer as an HTTP response, providing the necessary information for driving in real time.

[0887] Hardware and software used

[0888] Hardware:

[0889] On-board camera: a device that takes images of the environment

[0890] On-board computer: A device that manages the temporary storage and transmission of data.

[0891] Storage device: Data storage device

[0892] software:

[0893] OpenCV: Image processing library

[0894] Tesseract OCR: Character Recognition Engine

[0895] TensorFlow: A framework for running generative AI models

[0896] HTTP client: Software that manages the sending and receiving of data

[0897] Specific examples

[0898] For example, when an autonomous vehicle approaches an intersection, a camera collects information on signs and traffic lights. When the user presses the "upload" button, the image data is sent to a server. The server processes the image and generates navigation information such as "Follow the signs near the intersection to confirm your direction of travel." This navigation information is sent back to the user's device in real time and displayed on the driver's seat display.

[0899] Prompt Sentence Examples

[0900] Below are some example prompts to input to the generative AI model:

[0901] (prompt statement)

[0902] Please create a navigation notebook based on this video data, including important points, optimal route information, and recovery procedures in case of trouble.

[0903] Video data:

[0904] 1. Video file name | / path / to / data / video1.mp4

[0905] 2. Video file name | / path / to / data / video2.mp4

[0906] The above is an embodiment of this system.

[0907] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0908] Step 1:

[0909] The terminal captures images of the environment using a camera mounted on the autonomous vehicle and temporarily stores them in its internal memory. It receives raw image data captured by the camera as input and obtains image data stored in temporary storage as output. Specific operations include compressing the captured images and transferring them to temporary storage.

[0910] Step 2:

[0911] When the user presses the "Upload" button, the device sends the saved image data to the server as an HTTP request. The image data in temporary storage is used as input, and the data sent to the server is obtained as output. Specifically, the device performs a process to format the image data into an HTTP request.

[0912] Step 3:

[0913] The server receives image data sent from the terminal and saves it in a storage device. It receives image data sent from the terminal as input and obtains the saved data as output. Specifically, the server safely stores the data it receives in storage.

[0914] Step 4:

[0915] The server performs preprocessing on the stored image data using OpenCV. It receives the stored image data as input and obtains preprocessed data as output. Specifically, this includes processes such as grayscale conversion and noise removal.

[0916] Step 5:

[0917] The preprocessed image data is input into Tesseract OCR to extract text information. The preprocessed image data is used as input, and extracted text information is obtained as output. Specifically, the text in the image is analyzed and extracted as character data.

[0918] Step 6:

[0919] The extracted text information is input into a generative AI model using TensorFlow to generate navigation information. The extracted text information is used as input, and generated navigation information is obtained as output. Specific operations include the generative AI model analyzing the input data and generating the necessary information.

[0920] Step 7:

[0921] The generated navigation information is stored in a data storage device together with the identification information. The generated navigation information and the identification information are used as input, and information stored in a database is obtained as output. Specific operations include formatting the information into a database format.

[0922] Step 8:

[0923] The server returns the saved navigation information to the terminal as an HTTP response. It uses the navigation information saved in the database as input and obtains the data sent to the terminal as output. Specific operations include formatting the navigation information as an HTTP response and sending it to the terminal.

[0924] Step 9:

[0925] The terminal displays the received navigation information to the user. It uses the navigation information received from the server as input and obtains the information displayed on the user display as output. Specific operations include visually displaying the navigation information.

[0926] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0927] This invention is a system that allows users to take images of educational situations using devices such as smartphones or tablets, upload the image data to a server, automatically generate class notes, and further recognize the user's emotions and reflect the results in the notes, thereby improving the quality and efficiency of users' learning.

[0928] Client side (terminal)

[0929] Taking and saving photos

[0930] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[0931] Prepare to upload

[0932] The device moves the saved photos to the designated temporary upload storage, and then displays an "Upload" button to the user, giving them the option to send the photos to the server.

[0933] Upload Process

[0934] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0935] View notes

[0936] When the device receives the class notes returned from the server, it provides the user with an interface to display the notes. Through this interface, the user can view the generated notes and use them for study or review. In addition, the user's emotional information analyzed by the emotion engine is also reflected in the notes.

[0937] Server side

[0938] Receiving and saving images

[0939] The server receives the photo data sent from the device and temporarily stores it in the server's storage.

[0940] Image analysis and OCR processing

[0941] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used in this process, specifically by pre-processing the images (converting them to grayscale and removing noise) and then applying character recognition algorithms.

[0942] Emotion recognition by emotion engine

[0943] The server further executes an emotion engine that recognizes the user's emotion from image and text information, and collects the user's emotion data to identify the user's emotional state, such as relaxed, focused, or distressed.

[0944] Note generation

[0945] The server automatically generates class notes using a generation AI based on the extracted text information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes.

[0946] Saving and returning to the database

[0947] The generated class notes and the user's emotion data are linked to the user ID and stored in a database. Furthermore, an HTTP response is created and sent to return the generated notes to the device.

[0948] Specific examples

[0949] For example, imagine a user is taking a chemistry class and takes a photo of a chemical formula or explanation written on the blackboard with their smartphone. When the user takes a photo and presses the "upload" button, the photo is sent to the server. The server analyzes the received photo and uses OCR technology to extract the chemical formula and text. Next, the generative AI creates notes based on the extracted text data. Specifically, it formats the chemical formula, adds paragraphs, and emphasizes important points.

[0950] Furthermore, the emotion engine recognizes the user's emotions. For example, if the user feels "difficult to understand," the engine reflects this information in the notes. This information is recorded in a specific place in the notes, allowing the user to identify which part of the text they found difficult. This information is also stored in a database for future reference.

[0951] In this way, this system efficiently analyzes images of educational scenes taken by the user, automatically generates lesson notes, and uses an emotion engine to reflect the user's emotional information in the notes, thereby significantly improving learning efficiency.

[0952] The processing flow will be explained below.

[0953] Step 1:

[0954] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[0955] Step 2:

[0956] The device moves the saved photos to the specified temporary upload storage. The device displays an "Upload" button, giving the user the option to send the photos to the server.

[0957] Step 3:

[0958] The user presses the "Upload" button. The device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[0959] Step 4:

[0960] The server receives the HTTP request sent from the device, analyzes the photo data in the request, and temporarily stores it.

[0961] Step 5:

[0962] The server applies image analysis algorithms to the stored photos, including pre-processing the images (grayscale conversion and noise reduction).

[0963] Step 6:

[0964] The server uses OCR technology to extract text from the image, which involves detecting text regions and running character recognition algorithms.

[0965] Step 7:

[0966] The server sends the extracted text data to the emotion engine, which evaluates the user's emotional state (e.g., relaxed, focused, or distressed) based on the analysis of the text and images. Specific operations include performing text analysis and facial recognition technology.

[0967] Step 8:

[0968] The server then sends the text data to a generation AI, which automatically generates class notes. The generation AI formats the received text data into notes, adding headings and bullet points. Emotion data from the emotion engine is also reflected in the notes.

[0969] Step 9:

[0970] The server associates the generated class notes with the user's emotion data and stores them in a database. The server then returns the generated notes to the device as an HTTP response.

[0971] Step 10:

[0972] The device analyzes the HTTP response received from the server and extracts the generated note data and emotional information. The device then displays the note on the user interface, along with emotional information such as which parts the user found difficult to understand.

[0973] Example 2

[0974] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0975] In modern education, detailed notes are necessary to effectively review the content of lessons, but traditional methods are time-consuming and burdensome for students. Furthermore, there is a lack of means to grasp students' understanding and emotions in real time and reflect them in their notes, making it difficult to optimize learning outcomes. Therefore, there is a need for the development of a system that can efficiently generate lesson notes and also reflect students' emotional information.

[0976] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0977] In this invention, the server includes means for receiving images of educational scenes taken by a user, means for performing image analysis on the received images and extracting text information, means for automatically generating class notes based on the extracted text information, means for returning the generated class notes to the user terminal, and means for recognizing the user's emotions and reflecting the emotional information in the class notes. This enables the automatic generation of class notes and the reflection of emotional information, allowing students to efficiently review the content of lessons and improving learning effectiveness.

[0978] "User" refers to an individual who uses this system to capture images of educational situations and generate lesson notes.

[0979] "Educational setting" refers to a situation in which learning or education is taking place, such as in classes, lectures, or training.

[0980] "Images" refers to photographs or scanned data of educational scenes taken by users.

[0981] "Image analysis" refers to the process of extracting text information from received image data.

[0982] "Text information" refers to character data or text data extracted by image analysis.

[0983] "Class notes" refers to documents that record learning content and are automatically generated based on extracted text information.

[0984] "Means for receiving" refers to the function of the server to receive and store image data sent from the user terminal.

[0985] "Means for extracting" refers to the function of extracting text information from an image through image analysis.

[0986] "Means of automatic generation" refers to algorithms or programs that create class notes based on extracted text information.

[0987] "Means for returning" refers to the function of sending the generated lesson notes to the user's terminal and displaying them.

[0988] "Means for recognizing emotions" refers to a function that analyzes the user's emotional state and collects that information.

[0989] "Means for reflecting emotional information" refers to the function of incorporating the recognized emotions of the user into class notes.

[0990] "Temporary storage means" refers to a function for temporarily storing images taken by a user terminal and later transmitting them to a server.

[0991] "Means for uploading" refers to a function for sending image data from a user terminal to a server.

[0992] "Means of saving to a database" refers to the function for associating generated class notes with user IDs and storing them for a long period of time.

[0993] This invention is a system that allows users to take images of educational situations using devices such as smartphones or tablets, upload the image data to a server, automatically generate class notes, recognize the user's emotions, and reflect the results in the notes. This system can improve the quality and efficiency of users' learning.

[0994] Hardware and Software Configuration

[0995] Device: The user captures the classroom scene using a smartphone or tablet (iOS, Android). The device includes a camera, memory, and network module.

[0996] Server: A server with high-performance storage and processing power. Key software includes image analysis algorithms, OCR technology, emotion recognition engines, and generative AI models.

[0997] Image analysis algorithms: Libraries such as Python and OpenCV are used to extract text information from images.

[0998] OCR technology: OCR software such as Tesseract is used to extract text.

[0999] Emotion recognition engine: Sentiment analysis uses IBM Watson and Microsoft Azure's Emotion API.

[1000] Generative AI models: Generative AI models such as GPT-3 are used to format text data into notes.

[1001] Overview of system processing

[1002] Client side (terminal)

[1003] Taking and saving photos: A user takes a photo of a class or training session using their smartphone camera. The device temporarily stores the photo in its internal memory. For example, a user may use their smartphone camera to take a photo of a chemical formula drawn on a blackboard during class.

[1004] Prepare for upload: The device moves the saved photos to the specified temporary upload storage, and displays an "Upload" button to the user, giving them the option to send the photos to the server. The moved photos are saved in " / storage / emulated / 0 / AppUploads / ".

[1005] Upload process: When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo. For example, an HTTP POST request is sent to "https: / / example.com / upload".

[1006] Displaying notes: The device receives the class note data returned from the server and provides the user with an interface for displaying the notes. Through this interface, the user can view the generated notes and use them for study or review. In addition, the user's emotional information analyzed by the emotion engine is also reflected in the notes.

[1007] Server side

[1008] Image reception and storage: The server receives the photo data sent from the device and temporarily stores it in the server's storage. The received image data is stored in " / var / www / uploads / ".

[1009] Image analysis and OCR processing: Image analysis algorithms are run on the server to extract text information from images. Specifically, the Python OCR library "Tesseract" is used to convert images into text data.

[1010] Emotion recognition using emotion engine: Implement an emotion engine that recognizes user emotions from images and text information. Call emotion recognition APIs (e.g., IBM Watson, Microsoft Azure Emotion API) to analyze emotions from the user's facial expressions and context.

[1011] Note generation: Class notes are automatically generated by a generation AI based on the extracted text information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the user's emotional information. For example, the AI ​​model can be given a prompt such as, "Please highlight the chemical formula and indicate any difficult parts to understand."

[1012] Saving to database and returning: The generated class notes and the user's emotion data are linked to the user ID and stored in a database. The generated notes are then returned to the device as an HTTP response. For example, the note data and user ID are saved in a MySQL database, and JSON data is returned as a response.

[1013] Specific examples

[1014] For example, suppose a user takes a photo of a chemical formula or explanation written on a blackboard during a chemistry class using their smartphone. After taking the photo and pressing the "upload" button, the photo is sent to a server. The server analyzes the received photo and extracts the chemical formula and text using OCR technology. Next, a generative AI creates class notes based on the extracted text data. Specifically, it formats the chemical formula, adds paragraphs, and emphasizes important points. Furthermore, an emotion engine recognizes the user's emotions and, for example, if the user finds the text difficult to understand, reflects that information in the notes. This information is recorded in specific places in the notes, allowing the user to identify which parts were difficult. This information is also stored in a database for future reference. This system efficiently analyzes images of educational situations taken by users, automatically generates class notes, and uses the emotion engine to reflect the user's emotional information in the notes, significantly improving learning efficiency.

[1015] Example prompts to input to the generative AI model

[1016] "Extract text information from this photo and format it into a note. In particular, highlight the chemical formula and important points. Also, reflect the user's sentiment that it is difficult to understand."

[1017] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1018] Step 1: Take and save a photo

[1019] User: Uses a camera app on a smartphone or tablet to capture educational scenes.

[1020] Terminal: The captured photo is temporarily saved in the internal memory. The input is an image taken by the user, and the output is an image file saved in the internal memory. Specifically, the user takes a photo of a chemical formula drawn on a blackboard with the smartphone camera, and the image file is saved in the " / storage / emulated / 0 / Pictures / " directory.

[1021] Step 2: Prepare to upload

[1022] On device: Moves saved photos to the specified temporary upload storage. The input is an image file stored in the internal memory, and the output is an image file moved to the upload storage.

[1023] On the device: The user is presented with an "Upload" button, giving them the option to send the photo to the server. Specifically, the image file is moved to the " / storage / emulated / 0 / AppUploads / " directory, and then the "Upload" button is displayed in the UI.

[1024] Step 3: Upload Process

[1025] User: Presses the "Upload" button.

[1026] Terminal: Create an HTTP request, attach the saved photo, and send it to the specified endpoint on the server. The input is the image file saved in the upload storage, and the output is an HTTP request to the server. Specifically, an HTTP POST request is sent to "https: / / example.com / upload" with the photo data attached in multipart format.

[1027] Step 4: Receive and save images

[1028] Server: Receives photo data sent from the device. The received data is temporarily stored in the server's storage. The input is the image data sent from the device, and the output is an image file stored in the server's storage. Specifically, the image file is saved in the " / var / www / uploads / " directory.

[1029] Step 5: Image analysis and OCR processing

[1030] Server: Executes image analysis algorithms and extracts text information from images. After preprocessing the image (grayscale conversion and noise removal), character recognition is performed using OCR technology. The input is the received image file, and the output is the extracted text data. Specifically, the image is converted to text data using the Python OCR library "Tesseract."

[1031] Step 6: Emotion Recognition with the Emotion Engine

[1032] Server: Runs the emotion engine that recognizes the user's emotions from image and text information. The emotion engine collects the user's emotional data and identifies their emotional state, such as relaxed, focused, or difficult. The input is image and text data, and the output is the recognized emotion data. Specifically, it calls an emotion recognition API (e.g., IBM Watson, Microsoft Azure Emotion API) to analyze emotions from the user's facial expressions and context.

[1033] Step 7: Generate notes

[1034] Server: Class notes are automatically generated using a generative AI based on the extracted text information. The generative AI formats the text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes. The input is text data and emotion data, and the output is the generated class notes. Specifically, it uses a generative AI model such as GPT-3 to generate notes based on the input prompt.

[1035] Step 8: Saving and returning to the database

[1036] Server: The generated class notes and the user's emotion data are linked to the user ID and stored in a database. In addition, an HTTP response is created and sent to return the generated notes to the device. The input is the generated class notes and the user ID, and the output is the information stored in the database and the HTTP response for return. Specifically, the server saves the note data and the corresponding user ID in a MySQL database, and returns JSON data as a response to the device.

[1037] Step 9: Viewing Notes

[1038] Device: Receives class notes returned from the server and provides the user with an interface to display the notes. The input is the HTTP response from the server, and the output is the displayed note information. Specifically, the device analyzes the note data received in JSON format and formats it in an easy-to-read format within the app before displaying it.

[1039] (Application example 2)

[1040] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1041] In modern factories, product quality control and improved work efficiency are extremely important issues. However, traditional manual inspection records and worker emotional management are cumbersome and inefficient. Furthermore, because workers' emotional states are not reflected in inspection records, they often do not lead to improvements in the work environment. For this reason, there was a need for a system that automatically analyzes product images, generates accurate inspection records, and also reflects workers' emotional states.

[1042] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving images, means for performing image analysis and extracting text information, means for automatically generating notes, and means for recognizing the user's emotions and reflecting the information in the notes. This enables efficient quality control of products in factories, and further enables improvements to the work environment and early detection of problems by reflecting the emotional information of workers.

[1043] The "means for receiving images" refers to a function or process by which a user sends an image taken by the user to the server and the server receives the image.

[1044] "Means for performing image analysis and extracting text information" refers to a function or process that uses digital image processing technology on a received image to identify text contained in the image and extract it as characters.

[1045] "Means for automatically generating notes" refers to functions and processes for automatically generating structured notes based on extracted text information.

[1046] "Means for recognizing the user's emotions and reflecting the information in the notebook" refers to the functions and processes for analyzing the user's emotional state from images and surrounding data and reflecting the results in the notebook. For example, it could identify areas where the user is having difficulty and record that information in the notebook.

[1047] "Inside the factory" refers to the work area where a series of operations such as product manufacturing, assembly, inspection, and quality control are carried out.

[1048] "Quality control" refers to activities to check and ensure that the manufacturing process and finished products comply with prescribed standards and specifications.

[1049] "Working efficiently" means carrying out work in a way that maximizes effectiveness while minimizing the use of time and resources.

[1050] "Reflecting the emotional information of workers" means understanding the emotional state of workers and providing feedback to improve the work environment and methods based on that information.

[1051] "Improving the working environment" refers to activities that improve the physical and psychological working environment so that workers can work safely and efficiently.

[1052] "Early problem detection" means detecting product defects or problems with work procedures in advance and responding quickly.

[1053] The system for carrying out the present invention is for supporting quality control within a factory. Specific embodiments and processes thereof will be described below.

[1054] Client side (terminal)

[1055] 1. Taking and saving images

[1056] A robot in the factory takes pictures of the products, and the device temporarily stores the pictures in its internal memory.

[1057] 2. Prepare for upload

[1058] The device moves the saved image to a predetermined temporary storage location and then displays an "Upload" button to the user, providing them with the option to send the image to the server.

[1059] 3. Upload Process

[1060] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved image.

[1061] Server side

[1062] 1. Receiving and saving images

[1063] The server receives the image data sent from the device and temporarily stores the received data in the server's storage.

[1064] 2. Image analysis and OCR processing

[1065] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used here, specifically image preprocessing (grayscale conversion and noise removal) and text recognition algorithms are applied. Specific software includes OpenCV and Tesseract OCR.

[1066] 3. Emotion Recognition by Emotion Engine

[1067] The server also runs an emotion engine that recognizes user emotions from images and text information. The emotion engine collects the worker's emotional data and identifies their emotional states, such as relaxed, focused, or difficult. The emotion recognition model used here is trained with Keras.

[1068] 4. Creating Notes

[1069] The server automatically generates quality inspection notes using a generation AI based on the extracted text and emotion information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes.

[1070] 5. Saving to database and returning

[1071] The generated quality inspection note and emotion information are linked to the user ID and stored in a database. Furthermore, an HTTP response is created and sent to return the generated note to the device.

[1072] Specific examples

[1073] For example, a robot in a factory takes a picture of a product and automatically generates a quality inspection note based on that image. Text information (such as product dimensions and surface condition) is extracted from the captured image using OCR technology, and an emotion recognition model is used to recognize the worker's emotional state (such as "difficulty"). The generation AI uses this information to format the quality inspection note, creating a note that highlights the points where the worker found it difficult.

[1074] Prompt Sentence Examples

[1075] For example, the following prompt sentence could be input to a generative AI model:

[1076] prompt:

[1077] Please format the following quality inspection record as an inspection note and reflect the worker's emotional information.

[1078] Product condition: Surface scratches

[1079] Test result: Fail

[1080] Quality information: Further inspection required

[1081] Worker Emotions: Difficulty

[1082] This automatically generates quality inspection notes that are easy for users to understand, and provides information that is useful for improving the work environment and enhancing quality.

[1083] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1084] Step 1:

[1085] A robot takes images of products in a factory. Specifically, it acquires image data of the products using a camera mounted on the robot. The input is the physical appearance of the product, and the output is image data (JPEG or PNG format).

[1086] Step 2:

[1087] The captured image data is temporarily saved in the device's internal memory. Specifically, the image file is saved in the device's storage and its file path is recorded. The input is the image data captured in step 1, and the output is the file path of the saved image file.

[1088] Step 3:

[1089] The user presses the "Upload" button to send image data to the server. Specifically, the device creates an HTTP request, attaches the saved image file, and sends it to the specified endpoint on the server. The input is the file path of the saved image file, and the output is an HTTP request to the server.

[1090] Step 4:

[1091] The server receives image data sent from the terminal and temporarily stores the image data in the server's storage. The input is an HTTP request, and the output is an image file stored on the server.

[1092] Step 5:

[1093] The server runs an image analysis algorithm to extract text information from the image. Specifically, the image is converted to grayscale, noise is removed, and then Tesseract OCR is used to recognize text in the image. The input is the saved image file, and the output is the extracted text information.

[1094] Step 6:

[1095] The server uses an emotion recognition engine to analyze the user's emotional state. Specifically, it detects the worker's face in the image and labels the emotion using an emotion recognition model trained with Keras. The input is the saved image file, and the output is the recognized emotional state information.

[1096] Step 7:

[1097] The server uses the generative AI model to generate a quality inspection note based on the extracted text information and analyzed emotion information. Specifically, a prompt sentence is input to the generative AI model, and note-format text is output. The input is text information and emotion information, and the output is the generated quality inspection note.

[1098] Step 8:

[1099] The server associates the generated quality inspection note with the user ID and stores it in the database. Specifically, it associates the note text with the user ID, registers it in the database, and also stores the necessary metadata. The input is the generated quality inspection note and the user ID, and the output is the note record stored in the database.

[1100] Step 9:

[1101] The server returns the generated note to the terminal as an HTTP response. Specifically, it creates an HTTP response and sends it with the text of the generated note included as a payload. The input is the generated quality inspection note, and the output is the HTTP response.

[1102] Step 10:

[1103] The quality inspection note received by the terminal is displayed through the user interface. Specifically, the note text is displayed on the screen so that the user can check the contents. The input is the note text received as an HTTP response, and the output is the inspection note displayed on the terminal screen.

[1104] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1105] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1106] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1107] [Fourth embodiment]

[1108] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1109] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1110] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1111] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1112] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1113] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1114] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1115] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1116] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1117] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1118] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1119] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1120] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1121] This invention is a system that automatically generates lesson notes by allowing a user to take images of a teaching scene using a device such as a smartphone or tablet and upload the images to a server. A specific embodiment of this system will be described here.

[1122] Client side (terminal)

[1123] Taking and saving photos

[1124] A user takes a picture of a class or training session using a smartphone camera, and the device temporarily stores the photograph in the smartphone's internal memory.

[1125] Prepare to upload

[1126] The device moves the captured photo to the designated temporary storage for uploading, and then displays an "Upload" button to the user, offering them the option to send the photo to the server.

[1127] Upload Process

[1128] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[1129] View notes

[1130] When the terminal receives the class notes returned from the server, it provides the user with an interface that displays the notes. Through this interface, the user can view the generated notes and use them for study or review.

[1131] Server side

[1132] Receiving and saving images

[1133] The server receives the photo data sent from the device and temporarily stores it in the server's storage.

[1134] Image analysis and OCR processing

[1135] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used in this process, specifically by pre-processing the images (converting them to grayscale and removing noise) and then applying character recognition algorithms.

[1136] Note generation

[1137] The server passes the extracted text information to a generation AI, which then automatically generates class notes. The generation AI then formats the received text data into notes, adding headings and bullet points.

[1138] Saving and returning to the database

[1139] The generated class notes are linked to the user ID and stored in a database. An HTTP response is also created and sent to return the generated notes to the device.

[1140] Specific examples

[1141] For example, imagine a user taking a picture of a formula or explanation written on a blackboard during a math class with their smartphone. When the user takes a photo and presses the "upload" button, the photo is sent to a server. The server analyzes the received photo and uses OCR technology to extract the formula and text. Next, the generative AI creates notes based on the extracted text data. Specifically, it formats the formula, adds paragraphs, and emphasizes important points.

[1142] The generated math notes are stored in a database and then sent back to the device. The user can view the notes on their smartphone app and use them for review or summarizing after class.

[1143] In this way, this system efficiently analyzes images of educational scenes taken by the user and automatically generates and provides lesson notes, thereby significantly improving learning efficiency.

[1144] The processing flow will be explained below.

[1145] Step 1:

[1146] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[1147] Step 2:

[1148] The device moves the saved photos to the specified temporary upload storage. The device displays an "Upload" button, giving the user the option to send the photos to the server.

[1149] Step 3:

[1150] The user presses the "Upload" button. The device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[1151] Step 4:

[1152] The server receives the HTTP request sent from the device, analyzes the photo data in the request, and temporarily stores it.

[1153] Step 5:

[1154] The server applies image analysis algorithms to the stored photos, including pre-processing the images (grayscale conversion and noise reduction).

[1155] Step 6:

[1156] The server uses OCR technology to extract text from the image, which involves detecting text regions and running character recognition algorithms.

[1157] Step 7:

[1158] The server sends the extracted text data to the generation AI, which analyzes the text data and automatically generates class notes. Specific operations include formatting the text, adding headings, and generating bullet points.

[1159] Step 8:

[1160] The server associates the generated class notes with the user ID and stores them in a database. The server then returns the generated notes to the device as an HTTP response.

[1161] Step 9:

[1162] The device analyzes the HTTP response received from the server, extracts the generated note data, and displays the note on the user interface.

[1163] Example 1

[1164] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1165] Traditional methods of taking class notes require students to take notes by hand during lectures, which requires a lot of time and effort. Furthermore, there is a risk of overlooking information or making typos, which can lead to reduced learning effectiveness. Furthermore, there are few efficient ways to digitize lecture content, making it difficult to search and share later.

[1166] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1167] In this invention, the server includes means for receiving images of educational scenes taken by a user, means for temporarily saving the images in a storage device of the terminal, means for uploading the saved images to the server, means for performing image analysis on the received images to extract text information, means for inputting the extracted text information into a generative AI model to automatically generate class notes, means for returning the generated class notes to the user terminal, and means for displaying the generated class notes. This makes it possible to automatically digitize lecture content and efficiently generate accurate class notes.

[1168] "User" refers to a person who uses this system to take images of educational scenes and generate notes.

[1169] "Terminal" refers to a device, such as a smartphone or tablet, that a user uses to take images and upload them to a server.

[1170] "Server" refers to a computer system that receives and analyzes image data sent from a terminal and has the function of generating notes and returning them to the user.

[1171] "Image analysis" refers to a series of processes in which a terminal or server processes image data received and extracts useful information from the image.

[1172] "Text information" refers to information such as character data and mathematical formulas extracted from an image through image analysis.

[1173] "Generative AI model" refers to the artificial intelligence algorithm used to automatically generate notes based on text information.

[1174] A "prompt sentence" refers to an input sentence that instructs the generative AI model on what kind of note to generate.

[1175] "Class notes" refers to documents that organize the contents of lectures and are automatically generated by a generative AI model based on images of educational scenes taken by users.

[1176] "Storage device" refers to hardware for temporarily or long-term storage of image data and generated note data.

[1177] "Upload" refers to the process by which a device sends stored image data to a server.

[1178] A "user identifier" is information used to uniquely identify a user and is used to associate the user with the notes created in the database.

[1179] An "end-user terminal" refers to a terminal that is directly operated by a user and is responsible for taking images and displaying notes.

[1180] This invention is a system that automatically generates lesson notes by allowing a user to take images of a teaching scene using a device such as a smartphone or tablet and upload the images to a server. A specific embodiment of this system will be described here.

[1181] Client side (terminal)

[1182] 1. Taking and saving photos

[1183] Users use their smartphone camera to take pictures of classes and training sessions, and the photos are first temporarily saved in the device's internal memory.

[1184] 2. Prepare for upload

[1185] The device moves the temporarily saved photo data to temporary storage for uploading, and then the application displays an "Upload" button to the user, which the user can press to prepare the photo data for transmission to the server.

[1186] 3. Upload Process

[1187] When a user presses the "Upload" button, the device creates an HTTP request, attaches the saved photo, and sends it to the server. This process uses the smartphone's communication function and specifies the destination endpoint URL.

[1188] 4. View notes

[1189] The terminal receives the class notes returned from the server and provides an interface that allows users to view the notes. Users can use this interface to view the generated notes and use them for study or review.

[1190] Server side

[1191] 1. Receiving and saving images

[1192] The server receives the photo data sent from the device and temporarily stores it in the server's storage. The format and size of the received data are verified.

[1193] 2. Image analysis and OCR processing

[1194] The server analyzes the stored photo data and uses OCR technology to extract text information from the images. Specifically, the images are pre-processed (by converting them to grayscale and removing noise), and then a character recognition algorithm is applied.

[1195] 3. Creating notes

[1196] The server inputs the text information extracted by OCR into a generative AI model, which then automatically generates class notes. The generative AI model then formats the extracted text data into notes, adding headings and bullet points.

[1197] 4. Saving to database and returning

[1198] The generated class notes are linked to the user ID and stored in a database. An HTTP response is also created and sent to return the generated notes to the device.

[1199] Specific examples

[1200] For example, consider the case where a user takes a photo of a math class using their smartphone. During class, the user takes a photo of the formulas and explanations on the blackboard with their smartphone and presses the "upload" button, sending the photo to a server. The server analyzes the received photo and uses OCR technology to extract the formulas and characters. Next, a generative AI model creates class notes based on the extracted text data. Specifically, it formats formulas, adds paragraphs, and emphasizes important points.

[1201] The generated math notes are stored in a database and then sent back to the device, where users can view them using a smartphone application and use them for review or summarizing after class.

[1202] Examples of prompt statements

[1203] An example of a prompt sentence to input to the generative AI model is, "Please extract text data from the photo below and create class notes." By inputting this prompt sentence and the text data extracted by OCR into the AI ​​model, specific class notes are generated.

[1204] In this way, this system efficiently analyzes images of educational scenes taken by the user and automatically generates and provides lesson notes, thereby significantly improving learning efficiency.

[1205] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1206] Step 1:

[1207] A user takes a picture of a class or training session using the smartphone camera. The captured image is the input, and the device temporarily stores this image in its internal memory. Specifically, the smartphone's camera app accepts the operation of the capture button and saves the captured image data in a temporary directory within the camera app.

[1208] Step 2:

[1209] The device moves images stored in internal memory to temporary storage for uploading. The input to this process is image data stored in internal memory, and the output is image data moved to temporary storage for uploading. Specifically, the file is moved within the application, and an "Upload" button is displayed on the screen.

[1210] Step 3:

[1211] When a user presses the "Upload" button in the application, the device creates an HTTP request and sends the image data to the server. The input to this process is the image data in temporary upload storage, and the output is the HTTP request sent to the server. Specifically, the application constructs an HTTP request in the background and sends the request with the image data attached in multipart format to the specified server endpoint.

[1212] Step 4:

[1213] The server receives image data sent from the device and temporarily stores it in the server's storage. The input to this process is the HTTP request, and the output is image data stored in the server's temporary storage. Specifically, the server analyzes the request and stores the received image data in a specific directory.

[1214] Step 5:

[1215] The server performs image analysis and OCR processing on the stored image data to extract text information. The input to this process is the stored image data, and the output is the extracted text information. Specifically, the OCR software preprocesses the image (such as grayscale conversion and noise removal) and applies character recognition algorithms to generate the text information.

[1216] Step 6:

[1217] The server inputs the extracted text information into a generative AI model to automatically generate class notes. The input for this process is the text information extracted by OCR, and the output is the generated class notes. Specifically, the AI ​​model (such as the GPT series) receives the prompt and text information and generates formatted class notes.

[1218] Step 7:

[1219] The server associates the generated class notes with the user ID, stores them in a database, and returns them to the user's device. The input to this process is the generated class notes and the user ID, and the output is storage in the database and an HTTP response to the device. Specifically, the server stores the notes in the database, creates an HTTP response, and sends it to the device.

[1220] Step 8:

[1221] The device receives the class notes returned from the server and provides an interface for the user to view them. The input to this process is the HTTP response returned from the server, and the output is the class notes displayed in the user interface. Specifically, the application analyzes the received data and renders the note contents in the display area within the app.

[1222] (Application example 1)

[1223] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1224] Autonomous vehicles face the challenge of efficiently analyzing the large amounts of video and sensor data collected during operation, and providing the navigation information necessary for operation and a rapid response method when a problem occurs. This will reduce the risk of accidents and improve passenger safety and vehicle efficiency.

[1225] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1226] In this invention, the server includes means for receiving images of the environment taken by a user, means for performing image processing on the received images to extract information, and means for automatically generating navigation information based on the extracted information. This makes it possible to quickly extract important information from the taken images and provide necessary navigation information and response methods to autonomously driven vehicles in real time.

[1227] "User" refers to the operator or manager of the autonomous vehicle.

[1228] "Environmental images" are video data of the surroundings collected by cameras mounted on autonomous vehicles.

[1229] "Image processing" is a technique for extracting necessary information from video data captured by a camera, and specifically includes grayscale conversion and noise removal.

[1230] "Information" refers to text data and important point information extracted through image processing.

[1231] "Navigation information" is data that includes route information useful for operating an autonomous vehicle and instructions on how to respond in the event of an accident.

[1232] A "terminal" is a device for using navigation information, specifically an in-vehicle computer or a smartphone.

[1233] "Identification information" is information for identifying an autonomous vehicle or a user.

[1234] "Data storage device" refers to a database or storage device installed within a server.

[1235] A "mobile terminal" is an information device that a user can carry around, and specifically includes a smartphone or tablet.

[1236] A "server" is a computer system that receives images, processes them, and generates the necessary information.

[1237] This invention is a system that efficiently processes video data captured by an autonomous vehicle while it is in operation, and generates and provides navigation information in real time. First, the system program will be explained.

[1238] Overview of the program

[1239] This system involves a series of steps: an in-vehicle camera collects images of the surrounding environment and sends them to a server, which processes the images, extracts the information, generates navigation information, and sends it back to the terminal.

[1240] Program processing overview

[1241] 1. Data collection and transmission

[1242] The device captures images of the environment using a camera mounted on the autonomous vehicle and temporarily stores them in its internal memory. At this point, the device compresses the image data as needed and moves it to temporary storage.

[1243] 2. Image analysis and information extraction on the server

[1244] The server receives the image data sent from the device and performs image processing. In this process, preprocessing (grayscale conversion, noise removal, etc.) is performed using OpenCV, and text information is extracted using Tesseract OCR. The extracted information is just raw data, so it needs to be processed in the next step.

[1245] 3. Generating navigation information

[1246] The server inputs the extracted text information into a generative AI model using TensorFlow to generate navigation information, which includes important information on points of interest and how to respond when trouble occurs.

[1247] 4. Data storage and return

[1248] The generated navigation information is stored in a data storage device along with the identification information, and then sent back to the vehicle's onboard computer as an HTTP response, providing the necessary information for driving in real time.

[1249] Hardware and software used

[1250] Hardware:

[1251] On-board camera: a device that takes images of the environment

[1252] On-board computer: A device that manages the temporary storage and transmission of data.

[1253] Storage device: Data storage device

[1254] software:

[1255] OpenCV: Image processing library

[1256] Tesseract OCR: Character Recognition Engine

[1257] TensorFlow: A framework for running generative AI models

[1258] HTTP client: Software that manages the sending and receiving of data

[1259] Specific examples

[1260] For example, when an autonomous vehicle approaches an intersection, a camera collects information on signs and traffic lights. When the user presses the "upload" button, the image data is sent to a server. The server processes the image and generates navigation information such as "Follow the signs near the intersection to confirm your direction of travel." This navigation information is sent back to the user's device in real time and displayed on the driver's seat display.

[1261] Prompt Sentence Examples

[1262] Below are some example prompts to input to the generative AI model:

[1263] (prompt statement)

[1264] Please create a navigation notebook based on this video data, including important points, optimal route information, and recovery procedures in case of trouble.

[1265] Video data:

[1266] 1. Video file name | / path / to / data / video1.mp4

[1267] 2. Video file name | / path / to / data / video2.mp4

[1268] The above is an embodiment of this system.

[1269] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1270] Step 1:

[1271] The terminal captures images of the environment using a camera mounted on the autonomous vehicle and temporarily stores them in its internal memory. It receives raw image data captured by the camera as input and obtains image data stored in temporary storage as output. Specific operations include compressing the captured images and transferring them to temporary storage.

[1272] Step 2:

[1273] When the user presses the "Upload" button, the device sends the saved image data to the server as an HTTP request. The image data in temporary storage is used as input, and the data sent to the server is obtained as output. Specifically, the device performs a process to format the image data into an HTTP request.

[1274] Step 3:

[1275] The server receives image data sent from the terminal and saves it in a storage device. It receives image data sent from the terminal as input and obtains the saved data as output. Specifically, the server safely stores the data it receives in storage.

[1276] Step 4:

[1277] The server performs preprocessing on the stored image data using OpenCV. It receives the stored image data as input and obtains preprocessed data as output. Specifically, this includes processes such as grayscale conversion and noise removal.

[1278] Step 5:

[1279] The preprocessed image data is input into Tesseract OCR to extract text information. The preprocessed image data is used as input, and extracted text information is obtained as output. Specifically, the text in the image is analyzed and extracted as character data.

[1280] Step 6:

[1281] The extracted text information is input into a generative AI model using TensorFlow to generate navigation information. The extracted text information is used as input, and generated navigation information is obtained as output. Specific operations include the generative AI model analyzing the input data and generating the necessary information.

[1282] Step 7:

[1283] The generated navigation information is stored in a data storage device together with the identification information. The generated navigation information and the identification information are used as input, and information stored in a database is obtained as output. Specific operations include formatting the information into a database format.

[1284] Step 8:

[1285] The server returns the saved navigation information to the terminal as an HTTP response. It uses the navigation information saved in the database as input and obtains the data sent to the terminal as output. Specific operations include formatting the navigation information as an HTTP response and sending it to the terminal.

[1286] Step 9:

[1287] The terminal displays the received navigation information to the user. It uses the navigation information received from the server as input and obtains the information displayed on the user display as output. Specific operations include visually displaying the navigation information.

[1288] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1289] This invention is a system that allows users to take images of educational situations using devices such as smartphones or tablets, upload the image data to a server, automatically generate class notes, and further recognize the user's emotions and reflect the results in the notes, thereby improving the quality and efficiency of users' learning.

[1290] Client side (terminal)

[1291] Taking and saving photos

[1292] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[1293] Prepare to upload

[1294] The device moves the saved photos to the designated temporary upload storage, and then displays an "Upload" button to the user, giving them the option to send the photos to the server.

[1295] Upload Process

[1296] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[1297] View notes

[1298] When the device receives the class notes returned from the server, it provides the user with an interface to display the notes. Through this interface, the user can view the generated notes and use them for study or review. In addition, the user's emotional information analyzed by the emotion engine is also reflected in the notes.

[1299] Server side

[1300] Receiving and saving images

[1301] The server receives the photo data sent from the device and temporarily stores it in the server's storage.

[1302] Image analysis and OCR processing

[1303] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used in this process, specifically by pre-processing the images (converting them to grayscale and removing noise) and then applying character recognition algorithms.

[1304] Emotion recognition by emotion engine

[1305] The server further executes an emotion engine that recognizes the user's emotion from image and text information, and collects the user's emotion data to identify the user's emotional state, such as relaxed, focused, or distressed.

[1306] Note generation

[1307] The server automatically generates class notes using a generation AI based on the extracted text information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes.

[1308] Saving and returning to the database

[1309] The generated class notes and the user's emotion data are linked to the user ID and stored in a database. Furthermore, an HTTP response is created and sent to return the generated notes to the device.

[1310] Specific examples

[1311] For example, imagine a user is taking a chemistry class and takes a photo of a chemical formula or explanation written on the blackboard with their smartphone. When the user takes a photo and presses the "upload" button, the photo is sent to the server. The server analyzes the received photo and uses OCR technology to extract the chemical formula and text. Next, the generative AI creates notes based on the extracted text data. Specifically, it formats the chemical formula, adds paragraphs, and emphasizes important points.

[1312] Furthermore, the emotion engine recognizes the user's emotions. For example, if the user feels "difficult to understand," the engine reflects this information in the notes. This information is recorded in a specific place in the notes, allowing the user to identify which part of the text they found difficult. This information is also stored in a database for future reference.

[1313] In this way, this system efficiently analyzes images of educational scenes taken by the user, automatically generates lesson notes, and uses an emotion engine to reflect the user's emotional information in the notes, thereby significantly improving learning efficiency.

[1314] The processing flow will be explained below.

[1315] Step 1:

[1316] The user takes a photo of a class or training session using a smartphone camera, and the device temporarily stores the photo in its internal memory.

[1317] Step 2:

[1318] The device moves the saved photos to the specified temporary upload storage. The device displays an "Upload" button, giving the user the option to send the photos to the server.

[1319] Step 3:

[1320] The user presses the "Upload" button. The device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo.

[1321] Step 4:

[1322] The server receives the HTTP request sent from the device, analyzes the photo data in the request, and temporarily stores it.

[1323] Step 5:

[1324] The server applies image analysis algorithms to the stored photos, including pre-processing the images (grayscale conversion and noise reduction).

[1325] Step 6:

[1326] The server uses OCR technology to extract text from the image, which involves detecting text regions and running character recognition algorithms.

[1327] Step 7:

[1328] The server sends the extracted text data to the emotion engine, which evaluates the user's emotional state (e.g., relaxed, focused, or distressed) based on the analysis of the text and images. Specific operations include performing text analysis and facial recognition technology.

[1329] Step 8:

[1330] The server then sends the text data to a generation AI, which automatically generates class notes. The generation AI formats the received text data into notes, adding headings and bullet points. Emotion data from the emotion engine is also reflected in the notes.

[1331] Step 9:

[1332] The server associates the generated class notes with the user's emotion data and stores them in a database. The server then returns the generated notes to the device as an HTTP response.

[1333] Step 10:

[1334] The device analyzes the HTTP response received from the server and extracts the generated note data and emotional information. The device then displays the note on the user interface, along with emotional information such as which parts the user found difficult to understand.

[1335] Example 2

[1336] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1337] In modern education, detailed notes are necessary to effectively review the content of lessons, but traditional methods are time-consuming and burdensome for students. Furthermore, there is a lack of means to grasp students' understanding and emotions in real time and reflect them in their notes, making it difficult to optimize learning outcomes. Therefore, there is a need for the development of a system that can efficiently generate lesson notes and also reflect students' emotional information.

[1338] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1339] In this invention, the server includes means for receiving images of educational scenes taken by a user, means for performing image analysis on the received images and extracting text information, means for automatically generating class notes based on the extracted text information, means for returning the generated class notes to the user terminal, and means for recognizing the user's emotions and reflecting the emotional information in the class notes. This enables the automatic generation of class notes and the reflection of emotional information, allowing students to efficiently review the content of lessons and improving learning effectiveness.

[1340] "User" refers to an individual who uses this system to capture images of educational situations and generate lesson notes.

[1341] "Educational setting" refers to a situation in which learning or education is taking place, such as in classes, lectures, or training.

[1342] "Images" refers to photographs or scanned data of educational scenes taken by users.

[1343] "Image analysis" refers to the process of extracting text information from received image data.

[1344] "Text information" refers to character data or text data extracted by image analysis.

[1345] "Class notes" refers to documents that record learning content and are automatically generated based on extracted text information.

[1346] "Means for receiving" refers to the function of the server to receive and store image data sent from the user terminal.

[1347] "Means for extracting" refers to the function of extracting text information from an image through image analysis.

[1348] "Means of automatic generation" refers to algorithms or programs that create class notes based on extracted text information.

[1349] "Means for returning" refers to the function of sending the generated lesson notes to the user's terminal and displaying them.

[1350] "Means for recognizing emotions" refers to a function that analyzes the user's emotional state and collects that information.

[1351] "Means for reflecting emotional information" refers to the function of incorporating the recognized emotions of the user into class notes.

[1352] "Temporary storage means" refers to a function for temporarily storing images taken by a user terminal and later transmitting them to a server.

[1353] "Means for uploading" refers to a function for sending image data from a user terminal to a server.

[1354] "Means of saving to a database" refers to the function for associating generated class notes with user IDs and storing them for a long period of time.

[1355] This invention is a system that allows users to take images of educational situations using devices such as smartphones or tablets, upload the image data to a server, automatically generate class notes, recognize the user's emotions, and reflect the results in the notes. This system can improve the quality and efficiency of users' learning.

[1356] Hardware and Software Configuration

[1357] Device: The user captures the classroom scene using a smartphone or tablet (iOS, Android). The device includes a camera, memory, and network module.

[1358] Server: A server with high-performance storage and processing power. Key software includes image analysis algorithms, OCR technology, emotion recognition engines, and generative AI models.

[1359] Image analysis algorithms: Libraries such as Python and OpenCV are used to extract text information from images.

[1360] OCR technology: OCR software such as Tesseract is used to extract text.

[1361] Emotion recognition engine: Sentiment analysis uses IBM Watson and Microsoft Azure's Emotion API.

[1362] Generative AI models: Generative AI models such as GPT-3 are used to format text data into notes.

[1363] Overview of system processing

[1364] Client side (terminal)

[1365] Taking and saving photos: A user takes a photo of a class or training session using their smartphone camera. The device temporarily stores the photo in its internal memory. For example, a user may use their smartphone camera to take a photo of a chemical formula drawn on a blackboard during class.

[1366] Prepare for upload: The device moves the saved photos to the specified temporary upload storage, and displays an "Upload" button to the user, giving them the option to send the photos to the server. The moved photos are saved in " / storage / emulated / 0 / AppUploads / ".

[1367] Upload process: When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved photo. For example, an HTTP POST request is sent to "https: / / example.com / upload".

[1368] Displaying notes: The device receives the class note data returned from the server and provides the user with an interface for displaying the notes. Through this interface, the user can view the generated notes and use them for study or review. In addition, the user's emotional information analyzed by the emotion engine is also reflected in the notes.

[1369] Server side

[1370] Image reception and storage: The server receives the photo data sent from the device and temporarily stores it in the server's storage. The received image data is stored in " / var / www / uploads / ".

[1371] Image analysis and OCR processing: Image analysis algorithms are run on the server to extract text information from images. Specifically, the Python OCR library "Tesseract" is used to convert images into text data.

[1372] Emotion recognition using emotion engine: Implement an emotion engine that recognizes user emotions from images and text information. Call emotion recognition APIs (e.g., IBM Watson, Microsoft Azure Emotion API) to analyze emotions from the user's facial expressions and context.

[1373] Note generation: Class notes are automatically generated by a generation AI based on the extracted text information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the user's emotional information. For example, the AI ​​model can be given a prompt such as, "Please highlight the chemical formula and indicate any difficult parts to understand."

[1374] Saving to database and returning: The generated class notes and the user's emotion data are linked to the user ID and stored in a database. The generated notes are then returned to the device as an HTTP response. For example, the note data and user ID are saved in a MySQL database, and JSON data is returned as a response.

[1375] Specific examples

[1376] For example, suppose a user takes a photo of a chemical formula or explanation written on a blackboard during a chemistry class using their smartphone. After taking the photo and pressing the "upload" button, the photo is sent to a server. The server analyzes the received photo and extracts the chemical formula and text using OCR technology. Next, a generative AI creates class notes based on the extracted text data. Specifically, it formats the chemical formula, adds paragraphs, and emphasizes important points. Furthermore, an emotion engine recognizes the user's emotions and, for example, if the user finds the text difficult to understand, reflects that information in the notes. This information is recorded in specific places in the notes, allowing the user to identify which parts were difficult. This information is also stored in a database for future reference. This system efficiently analyzes images of educational situations taken by users, automatically generates class notes, and uses the emotion engine to reflect the user's emotional information in the notes, significantly improving learning efficiency.

[1377] Example prompts to input to the generative AI model

[1378] "Extract text information from this photo and format it into a note. In particular, highlight the chemical formula and important points. Also, reflect the user's sentiment that it is difficult to understand."

[1379] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1380] Step 1: Take and save a photo

[1381] User: Uses a camera app on a smartphone or tablet to capture educational scenes.

[1382] Terminal: The captured photo is temporarily saved in the internal memory. The input is an image taken by the user, and the output is an image file saved in the internal memory. Specifically, the user takes a photo of a chemical formula drawn on a blackboard with the smartphone camera, and the image file is saved in the " / storage / emulated / 0 / Pictures / " directory.

[1383] Step 2: Prepare to upload

[1384] On device: Moves saved photos to the specified temporary upload storage. The input is an image file stored in the internal memory, and the output is an image file moved to the upload storage.

[1385] On the device: The user is presented with an "Upload" button, giving them the option to send the photo to the server. Specifically, the image file is moved to the " / storage / emulated / 0 / AppUploads / " directory, and then the "Upload" button is displayed in the UI.

[1386] Step 3: Upload Process

[1387] User: Presses the "Upload" button.

[1388] Terminal: Create an HTTP request, attach the saved photo, and send it to the specified endpoint on the server. The input is the image file saved in the upload storage, and the output is an HTTP request to the server. Specifically, an HTTP POST request is sent to "https: / / example.com / upload" with the photo data attached in multipart format.

[1389] Step 4: Receive and save images

[1390] Server: Receives photo data sent from the device. The received data is temporarily stored in the server's storage. The input is the image data sent from the device, and the output is an image file stored in the server's storage. Specifically, the image file is saved in the " / var / www / uploads / " directory.

[1391] Step 5: Image analysis and OCR processing

[1392] Server: Executes image analysis algorithms and extracts text information from images. After preprocessing the image (grayscale conversion and noise removal), character recognition is performed using OCR technology. The input is the received image file, and the output is the extracted text data. Specifically, the image is converted to text data using the Python OCR library "Tesseract."

[1393] Step 6: Emotion Recognition with the Emotion Engine

[1394] Server: Runs the emotion engine that recognizes the user's emotions from image and text information. The emotion engine collects the user's emotional data and identifies their emotional state, such as relaxed, focused, or difficult. The input is image and text data, and the output is the recognized emotion data. Specifically, it calls an emotion recognition API (e.g., IBM Watson, Microsoft Azure Emotion API) to analyze emotions from the user's facial expressions and context.

[1395] Step 7: Generate notes

[1396] Server: Class notes are automatically generated using a generative AI based on the extracted text information. The generative AI formats the text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes. The input is text data and emotion data, and the output is the generated class notes. Specifically, it uses a generative AI model such as GPT-3 to generate notes based on the input prompt.

[1397] Step 8: Saving and returning to the database

[1398] Server: The generated class notes and the user's emotion data are linked to the user ID and stored in a database. In addition, an HTTP response is created and sent to return the generated notes to the device. The input is the generated class notes and the user ID, and the output is the information stored in the database and the HTTP response for return. Specifically, the server saves the note data and the corresponding user ID in a MySQL database, and returns JSON data as a response to the device.

[1399] Step 9: Viewing Notes

[1400] Device: Receives class notes returned from the server and provides the user with an interface to display the notes. The input is the HTTP response from the server, and the output is the displayed note information. Specifically, the device analyzes the note data received in JSON format and formats it in an easy-to-read format within the app before displaying it.

[1401] (Application example 2)

[1402] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1403] In modern factories, product quality control and improved work efficiency are extremely important issues. However, traditional manual inspection records and worker emotional management are cumbersome and inefficient. Furthermore, because workers' emotional states are not reflected in inspection records, they often do not lead to improvements in the work environment. For this reason, there was a need for a system that automatically analyzes product images, generates accurate inspection records, and also reflects workers' emotional states.

[1404] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving images, means for performing image analysis and extracting text information, means for automatically generating notes, and means for recognizing the user's emotions and reflecting the information in the notes. This enables efficient quality control of products in factories, and further enables improvements to the work environment and early detection of problems by reflecting the emotional information of workers.

[1405] The "means for receiving images" refers to a function or process by which a user sends an image taken by the user to the server and the server receives the image.

[1406] "Means for performing image analysis and extracting text information" refers to a function or process that uses digital image processing technology on a received image to identify text contained in the image and extract it as characters.

[1407] "Means for automatically generating notes" refers to functions and processes for automatically generating structured notes based on extracted text information.

[1408] "Means for recognizing the user's emotions and reflecting the information in the notebook" refers to the functions and processes for analyzing the user's emotional state from images and surrounding data and reflecting the results in the notebook. For example, it could identify areas where the user is having difficulty and record that information in the notebook.

[1409] "Inside the factory" refers to the work area where a series of operations such as product manufacturing, assembly, inspection, and quality control are carried out.

[1410] "Quality control" refers to activities to check and ensure that the manufacturing process and finished products comply with prescribed standards and specifications.

[1411] "Working efficiently" means carrying out work in a way that maximizes effectiveness while minimizing the use of time and resources.

[1412] "Reflecting the emotional information of workers" means understanding the emotional state of workers and providing feedback to improve the work environment and methods based on that information.

[1413] "Improving the working environment" refers to activities that improve the physical and psychological working environment so that workers can work safely and efficiently.

[1414] "Early problem detection" means detecting product defects or problems with work procedures in advance and responding quickly.

[1415] The system for carrying out the present invention is for supporting quality control within a factory. Specific embodiments and processes thereof will be described below.

[1416] Client side (terminal)

[1417] 1. Taking and saving images

[1418] A robot in the factory takes pictures of the products, and the device temporarily stores the pictures in its internal memory.

[1419] 2. Prepare for upload

[1420] The device moves the saved image to a predetermined temporary storage location and then displays an "Upload" button to the user, providing them with the option to send the image to the server.

[1421] 3. Upload Process

[1422] When the user presses the "Upload" button, the device creates an HTTP request and sends it to the specified endpoint on the server, attaching the saved image.

[1423] Server side

[1424] 1. Receiving and saving images

[1425] The server receives the image data sent from the device and temporarily stores the received data in the server's storage.

[1426] 2. Image analysis and OCR processing

[1427] Image analysis algorithms are run on the server to extract text information from images. OCR technology is used here, specifically image preprocessing (grayscale conversion and noise removal) and text recognition algorithms are applied. Specific software includes OpenCV and Tesseract OCR.

[1428] 3. Emotion Recognition by Emotion Engine

[1429] The server also runs an emotion engine that recognizes user emotions from images and text information. The emotion engine collects the worker's emotional data and identifies their emotional states, such as relaxed, focused, or difficult. The emotion recognition model used here is trained with Keras.

[1430] 4. Creating Notes

[1431] The server automatically generates quality inspection notes using a generation AI based on the extracted text and emotion information. The generation AI formats the received text data into notes, adding headings and bullet points. It also reflects the analysis results of the emotion engine in the notes.

[1432] 5. Saving to database and returning

[1433] The generated quality inspection note and emotion information are linked to the user ID and stored in a database. Furthermore, an HTTP response is created and sent to return the generated note to the device.

[1434] Specific examples

[1435] For example, a robot in a factory takes a picture of a product and automatically generates a quality inspection note based on that image. Text information (such as product dimensions and surface condition) is extracted from the captured image using OCR technology, and an emotion recognition model is used to recognize the worker's emotional state (such as "difficulty"). The generation AI uses this information to format the quality inspection note, creating a note that highlights the points where the worker found it difficult.

[1436] Prompt Sentence Examples

[1437] For example, the following prompt sentence could be input to a generative AI model:

[1438] prompt:

[1439] Please format the following quality inspection record as an inspection note and reflect the worker's emotional information.

[1440] Product condition: Surface scratches

[1441] Test result: Fail

[1442] Quality information: Further inspection required

[1443] Worker Emotions: Difficulty

[1444] This automatically generates quality inspection notes that are easy for users to understand, and provides information that is useful for improving the work environment and enhancing quality.

[1445] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1446] Step 1:

[1447] A robot takes images of products in a factory. Specifically, it acquires image data of the products using a camera mounted on the robot. The input is the physical appearance of the product, and the output is image data (JPEG or PNG format).

[1448] Step 2:

[1449] The captured image data is temporarily saved in the device's internal memory. Specifically, the image file is saved in the device's storage and its file path is recorded. The input is the image data captured in step 1, and the output is the file path of the saved image file.

[1450] Step 3:

[1451] The user presses the "Upload" button to send image data to the server. Specifically, the device creates an HTTP request, attaches the saved image file, and sends it to the specified endpoint on the server. The input is the file path of the saved image file, and the output is an HTTP request to the server.

[1452] Step 4:

[1453] The server receives image data sent from the terminal and temporarily stores the image data in the server's storage. The input is an HTTP request, and the output is an image file stored on the server.

[1454] Step 5:

[1455] The server runs an image analysis algorithm to extract text information from the image. Specifically, the image is converted to grayscale, noise is removed, and then Tesseract OCR is used to recognize text in the image. The input is the saved image file, and the output is the extracted text information.

[1456] Step 6:

[1457] The server uses an emotion recognition engine to analyze the user's emotional state. Specifically, it detects the worker's face in the image and labels the emotion using an emotion recognition model trained with Keras. The input is the saved image file, and the output is the recognized emotional state information.

[1458] Step 7:

[1459] The server uses the generative AI model to generate a quality inspection note based on the extracted text information and analyzed emotion information. Specifically, a prompt sentence is input to the generative AI model, and note-format text is output. The input is text information and emotion information, and the output is the generated quality inspection note.

[1460] Step 8:

[1461] The server associates the generated quality inspection note with the user ID and stores it in the database. Specifically, it associates the note text with the user ID, registers it in the database, and also stores the necessary metadata. The input is the generated quality inspection note and the user ID, and the output is the note record stored in the database.

[1462] Step 9:

[1463] The server returns the generated note to the terminal as an HTTP response. Specifically, it creates an HTTP response and sends it with the text of the generated note included as a payload. The input is the generated quality inspection note, and the output is the HTTP response.

[1464] Step 10:

[1465] The quality inspection note received by the terminal is displayed through the user interface. Specifically, the note text is displayed on the screen so that the user can check the contents. The input is the note text received as an HTTP response, and the output is the inspection note displayed on the terminal screen.

[1466] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1467] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1468] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1469] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1470] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1471] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1472] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1473] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1474] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1475] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1476] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1477] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1478] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1479] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1480] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1481] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1482] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1483] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1484] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1485] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1486] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1487] The following is further disclosed regarding the above embodiment.

[1488] (Claim 1)

[1489] means for receiving images of educational scenes captured by a user;

[1490] means for performing image analysis on the received image and extracting text information;

[1491] A means for automatically generating class notes based on the extracted text information;

[1492] The system includes a means for returning the generated class notes to the user terminal.

[1493] (Claim 2)

[1494] 10. The system of claim 1,

[1495] A system that includes a means for linking generated class notes with user IDs and storing them in a database.

[1496] (Claim 3)

[1497] 10. The system of claim 1,

[1498] A system that includes a means for temporarily storing images taken by a user on a smartphone terminal and sending them to a server when they are uploaded.

[1499] "Example 1"

[1500] (Claim 1)

[1501] means for receiving images of educational scenes captured by a user;

[1502] means for temporarily storing the received image in a storage device of the terminal;

[1503] means for uploading the stored images to a server;

[1504] means for performing image analysis on the received image and extracting text information;

[1505] A means to input the extracted text information into a generative AI model and automatically generate class notes;

[1506] A means for returning the generated class notes to the user terminal;

[1507] The system includes a means for displaying the generated lesson notes.

[1508] (Claim 2)

[1509] 2. The system according to claim 1, further comprising means for linking the generated lesson notes with the user identifier and storing them in a database.

[1510] (Claim 3)

[1511] 2. The system according to claim 1, further comprising means for temporarily storing images taken by a user in an end-user terminal and transmitting the images to a server when uploaded.

[1512] "Application Example 1"

[1513] (Claim 1)

[1514] means for receiving an image of the environment captured by a user;

[1515] means for performing image processing on the received image and extracting information;

[1516] means for automatically generating navigation information based on the extracted information;

[1517] means for returning the generated navigation information to the terminal;

[1518] A system including:

[1519] (Claim 2)

[1520] 10. The system of claim 1, further comprising means for storing the generated navigation information and the identification information in association with each other in a data storage device.

[1521] (Claim 3)

[1522] 10. The system of claim 1, further comprising means for temporarily storing images taken by a user on a mobile device and transmitting the images to a server upon upload.

[1523] "Example 2: Combining Emotion Engines"

[1524] (Claim 1)

[1525] means for receiving images of educational scenes captured by a user;

[1526] means for performing image analysis on the received image and extracting text information;

[1527] A means for automatically generating class notes based on the extracted text information;

[1528] A means for returning the generated class notes to the user terminal;

[1529] A system that includes a means for recognizing a user's emotions and reflecting that emotional information in class notes.

[1530] (Claim 2)

[1531] 2. The system according to claim 1, further comprising means for linking the generated class notes with the user ID and storing them in a database.

[1532] (Claim 3)

[1533] The system according to claim 1, further comprising means for temporarily storing images taken by a user on a smartphone terminal and transmitting the images to a server when the images are uploaded.

[1534] "Application example 2 when combining emotion engines"

[1535] (Claim 1)

[1536] means for receiving an image captured by a user;

[1537] means for performing image analysis on the received image and extracting text information;

[1538] A means for automatically generating notes based on the extracted text information;

[1539] A means for recognizing the user's emotions and reflecting that information in the notes;

[1540] The system includes means for returning the generated notes to the user terminal.

[1541] (Claim 2)

[1542] 2. The system according to claim 1, further comprising means for linking the generated note with the user ID and storing the linked note in a database.

[1543] (Claim 3)

[1544] 2. The system according to claim 1, further comprising means for temporarily storing images taken by a user in a terminal and transmitting the images to a server when the images are uploaded. [Explanation of symbols]

[1545] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving images of educational scenes captured by a user; means for performing image analysis on the received image and extracting text information; A means for automatically generating class notes based on the extracted text information; The system includes a means for returning the generated class notes to the user terminal.

2. 10. The system of claim 1, A system that includes a means for linking generated class notes with user IDs and storing them in a database.

3. 10. The system of claim 1, A system that includes a means for temporarily storing images taken by a user on a smartphone terminal and sending them to a server when they are uploaded.

Images