System

Abstract

To provide a system capable of improving productivity by automating confirmation work of a document.SOLUTION: The system includes means for scanning various documents to generate images, means for transmitting the images to a server for analysis by a AI system, means for extracting text information from the images and detecting specific elements by the AI system, means for automatically determining the integrity and validity of the documents based on the text information by the server, and means for notifying the user of the determination and receiving feedback.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, when applying for various services, application forms and confirmation documents must be obtained and then reviewed by a dedicated team for consistency and validity. This process requires a lot of time and effort, resulting in reduced productivity and increased costs. Furthermore, human errors can occur, making ensuring reliability a challenge. The present invention aims to solve these problems and automate the document review process, thereby reducing operation time and eliminating judgment errors. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means. A system is provided that includes means for scanning various documents with a terminal camera and generating image data, means for transmitting the image data to a server and analyzing it with an AI system, means for the AI ​​system to extract text information from the image data and detect specific elements (such as a seal or expiration date), means for the server to automatically determine the integrity and validity of the document based on the text information, and means for notifying the user of the determination result and receiving feedback. This makes it possible to automate document verification work and improve productivity.

[0006] "Various documents" refers to documents required when using or signing a contract for a service, such as application forms and confirmation documents.

[0007] "Device" refers to a portable electronic device, such as a smartphone or tablet, that can be operated by a user to utilize a camera function.

[0008] "Camera" refers to an optical device for photography built into a terminal that is used to acquire image data.

[0009] "Image data" refers to digital data of still images and videos taken by a camera.

[0010] "Server" refers to a remote computer system that communicates with terminals over a network and receives, stores, and processes data.

[0011] "AI system" refers to software and hardware that uses artificial intelligence technology to automatically analyze data and make decisions.

[0012] "Text information" refers to character string data extracted from image data.

[0013] "Specific elements" refer to elements that are necessary when determining the integrity and validity of a document, such as a seal or expiration date that the AI ​​system detects from image data.

[0014] "Integrity" refers to a state in which multiple data or documents are consistent with each other and do not contradict each other.

[0015] "Validity" refers to the state in which confirmation documents, etc. are confirmed to be legal and within the valid period.

[0016] "Judgment result" refers to the evaluation and judgment made by the AI ​​system and server based on text information and specific elements.

[0017] "Feedback" refers to information or responses returned from a user to a server. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention relates to an AI system for streamlining the document review process. This AI system uses a device's camera to read application and confirmation documents, generate image data, and send it to a server. The AI ​​system in the server analyzes this image data to automatically determine the integrity and validity of the documents and provide feedback on the results to the user.

[0040] Specific system configuration and program processing

[0041] Image data generation and transmission

[0042] The terminal is operated by the user, who takes a photo of the application or confirmation document with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, the image data is generated and sent to the server.

[0043] As a concrete example, consider the case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear photo of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[0044] Receiving and analyzing image data

[0045] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image and detect specific elements (such as seals and expiration dates).

[0046] For example, an AI system can extract information such as the name, date of birth, and expiration date from an image of a driver's license. It also checks whether the image contains a stamp and determines whether the expiration date is past the current date.

[0047] Consistency determination and notification

[0048] The server checks the consistency of the application and confirmation documents based on the extracted text information. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal has been affixed and whether the expiration date is valid.

[0049] For example, the server checks whether there are any discrepancies between the "applicant name" on the application form and the "name" on the driver's license. At the same time, it checks whether the required seal has been properly stamped and whether the expiration date has passed the current date.

[0050] Feedback and final approval

[0051] The server notifies the user of the results of the AI's automated assessment. The user receives the notification and, if necessary, submits revised documents. By repeating this process, if all conditions are met, the server stores the application and confirmation document information in a database and takes steps to proceed to the next step in the process.

[0052] For example, the server may notify the user of an error message such as "Name does not match" or "Validity period has expired." The user may then correct the error and submit the document again. If the server determines that all information is correct and consistent, it will send a notification to the user that the process has been completed. The user will receive this notification and confirm that the process has been completed successfully.

[0053] This system automates the process of reviewing various documents, shortening operation time and eliminating judgment errors, thereby improving productivity and is expected to be used in many industries.

[0054] The processing flow will be explained below.

[0055] Program processing steps

[0056] Step 1:

[0057] The terminal starts a mobile application, and the user selects the camera function.

[0058] Specific operation: The user taps the "Launch Camera" button in the app to launch the camera. A shooting guide will be displayed on the screen.

[0059] Step 2:

[0060] The user takes a photo of the application or confirmation document with a camera.

[0061] Specific actions: Place the application form or confirmation document within the camera frame and press the shutter button to take the photo.

[0062] Step 3:

[0063] The device displays a preview of the captured image and prompts the user to confirm it.

[0064] Specific operation: A preview screen is displayed and a "Use this image" button is displayed. The user confirms the image and presses the OK button.

[0065] Step 4:

[0066] The device compresses and encrypts the image data and sends it to the server.

[0067] What happens: Image data is processed within the app and sent to a server using a secure communication protocol.

[0068] Step 5:

[0069] The server temporarily stores the image data received from the terminal.

[0070] Specific operation: The received image data is stored in the server storage and prepared for analysis.

[0071] Step 6:

[0072] The server requests the AI ​​system to analyze the image data.

[0073] Specific operation: The saved image data is passed to the AI ​​analysis module, and the analysis task is initiated.

[0074] Step 7:

[0075] The AI ​​system uses OCR technology to extract text information from images.

[0076] Specific operation: Identifies text areas within an image and extracts text information such as name and date of birth as digital data.

[0077] Step 8:

[0078] The AI ​​system uses image recognition algorithms to detect specific elements (stamps, expiration dates, etc.).

[0079] Specific operation: Check whether the stamp position and expiration date are correct. Analyze other detection elements in the same way.

[0080] Step 9:

[0081] The server checks the consistency of the application documents and confirmation documents based on the extracted text information.

[0082] Specific operations: For example, compare the "applicant name" written on the application form with the "name" on the verification document to check for any discrepancies.

[0083] Step 10:

[0084] The server receives the results of the AI ​​system and generates feedback.

[0085] Specific operation: Set a status of "pass" or "fail" for each check item and summarize the results.

[0086] Step 11:

[0087] The server notifies the user of the result of the determination.

[0088] Specific operation: Notify the user of the results in real time via a mobile application.

[0089] Step 12:

[0090] The user receives a notification and, if necessary, re-photographs and submits the corrected document.

[0091] Specific operation: If an error occurs, the user re-photographs the corrected document and sends it to the server again.

[0092] Step 13:

[0093] If all conditions are met, the server stores the application and confirmation documents in a database.

[0094] What it does: Securely stores the final data in a database and sets a flag to start the next process.

[0095] Step 14:

[0096] The server sends a notification to the user that the procedure is complete.

[0097] Specific operation: The application sends a notification to the user that the procedure has been completed.

[0098] Step 15:

[0099] The user receives a final notification and confirms that the transaction is complete.

[0100] What happens: The user checks the in-app notification to confirm the transaction is complete.

[0101] Example 1

[0102] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0103] The document review process requires manual review of documents to determine their consistency and validity, which requires a great deal of time and effort. Furthermore, manual reviewing is prone to human error, and deficiencies and errors in documents can be overlooked. Therefore, there is a need to improve the efficiency and accuracy of the review process.

[0104] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0105] In this invention, the server includes: a means for scanning various documents with a device for taking photos of the terminal and generating image data; a means for transmitting the image data to the server via a communication device and analyzing it with an AI system; a means for the AI ​​system to extract text information from the image data using optical character recognition technology and detect specific elements (such as an approval seal or validity period); a means for the server to automatically determine the consistency and validity of the document based on the text information; and a means for notifying the user of the determination result via the communication device and receiving feedback. This automates the document review process, shortening operation time and eliminating judgment errors. Furthermore, the user can receive feedback and make quick corrections, improving the efficiency and accuracy of the entire review process.

[0106] A "terminal" is a device for taking pictures of various documents and transmitting the generated image data to a server via a communication device.

[0107] The "photography device" is a device such as a camera mounted on a terminal, which is used to record documents as digital images.

[0108] "Image data" refers to digital document image information generated by a photographing device.

[0109] A "communication device" is a network communication device for transmitting image data to a server.

[0110] A "server" is a computing device that receives image data sent from a terminal, temporarily stores it, and then analyzes it using an AI system.

[0111] "AI system" refers to artificial intelligence technology that analyzes image data, extracts text information using optical character recognition technology, and detects specific elements of documents (such as stamps of approval or validity dates).

[0112] Optical character recognition (OCR) is a technology that automatically reads text information from image data and converts it into digital text.

[0113] "Text information" refers to character data extracted from image data using optical character recognition technology.

[0114] "Specific elements" refer to important information items such as the document's stamp of approval, validity period, name, etc.

[0115] "Consistency" refers to a state in which the contents of the application documents and confirmation documents match and are free of contradictions.

[0116] "Validity" refers to the state in which a document is within its legal validity period and meets all necessary requirements.

[0117] "Means for receiving feedback" refers to the function by which the server notifies the user of the result of the judgment and receives corrections or resubmissions from the user.

[0118] This invention relates to an AI system for streamlining the document review process. This AI system involves a series of processes: scanning application and confirmation documents using a device's camera, generating image data, and transmitting it to a server via a communication device. The AI ​​system in the server then analyzes the image data, automatically determining the consistency and validity of the documents, and providing feedback on the results to the user.

[0119] Hardware and software used:

[0120] Device:

[0121] Devices for taking photos of various documents (e.g. smartphones, tablets, etc.)

[0122] Camera as a photographic device

[0123] Application software with image data preview and confirmation functions

[0124] server:

[0125] Storage for receiving and temporarily storing image data

[0126] AI systems (including optical character recognition (OCR) and image recognition algorithms)

[0127] Software for analyzing text information and determining document integrity

[0128] Specific examples:

[0129] Image data is generated when a user takes a photo of an application or confirmation document using the device's camera. At this time, the device displays a preview of the captured image, allowing the user to confirm that the image is clear. For example, a user can take a photo of the front and back of a driver's license and check the image on the preview screen within the application. Once the user has completed the verification, the image data is compressed (e.g., in JPEG format) and encrypted (e.g., using AES) and sent to the server by the communication device.

[0130] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR technology to extract text information from the image and detect specific elements (e.g., name, date of birth, expiration date, whether or not a seal is attached, etc.). During this process, the AI ​​system automatically reads information such as "name," "date of birth," and "expiration date" from the image of the driver's license.

[0131] The server uses the extracted text information to check whether the information on the application and confirmation documents is consistent. For example, it checks whether the "applicant name" on the application form matches the "name" on the driver's license, and whether the expiration date is within the current date. It also checks whether the required seals have been properly stamped. Based on the results, the server sends an error message or a success message to the user.

[0132] The user receives a notification from the server and makes corrections as necessary. This process is repeated until all conditions are met, at which point the server saves the application and confirmation document information in its database and takes steps to proceed to the next step in the process. For example, the server may notify the user with an error message such as "Name does not match" or "Expired date has passed," prompting the user to correct the information and submit it again. Finally, if the server determines that all information is accurate and matches, it will send a notification to the user that the process is complete.

[0133] Example prompts for generative AI models:

[0134] "Extract the name and expiration date from the driver's license."

[0135] Please check that the information on your application matches your verification documents.

[0136] "Please check whether the stamp is properly stamped in the required places on the document."

[0137] This invention automates the document review process, shortens operation time, and eliminates judgment errors, improving productivity and is expected to be used in many industries.

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] Step 1:

[0140] The terminal is operated by the user, who activates the device's image capture to take a photo of the application or confirmation document. The terminal displays the captured image to the user on a preview screen, and after the user checks the image's clarity and content, it processes the data to generate image data. The input is the image of the document captured by the user, and the output is the confirmation image displayed on the preview screen. In concrete terms, the user uses the camera function to take a photo of their driver's license and checks the image on the preview screen.

[0141] Step 2:

[0142] When the user has finished confirming the image, the device recognizes that a button has been pressed on the preview screen requesting confirmation by the user. The image data is then compressed (e.g., JPEG format) and encrypted (e.g., AES encryption). The input is the image data after the user's confirmation, and the output is the compressed and encrypted image data. Specifically, when the user taps the "Send Image" button, the device compresses the image data, encrypts it, and sends it to the server.

[0143] Step 3:

[0144] The server receives image data sent from the terminal and temporarily stores it. The input is the compressed and encrypted image data sent from the terminal, and the output is the image data stored in the server's storage. Specifically, the server receives the image data, checks the integrity of the data, and stores it in storage.

[0145] Step 4:

[0146] The server passes the stored image data to the AI ​​system and requests it to analyze it. The AI ​​system uses OCR technology to extract text information from the image data and detects specific elements (e.g., name, expiration date, whether or not a seal has been affixed). The input is the image data stored on the server, and the output is the analyzed text information. Specifically, the image data is input into the AI ​​system, and text information such as "name," "date of birth," and "expiration date" is extracted through OCR processing.

[0147] Step 5:

[0148] The server uses the extracted text information to check the consistency and validity of the application and confirmation documents. For example, it determines whether the "applicant name" written on the application form matches the "name" extracted from the image, and whether the expiration date is beyond the current date. The input is the text information provided by the AI ​​system, and the output is the result of the consistency and validity determination. Specifically, it compares the text information with the application form information to check for inconsistencies or omissions.

[0149] Step 6:

[0150] The server sends a notification message via a communication device to notify the user of the result of the judgment. The user receives the notification and makes corrections as necessary. The input is the result of the consistency and validity check, and the output is a notification message to the user. Specifically, the user receives an error message on their smartphone or tablet, such as "Name does not match" or "Expired date."

[0151] Step 7:

[0152] The user receives a notification from the server, makes any necessary corrections, photographs the documents again, and submits them. By repeating this process, if all conditions are met, the server saves the application and confirmation document information in its database and proceeds to the next process. The input is new image data of the documents revised by the user, and the output is text information reanalyzed by the server and the final judgment result. In concrete terms, the user photographs the documents again, submits them to the server, and finally receives a notification that the procedure has been completed.

[0153] (Application example 1)

[0154] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0155] The present invention aims to streamline the screening and authentication process for membership cards, coupons, point cards, etc. in physical stores, thereby reducing the burden on users and stores. Traditionally, these screening and authentication processes have often been performed manually, which requires time and effort and is prone to human error. Furthermore, there are issues with physical cards that users carry with them, such as loss or damage, so a solution to these issues is needed.

[0156] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0157] In this invention, the server includes: a means for reading various information with a device camera and generating image data; a means for transmitting the image data to a network server and analyzing it with an artificial intelligence system; a means for the artificial intelligence system to extract text information from the image data and detect specific elements (such as a seal or expiration date); a means for the server to automatically determine the integrity and validity of the document based on the text information; a means for notifying the user of the determination result and receiving feedback; and a means for authenticating the use of membership cards, coupons, and point cards using the feedback. This automates the review and authentication process for membership cards, coupons, and point cards, significantly reducing time and labor. It also reduces the risk of human error and improves user convenience.

[0158] A "device" is an electronic device that is operated by a user and has a camera mounted thereon for reading various types of information.

[0159] "Image data" is digital data generated based on information captured by the device's camera.

[0160] "Network Server" is a central processing unit that receives image data sent from the device and interfaces with the artificial intelligence system for analysis.

[0161] An "artificial intelligence system" is a software system equipped with AI technology that is used to extract textual information from image data and detect specific elements.

[0162] "Text information" is text data extracted from image data by an artificial intelligence system.

[0163] A "seal" is an impression required to indicate the validity of a document.

[0164] "Expiration date" refers to the period during which a document, membership card, coupon, etc. is legally valid.

[0165] "Integrity" is a concept that refers to documents and data being consistent and free of contradictions.

[0166] "Validity" is the state of a document or data that indicates it is suitable for its purpose and can be used.

[0167] "Feedback" refers to response information including notification results and correction instructions for the user.

[0168] A "membership card" is a card or digital certificate used to prove membership in a particular organization or service.

[0169] A "coupon" is a voucher that provides discounts or special offers for specific services or products.

[0170] A "point card" is a card that accumulates points with each purchase and can be used for special offers or discounts.

[0171] "Authenticating use" is the process of verifying the validity of a membership card, coupon, or point card and authorizing its use.

[0172] To implement the present invention, a device used by a user, a network server, and an artificial intelligence system are required. The details of each component and process are as follows.

[0173] 1. Device-side processing

[0174] The device is operated by the user and uses the camera to capture information such as membership cards, coupons, and point cards. At this time, the device displays a preview of the captured image and asks the user to confirm. Once the user has completed the confirmation, image data is generated.

[0175] 2. Sending image data

[0176] The generated image data is sent from the device to a network server, where it is encrypted to ensure confidentiality.

[0177] 3. Server-side processing

[0178] The server receives the image data sent from the device, temporarily stores it in storage, and then requests an AI system to analyze the image data.

[0179] 4. Image analysis using artificial intelligence systems

[0180] The AI ​​system uses OCR (optical character recognition) technology to extract text information from the received image data. Specifically, it detects specific elements such as ID numbers, expiration dates, and stamps on membership cards and coupons. The software used here is, for example, OpenCV and pytesseract.

[0181] 5. Checking data integrity and validity

[0182] The server checks whether the membership card or coupon is valid based on the character information extracted by the AI ​​system. This involves checking against information stored in a database. A specific example is a process that checks whether the extracted expiration date is beyond the current date.

[0183] 6. Providing Feedback

[0184] The results of the review are fed back to the user in real time. For example, a message such as "This is a valid membership card" or "This card has expired" is displayed. The user can receive this feedback, correct the information as needed, and submit it again.

[0185] Specific examples

[0186] The user launches the smartphone app, presses the "Membership Card Photo" button to take a photo of their membership card, checks the image on the preview screen, and then presses the "Send" button, which sends the image data to the server. The server analyzes this image data and notifies the user of the results. Specific examples of screen displays and notifications that can be used include prompts such as the following:

[0187] Prompt Sentence Examples

[0188] User: Press the "Membership Card Photo" button to take a photo of your membership card, check it on the preview screen, and then press the "Send" button.

[0189] This will streamline the review and authentication process for membership cards, coupons, and point cards at physical stores, while also improving user convenience.

[0190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0191] Step 1:

[0192] The user takes a photo of a membership card or coupon with the camera. The input to the device is the user pressing the capture button, and image data is generated as output. The device displays a preview of this image data and asks the user to confirm it. When the user presses the confirmation button, the device confirms the image data.

[0193] Step 2:

[0194] The device sends the determined image data to the network server. The input is the generated image data, and the output is the data sent over the network. At this time, the data is encrypted to maintain security.

[0195] Step 3:

[0196] The server receives image data sent from the device and temporarily stores it in storage. The input is the sent image data, and the data temporarily stored in storage is the output.

[0197] Step 4:

[0198] The server passes the stored image data to the AI ​​system and requests it to analyze it. The input is the image data stored in the storage, and the output is the image data that the AI ​​system receives.

[0199] Step 5:

[0200] The artificial intelligence system uses OCR technology to extract text information from the received image data. Specifically, it detects ID numbers, expiration dates, and seals on membership cards and coupons contained in the image. The input is image data, and the output is the extracted text information. Software such as OpenCV and pytesseract is used for this processing.

[0201] Step 6:

[0202] The server checks the consistency and validity of the corresponding data based on the text information provided by the AI ​​system. Specifically, it compares it with the information registered in the database to see if it matches and if it has expired. The input is the extracted text information, and the output is the result of the consistency and validity determination.

[0203] Step 7:

[0204] The server notifies the user of the result of the determination, displaying real-time feedback on the user's device and sending messages such as "This is a valid membership card" or "This card has expired." The input is the determination result, and the output is a notification message displayed on the user's device.

[0205] Step 8:

[0206] The user receives the notification message and, if necessary, takes another photo of the membership card or coupon and submits it. This process loops back to step 1 until the server determines that all information is accurate and consistent. Finally, if the information is authenticated, the membership card or coupon is authorized for use.

[0207] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0208] This invention relates to a system that improves user experience by combining an AI system for streamlining the document review process with an emotion engine that recognizes user emotions. This system uses the device's camera to read application and confirmation documents, generate image data, and send it to a server. The AI ​​system on the server analyzes this image data and automatically determines the integrity and validity of the documents. It also uses the emotion engine to analyze the user's emotions when entering input and adjust the feedback content.

[0209] Specific system configuration and program processing

[0210] Image data generation and transmission

[0211] The terminal is operated by the user, who takes a photo of the application or confirmation document with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, the image data is generated and sent to the server.

[0212] As a concrete example, consider the case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear photo of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[0213] Receiving and analyzing image data

[0214] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image and detect specific elements (such as seals and expiration dates).

[0215] For example, an AI system can extract information such as the name, date of birth, and expiration date from an image of a driver's license. It also checks whether the image contains a stamp and determines whether the expiration date is past the current date.

[0216] Consistency determination and notification

[0217] The server checks the consistency of the application and confirmation documents based on the extracted text information. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal has been affixed and whether the expiration date is valid.

[0218] For example, the server checks whether there are any discrepancies between the "applicant name" on the application form and the "name" on the driver's license. At the same time, it checks whether the required seal has been properly stamped and whether the expiration date has passed the current date.

[0219] Introducing an emotion engine and adjusting feedback

[0220] The server is equipped with an emotion engine that analyzes the user's emotions based on their input content and speed. The emotion engine detects stress and anxiety when the user submits the application and generates feedback accordingly.

[0221] For example, if the emotion engine determines that the user is feeling anxious or stressed when the user's input speed is extremely slow or when the input content has been corrected many times, the server will send a friendly message such as, "Was it difficult to input? We are here to help if you need any help."

[0222] Final approval and saving

[0223] If all conditions are met, the server stores the application and confirmation documents in the database and takes steps to proceed to the next step. The emotion engine also further analyzes the user's feedback and stores it as data to improve the user experience.

[0224] For example, the server notifies the user with error messages such as "Name does not match" or "Expired date has passed," and the emotion engine analyzes the user's reaction and adjusts the feedback. If the server determines that all information is accurate and matches after the user resubmits the corrected document, it sends a notification to the user that the process has been completed. The user receives the notification and confirms that the process has been completed successfully.

[0225] This system automates the process of reviewing various documents, shortening operation time and eliminating judgment errors. Furthermore, an emotion engine provides feedback that takes into account the user's emotions, improving the user experience. This improves productivity and is expected to be used in many industries.

[0226] The processing flow will be explained below.

[0227] Program processing steps

[0228] Step 1:

[0229] The terminal starts a mobile application, and the user selects the camera function.

[0230] Specific operation: The user taps the "Launch Camera" button in the app to launch the camera. A shooting guide will be displayed on the screen.

[0231] Step 2:

[0232] The user takes a photo of the application or confirmation document with a camera.

[0233] Specific actions: Place the application form or confirmation document within the camera frame and press the shutter button to take the photo.

[0234] Step 3:

[0235] The device displays a preview of the captured image and prompts the user to confirm it.

[0236] Specific operation: A preview screen is displayed and a "Use this image" button is displayed. The user confirms the image and presses the OK button.

[0237] Step 4:

[0238] The device compresses and encrypts the image data and sends it to the server.

[0239] What happens: Image data is processed within the app and sent to a server using a secure communication protocol.

[0240] Step 5:

[0241] The server temporarily stores the image data received from the terminal.

[0242] Specific operation: The received image data is stored in the server storage and prepared for analysis.

[0243] Step 6:

[0244] The server requests the AI ​​system to analyze the image data.

[0245] Specific operation: The saved image data is passed to the AI ​​analysis module, and the analysis task is initiated.

[0246] Step 7:

[0247] The AI ​​system uses OCR technology to extract text information from images.

[0248] Specific operation: Identifies text areas within an image and extracts text information such as name and date of birth as digital data.

[0249] Step 8:

[0250] The AI ​​system uses image recognition algorithms to detect specific elements (stamps, expiration dates, etc.).

[0251] Specific operation: Check whether the stamp position and expiration date are correct. Analyze other detection elements in the same way.

[0252] Step 9:

[0253] The server checks the consistency of the application documents and confirmation documents based on the extracted text information.

[0254] Specific operations: For example, compare the "applicant name" written on the application form with the "name" on the verification document to check for any discrepancies.

[0255] Step 10:

[0256] The server receives the results of the AI ​​system and generates feedback.

[0257] Specific operation: Set a status of "pass" or "fail" for each check item and summarize the results.

[0258] Step 11:

[0259] The server notifies the user of the result of the determination.

[0260] Specific operation: Notify the user of the results in real time via a mobile application.

[0261] Step 12:

[0262] The user receives a notification and, if necessary, re-photographs and submits the corrected document.

[0263] Specific operation: If an error occurs, the user re-photographs the corrected document and sends it to the server again.

[0264] Step 13:

[0265] The server uses an emotion engine to analyze the user's emotions.

[0266] Specific operation: Analyzes the user's input speed and frequency of content revisions to determine whether the user is feeling stressed or anxious.

[0267] Step 14:

[0268] The server adjusts the feedback content based on the results analyzed by the emotion engine.

[0269] Specific operation: For example, if it is determined that the user is in a stressful state, a friendly follow-up message is generated and sent.

[0270] Step 15:

[0271] If all conditions are met, the server stores the application and confirmation documents in a database.

[0272] What it does: Securely stores the final data in a database and sets a flag to start the next process.

[0273] Step 16:

[0274] The server sends a notification to the user that the procedure is complete.

[0275] Specific operation: The application sends a notification to the user that the procedure has been completed.

[0276] Step 17:

[0277] The user receives a final notification and confirms that the transaction is complete.

[0278] What happens: The user checks the in-app notification to confirm the transaction is complete.

[0279] Example 2

[0280] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] Previous document review processes were often manual, which was time-consuming, labor-intensive, and prone to errors. Furthermore, there was little technology available to reduce the stress and anxiety users felt when submitting documents. Therefore, there was a need for a system that could review documents efficiently and accurately. Furthermore, there was a need to improve the user experience by analyzing user emotions and providing appropriate feedback.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0283] In this invention, the server includes means for temporarily storing image data and analyzing it using AI technology, means for extracting text information from the image data and detecting specific elements (such as a seal or expiration date) using AI technology, and means for detecting the user's emotional state using emotion analysis technology, generating appropriate feedback, and sending it to the user. This not only automates the document review process efficiently and accurately, but also makes it possible to provide feedback that takes the user's emotions into consideration.

[0284] A "terminal" is a device that is operated by a user to photograph a document and generate image data.

[0285] A "camera" is hardware with a photographing function that is installed on a terminal.

[0286] A "server" is a computing system that receives, stores, analyzes, and generates feedback from image data.

[0287] "Image data" is a digital image file of a document captured by the device's camera.

[0288] "AI technology" refers to artificial intelligence technology that analyzes image data using optical character recognition technology and image recognition algorithms.

[0289] "Text information" is character information extracted from image data using AI technology.

[0290] "Specific elements" refer to specific information such as a seal or expiration date on a document.

[0291] "Consistency" is a criterion for evaluating whether the information in the application documents and the confirmation documents match.

[0292] "Validity" is a criterion for assessing whether a document is currently valid.

[0293] "Emotion analysis technology" is a technology that analyzes the content and speed of a user's input and infers the user's emotional state.

[0294] "Feedback" refers to the response or message that the server provides to the user based on the analysis results.

[0295] "Temporary storage" is a process in which the server temporarily stores the image data received in storage.

[0296] This invention is a system that streamlines the document review process and combines it with an emotion engine that analyzes user emotions. This system uses the device's camera to read application and confirmation documents, generate image data, and send it to a server. An AI system on the server analyzes this image data and automatically determines the integrity and validity of the documents. The emotion engine is also used to analyze the user's emotions when they enter information, and the feedback content is adjusted to improve the user experience.

[0297] Generation and transmission of image data by the terminal

[0298] The terminal is operated by the user, who takes a photo of the application or confirmation documents with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, image data is generated, compressed, encrypted, and sent to the server. This process uses the camera function of a smartphone or tablet, and the mobile application that controls it.

[0299] Receipt of image data by the server

[0300] The server receives and temporarily stores image data sent from the device. This data is received via API, temporarily stored in storage, and then passed to the AI ​​system.

[0301] Image data analysis by the server

[0302] The AI ​​system in the server uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from images and detect specific elements (such as seals, expiration dates, etc.) The technologies used include Python, TensorFlow, Tesseract OCR, and OpenCV.

[0303] Determining document integrity

[0304] The server checks the consistency of the application and confirmation documents based on the text information extracted from the AI ​​system. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal is affixed and whether the expiration date is valid.

[0305] Emotion analysis using an emotion engine

[0306] The server is equipped with an emotion engine that analyzes the user's emotions based on their input and typing speed. It uses natural language processing libraries (e.g., spaCy, NLTK) and machine learning models (e.g., BERT) to detect stress and anxiety when the user submits their application and generates feedback accordingly.

[0307] Generate and send feedback

[0308] The server generates an appropriate feedback message based on the analysis results of the emotion engine and sends it to the user, such as "Did you have difficulty entering the information? We are here to help if you need any assistance."

[0309] Final approval and data storage

[0310] If all conditions are met, the server stores the application and confirmation documents in the database and takes steps to proceed to the next step. The emotion engine also further analyzes the user's feedback and stores it as data to improve the user experience. Finally, a notification of the completion of the process is sent to the user, who then receives the notification and confirms that the process has been completed.

[0311] Examples of specific examples and prompts

[0312] As a concrete example, let's consider a case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear picture of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[0313] Example prompts to input to a generative AI model:

[0314] "Take a photo of the front and back of your driver's license and send the image data to our server."

[0315] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0316] Step 1: Take a photo of the document and preview the image

[0317] The user takes a photo of the application or confirmation document using the device's camera. The user launches the device's mobile application and uses the camera function to take a clear photo of the front of the driver's license. The application then displays the captured image on a preview screen and asks the user to confirm. The user checks the captured image on the preview screen and taps the "Confirm" button to proceed.

[0318] Input: Document image taken by camera

[0319] Output: Image displayed on the preview screen

[0320] Step 2: Generate image data and send it to the server

[0321] The device generates image data based on the image viewed by the user. This image data is compressed, encrypted, and sent to the server. Specifically, the image captured by the device's internal processing is saved in JPEG or PNG format, encrypted with AES, and sent to the server using the HTTPS protocol.

[0322] Input: Image confirmed by user

[0323] Output: Encrypted image data (compressed)

[0324] Step 3: Receiving and temporarily saving image data

[0325] The server receives the image data sent from the device and temporarily stores it. The server decrypts the encrypted image data received via HTTPS communication and temporarily stores it in storage. This temporarily stored data is used in the next analysis step.

[0326] Input: Encrypted image data

[0327] Output: Temporarily saved decrypted image data

[0328] Step 4: Analyzing the image data

[0329] The AI ​​system on the server analyzes the stored image data. It uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image data and detect specific elements (such as seals and expiration dates). Specifically, it runs a Python script to extract text information using TensorFlow and the Tesseract OCR library, and performs image recognition using OpenCV.

[0330] Input: Temporarily saved image data

[0331] Output: Extracted text information (name, expiration date, whether or not it has a seal, etc.)

[0332] Step 5: Determine consistency and validity

[0333] The server determines the consistency and validity of the application and confirmation documents based on the text information extracted from the AI ​​system. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document, whether there is a seal, and whether the expiration date is beyond the current date. This process requires connection to a database, and the check is performed using an SQL query.

[0334] Input: Extracted text information

[0335] Output: Consistency and validity determination results

[0336] Step 6: Sentiment analysis and feedback generation

[0337] The server uses sentiment analysis technology to analyze the user's input content and input speed and infer the user's emotional state. Natural language processing libraries (e.g., spaCy, NLTK) and machine learning models (e.g., BERT) are used for sentiment analysis. Based on the results of this analysis, the server generates an appropriate feedback message. For example, if the input speed is extremely slow or there are many corrections, the server can determine that the user is feeling anxious or stressed and generate a message such as, "Was it difficult to input? We are here to help if you need any help."

[0338] Input: User input and speed data

[0339] Output: The generated feedback message

[0340] Step 7: Submit feedback and final approval

[0341] The server sends the generated feedback message to the user. If all conditions are met, the server saves the application and confirmation documents in the database. Finally, it sends a notification of the completion of the procedure to the user's device, and the user receives the notification and confirms that the procedure has been completed.

[0342] Input: Feedback message and final approval information

[0343] Output: Notify user and save to database

[0344] (Application example 2)

[0345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] In the conventional document review process, manual review required a lot of time and effort. Furthermore, there was a lack of mechanisms to reduce the anxiety and stress users felt when submitting documents. A system to address this issue is needed, and it is necessary to develop a system that not only streamlines the review process but also provides feedback that takes users' emotions into account.

[0347] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0348] In this invention, the server includes: means for scanning various documents with a photographing device of a terminal and generating image data; means for transmitting the image data to a data processing device and analyzing it with an artificial intelligence system; means for the artificial intelligence system to extract text information from the image data and detect specific information (such as a seal or expiration date); means for the data processing device to automatically determine the integrity and validity of the document based on the text information; means for notifying the user of the determination result and receiving feedback; means including an emotion analysis engine that analyzes emotions based on the user's input content and input speed and adjusts the feedback content; and means for installing and operating the system on a smartphone or computer. This makes it possible to streamline the review process and provide appropriate feedback that takes the user's emotions into consideration.

[0349] "Documents" refers to multiple different types of paper or electronic documents, such as contracts, applications, confirmations, etc.

[0350] "Terminal" means a device that can be operated by a user, such as a smartphone, tablet, laptop, or desktop computer.

[0351] The term "photography device" refers to a device for capturing images, such as a camera mounted on a terminal.

[0352] "Image data" refers to data that digitally represents an image captured by an imaging device.

[0353] "Data processing device" refers to a computer system for analyzing and processing acquired image data and text information.

[0354] An "artificial intelligence system" is a system that uses algorithms such as machine learning and deep learning to automatically analyze data and extract and judge specific information.

[0355] "Text information" refers to text data extracted from image data.

[0356] "Specific information" refers to specific elements written on a document, such as a seal or expiration date.

[0357] "Document integrity and validity" refers to determining whether the application details are correct and consistent, and whether the documents are legally valid.

[0358] "Judgment result" refers to information obtained as a result of analysis by an artificial intelligence system and evaluation by a data processing device.

[0359] "Feedback" refers to answers or advice provided to users based on the results of their decisions.

[0360] An "emotion analysis engine" is software that analyzes the speed and content of a user's input and estimates that person's emotions.

[0361] The present invention provides a system for improving the efficiency of the review process of various documents and adjusting the feedback content by analyzing the user's emotions. Hereinafter, a specific embodiment of the present invention will be described.

[0362] System configuration

[0363] The system is broadly composed of the following components:

[0364] 1. Terminal

[0365] 2. Data Processing Device

[0366] 3. Artificial Intelligence Systems

[0367] 4. Sentiment Analysis Engine

[0368] 1. Terminal

[0369] Users use devices such as smartphones, tablets, and laptops. The devices are equipped with a camera, which is used to capture image data of various documents.

[0370] The user uses an application on the device to take a photo of a document. The captured image is previewed on the device and the user is asked to confirm. Once the user confirms, the image data is compressed, encrypted, and sent to the data processing device.

[0371] 2. Data Processing Device

[0372] The data processing device includes a storage device for temporarily storing the received image data. The data processing device functions as a server and requests the artificial intelligence system to analyze the image data.

[0373] 3. Artificial Intelligence Systems

[0374] The artificial intelligence system uses TensorFlow, an open-source framework, to build a machine learning model. The system uses OCR technology to extract text from images and detect specific information (such as seals and expiration dates).

[0375] As a specific example, an image of a driver's license can be analyzed to extract information such as the name, date of birth, and expiration date. This can be done with high accuracy using a pre-trained model.

[0376] 4. Sentiment Analysis Engine

[0377] The sentiment analysis engine uses natural language processing tools such as TextBlob to analyze the content and speed of the user's input to detect stress or anxiety, allowing the system to generate appropriate feedback for the user.

[0378] For example, consider the following prompt:

[0379] "I've been a little concerned about the service here lately."

[0380] The sentiment analysis engine analyzes this input, and if the emotion is strongly negative, it generates a friendly message such as, "Was it difficult to enter? We are here to help if you need any help."

[0381] These elements work together to improve the efficiency of the review process and the user experience as a whole system.

[0382] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0383] Step 1:

[0384] The user takes an image of a document on the device. The input is the image captured through the device's camera function, and the output is the captured document image data. After taking the image, the user can check the image on the preview screen.

[0385] Step 2:

[0386] The terminal compresses and encrypts the image data of the document that has been confirmed by the user and sends it to the server. The input is the confirmed image data, and the output is the compressed and encrypted image data. This ensures that the data is securely transferred to the server.

[0387] Step 3:

[0388] The server receives the image data sent from the terminal and temporarily stores it in storage. The input is compressed and encrypted image data, and the output is data stored in the server's storage.

[0389] Step 4:

[0390] The server requests the AI ​​system to analyze the stored image data. The input is the image data stored in the storage, and the output is the data sent to the AI ​​system.

[0391] Step 5:

[0392] The AI ​​system extracts text information from image data and detects specific information (such as a seal or expiration date). The input is image data, and the output is the extracted text information and specific information. The AI ​​system uses the TensorFlow framework.

[0393] Step 6:

[0394] The server determines the integrity and validity of the document based on the text information received from the AI ​​system. The input is the text information extracted by the AI ​​system, and the output is the judgment result. The server performs this automatically.

[0395] Step 7:

[0396] The server notifies the user of the decision result and receives feedback. The input is the decision result, and the output is the notification to the user and the feedback from the user.

[0397] Step 8:

[0398] The device analyzes the user's input content and input speed using a sentiment analysis engine. The input is the user's feedback content, and the output is the sentiment analysis result. The sentiment analysis engine uses TextBlob.

[0399] Step 9:

[0400] The server adjusts the feedback content based on the emotion analysis results and provides it to the user. The input is the emotion analysis results, and the output is the adjusted feedback. This improves user satisfaction.

[0401] This series of processes streamlines the document review process while providing appropriate feedback that takes users' emotions into consideration.

[0402] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0403] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0404] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0405] [Second embodiment]

[0406] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0407] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0408] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0409] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0410] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0412] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0413] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0414] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0415] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0416] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0417] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0418] This invention relates to an AI system for streamlining the document review process. This AI system uses a device's camera to read application and confirmation documents, generate image data, and send it to a server. The AI ​​system in the server analyzes this image data to automatically determine the integrity and validity of the documents and provide feedback on the results to the user.

[0419] Specific system configuration and program processing

[0420] Image data generation and transmission

[0421] The terminal is operated by the user, who takes a photo of the application or confirmation document with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, the image data is generated and sent to the server.

[0422] As a concrete example, consider the case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear photo of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[0423] Receiving and analyzing image data

[0424] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image and detect specific elements (such as seals and expiration dates).

[0425] For example, an AI system can extract information such as the name, date of birth, and expiration date from an image of a driver's license. It also checks whether the image contains a stamp and determines whether the expiration date is past the current date.

[0426] Consistency determination and notification

[0427] The server checks the consistency of the application and confirmation documents based on the extracted text information. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal has been affixed and whether the expiration date is valid.

[0428] For example, the server checks whether there are any discrepancies between the "applicant name" on the application form and the "name" on the driver's license. At the same time, it checks whether the required seal has been properly stamped and whether the expiration date has passed the current date.

[0429] Feedback and final approval

[0430] The server notifies the user of the results of the AI's automated assessment. The user receives the notification and, if necessary, submits revised documents. By repeating this process, if all conditions are met, the server stores the application and confirmation document information in a database and takes steps to proceed to the next step in the process.

[0431] For example, the server may notify the user of an error message such as "Name does not match" or "Validity period has expired." The user may then correct the error and submit the document again. If the server determines that all information is correct and consistent, it will send a notification to the user that the process has been completed. The user will receive this notification and confirm that the process has been completed successfully.

[0432] This system automates the process of reviewing various documents, shortening operation time and eliminating judgment errors, thereby improving productivity and is expected to be used in many industries.

[0433] The processing flow will be explained below.

[0434] Program processing steps

[0435] Step 1:

[0436] The terminal starts a mobile application, and the user selects the camera function.

[0437] Specific operation: The user taps the "Launch Camera" button in the app to launch the camera. A shooting guide will be displayed on the screen.

[0438] Step 2:

[0439] The user takes a photo of the application or confirmation document with a camera.

[0440] Specific actions: Place the application form or confirmation document within the camera frame and press the shutter button to take the photo.

[0441] Step 3:

[0442] The device displays a preview of the captured image and prompts the user to confirm it.

[0443] Specific operation: A preview screen is displayed and a "Use this image" button is displayed. The user confirms the image and presses the OK button.

[0444] Step 4:

[0445] The device compresses and encrypts the image data and sends it to the server.

[0446] What happens: Image data is processed within the app and sent to a server using a secure communication protocol.

[0447] Step 5:

[0448] The server temporarily stores the image data received from the terminal.

[0449] Specific operation: The received image data is stored in the server storage and prepared for analysis.

[0450] Step 6:

[0451] The server requests the AI ​​system to analyze the image data.

[0452] Specific operation: The saved image data is passed to the AI ​​analysis module, and the analysis task is initiated.

[0453] Step 7:

[0454] The AI ​​system uses OCR technology to extract text information from images.

[0455] Specific operation: Identifies text areas within an image and extracts text information such as name and date of birth as digital data.

[0456] Step 8:

[0457] The AI ​​system uses image recognition algorithms to detect specific elements (stamps, expiration dates, etc.).

[0458] Specific operation: Check whether the stamp position and expiration date are correct. Analyze other detection elements in the same way.

[0459] Step 9:

[0460] The server checks the consistency of the application documents and confirmation documents based on the extracted text information.

[0461] Specific operations: For example, compare the "applicant name" written on the application form with the "name" on the verification document to check for any discrepancies.

[0462] Step 10:

[0463] The server receives the results of the AI ​​system and generates feedback.

[0464] Specific operation: Set a status of "pass" or "fail" for each check item and summarize the results.

[0465] Step 11:

[0466] The server notifies the user of the result of the determination.

[0467] Specific operation: Notify the user of the results in real time via a mobile application.

[0468] Step 12:

[0469] The user receives a notification and, if necessary, re-photographs and submits the corrected document.

[0470] Specific operation: If an error occurs, the user re-photographs the corrected document and sends it to the server again.

[0471] Step 13:

[0472] If all conditions are met, the server stores the application and confirmation documents in a database.

[0473] What it does: Securely stores the final data in a database and sets a flag to start the next process.

[0474] Step 14:

[0475] The server sends a notification to the user that the procedure is complete.

[0476] Specific operation: The application sends a notification to the user that the procedure has been completed.

[0477] Step 15:

[0478] The user receives a final notification and confirms that the transaction is complete.

[0479] What happens: The user checks the in-app notification to confirm the transaction is complete.

[0480] Example 1

[0481] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0482] The document review process requires manual review of documents to determine their consistency and validity, which requires a great deal of time and effort. Furthermore, manual reviewing is prone to human error, and deficiencies and errors in documents can be overlooked. Therefore, there is a need to improve the efficiency and accuracy of the review process.

[0483] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0484] In this invention, the server includes: a means for scanning various documents with a device for taking photos of the terminal and generating image data; a means for transmitting the image data to the server via a communication device and analyzing it with an AI system; a means for the AI ​​system to extract text information from the image data using optical character recognition technology and detect specific elements (such as an approval seal or validity period); a means for the server to automatically determine the consistency and validity of the document based on the text information; and a means for notifying the user of the determination result via the communication device and receiving feedback. This automates the document review process, shortening operation time and eliminating judgment errors. Furthermore, the user can receive feedback and make quick corrections, improving the efficiency and accuracy of the entire review process.

[0485] A "terminal" is a device for taking pictures of various documents and transmitting the generated image data to a server via a communication device.

[0486] The "photography device" is a device such as a camera mounted on a terminal, which is used to record documents as digital images.

[0487] "Image data" refers to digital document image information generated by a photographing device.

[0488] A "communication device" is a network communication device for transmitting image data to a server.

[0489] A "server" is a computing device that receives image data sent from a terminal, temporarily stores it, and then analyzes it using an AI system.

[0490] "AI system" refers to artificial intelligence technology that analyzes image data, extracts text information using optical character recognition technology, and detects specific elements of documents (such as stamps of approval or validity dates).

[0491] Optical character recognition (OCR) is a technology that automatically reads text information from image data and converts it into digital text.

[0492] "Text information" refers to character data extracted from image data using optical character recognition technology.

[0493] "Specific elements" refer to important information items such as the document's stamp of approval, validity period, name, etc.

[0494] "Consistency" refers to a state in which the contents of the application documents and confirmation documents match and are free of contradictions.

[0495] "Validity" refers to the state in which a document is within its legal validity period and meets all necessary requirements.

[0496] "Means for receiving feedback" refers to the function by which the server notifies the user of the result of the judgment and receives corrections or resubmissions from the user.

[0497] This invention relates to an AI system for streamlining the document review process. This AI system involves a series of processes: scanning application and confirmation documents using a device's camera, generating image data, and transmitting it to a server via a communication device. The AI ​​system in the server then analyzes the image data, automatically determining the consistency and validity of the documents, and providing feedback on the results to the user.

[0498] Hardware and software used:

[0499] Device:

[0500] Devices for taking photos of various documents (e.g. smartphones, tablets, etc.)

[0501] Camera as a photographic device

[0502] Application software with image data preview and confirmation functions

[0503] server:

[0504] Storage for receiving and temporarily storing image data

[0505] AI systems (including optical character recognition (OCR) and image recognition algorithms)

[0506] Software for analyzing text information and determining document integrity

[0507] Specific examples:

[0508] Image data is generated when a user takes a photo of an application or confirmation document using the device's camera. At this time, the device displays a preview of the captured image, allowing the user to confirm that the image is clear. For example, a user can take a photo of the front and back of a driver's license and check the image on the preview screen within the application. Once the user has completed the verification, the image data is compressed (e.g., in JPEG format) and encrypted (e.g., using AES) and sent to the server by the communication device.

[0509] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR technology to extract text information from the image and detect specific elements (e.g., name, date of birth, expiration date, whether or not a seal is attached, etc.). During this process, the AI ​​system automatically reads information such as "name," "date of birth," and "expiration date" from the image of the driver's license.

[0510] The server uses the extracted text information to check whether the information on the application and confirmation documents is consistent. For example, it checks whether the "applicant name" on the application form matches the "name" on the driver's license, and whether the expiration date is within the current date. It also checks whether the required seals have been properly stamped. Based on the results, the server sends an error message or a success message to the user.

[0511] The user receives a notification from the server and makes corrections as necessary. This process is repeated until all conditions are met, at which point the server saves the application and confirmation document information in its database and takes steps to proceed to the next step in the process. For example, the server may notify the user with an error message such as "Name does not match" or "Expired date has passed," prompting the user to correct the information and submit it again. Finally, if the server determines that all information is accurate and matches, it will send a notification to the user that the process is complete.

[0512] Example prompts for generative AI models:

[0513] "Extract the name and expiration date from the driver's license."

[0514] Please check that the information on your application matches your verification documents.

[0515] "Please check whether the stamp is properly stamped in the required places on the document."

[0516] This invention automates the document review process, shortens operation time, and eliminates judgment errors, improving productivity and is expected to be used in many industries.

[0517] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0518] Step 1:

[0519] The terminal is operated by the user, who activates the device's image capture to take a photo of the application or confirmation document. The terminal displays the captured image to the user on a preview screen, and after the user checks the image's clarity and content, it processes the data to generate image data. The input is the image of the document captured by the user, and the output is the confirmation image displayed on the preview screen. In concrete terms, the user uses the camera function to take a photo of their driver's license and checks the image on the preview screen.

[0520] Step 2:

[0521] When the user has finished confirming the image, the device recognizes that a button has been pressed on the preview screen requesting confirmation by the user. The image data is then compressed (e.g., JPEG format) and encrypted (e.g., AES encryption). The input is the image data after the user's confirmation, and the output is the compressed and encrypted image data. Specifically, when the user taps the "Send Image" button, the device compresses the image data, encrypts it, and sends it to the server.

[0522] Step 3:

[0523] The server receives image data sent from the terminal and temporarily stores it. The input is the compressed and encrypted image data sent from the terminal, and the output is the image data stored in the server's storage. Specifically, the server receives the image data, checks the integrity of the data, and stores it in storage.

[0524] Step 4:

[0525] The server passes the stored image data to the AI ​​system and requests it to analyze it. The AI ​​system uses OCR technology to extract text information from the image data and detects specific elements (e.g., name, expiration date, whether or not a seal has been affixed). The input is the image data stored on the server, and the output is the analyzed text information. Specifically, the image data is input into the AI ​​system, and text information such as "name," "date of birth," and "expiration date" is extracted through OCR processing.

[0526] Step 5:

[0527] The server uses the extracted text information to check the consistency and validity of the application and confirmation documents. For example, it determines whether the "applicant name" written on the application form matches the "name" extracted from the image, and whether the expiration date is beyond the current date. The input is the text information provided by the AI ​​system, and the output is the result of the consistency and validity determination. Specifically, it compares the text information with the application form information to check for inconsistencies or omissions.

[0528] Step 6:

[0529] The server sends a notification message via a communication device to notify the user of the result of the judgment. The user receives the notification and makes corrections as necessary. The input is the result of the consistency and validity check, and the output is a notification message to the user. Specifically, the user receives an error message on their smartphone or tablet, such as "Name does not match" or "Expired date."

[0530] Step 7:

[0531] The user receives a notification from the server, makes any necessary corrections, photographs the documents again, and submits them. By repeating this process, if all conditions are met, the server saves the application and confirmation document information in its database and proceeds to the next process. The input is new image data of the documents revised by the user, and the output is text information reanalyzed by the server and the final judgment result. In concrete terms, the user photographs the documents again, submits them to the server, and finally receives a notification that the procedure has been completed.

[0532] (Application example 1)

[0533] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0534] The present invention aims to streamline the screening and authentication process for membership cards, coupons, point cards, etc. in physical stores, thereby reducing the burden on users and stores. Traditionally, these screening and authentication processes have often been performed manually, which requires time and effort and is prone to human error. Furthermore, there are issues with physical cards that users carry with them, such as loss or damage, so a solution to these issues is needed.

[0535] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0536] In this invention, the server includes: a means for reading various information with a device camera and generating image data; a means for transmitting the image data to a network server and analyzing it with an artificial intelligence system; a means for the artificial intelligence system to extract text information from the image data and detect specific elements (such as a seal or expiration date); a means for the server to automatically determine the integrity and validity of the document based on the text information; a means for notifying the user of the determination result and receiving feedback; and a means for authenticating the use of membership cards, coupons, and point cards using the feedback. This automates the review and authentication process for membership cards, coupons, and point cards, significantly reducing time and labor. It also reduces the risk of human error and improves user convenience.

[0537] A "device" is an electronic device that is operated by a user and has a camera mounted thereon for reading various types of information.

[0538] "Image data" is digital data generated based on information captured by the device's camera.

[0539] "Network Server" is a central processing unit that receives image data sent from the device and interfaces with the artificial intelligence system for analysis.

[0540] An "artificial intelligence system" is a software system equipped with AI technology that is used to extract textual information from image data and detect specific elements.

[0541] "Text information" is text data extracted from image data by an artificial intelligence system.

[0542] A "seal" is an impression required to indicate the validity of a document.

[0543] "Expiration date" refers to the period during which a document, membership card, coupon, etc. is legally valid.

[0544] "Integrity" is a concept that refers to documents and data being consistent and free of contradictions.

[0545] "Validity" is the state of a document or data that indicates it is suitable for its purpose and can be used.

[0546] "Feedback" refers to response information including notification results and correction instructions for the user.

[0547] A "membership card" is a card or digital certificate used to prove membership in a particular organization or service.

[0548] A "coupon" is a voucher that provides discounts or special offers for specific services or products.

[0549] A "point card" is a card that accumulates points with each purchase and can be used for special offers or discounts.

[0550] "Authenticating use" is the process of verifying the validity of a membership card, coupon, or point card and authorizing its use.

[0551] To implement the present invention, a device used by a user, a network server, and an artificial intelligence system are required. The details of each component and process are as follows.

[0552] 1. Device-side processing

[0553] The device is operated by the user and uses the camera to capture information such as membership cards, coupons, and point cards. At this time, the device displays a preview of the captured image and asks the user to confirm. Once the user has completed the confirmation, image data is generated.

[0554] 2. Sending image data

[0555] The generated image data is sent from the device to a network server, where it is encrypted to ensure confidentiality.

[0556] 3. Server-side processing

[0557] The server receives the image data sent from the device, temporarily stores it in storage, and then requests an AI system to analyze the image data.

[0558] 4. Image analysis using artificial intelligence systems

[0559] The AI ​​system uses OCR (optical character recognition) technology to extract text information from the received image data. Specifically, it detects specific elements such as ID numbers, expiration dates, and stamps on membership cards and coupons. The software used here is, for example, OpenCV and pytesseract.

[0560] 5. Checking data integrity and validity

[0561] The server checks whether the membership card or coupon is valid based on the character information extracted by the AI ​​system. This involves checking against information stored in a database. A specific example is a process that checks whether the extracted expiration date is beyond the current date.

[0562] 6. Providing Feedback

[0563] The results of the review are fed back to the user in real time. For example, a message such as "This is a valid membership card" or "This card has expired" is displayed. The user can receive this feedback, correct the information as needed, and submit it again.

[0564] Specific examples

[0565] The user launches the smartphone app, presses the "Membership Card Photo" button to take a photo of their membership card, checks the image on the preview screen, and then presses the "Send" button, which sends the image data to the server. The server analyzes this image data and notifies the user of the results. Specific examples of screen displays and notifications that can be used include prompts such as the following:

[0566] Prompt Sentence Examples

[0567] User: Press the "Membership Card Photo" button to take a photo of your membership card, check it on the preview screen, and then press the "Send" button.

[0568] This will streamline the review and authentication process for membership cards, coupons, and point cards at physical stores, while also improving user convenience.

[0569] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0570] Step 1:

[0571] The user takes a photo of a membership card or coupon with the camera. The input to the device is the user pressing the capture button, and image data is generated as output. The device displays a preview of this image data and asks the user to confirm it. When the user presses the confirmation button, the device confirms the image data.

[0572] Step 2:

[0573] The device sends the determined image data to the network server. The input is the generated image data, and the output is the data sent over the network. At this time, the data is encrypted to maintain security.

[0574] Step 3:

[0575] The server receives image data sent from the device and temporarily stores it in storage. The input is the sent image data, and the data temporarily stored in storage is the output.

[0576] Step 4:

[0577] The server passes the stored image data to the AI ​​system and requests it to analyze it. The input is the image data stored in the storage, and the output is the image data that the AI ​​system receives.

[0578] Step 5:

[0579] The artificial intelligence system uses OCR technology to extract text information from the received image data. Specifically, it detects ID numbers, expiration dates, and seals on membership cards and coupons contained in the image. The input is image data, and the output is the extracted text information. Software such as OpenCV and pytesseract is used for this processing.

[0580] Step 6:

[0581] The server checks the consistency and validity of the corresponding data based on the text information provided by the AI ​​system. Specifically, it compares it with the information registered in the database to see if it matches and if it has expired. The input is the extracted text information, and the output is the result of the consistency and validity determination.

[0582] Step 7:

[0583] The server notifies the user of the result of the determination, displaying real-time feedback on the user's device and sending messages such as "This is a valid membership card" or "This card has expired." The input is the determination result, and the output is a notification message displayed on the user's device.

[0584] Step 8:

[0585] The user receives the notification message and, if necessary, takes another photo of the membership card or coupon and submits it. This process loops back to step 1 until the server determines that all information is accurate and consistent. Finally, if the information is authenticated, the membership card or coupon is authorized for use.

[0586] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0587] This invention relates to a system that improves user experience by combining an AI system for streamlining the document review process with an emotion engine that recognizes user emotions. This system uses the device's camera to read application and confirmation documents, generate image data, and send it to a server. The AI ​​system on the server analyzes this image data and automatically determines the integrity and validity of the documents. It also uses the emotion engine to analyze the user's emotions when entering input and adjust the feedback content.

[0588] Specific system configuration and program processing

[0589] Image data generation and transmission

[0590] The terminal is operated by the user, who takes a photo of the application or confirmation document with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, the image data is generated and sent to the server.

[0591] As a concrete example, consider the case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear photo of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[0592] Receiving and analyzing image data

[0593] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image and detect specific elements (such as seals and expiration dates).

[0594] For example, an AI system can extract information such as the name, date of birth, and expiration date from an image of a driver's license. It also checks whether the image contains a stamp and determines whether the expiration date is past the current date.

[0595] Consistency determination and notification

[0596] The server checks the consistency of the application and confirmation documents based on the extracted text information. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal has been affixed and whether the expiration date is valid.

[0597] For example, the server checks whether there are any discrepancies between the "applicant name" on the application form and the "name" on the driver's license. At the same time, it checks whether the required seal has been properly stamped and whether the expiration date has passed the current date.

[0598] Introducing an emotion engine and adjusting feedback

[0599] The server is equipped with an emotion engine that analyzes the user's emotions based on their input content and speed. The emotion engine detects stress and anxiety when the user submits the application and generates feedback accordingly.

[0600] For example, if the emotion engine determines that the user is feeling anxious or stressed when the user's input speed is extremely slow or when the input content has been corrected many times, the server will send a friendly message such as, "Was it difficult to input? We are here to help if you need any help."

[0601] Final approval and saving

[0602] If all conditions are met, the server stores the application and confirmation documents in the database and takes steps to proceed to the next step. The emotion engine also further analyzes the user's feedback and stores it as data to improve the user experience.

[0603] For example, the server notifies the user with error messages such as "Name does not match" or "Expired date has passed," and the emotion engine analyzes the user's reaction and adjusts the feedback. If the server determines that all information is accurate and matches after the user resubmits the corrected document, it sends a notification to the user that the process has been completed. The user receives the notification and confirms that the process has been completed successfully.

[0604] This system automates the process of reviewing various documents, shortening operation time and eliminating judgment errors. Furthermore, an emotion engine provides feedback that takes into account the user's emotions, improving the user experience. This improves productivity and is expected to be used in many industries.

[0605] The processing flow will be explained below.

[0606] Program processing steps

[0607] Step 1:

[0608] The terminal starts a mobile application, and the user selects the camera function.

[0609] Specific operation: The user taps the "Launch Camera" button in the app to launch the camera. A shooting guide will be displayed on the screen.

[0610] Step 2:

[0611] The user takes a photo of the application or confirmation document with a camera.

[0612] Specific actions: Place the application form or confirmation document within the camera frame and press the shutter button to take the photo.

[0613] Step 3:

[0614] The device displays a preview of the captured image and prompts the user to confirm it.

[0615] Specific operation: A preview screen is displayed and a "Use this image" button is displayed. The user confirms the image and presses the OK button.

[0616] Step 4:

[0617] The device compresses and encrypts the image data and sends it to the server.

[0618] What happens: Image data is processed within the app and sent to a server using a secure communication protocol.

[0619] Step 5:

[0620] The server temporarily stores the image data received from the terminal.

[0621] Specific operation: The received image data is stored in the server storage and prepared for analysis.

[0622] Step 6:

[0623] The server requests the AI ​​system to analyze the image data.

[0624] Specific operation: The saved image data is passed to the AI ​​analysis module, and the analysis task is initiated.

[0625] Step 7:

[0626] The AI ​​system uses OCR technology to extract text information from images.

[0627] Specific operation: Identifies text areas within an image and extracts text information such as name and date of birth as digital data.

[0628] Step 8:

[0629] The AI ​​system uses image recognition algorithms to detect specific elements (stamps, expiration dates, etc.).

[0630] Specific operation: Check whether the stamp position and expiration date are correct. Analyze other detection elements in the same way.

[0631] Step 9:

[0632] The server checks the consistency of the application documents and confirmation documents based on the extracted text information.

[0633] Specific operations: For example, compare the "applicant name" written on the application form with the "name" on the verification document to check for any discrepancies.

[0634] Step 10:

[0635] The server receives the results of the AI ​​system and generates feedback.

[0636] Specific operation: Set a status of "pass" or "fail" for each check item and summarize the results.

[0637] Step 11:

[0638] The server notifies the user of the result of the determination.

[0639] Specific operation: Notify the user of the results in real time via a mobile application.

[0640] Step 12:

[0641] The user receives a notification and, if necessary, re-photographs and submits the corrected document.

[0642] Specific operation: If an error occurs, the user re-photographs the corrected document and sends it to the server again.

[0643] Step 13:

[0644] The server uses an emotion engine to analyze the user's emotions.

[0645] Specific operation: Analyzes the user's input speed and frequency of content revisions to determine whether the user is feeling stressed or anxious.

[0646] Step 14:

[0647] The server adjusts the feedback content based on the results analyzed by the emotion engine.

[0648] Specific operation: For example, if it is determined that the user is in a stressful state, a friendly follow-up message is generated and sent.

[0649] Step 15:

[0650] If all conditions are met, the server stores the application and confirmation documents in a database.

[0651] What it does: Securely stores the final data in a database and sets a flag to start the next process.

[0652] Step 16:

[0653] The server sends a notification to the user that the procedure is complete.

[0654] Specific operation: The application sends a notification to the user that the procedure has been completed.

[0655] Step 17:

[0656] The user receives a final notification and confirms that the transaction is complete.

[0657] What happens: The user checks the in-app notification to confirm the transaction is complete.

[0658] Example 2

[0659] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0660] Previous document review processes were often manual, which was time-consuming, labor-intensive, and prone to errors. Furthermore, there was little technology available to reduce the stress and anxiety users felt when submitting documents. Therefore, there was a need for a system that could review documents efficiently and accurately. Furthermore, there was a need to improve the user experience by analyzing user emotions and providing appropriate feedback.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0662] In this invention, the server includes means for temporarily storing image data and analyzing it using AI technology, means for extracting text information from the image data and detecting specific elements (such as a seal or expiration date) using AI technology, and means for detecting the user's emotional state using emotion analysis technology, generating appropriate feedback, and sending it to the user. This not only automates the document review process efficiently and accurately, but also makes it possible to provide feedback that takes the user's emotions into consideration.

[0663] A "terminal" is a device that is operated by a user to photograph a document and generate image data.

[0664] A "camera" is hardware with a photographing function that is installed on a terminal.

[0665] A "server" is a computing system that receives, stores, analyzes, and generates feedback from image data.

[0666] "Image data" is a digital image file of a document captured by the device's camera.

[0667] "AI technology" refers to artificial intelligence technology that analyzes image data using optical character recognition technology and image recognition algorithms.

[0668] "Text information" is character information extracted from image data using AI technology.

[0669] "Specific elements" refer to specific information such as a seal or expiration date on a document.

[0670] "Consistency" is a criterion for evaluating whether the information in the application documents and the confirmation documents match.

[0671] "Validity" is a criterion for assessing whether a document is currently valid.

[0672] "Emotion analysis technology" is a technology that analyzes the content and speed of a user's input and infers the user's emotional state.

[0673] "Feedback" refers to the response or message that the server provides to the user based on the analysis results.

[0674] "Temporary storage" is a process in which the server temporarily stores the image data received in storage.

[0675] This invention is a system that streamlines the document review process and combines it with an emotion engine that analyzes user emotions. This system uses the device's camera to read application and confirmation documents, generate image data, and send it to a server. An AI system on the server analyzes this image data and automatically determines the integrity and validity of the documents. The emotion engine is also used to analyze the user's emotions when they enter information, and the feedback content is adjusted to improve the user experience.

[0676] Generation and transmission of image data by the terminal

[0677] The terminal is operated by the user, who takes a photo of the application or confirmation documents with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, image data is generated, compressed, encrypted, and sent to the server. This process uses the camera function of a smartphone or tablet, and the mobile application that controls it.

[0678] Receipt of image data by the server

[0679] The server receives and temporarily stores image data sent from the device. This data is received via API, temporarily stored in storage, and then passed to the AI ​​system.

[0680] Image data analysis by the server

[0681] The AI ​​system in the server uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from images and detect specific elements (such as seals, expiration dates, etc.) The technologies used include Python, TensorFlow, Tesseract OCR, and OpenCV.

[0682] Determining document integrity

[0683] The server checks the consistency of the application and confirmation documents based on the text information extracted from the AI ​​system. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal is affixed and whether the expiration date is valid.

[0684] Emotion analysis using an emotion engine

[0685] The server is equipped with an emotion engine that analyzes the user's emotions based on their input and typing speed. It uses natural language processing libraries (e.g., spaCy, NLTK) and machine learning models (e.g., BERT) to detect stress and anxiety when the user submits their application and generates feedback accordingly.

[0686] Generate and send feedback

[0687] The server generates an appropriate feedback message based on the analysis results of the emotion engine and sends it to the user, such as "Did you have difficulty entering the information? We are here to help if you need any assistance."

[0688] Final approval and data storage

[0689] If all conditions are met, the server stores the application and confirmation documents in the database and takes steps to proceed to the next step. The emotion engine also further analyzes the user's feedback and stores it as data to improve the user experience. Finally, a notification of the completion of the process is sent to the user, who then receives the notification and confirms that the process has been completed.

[0690] Examples of specific examples and prompts

[0691] As a concrete example, let's consider a case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear picture of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[0692] Example prompts to input to a generative AI model:

[0693] "Take a photo of the front and back of your driver's license and send the image data to our server."

[0694] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0695] Step 1: Take a photo of the document and preview the image

[0696] The user takes a photo of the application or confirmation document using the device's camera. The user launches the device's mobile application and uses the camera function to take a clear photo of the front of the driver's license. The application then displays the captured image on a preview screen and asks the user to confirm. The user checks the captured image on the preview screen and taps the "Confirm" button to proceed.

[0697] Input: Document image taken by camera

[0698] Output: Image displayed on the preview screen

[0699] Step 2: Generate image data and send it to the server

[0700] The device generates image data based on the image viewed by the user. This image data is compressed, encrypted, and sent to the server. Specifically, the image captured by the device's internal processing is saved in JPEG or PNG format, encrypted with AES, and sent to the server using the HTTPS protocol.

[0701] Input: Image confirmed by user

[0702] Output: Encrypted image data (compressed)

[0703] Step 3: Receiving and temporarily saving image data

[0704] The server receives the image data sent from the device and temporarily stores it. The server decrypts the encrypted image data received via HTTPS communication and temporarily stores it in storage. This temporarily stored data is used in the next analysis step.

[0705] Input: Encrypted image data

[0706] Output: Temporarily saved decrypted image data

[0707] Step 4: Analyzing the image data

[0708] The AI ​​system on the server analyzes the stored image data. It uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image data and detect specific elements (such as seals and expiration dates). Specifically, it runs a Python script to extract text information using TensorFlow and the Tesseract OCR library, and performs image recognition using OpenCV.

[0709] Input: Temporarily saved image data

[0710] Output: Extracted text information (name, expiration date, whether or not it has a seal, etc.)

[0711] Step 5: Determine consistency and validity

[0712] The server determines the consistency and validity of the application and confirmation documents based on the text information extracted from the AI ​​system. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document, whether there is a seal, and whether the expiration date is beyond the current date. This process requires connection to a database, and the check is performed using an SQL query.

[0713] Input: Extracted text information

[0714] Output: Consistency and validity determination results

[0715] Step 6: Sentiment analysis and feedback generation

[0716] The server uses sentiment analysis technology to analyze the user's input content and input speed and infer the user's emotional state. Natural language processing libraries (e.g., spaCy, NLTK) and machine learning models (e.g., BERT) are used for sentiment analysis. Based on the results of this analysis, the server generates an appropriate feedback message. For example, if the input speed is extremely slow or there are many corrections, the server can determine that the user is feeling anxious or stressed and generate a message such as, "Was it difficult to input? We are here to help if you need any help."

[0717] Input: User input and speed data

[0718] Output: The generated feedback message

[0719] Step 7: Submit feedback and final approval

[0720] The server sends the generated feedback message to the user. If all conditions are met, the server saves the application and confirmation documents in the database. Finally, it sends a notification of the completion of the procedure to the user's device, and the user receives the notification and confirms that the procedure has been completed.

[0721] Input: Feedback message and final approval information

[0722] Output: Notify user and save to database

[0723] (Application example 2)

[0724] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0725] In the conventional document review process, manual review required a lot of time and effort. Furthermore, there was a lack of mechanisms to reduce the anxiety and stress users felt when submitting documents. A system to address this issue is needed, and it is necessary to develop a system that not only streamlines the review process but also provides feedback that takes users' emotions into account.

[0726] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0727] In this invention, the server includes: means for scanning various documents with a photographing device of a terminal and generating image data; means for transmitting the image data to a data processing device and analyzing it with an artificial intelligence system; means for the artificial intelligence system to extract text information from the image data and detect specific information (such as a seal or expiration date); means for the data processing device to automatically determine the integrity and validity of the document based on the text information; means for notifying the user of the determination result and receiving feedback; means including an emotion analysis engine that analyzes emotions based on the user's input content and input speed and adjusts the feedback content; and means for installing and operating the system on a smartphone or computer. This makes it possible to streamline the review process and provide appropriate feedback that takes the user's emotions into consideration.

[0728] "Documents" refers to multiple different types of paper or electronic documents, such as contracts, applications, confirmations, etc.

[0729] "Terminal" means a device that can be operated by a user, such as a smartphone, tablet, laptop, or desktop computer.

[0730] The term "photography device" refers to a device for capturing images, such as a camera mounted on a terminal.

[0731] "Image data" refers to data that digitally represents an image captured by an imaging device.

[0732] "Data processing device" refers to a computer system for analyzing and processing acquired image data and text information.

[0733] An "artificial intelligence system" is a system that uses algorithms such as machine learning and deep learning to automatically analyze data and extract and judge specific information.

[0734] "Text information" refers to text data extracted from image data.

[0735] "Specific information" refers to specific elements written on a document, such as a seal or expiration date.

[0736] "Document integrity and validity" refers to determining whether the application details are correct and consistent, and whether the documents are legally valid.

[0737] "Judgment result" refers to information obtained as a result of analysis by an artificial intelligence system and evaluation by a data processing device.

[0738] "Feedback" refers to answers or advice provided to users based on the results of their decisions.

[0739] An "emotion analysis engine" is software that analyzes the speed and content of a user's input and estimates that person's emotions.

[0740] The present invention provides a system for improving the efficiency of the review process of various documents and adjusting the feedback content by analyzing the user's emotions. Hereinafter, a specific embodiment of the present invention will be described.

[0741] System configuration

[0742] The system is broadly composed of the following components:

[0743] 1. Terminal

[0744] 2. Data Processing Device

[0745] 3. Artificial Intelligence Systems

[0746] 4. Sentiment Analysis Engine

[0747] 1. Terminal

[0748] Users use devices such as smartphones, tablets, and laptops. The devices are equipped with a camera, which is used to capture image data of various documents.

[0749] The user uses an application on the device to take a photo of a document. The captured image is previewed on the device and the user is asked to confirm. Once the user confirms, the image data is compressed, encrypted, and sent to the data processing device.

[0750] 2. Data Processing Device

[0751] The data processing device includes a storage device for temporarily storing the received image data. The data processing device functions as a server and requests the artificial intelligence system to analyze the image data.

[0752] 3. Artificial Intelligence Systems

[0753] The artificial intelligence system uses TensorFlow, an open-source framework, to build a machine learning model. The system uses OCR technology to extract text from images and detect specific information (such as seals and expiration dates).

[0754] As a specific example, an image of a driver's license can be analyzed to extract information such as the name, date of birth, and expiration date. This can be done with high accuracy using a pre-trained model.

[0755] 4. Sentiment Analysis Engine

[0756] The sentiment analysis engine uses natural language processing tools such as TextBlob to analyze the content and speed of the user's input to detect stress or anxiety, allowing the system to generate appropriate feedback for the user.

[0757] For example, consider the following prompt:

[0758] "I've been a little concerned about the service here lately."

[0759] The sentiment analysis engine analyzes this input, and if the emotion is strongly negative, it generates a friendly message such as, "Was it difficult to enter? We are here to help if you need any help."

[0760] These elements work together to improve the efficiency of the review process and the user experience as a whole system.

[0761] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0762] Step 1:

[0763] The user takes an image of a document on the device. The input is the image captured through the device's camera function, and the output is the captured document image data. After taking the image, the user can check the image on the preview screen.

[0764] Step 2:

[0765] The terminal compresses and encrypts the image data of the document that has been confirmed by the user and sends it to the server. The input is the confirmed image data, and the output is the compressed and encrypted image data. This ensures that the data is securely transferred to the server.

[0766] Step 3:

[0767] The server receives the image data sent from the terminal and temporarily stores it in storage. The input is compressed and encrypted image data, and the output is data stored in the server's storage.

[0768] Step 4:

[0769] The server requests the AI ​​system to analyze the stored image data. The input is the image data stored in the storage, and the output is the data sent to the AI ​​system.

[0770] Step 5:

[0771] The AI ​​system extracts text information from image data and detects specific information (such as a seal or expiration date). The input is image data, and the output is the extracted text information and specific information. The AI ​​system uses the TensorFlow framework.

[0772] Step 6:

[0773] The server determines the integrity and validity of the document based on the text information received from the AI ​​system. The input is the text information extracted by the AI ​​system, and the output is the judgment result. The server performs this automatically.

[0774] Step 7:

[0775] The server notifies the user of the decision result and receives feedback. The input is the decision result, and the output is the notification to the user and the feedback from the user.

[0776] Step 8:

[0777] The device analyzes the user's input content and input speed using a sentiment analysis engine. The input is the user's feedback content, and the output is the sentiment analysis result. The sentiment analysis engine uses TextBlob.

[0778] Step 9:

[0779] The server adjusts the feedback content based on the emotion analysis results and provides it to the user. The input is the emotion analysis results, and the output is the adjusted feedback. This improves user satisfaction.

[0780] This series of processes streamlines the document review process while providing appropriate feedback that takes users' emotions into consideration.

[0781] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0782] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0783] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0784] [Third embodiment]

[0785] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0786] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0787] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0788] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0789] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0790] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0791] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0792] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0793] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0794] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0795] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0796] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0797] This invention relates to an AI system for streamlining the document review process. This AI system uses a device's camera to read application and confirmation documents, generate image data, and send it to a server. The AI ​​system in the server analyzes this image data to automatically determine the integrity and validity of the documents and provide feedback on the results to the user.

[0798] Specific system configuration and program processing

[0799] Image data generation and transmission

[0800] The terminal is operated by the user, who takes a photo of the application or confirmation document with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, the image data is generated and sent to the server.

[0801] As a concrete example, consider the case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear photo of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[0802] Receiving and analyzing image data

[0803] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image and detect specific elements (such as seals and expiration dates).

[0804] For example, an AI system can extract information such as the name, date of birth, and expiration date from an image of a driver's license. It also checks whether the image contains a stamp and determines whether the expiration date is past the current date.

[0805] Consistency determination and notification

[0806] The server checks the consistency of the application and confirmation documents based on the extracted text information. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal has been affixed and whether the expiration date is valid.

[0807] For example, the server checks whether there are any discrepancies between the "applicant name" on the application form and the "name" on the driver's license. At the same time, it checks whether the required seal has been properly stamped and whether the expiration date has passed the current date.

[0808] Feedback and final approval

[0809] The server notifies the user of the results of the AI's automated assessment. The user receives the notification and, if necessary, submits revised documents. By repeating this process, if all conditions are met, the server stores the application and confirmation document information in a database and takes steps to proceed to the next step in the process.

[0810] For example, the server may notify the user of an error message such as "Name does not match" or "Validity period has expired." The user may then correct the error and submit the document again. If the server determines that all information is correct and consistent, it will send a notification to the user that the process has been completed. The user will receive this notification and confirm that the process has been completed successfully.

[0811] This system automates the process of reviewing various documents, shortening operation time and eliminating judgment errors, thereby improving productivity and is expected to be used in many industries.

[0812] The processing flow will be explained below.

[0813] Program processing steps

[0814] Step 1:

[0815] The terminal starts a mobile application, and the user selects the camera function.

[0816] Specific operation: The user taps the "Launch Camera" button in the app to launch the camera. A shooting guide will be displayed on the screen.

[0817] Step 2:

[0818] The user takes a photo of the application or confirmation document with a camera.

[0819] Specific actions: Place the application form or confirmation document within the camera frame and press the shutter button to take the photo.

[0820] Step 3:

[0821] The device displays a preview of the captured image and prompts the user to confirm it.

[0822] Specific operation: A preview screen is displayed and a "Use this image" button is displayed. The user confirms the image and presses the OK button.

[0823] Step 4:

[0824] The device compresses and encrypts the image data and sends it to the server.

[0825] What happens: Image data is processed within the app and sent to a server using a secure communication protocol.

[0826] Step 5:

[0827] The server temporarily stores the image data received from the terminal.

[0828] Specific operation: The received image data is stored in the server storage and prepared for analysis.

[0829] Step 6:

[0830] The server requests the AI ​​system to analyze the image data.

[0831] Specific operation: The saved image data is passed to the AI ​​analysis module, and the analysis task is initiated.

[0832] Step 7:

[0833] The AI ​​system uses OCR technology to extract text information from images.

[0834] Specific operation: Identifies text areas within an image and extracts text information such as name and date of birth as digital data.

[0835] Step 8:

[0836] The AI ​​system uses image recognition algorithms to detect specific elements (stamps, expiration dates, etc.).

[0837] Specific operation: Check whether the stamp position and expiration date are correct. Analyze other detection elements in the same way.

[0838] Step 9:

[0839] The server checks the consistency of the application documents and confirmation documents based on the extracted text information.

[0840] Specific operations: For example, compare the "applicant name" written on the application form with the "name" on the verification document to check for any discrepancies.

[0841] Step 10:

[0842] The server receives the results of the AI ​​system and generates feedback.

[0843] Specific operation: Set a status of "pass" or "fail" for each check item and summarize the results.

[0844] Step 11:

[0845] The server notifies the user of the result of the determination.

[0846] Specific operation: Notify the user of the results in real time via a mobile application.

[0847] Step 12:

[0848] The user receives a notification and, if necessary, re-photographs and submits the corrected document.

[0849] Specific operation: If an error occurs, the user re-photographs the corrected document and sends it to the server again.

[0850] Step 13:

[0851] If all conditions are met, the server stores the application and confirmation documents in a database.

[0852] What it does: Securely stores the final data in a database and sets a flag to start the next process.

[0853] Step 14:

[0854] The server sends a notification to the user that the procedure is complete.

[0855] Specific operation: The application sends a notification to the user that the procedure has been completed.

[0856] Step 15:

[0857] The user receives a final notification and confirms that the transaction is complete.

[0858] What happens: The user checks the in-app notification to confirm the transaction is complete.

[0859] Example 1

[0860] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0861] The document review process requires manual review of documents to determine their consistency and validity, which requires a great deal of time and effort. Furthermore, manual reviewing is prone to human error, and deficiencies and errors in documents can be overlooked. Therefore, there is a need to improve the efficiency and accuracy of the review process.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0863] In this invention, the server includes: a means for scanning various documents with a device for taking photos of the terminal and generating image data; a means for transmitting the image data to the server via a communication device and analyzing it with an AI system; a means for the AI ​​system to extract text information from the image data using optical character recognition technology and detect specific elements (such as an approval seal or validity period); a means for the server to automatically determine the consistency and validity of the document based on the text information; and a means for notifying the user of the determination result via the communication device and receiving feedback. This automates the document review process, shortening operation time and eliminating judgment errors. Furthermore, the user can receive feedback and make quick corrections, improving the efficiency and accuracy of the entire review process.

[0864] A "terminal" is a device for taking pictures of various documents and transmitting the generated image data to a server via a communication device.

[0865] The "photography device" is a device such as a camera mounted on a terminal, which is used to record documents as digital images.

[0866] "Image data" refers to digital document image information generated by a photographing device.

[0867] A "communication device" is a network communication device for transmitting image data to a server.

[0868] A "server" is a computing device that receives image data sent from a terminal, temporarily stores it, and then analyzes it using an AI system.

[0869] "AI system" refers to artificial intelligence technology that analyzes image data, extracts text information using optical character recognition technology, and detects specific elements of documents (such as stamps of approval or validity dates).

[0870] Optical character recognition (OCR) is a technology that automatically reads text information from image data and converts it into digital text.

[0871] "Text information" refers to character data extracted from image data using optical character recognition technology.

[0872] "Specific elements" refer to important information items such as the document's stamp of approval, validity period, name, etc.

[0873] "Consistency" refers to a state in which the contents of the application documents and confirmation documents match and are free of contradictions.

[0874] "Validity" refers to the state in which a document is within its legal validity period and meets all necessary requirements.

[0875] "Means for receiving feedback" refers to the function by which the server notifies the user of the result of the judgment and receives corrections or resubmissions from the user.

[0876] This invention relates to an AI system for streamlining the document review process. This AI system involves a series of processes: scanning application and confirmation documents using a device's camera, generating image data, and transmitting it to a server via a communication device. The AI ​​system in the server then analyzes the image data, automatically determining the consistency and validity of the documents, and providing feedback on the results to the user.

[0877] Hardware and software used:

[0878] Device:

[0879] Devices for taking photos of various documents (e.g. smartphones, tablets, etc.)

[0880] Camera as a photographic device

[0881] Application software with image data preview and confirmation functions

[0882] server:

[0883] Storage for receiving and temporarily storing image data

[0884] AI systems (including optical character recognition (OCR) and image recognition algorithms)

[0885] Software for analyzing text information and determining document integrity

[0886] Specific examples:

[0887] Image data is generated when a user takes a photo of an application or confirmation document using the device's camera. At this time, the device displays a preview of the captured image, allowing the user to confirm that the image is clear. For example, a user can take a photo of the front and back of a driver's license and check the image on the preview screen within the application. Once the user has completed the verification, the image data is compressed (e.g., in JPEG format) and encrypted (e.g., using AES) and sent to the server by the communication device.

[0888] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR technology to extract text information from the image and detect specific elements (e.g., name, date of birth, expiration date, whether or not a seal is attached, etc.). During this process, the AI ​​system automatically reads information such as "name," "date of birth," and "expiration date" from the image of the driver's license.

[0889] The server uses the extracted text information to check whether the information on the application and confirmation documents is consistent. For example, it checks whether the "applicant name" on the application form matches the "name" on the driver's license, and whether the expiration date is within the current date. It also checks whether the required seals have been properly stamped. Based on the results, the server sends an error message or a success message to the user.

[0890] The user receives a notification from the server and makes corrections as necessary. This process is repeated until all conditions are met, at which point the server saves the application and confirmation document information in its database and takes steps to proceed to the next step in the process. For example, the server may notify the user with an error message such as "Name does not match" or "Expired date has passed," prompting the user to correct the information and submit it again. Finally, if the server determines that all information is accurate and matches, it will send a notification to the user that the process is complete.

[0891] Example prompts for generative AI models:

[0892] "Extract the name and expiration date from the driver's license."

[0893] Please check that the information on your application matches your verification documents.

[0894] "Please check whether the stamp is properly stamped in the required places on the document."

[0895] This invention automates the document review process, shortens operation time, and eliminates judgment errors, improving productivity and is expected to be used in many industries.

[0896] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0897] Step 1:

[0898] The terminal is operated by the user, who activates the device's image capture to take a photo of the application or confirmation document. The terminal displays the captured image to the user on a preview screen, and after the user checks the image's clarity and content, it processes the data to generate image data. The input is the image of the document captured by the user, and the output is the confirmation image displayed on the preview screen. In concrete terms, the user uses the camera function to take a photo of their driver's license and checks the image on the preview screen.

[0899] Step 2:

[0900] When the user has finished confirming the image, the device recognizes that a button has been pressed on the preview screen requesting confirmation by the user. The image data is then compressed (e.g., JPEG format) and encrypted (e.g., AES encryption). The input is the image data after the user's confirmation, and the output is the compressed and encrypted image data. Specifically, when the user taps the "Send Image" button, the device compresses the image data, encrypts it, and sends it to the server.

[0901] Step 3:

[0902] The server receives image data sent from the terminal and temporarily stores it. The input is the compressed and encrypted image data sent from the terminal, and the output is the image data stored in the server's storage. Specifically, the server receives the image data, checks the integrity of the data, and stores it in storage.

[0903] Step 4:

[0904] The server passes the stored image data to the AI ​​system and requests it to analyze it. The AI ​​system uses OCR technology to extract text information from the image data and detects specific elements (e.g., name, expiration date, whether or not a seal has been affixed). The input is the image data stored on the server, and the output is the analyzed text information. Specifically, the image data is input into the AI ​​system, and text information such as "name," "date of birth," and "expiration date" is extracted through OCR processing.

[0905] Step 5:

[0906] The server uses the extracted text information to check the consistency and validity of the application and confirmation documents. For example, it determines whether the "applicant name" written on the application form matches the "name" extracted from the image, and whether the expiration date is beyond the current date. The input is the text information provided by the AI ​​system, and the output is the result of the consistency and validity determination. Specifically, it compares the text information with the application form information to check for inconsistencies or omissions.

[0907] Step 6:

[0908] The server sends a notification message via a communication device to notify the user of the result of the judgment. The user receives the notification and makes corrections as necessary. The input is the result of the consistency and validity check, and the output is a notification message to the user. Specifically, the user receives an error message on their smartphone or tablet, such as "Name does not match" or "Expired date."

[0909] Step 7:

[0910] The user receives a notification from the server, makes any necessary corrections, photographs the documents again, and submits them. By repeating this process, if all conditions are met, the server saves the application and confirmation document information in its database and proceeds to the next process. The input is new image data of the documents revised by the user, and the output is text information reanalyzed by the server and the final judgment result. In concrete terms, the user photographs the documents again, submits them to the server, and finally receives a notification that the procedure has been completed.

[0911] (Application example 1)

[0912] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0913] The present invention aims to streamline the screening and authentication process for membership cards, coupons, point cards, etc. in physical stores, thereby reducing the burden on users and stores. Traditionally, these screening and authentication processes have often been performed manually, which requires time and effort and is prone to human error. Furthermore, there are issues with physical cards that users carry with them, such as loss or damage, so a solution to these issues is needed.

[0914] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0915] In this invention, the server includes: a means for reading various information with a device camera and generating image data; a means for transmitting the image data to a network server and analyzing it with an artificial intelligence system; a means for the artificial intelligence system to extract text information from the image data and detect specific elements (such as a seal or expiration date); a means for the server to automatically determine the integrity and validity of the document based on the text information; a means for notifying the user of the determination result and receiving feedback; and a means for authenticating the use of membership cards, coupons, and point cards using the feedback. This automates the review and authentication process for membership cards, coupons, and point cards, significantly reducing time and labor. It also reduces the risk of human error and improves user convenience.

[0916] A "device" is an electronic device that is operated by a user and has a camera mounted thereon for reading various types of information.

[0917] "Image data" is digital data generated based on information captured by the device's camera.

[0918] "Network Server" is a central processing unit that receives image data sent from the device and interfaces with the artificial intelligence system for analysis.

[0919] An "artificial intelligence system" is a software system equipped with AI technology that is used to extract textual information from image data and detect specific elements.

[0920] "Text information" is text data extracted from image data by an artificial intelligence system.

[0921] A "seal" is an impression required to indicate the validity of a document.

[0922] "Expiration date" refers to the period during which a document, membership card, coupon, etc. is legally valid.

[0923] "Integrity" is a concept that refers to documents and data being consistent and free of contradictions.

[0924] "Validity" is the state of a document or data that indicates it is suitable for its purpose and can be used.

[0925] "Feedback" refers to response information including notification results and correction instructions for the user.

[0926] A "membership card" is a card or digital certificate used to prove membership in a particular organization or service.

[0927] A "coupon" is a voucher that provides discounts or special offers for specific services or products.

[0928] A "point card" is a card that accumulates points with each purchase and can be used for special offers or discounts.

[0929] "Authenticating use" is the process of verifying the validity of a membership card, coupon, or point card and authorizing its use.

[0930] To implement the present invention, a device used by a user, a network server, and an artificial intelligence system are required. The details of each component and process are as follows.

[0931] 1. Device-side processing

[0932] The device is operated by the user and uses the camera to capture information such as membership cards, coupons, and point cards. At this time, the device displays a preview of the captured image and asks the user to confirm. Once the user has completed the confirmation, image data is generated.

[0933] 2. Sending image data

[0934] The generated image data is sent from the device to a network server, where it is encrypted to ensure confidentiality.

[0935] 3. Server-side processing

[0936] The server receives the image data sent from the device, temporarily stores it in storage, and then requests an AI system to analyze the image data.

[0937] 4. Image analysis using artificial intelligence systems

[0938] The AI ​​system uses OCR (optical character recognition) technology to extract text information from the received image data. Specifically, it detects specific elements such as ID numbers, expiration dates, and stamps on membership cards and coupons. The software used here is, for example, OpenCV and pytesseract.

[0939] 5. Checking data integrity and validity

[0940] The server checks whether the membership card or coupon is valid based on the character information extracted by the AI ​​system. This involves checking against information stored in a database. A specific example is a process that checks whether the extracted expiration date is beyond the current date.

[0941] 6. Providing Feedback

[0942] The results of the review are fed back to the user in real time. For example, a message such as "This is a valid membership card" or "This card has expired" is displayed. The user can receive this feedback, correct the information as needed, and submit it again.

[0943] Specific examples

[0944] The user launches the smartphone app, presses the "Membership Card Photo" button to take a photo of their membership card, checks the image on the preview screen, and then presses the "Send" button, which sends the image data to the server. The server analyzes this image data and notifies the user of the results. Specific examples of screen displays and notifications that can be used include prompts such as the following:

[0945] Prompt Sentence Examples

[0946] User: Press the "Membership Card Photo" button to take a photo of your membership card, check it on the preview screen, and then press the "Send" button.

[0947] This will streamline the review and authentication process for membership cards, coupons, and point cards at physical stores, while also improving user convenience.

[0948] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0949] Step 1:

[0950] The user takes a photo of a membership card or coupon with the camera. The input to the device is the user pressing the capture button, and image data is generated as output. The device displays a preview of this image data and asks the user to confirm it. When the user presses the confirmation button, the device confirms the image data.

[0951] Step 2:

[0952] The device sends the determined image data to the network server. The input is the generated image data, and the output is the data sent over the network. At this time, the data is encrypted to maintain security.

[0953] Step 3:

[0954] The server receives image data sent from the device and temporarily stores it in storage. The input is the sent image data, and the data temporarily stored in storage is the output.

[0955] Step 4:

[0956] The server passes the stored image data to the AI ​​system and requests it to analyze it. The input is the image data stored in the storage, and the output is the image data that the AI ​​system receives.

[0957] Step 5:

[0958] The artificial intelligence system uses OCR technology to extract text information from the received image data. Specifically, it detects ID numbers, expiration dates, and seals on membership cards and coupons contained in the image. The input is image data, and the output is the extracted text information. Software such as OpenCV and pytesseract is used for this processing.

[0959] Step 6:

[0960] The server checks the consistency and validity of the corresponding data based on the text information provided by the AI ​​system. Specifically, it compares it with the information registered in the database to see if it matches and if it has expired. The input is the extracted text information, and the output is the result of the consistency and validity determination.

[0961] Step 7:

[0962] The server notifies the user of the result of the determination, displaying real-time feedback on the user's device and sending messages such as "This is a valid membership card" or "This card has expired." The input is the determination result, and the output is a notification message displayed on the user's device.

[0963] Step 8:

[0964] The user receives the notification message and, if necessary, takes another photo of the membership card or coupon and submits it. This process loops back to step 1 until the server determines that all information is accurate and consistent. Finally, if the information is authenticated, the membership card or coupon is authorized for use.

[0965] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0966] This invention relates to a system that improves user experience by combining an AI system for streamlining the document review process with an emotion engine that recognizes user emotions. This system uses the device's camera to read application and confirmation documents, generate image data, and send it to a server. The AI ​​system on the server analyzes this image data and automatically determines the integrity and validity of the documents. It also uses the emotion engine to analyze the user's emotions when entering input and adjust the feedback content.

[0967] Specific system configuration and program processing

[0968] Image data generation and transmission

[0969] The terminal is operated by the user, who takes a photo of the application or confirmation document with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, the image data is generated and sent to the server.

[0970] As a concrete example, consider the case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear photo of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[0971] Receiving and analyzing image data

[0972] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image and detect specific elements (such as seals and expiration dates).

[0973] For example, an AI system can extract information such as the name, date of birth, and expiration date from an image of a driver's license. It also checks whether the image contains a stamp and determines whether the expiration date is past the current date.

[0974] Consistency determination and notification

[0975] The server checks the consistency of the application and confirmation documents based on the extracted text information. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal has been affixed and whether the expiration date is valid.

[0976] For example, the server checks whether there are any discrepancies between the "applicant name" on the application form and the "name" on the driver's license. At the same time, it checks whether the required seal has been properly stamped and whether the expiration date has passed the current date.

[0977] Introducing an emotion engine and adjusting feedback

[0978] The server is equipped with an emotion engine that analyzes the user's emotions based on their input content and speed. The emotion engine detects stress and anxiety when the user submits the application and generates feedback accordingly.

[0979] For example, if the emotion engine determines that the user is feeling anxious or stressed when the user's input speed is extremely slow or when the input content has been corrected many times, the server will send a friendly message such as, "Was it difficult to input? We are here to help if you need any help."

[0980] Final approval and saving

[0981] If all conditions are met, the server stores the application and confirmation documents in the database and takes steps to proceed to the next step. The emotion engine also further analyzes the user's feedback and stores it as data to improve the user experience.

[0982] For example, the server notifies the user with error messages such as "Name does not match" or "Expired date has passed," and the emotion engine analyzes the user's reaction and adjusts the feedback. If the server determines that all information is accurate and matches after the user resubmits the corrected document, it sends a notification to the user that the process has been completed. The user receives the notification and confirms that the process has been completed successfully.

[0983] This system automates the process of reviewing various documents, shortening operation time and eliminating judgment errors. Furthermore, an emotion engine provides feedback that takes into account the user's emotions, improving the user experience. This improves productivity and is expected to be used in many industries.

[0984] The processing flow will be explained below.

[0985] Program processing steps

[0986] Step 1:

[0987] The terminal starts a mobile application, and the user selects the camera function.

[0988] Specific operation: The user taps the "Launch Camera" button in the app to launch the camera. A shooting guide will be displayed on the screen.

[0989] Step 2:

[0990] The user takes a photo of the application or confirmation document with a camera.

[0991] Specific actions: Place the application form or confirmation document within the camera frame and press the shutter button to take the photo.

[0992] Step 3:

[0993] The device displays a preview of the captured image and prompts the user to confirm it.

[0994] Specific operation: A preview screen is displayed and a "Use this image" button is displayed. The user confirms the image and presses the OK button.

[0995] Step 4:

[0996] The device compresses and encrypts the image data and sends it to the server.

[0997] What happens: Image data is processed within the app and sent to a server using a secure communication protocol.

[0998] Step 5:

[0999] The server temporarily stores the image data received from the terminal.

[1000] Specific operation: The received image data is stored in the server storage and prepared for analysis.

[1001] Step 6:

[1002] The server requests the AI ​​system to analyze the image data.

[1003] Specific operation: The saved image data is passed to the AI ​​analysis module, and the analysis task is initiated.

[1004] Step 7:

[1005] The AI ​​system uses OCR technology to extract text information from images.

[1006] Specific operation: Identifies text areas within an image and extracts text information such as name and date of birth as digital data.

[1007] Step 8:

[1008] The AI ​​system uses image recognition algorithms to detect specific elements (stamps, expiration dates, etc.).

[1009] Specific operation: Check whether the stamp position and expiration date are correct. Analyze other detection elements in the same way.

[1010] Step 9:

[1011] The server checks the consistency of the application documents and confirmation documents based on the extracted text information.

[1012] Specific operations: For example, compare the "applicant name" written on the application form with the "name" on the verification document to check for any discrepancies.

[1013] Step 10:

[1014] The server receives the results of the AI ​​system and generates feedback.

[1015] Specific operation: Set a status of "pass" or "fail" for each check item and summarize the results.

[1016] Step 11:

[1017] The server notifies the user of the result of the determination.

[1018] Specific operation: Notify the user of the results in real time via a mobile application.

[1019] Step 12:

[1020] The user receives a notification and, if necessary, re-photographs and submits the corrected document.

[1021] Specific operation: If an error occurs, the user re-photographs the corrected document and sends it to the server again.

[1022] Step 13:

[1023] The server uses an emotion engine to analyze the user's emotions.

[1024] Specific operation: Analyzes the user's input speed and frequency of content revisions to determine whether the user is feeling stressed or anxious.

[1025] Step 14:

[1026] The server adjusts the feedback content based on the results analyzed by the emotion engine.

[1027] Specific operation: For example, if it is determined that the user is in a stressful state, a friendly follow-up message is generated and sent.

[1028] Step 15:

[1029] If all conditions are met, the server stores the application and confirmation documents in a database.

[1030] What it does: Securely stores the final data in a database and sets a flag to start the next process.

[1031] Step 16:

[1032] The server sends a notification to the user that the procedure is complete.

[1033] Specific operation: The application sends a notification to the user that the procedure has been completed.

[1034] Step 17:

[1035] The user receives a final notification and confirms that the transaction is complete.

[1036] What happens: The user checks the in-app notification to confirm the transaction is complete.

[1037] Example 2

[1038] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1039] Previous document review processes were often manual, which was time-consuming, labor-intensive, and prone to errors. Furthermore, there was little technology available to reduce the stress and anxiety users felt when submitting documents. Therefore, there was a need for a system that could review documents efficiently and accurately. Furthermore, there was a need to improve the user experience by analyzing user emotions and providing appropriate feedback.

[1040] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1041] In this invention, the server includes means for temporarily storing image data and analyzing it using AI technology, means for extracting text information from the image data and detecting specific elements (such as a seal or expiration date) using AI technology, and means for detecting the user's emotional state using emotion analysis technology, generating appropriate feedback, and sending it to the user. This not only automates the document review process efficiently and accurately, but also makes it possible to provide feedback that takes the user's emotions into consideration.

[1042] A "terminal" is a device that is operated by a user to photograph a document and generate image data.

[1043] A "camera" is hardware with a photographing function that is installed on a terminal.

[1044] A "server" is a computing system that receives, stores, analyzes, and generates feedback from image data.

[1045] "Image data" is a digital image file of a document captured by the device's camera.

[1046] "AI technology" refers to artificial intelligence technology that analyzes image data using optical character recognition technology and image recognition algorithms.

[1047] "Text information" is character information extracted from image data using AI technology.

[1048] "Specific elements" refer to specific information such as a seal or expiration date on a document.

[1049] "Consistency" is a criterion for evaluating whether the information in the application documents and the confirmation documents match.

[1050] "Validity" is a criterion for assessing whether a document is currently valid.

[1051] "Emotion analysis technology" is a technology that analyzes the content and speed of a user's input and infers the user's emotional state.

[1052] "Feedback" refers to the response or message that the server provides to the user based on the analysis results.

[1053] "Temporary storage" is a process in which the server temporarily stores the image data received in storage.

[1054] This invention is a system that streamlines the document review process and combines it with an emotion engine that analyzes user emotions. This system uses the device's camera to read application and confirmation documents, generate image data, and send it to a server. An AI system on the server analyzes this image data and automatically determines the integrity and validity of the documents. The emotion engine is also used to analyze the user's emotions when they enter information, and the feedback content is adjusted to improve the user experience.

[1055] Generation and transmission of image data by the terminal

[1056] The terminal is operated by the user, who takes a photo of the application or confirmation documents with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, image data is generated, compressed, encrypted, and sent to the server. This process uses the camera function of a smartphone or tablet, and the mobile application that controls it.

[1057] Receipt of image data by the server

[1058] The server receives and temporarily stores image data sent from the device. This data is received via API, temporarily stored in storage, and then passed to the AI ​​system.

[1059] Image data analysis by the server

[1060] The AI ​​system in the server uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from images and detect specific elements (such as seals, expiration dates, etc.) The technologies used include Python, TensorFlow, Tesseract OCR, and OpenCV.

[1061] Determining document integrity

[1062] The server checks the consistency of the application and confirmation documents based on the text information extracted from the AI ​​system. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal is affixed and whether the expiration date is valid.

[1063] Emotion analysis using an emotion engine

[1064] The server is equipped with an emotion engine that analyzes the user's emotions based on their input and typing speed. It uses natural language processing libraries (e.g., spaCy, NLTK) and machine learning models (e.g., BERT) to detect stress and anxiety when the user submits their application and generates feedback accordingly.

[1065] Generate and send feedback

[1066] The server generates an appropriate feedback message based on the analysis results of the emotion engine and sends it to the user, such as "Did you have difficulty entering the information? We are here to help if you need any assistance."

[1067] Final approval and data storage

[1068] If all conditions are met, the server stores the application and confirmation documents in the database and takes steps to proceed to the next step. The emotion engine also further analyzes the user's feedback and stores it as data to improve the user experience. Finally, a notification of the completion of the process is sent to the user, who then receives the notification and confirms that the process has been completed.

[1069] Examples of specific examples and prompts

[1070] As a concrete example, let's consider a case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear picture of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[1071] Example prompts to input to a generative AI model:

[1072] "Take a photo of the front and back of your driver's license and send the image data to our server."

[1073] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1074] Step 1: Take a photo of the document and preview the image

[1075] The user takes a photo of the application or confirmation document using the device's camera. The user launches the device's mobile application and uses the camera function to take a clear photo of the front of the driver's license. The application then displays the captured image on a preview screen and asks the user to confirm. The user checks the captured image on the preview screen and taps the "Confirm" button to proceed.

[1076] Input: Document image taken by camera

[1077] Output: Image displayed on the preview screen

[1078] Step 2: Generate image data and send it to the server

[1079] The device generates image data based on the image viewed by the user. This image data is compressed, encrypted, and sent to the server. Specifically, the image captured by the device's internal processing is saved in JPEG or PNG format, encrypted with AES, and sent to the server using the HTTPS protocol.

[1080] Input: Image confirmed by user

[1081] Output: Encrypted image data (compressed)

[1082] Step 3: Receiving and temporarily saving image data

[1083] The server receives the image data sent from the device and temporarily stores it. The server decrypts the encrypted image data received via HTTPS communication and temporarily stores it in storage. This temporarily stored data is used in the next analysis step.

[1084] Input: Encrypted image data

[1085] Output: Temporarily saved decrypted image data

[1086] Step 4: Analyzing the image data

[1087] The AI ​​system on the server analyzes the stored image data. It uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image data and detect specific elements (such as seals and expiration dates). Specifically, it runs a Python script to extract text information using TensorFlow and the Tesseract OCR library, and performs image recognition using OpenCV.

[1088] Input: Temporarily saved image data

[1089] Output: Extracted text information (name, expiration date, whether or not it has a seal, etc.)

[1090] Step 5: Determine consistency and validity

[1091] The server determines the consistency and validity of the application and confirmation documents based on the text information extracted from the AI ​​system. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document, whether there is a seal, and whether the expiration date is beyond the current date. This process requires connection to a database, and the check is performed using an SQL query.

[1092] Input: Extracted text information

[1093] Output: Consistency and validity determination results

[1094] Step 6: Sentiment analysis and feedback generation

[1095] The server uses sentiment analysis technology to analyze the user's input content and input speed and infer the user's emotional state. Natural language processing libraries (e.g., spaCy, NLTK) and machine learning models (e.g., BERT) are used for sentiment analysis. Based on the results of this analysis, the server generates an appropriate feedback message. For example, if the input speed is extremely slow or there are many corrections, the server can determine that the user is feeling anxious or stressed and generate a message such as, "Was it difficult to input? We are here to help if you need any help."

[1096] Input: User input and speed data

[1097] Output: The generated feedback message

[1098] Step 7: Submit feedback and final approval

[1099] The server sends the generated feedback message to the user. If all conditions are met, the server saves the application and confirmation documents in the database. Finally, it sends a notification of the completion of the procedure to the user's device, and the user receives the notification and confirms that the procedure has been completed.

[1100] Input: Feedback message and final approval information

[1101] Output: Notify user and save to database

[1102] (Application example 2)

[1103] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1104] In the conventional document review process, manual review required a lot of time and effort. Furthermore, there was a lack of mechanisms to reduce the anxiety and stress users felt when submitting documents. A system to address this issue is needed, and it is necessary to develop a system that not only streamlines the review process but also provides feedback that takes users' emotions into account.

[1105] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1106] In this invention, the server includes: means for scanning various documents with a photographing device of a terminal and generating image data; means for transmitting the image data to a data processing device and analyzing it with an artificial intelligence system; means for the artificial intelligence system to extract text information from the image data and detect specific information (such as a seal or expiration date); means for the data processing device to automatically determine the integrity and validity of the document based on the text information; means for notifying the user of the determination result and receiving feedback; means including an emotion analysis engine that analyzes emotions based on the user's input content and input speed and adjusts the feedback content; and means for installing and operating the system on a smartphone or computer. This makes it possible to streamline the review process and provide appropriate feedback that takes the user's emotions into consideration.

[1107] "Documents" refers to multiple different types of paper or electronic documents, such as contracts, applications, confirmations, etc.

[1108] "Terminal" means a device that can be operated by a user, such as a smartphone, tablet, laptop, or desktop computer.

[1109] The term "photography device" refers to a device for capturing images, such as a camera mounted on a terminal.

[1110] "Image data" refers to data that digitally represents an image captured by an imaging device.

[1111] "Data processing device" refers to a computer system for analyzing and processing acquired image data and text information.

[1112] An "artificial intelligence system" is a system that uses algorithms such as machine learning and deep learning to automatically analyze data and extract and judge specific information.

[1113] "Text information" refers to text data extracted from image data.

[1114] "Specific information" refers to specific elements written on a document, such as a seal or expiration date.

[1115] "Document integrity and validity" refers to determining whether the application details are correct and consistent, and whether the documents are legally valid.

[1116] "Judgment result" refers to information obtained as a result of analysis by an artificial intelligence system and evaluation by a data processing device.

[1117] "Feedback" refers to answers or advice provided to users based on the results of their decisions.

[1118] An "emotion analysis engine" is software that analyzes the speed and content of a user's input and estimates that person's emotions.

[1119] The present invention provides a system for improving the efficiency of the review process of various documents and adjusting the feedback content by analyzing the user's emotions. Hereinafter, a specific embodiment of the present invention will be described.

[1120] System configuration

[1121] The system is broadly composed of the following components:

[1122] 1. Terminal

[1123] 2. Data Processing Device

[1124] 3. Artificial Intelligence Systems

[1125] 4. Sentiment Analysis Engine

[1126] 1. Terminal

[1127] Users use devices such as smartphones, tablets, and laptops. The devices are equipped with a camera, which is used to capture image data of various documents.

[1128] The user uses an application on the device to take a photo of a document. The captured image is previewed on the device and the user is asked to confirm. Once the user confirms, the image data is compressed, encrypted, and sent to the data processing device.

[1129] 2. Data Processing Device

[1130] The data processing device includes a storage device for temporarily storing the received image data. The data processing device functions as a server and requests the artificial intelligence system to analyze the image data.

[1131] 3. Artificial Intelligence Systems

[1132] The artificial intelligence system uses TensorFlow, an open-source framework, to build a machine learning model. The system uses OCR technology to extract text from images and detect specific information (such as seals and expiration dates).

[1133] As a specific example, an image of a driver's license can be analyzed to extract information such as the name, date of birth, and expiration date. This can be done with high accuracy using a pre-trained model.

[1134] 4. Sentiment Analysis Engine

[1135] The sentiment analysis engine uses natural language processing tools such as TextBlob to analyze the content and speed of the user's input to detect stress or anxiety, allowing the system to generate appropriate feedback for the user.

[1136] For example, consider the following prompt:

[1137] "I've been a little concerned about the service here lately."

[1138] The sentiment analysis engine analyzes this input, and if the emotion is strongly negative, it generates a friendly message such as, "Was it difficult to enter? We are here to help if you need any help."

[1139] These elements work together to improve the efficiency of the review process and the user experience as a whole system.

[1140] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1141] Step 1:

[1142] The user takes an image of a document on the device. The input is the image captured through the device's camera function, and the output is the captured document image data. After taking the image, the user can check the image on the preview screen.

[1143] Step 2:

[1144] The terminal compresses and encrypts the image data of the document that has been confirmed by the user and sends it to the server. The input is the confirmed image data, and the output is the compressed and encrypted image data. This ensures that the data is securely transferred to the server.

[1145] Step 3:

[1146] The server receives the image data sent from the terminal and temporarily stores it in storage. The input is compressed and encrypted image data, and the output is data stored in the server's storage.

[1147] Step 4:

[1148] The server requests the AI ​​system to analyze the stored image data. The input is the image data stored in the storage, and the output is the data sent to the AI ​​system.

[1149] Step 5:

[1150] The AI ​​system extracts text information from image data and detects specific information (such as a seal or expiration date). The input is image data, and the output is the extracted text information and specific information. The AI ​​system uses the TensorFlow framework.

[1151] Step 6:

[1152] The server determines the integrity and validity of the document based on the text information received from the AI ​​system. The input is the text information extracted by the AI ​​system, and the output is the judgment result. The server performs this automatically.

[1153] Step 7:

[1154] The server notifies the user of the decision result and receives feedback. The input is the decision result, and the output is the notification to the user and the feedback from the user.

[1155] Step 8:

[1156] The device analyzes the user's input content and input speed using a sentiment analysis engine. The input is the user's feedback content, and the output is the sentiment analysis result. The sentiment analysis engine uses TextBlob.

[1157] Step 9:

[1158] The server adjusts the feedback content based on the emotion analysis results and provides it to the user. The input is the emotion analysis results, and the output is the adjusted feedback. This improves user satisfaction.

[1159] This series of processes streamlines the document review process while providing appropriate feedback that takes users' emotions into consideration.

[1160] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1161] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1162] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1163] [Fourth embodiment]

[1164] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1165] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1166] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1167] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1168] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1169] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1170] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1171] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1172] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1173] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1174] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1175] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1176] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1177] This invention relates to an AI system for streamlining the document review process. This AI system uses a device's camera to read application and confirmation documents, generate image data, and send it to a server. The AI ​​system in the server analyzes this image data to automatically determine the integrity and validity of the documents and provide feedback on the results to the user.

[1178] Specific system configuration and program processing

[1179] Image data generation and transmission

[1180] The terminal is operated by the user, who takes a photo of the application or confirmation document with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, the image data is generated and sent to the server.

[1181] As a concrete example, consider the case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear photo of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[1182] Receiving and analyzing image data

[1183] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image and detect specific elements (such as seals and expiration dates).

[1184] For example, an AI system can extract information such as the name, date of birth, and expiration date from an image of a driver's license. It also checks whether the image contains a stamp and determines whether the expiration date is past the current date.

[1185] Consistency determination and notification

[1186] The server checks the consistency of the application and confirmation documents based on the extracted text information. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal has been affixed and whether the expiration date is valid.

[1187] For example, the server checks whether there are any discrepancies between the "applicant name" on the application form and the "name" on the driver's license. At the same time, it checks whether the required seal has been properly stamped and whether the expiration date has passed the current date.

[1188] Feedback and final approval

[1189] The server notifies the user of the results of the AI's automated assessment. The user receives the notification and, if necessary, submits revised documents. By repeating this process, if all conditions are met, the server stores the application and confirmation document information in a database and takes steps to proceed to the next step in the process.

[1190] For example, the server may notify the user of an error message such as "Name does not match" or "Validity period has expired." The user may then correct the error and submit the document again. If the server determines that all information is correct and consistent, it will send a notification to the user that the process has been completed. The user will receive this notification and confirm that the process has been completed successfully.

[1191] This system automates the process of reviewing various documents, shortening operation time and eliminating judgment errors, thereby improving productivity and is expected to be used in many industries.

[1192] The processing flow will be explained below.

[1193] Program processing steps

[1194] Step 1:

[1195] The terminal starts a mobile application, and the user selects the camera function.

[1196] Specific operation: The user taps the "Launch Camera" button in the app to launch the camera. A shooting guide will be displayed on the screen.

[1197] Step 2:

[1198] The user takes a photo of the application or confirmation document with a camera.

[1199] Specific actions: Place the application form or confirmation document within the camera frame and press the shutter button to take the photo.

[1200] Step 3:

[1201] The device displays a preview of the captured image and prompts the user to confirm it.

[1202] Specific operation: A preview screen is displayed and a "Use this image" button is displayed. The user confirms the image and presses the OK button.

[1203] Step 4:

[1204] The device compresses and encrypts the image data and sends it to the server.

[1205] What happens: Image data is processed within the app and sent to a server using a secure communication protocol.

[1206] Step 5:

[1207] The server temporarily stores the image data received from the terminal.

[1208] Specific operation: The received image data is stored in the server storage and prepared for analysis.

[1209] Step 6:

[1210] The server requests the AI ​​system to analyze the image data.

[1211] Specific operation: The saved image data is passed to the AI ​​analysis module, and the analysis task is initiated.

[1212] Step 7:

[1213] The AI ​​system uses OCR technology to extract text information from images.

[1214] Specific operation: Identifies text areas within an image and extracts text information such as name and date of birth as digital data.

[1215] Step 8:

[1216] The AI ​​system uses image recognition algorithms to detect specific elements (stamps, expiration dates, etc.).

[1217] Specific operation: Check whether the stamp position and expiration date are correct. Analyze other detection elements in the same way.

[1218] Step 9:

[1219] The server checks the consistency of the application documents and confirmation documents based on the extracted text information.

[1220] Specific operations: For example, compare the "applicant name" written on the application form with the "name" on the verification document to check for any discrepancies.

[1221] Step 10:

[1222] The server receives the results of the AI ​​system and generates feedback.

[1223] Specific operation: Set a status of "pass" or "fail" for each check item and summarize the results.

[1224] Step 11:

[1225] The server notifies the user of the result of the determination.

[1226] Specific operation: Notify the user of the results in real time via a mobile application.

[1227] Step 12:

[1228] The user receives a notification and, if necessary, re-photographs and submits the corrected document.

[1229] Specific operation: If an error occurs, the user re-photographs the corrected document and sends it to the server again.

[1230] Step 13:

[1231] If all conditions are met, the server stores the application and confirmation documents in a database.

[1232] What it does: Securely stores the final data in a database and sets a flag to start the next process.

[1233] Step 14:

[1234] The server sends a notification to the user that the procedure is complete.

[1235] Specific operation: The application sends a notification to the user that the procedure has been completed.

[1236] Step 15:

[1237] The user receives a final notification and confirms that the transaction is complete.

[1238] What happens: The user checks the in-app notification to confirm the transaction is complete.

[1239] Example 1

[1240] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1241] The document review process requires manual review of documents to determine their consistency and validity, which requires a great deal of time and effort. Furthermore, manual reviewing is prone to human error, and deficiencies and errors in documents can be overlooked. Therefore, there is a need to improve the efficiency and accuracy of the review process.

[1242] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1243] In this invention, the server includes: a means for scanning various documents with a device for taking photos of the terminal and generating image data; a means for transmitting the image data to the server via a communication device and analyzing it with an AI system; a means for the AI ​​system to extract text information from the image data using optical character recognition technology and detect specific elements (such as an approval seal or validity period); a means for the server to automatically determine the consistency and validity of the document based on the text information; and a means for notifying the user of the determination result via the communication device and receiving feedback. This automates the document review process, shortening operation time and eliminating judgment errors. Furthermore, the user can receive feedback and make quick corrections, improving the efficiency and accuracy of the entire review process.

[1244] A "terminal" is a device for taking pictures of various documents and transmitting the generated image data to a server via a communication device.

[1245] The "photography device" is a device such as a camera mounted on a terminal, which is used to record documents as digital images.

[1246] "Image data" refers to digital document image information generated by a photographing device.

[1247] A "communication device" is a network communication device for transmitting image data to a server.

[1248] A "server" is a computing device that receives image data sent from a terminal, temporarily stores it, and then analyzes it using an AI system.

[1249] "AI system" refers to artificial intelligence technology that analyzes image data, extracts text information using optical character recognition technology, and detects specific elements of documents (such as stamps of approval or validity dates).

[1250] Optical character recognition (OCR) is a technology that automatically reads text information from image data and converts it into digital text.

[1251] "Text information" refers to character data extracted from image data using optical character recognition technology.

[1252] "Specific elements" refer to important information items such as the document's stamp of approval, validity period, name, etc.

[1253] "Consistency" refers to a state in which the contents of the application documents and confirmation documents match and are free of contradictions.

[1254] "Validity" refers to the state in which a document is within its legal validity period and meets all necessary requirements.

[1255] "Means for receiving feedback" refers to the function by which the server notifies the user of the result of the judgment and receives corrections or resubmissions from the user.

[1256] This invention relates to an AI system for streamlining the document review process. This AI system involves a series of processes: scanning application and confirmation documents using a device's camera, generating image data, and transmitting it to a server via a communication device. The AI ​​system in the server then analyzes the image data, automatically determining the consistency and validity of the documents, and providing feedback on the results to the user.

[1257] Hardware and software used:

[1258] Device:

[1259] Devices for taking photos of various documents (e.g. smartphones, tablets, etc.)

[1260] Camera as a photographic device

[1261] Application software with image data preview and confirmation functions

[1262] server:

[1263] Storage for receiving and temporarily storing image data

[1264] AI systems (including optical character recognition (OCR) and image recognition algorithms)

[1265] Software for analyzing text information and determining document integrity

[1266] Specific examples:

[1267] Image data is generated when a user takes a photo of an application or confirmation document using the device's camera. At this time, the device displays a preview of the captured image, allowing the user to confirm that the image is clear. For example, a user can take a photo of the front and back of a driver's license and check the image on the preview screen within the application. Once the user has completed the verification, the image data is compressed (e.g., in JPEG format) and encrypted (e.g., using AES) and sent to the server by the communication device.

[1268] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR technology to extract text information from the image and detect specific elements (e.g., name, date of birth, expiration date, whether or not a seal is attached, etc.). During this process, the AI ​​system automatically reads information such as "name," "date of birth," and "expiration date" from the image of the driver's license.

[1269] The server uses the extracted text information to check whether the information on the application and confirmation documents is consistent. For example, it checks whether the "applicant name" on the application form matches the "name" on the driver's license, and whether the expiration date is within the current date. It also checks whether the required seals have been properly stamped. Based on the results, the server sends an error message or a success message to the user.

[1270] The user receives a notification from the server and makes corrections as necessary. This process is repeated until all conditions are met, at which point the server saves the application and confirmation document information in its database and takes steps to proceed to the next step in the process. For example, the server may notify the user with an error message such as "Name does not match" or "Expired date has passed," prompting the user to correct the information and submit it again. Finally, if the server determines that all information is accurate and matches, it will send a notification to the user that the process is complete.

[1271] Example prompts for generative AI models:

[1272] "Extract the name and expiration date from the driver's license."

[1273] Please check that the information on your application matches your verification documents.

[1274] "Please check whether the stamp is properly stamped in the required places on the document."

[1275] This invention automates the document review process, shortens operation time, and eliminates judgment errors, improving productivity and is expected to be used in many industries.

[1276] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1277] Step 1:

[1278] The terminal is operated by the user, who activates the device's image capture to take a photo of the application or confirmation document. The terminal displays the captured image to the user on a preview screen, and after the user checks the image's clarity and content, it processes the data to generate image data. The input is the image of the document captured by the user, and the output is the confirmation image displayed on the preview screen. In concrete terms, the user uses the camera function to take a photo of their driver's license and checks the image on the preview screen.

[1279] Step 2:

[1280] When the user has finished confirming the image, the device recognizes that a button has been pressed on the preview screen requesting confirmation by the user. The image data is then compressed (e.g., JPEG format) and encrypted (e.g., AES encryption). The input is the image data after the user's confirmation, and the output is the compressed and encrypted image data. Specifically, when the user taps the "Send Image" button, the device compresses the image data, encrypts it, and sends it to the server.

[1281] Step 3:

[1282] The server receives image data sent from the terminal and temporarily stores it. The input is the compressed and encrypted image data sent from the terminal, and the output is the image data stored in the server's storage. Specifically, the server receives the image data, checks the integrity of the data, and stores it in storage.

[1283] Step 4:

[1284] The server passes the stored image data to the AI ​​system and requests it to analyze it. The AI ​​system uses OCR technology to extract text information from the image data and detects specific elements (e.g., name, expiration date, whether or not a seal has been affixed). The input is the image data stored on the server, and the output is the analyzed text information. Specifically, the image data is input into the AI ​​system, and text information such as "name," "date of birth," and "expiration date" is extracted through OCR processing.

[1285] Step 5:

[1286] The server uses the extracted text information to check the consistency and validity of the application and confirmation documents. For example, it determines whether the "applicant name" written on the application form matches the "name" extracted from the image, and whether the expiration date is beyond the current date. The input is the text information provided by the AI ​​system, and the output is the result of the consistency and validity determination. Specifically, it compares the text information with the application form information to check for inconsistencies or omissions.

[1287] Step 6:

[1288] The server sends a notification message via a communication device to notify the user of the result of the judgment. The user receives the notification and makes corrections as necessary. The input is the result of the consistency and validity check, and the output is a notification message to the user. Specifically, the user receives an error message on their smartphone or tablet, such as "Name does not match" or "Expired date."

[1289] Step 7:

[1290] The user receives a notification from the server, makes any necessary corrections, photographs the documents again, and submits them. By repeating this process, if all conditions are met, the server saves the application and confirmation document information in its database and proceeds to the next process. The input is new image data of the documents revised by the user, and the output is text information reanalyzed by the server and the final judgment result. In concrete terms, the user photographs the documents again, submits them to the server, and finally receives a notification that the procedure has been completed.

[1291] (Application example 1)

[1292] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1293] The present invention aims to streamline the screening and authentication process for membership cards, coupons, point cards, etc. in physical stores, thereby reducing the burden on users and stores. Traditionally, these screening and authentication processes have often been performed manually, which requires time and effort and is prone to human error. Furthermore, there are issues with physical cards that users carry with them, such as loss or damage, so a solution to these issues is needed.

[1294] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1295] In this invention, the server includes: a means for reading various information with a device camera and generating image data; a means for transmitting the image data to a network server and analyzing it with an artificial intelligence system; a means for the artificial intelligence system to extract text information from the image data and detect specific elements (such as a seal or expiration date); a means for the server to automatically determine the integrity and validity of the document based on the text information; a means for notifying the user of the determination result and receiving feedback; and a means for authenticating the use of membership cards, coupons, and point cards using the feedback. This automates the review and authentication process for membership cards, coupons, and point cards, significantly reducing time and labor. It also reduces the risk of human error and improves user convenience.

[1296] A "device" is an electronic device that is operated by a user and has a camera mounted thereon for reading various types of information.

[1297] "Image data" is digital data generated based on information captured by the device's camera.

[1298] "Network Server" is a central processing unit that receives image data sent from the device and interfaces with the artificial intelligence system for analysis.

[1299] An "artificial intelligence system" is a software system equipped with AI technology that is used to extract textual information from image data and detect specific elements.

[1300] "Text information" is text data extracted from image data by an artificial intelligence system.

[1301] A "seal" is an impression required to indicate the validity of a document.

[1302] "Expiration date" refers to the period during which a document, membership card, coupon, etc. is legally valid.

[1303] "Integrity" is a concept that refers to documents and data being consistent and free of contradictions.

[1304] "Validity" is the state of a document or data that indicates it is suitable for its purpose and can be used.

[1305] "Feedback" refers to response information including notification results and correction instructions for the user.

[1306] A "membership card" is a card or digital certificate used to prove membership in a particular organization or service.

[1307] A "coupon" is a voucher that provides discounts or special offers for specific services or products.

[1308] A "point card" is a card that accumulates points with each purchase and can be used for special offers or discounts.

[1309] "Authenticating use" is the process of verifying the validity of a membership card, coupon, or point card and authorizing its use.

[1310] To implement the present invention, a device used by a user, a network server, and an artificial intelligence system are required. The details of each component and process are as follows.

[1311] 1. Device-side processing

[1312] The device is operated by the user and uses the camera to capture information such as membership cards, coupons, and point cards. At this time, the device displays a preview of the captured image and asks the user to confirm. Once the user has completed the confirmation, image data is generated.

[1313] 2. Sending image data

[1314] The generated image data is sent from the device to a network server, where it is encrypted to ensure confidentiality.

[1315] 3. Server-side processing

[1316] The server receives the image data sent from the device, temporarily stores it in storage, and then requests an AI system to analyze the image data.

[1317] 4. Image analysis using artificial intelligence systems

[1318] The AI ​​system uses OCR (optical character recognition) technology to extract text information from the received image data. Specifically, it detects specific elements such as ID numbers, expiration dates, and stamps on membership cards and coupons. The software used here is, for example, OpenCV and pytesseract.

[1319] 5. Checking data integrity and validity

[1320] The server checks whether the membership card or coupon is valid based on the character information extracted by the AI ​​system. This involves checking against information stored in a database. A specific example is a process that checks whether the extracted expiration date is beyond the current date.

[1321] 6. Providing Feedback

[1322] The results of the review are fed back to the user in real time. For example, a message such as "This is a valid membership card" or "This card has expired" is displayed. The user can receive this feedback, correct the information as needed, and submit it again.

[1323] Specific examples

[1324] The user launches the smartphone app, presses the "Membership Card Photo" button to take a photo of their membership card, checks the image on the preview screen, and then presses the "Send" button, which sends the image data to the server. The server analyzes this image data and notifies the user of the results. Specific examples of screen displays and notifications that can be used include prompts such as the following:

[1325] Prompt Sentence Examples

[1326] User: Press the "Membership Card Photo" button to take a photo of your membership card, check it on the preview screen, and then press the "Send" button.

[1327] This will streamline the review and authentication process for membership cards, coupons, and point cards at physical stores, while also improving user convenience.

[1328] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1329] Step 1:

[1330] The user takes a photo of a membership card or coupon with the camera. The input to the device is the user pressing the capture button, and image data is generated as output. The device displays a preview of this image data and asks the user to confirm it. When the user presses the confirmation button, the device confirms the image data.

[1331] Step 2:

[1332] The device sends the determined image data to the network server. The input is the generated image data, and the output is the data sent over the network. At this time, the data is encrypted to maintain security.

[1333] Step 3:

[1334] The server receives image data sent from the device and temporarily stores it in storage. The input is the sent image data, and the data temporarily stored in storage is the output.

[1335] Step 4:

[1336] The server passes the stored image data to the AI ​​system and requests it to analyze it. The input is the image data stored in the storage, and the output is the image data that the AI ​​system receives.

[1337] Step 5:

[1338] The artificial intelligence system uses OCR technology to extract text information from the received image data. Specifically, it detects ID numbers, expiration dates, and seals on membership cards and coupons contained in the image. The input is image data, and the output is the extracted text information. Software such as OpenCV and pytesseract is used for this processing.

[1339] Step 6:

[1340] The server checks the consistency and validity of the corresponding data based on the text information provided by the AI ​​system. Specifically, it compares it with the information registered in the database to see if it matches and if it has expired. The input is the extracted text information, and the output is the result of the consistency and validity determination.

[1341] Step 7:

[1342] The server notifies the user of the result of the determination, displaying real-time feedback on the user's device and sending messages such as "This is a valid membership card" or "This card has expired." The input is the determination result, and the output is a notification message displayed on the user's device.

[1343] Step 8:

[1344] The user receives the notification message and, if necessary, takes another photo of the membership card or coupon and submits it. This process loops back to step 1 until the server determines that all information is accurate and consistent. Finally, if the information is authenticated, the membership card or coupon is authorized for use.

[1345] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1346] This invention relates to a system that improves user experience by combining an AI system for streamlining the document review process with an emotion engine that recognizes user emotions. This system uses the device's camera to read application and confirmation documents, generate image data, and send it to a server. The AI ​​system on the server analyzes this image data and automatically determines the integrity and validity of the documents. It also uses the emotion engine to analyze the user's emotions when entering input and adjust the feedback content.

[1347] Specific system configuration and program processing

[1348] Image data generation and transmission

[1349] The terminal is operated by the user, who takes a photo of the application or confirmation document with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, the image data is generated and sent to the server.

[1350] As a concrete example, consider the case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear photo of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[1351] Receiving and analyzing image data

[1352] The server receives the image data sent from the device and temporarily stores it. It then requests the AI ​​system to analyze the image data. The AI ​​system uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image and detect specific elements (such as seals and expiration dates).

[1353] For example, an AI system can extract information such as the name, date of birth, and expiration date from an image of a driver's license. It also checks whether the image contains a stamp and determines whether the expiration date is past the current date.

[1354] Consistency determination and notification

[1355] The server checks the consistency of the application and confirmation documents based on the extracted text information. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal has been affixed and whether the expiration date is valid.

[1356] For example, the server checks whether there are any discrepancies between the "applicant name" on the application form and the "name" on the driver's license. At the same time, it checks whether the required seal has been properly stamped and whether the expiration date has passed the current date.

[1357] Introducing an emotion engine and adjusting feedback

[1358] The server is equipped with an emotion engine that analyzes the user's emotions based on their input content and speed. The emotion engine detects stress and anxiety when the user submits the application and generates feedback accordingly.

[1359] For example, if the emotion engine determines that the user is feeling anxious or stressed when the user's input speed is extremely slow or when the input content has been corrected many times, the server will send a friendly message such as, "Was it difficult to input? We are here to help if you need any help."

[1360] Final approval and saving

[1361] If all conditions are met, the server stores the application and confirmation documents in the database and takes steps to proceed to the next step. The emotion engine also further analyzes the user's feedback and stores it as data to improve the user experience.

[1362] For example, the server notifies the user with error messages such as "Name does not match" or "Expired date has passed," and the emotion engine analyzes the user's reaction and adjusts the feedback. If the server determines that all information is accurate and matches after the user resubmits the corrected document, it sends a notification to the user that the process has been completed. The user receives the notification and confirms that the process has been completed successfully.

[1363] This system automates the process of reviewing various documents, shortening operation time and eliminating judgment errors. Furthermore, an emotion engine provides feedback that takes into account the user's emotions, improving the user experience. This improves productivity and is expected to be used in many industries.

[1364] The processing flow will be explained below.

[1365] Program processing steps

[1366] Step 1:

[1367] The terminal starts a mobile application, and the user selects the camera function.

[1368] Specific operation: The user taps the "Launch Camera" button in the app to launch the camera. A shooting guide will be displayed on the screen.

[1369] Step 2:

[1370] The user takes a photo of the application or confirmation document with a camera.

[1371] Specific actions: Place the application form or confirmation document within the camera frame and press the shutter button to take the photo.

[1372] Step 3:

[1373] The device displays a preview of the captured image and prompts the user to confirm it.

[1374] Specific operation: A preview screen is displayed and a "Use this image" button is displayed. The user confirms the image and presses the OK button.

[1375] Step 4:

[1376] The device compresses and encrypts the image data and sends it to the server.

[1377] What happens: Image data is processed within the app and sent to a server using a secure communication protocol.

[1378] Step 5:

[1379] The server temporarily stores the image data received from the terminal.

[1380] Specific operation: The received image data is stored in the server storage and prepared for analysis.

[1381] Step 6:

[1382] The server requests the AI ​​system to analyze the image data.

[1383] Specific operation: The saved image data is passed to the AI ​​analysis module, and the analysis task is initiated.

[1384] Step 7:

[1385] The AI ​​system uses OCR technology to extract text information from images.

[1386] Specific operation: Identifies text areas within an image and extracts text information such as name and date of birth as digital data.

[1387] Step 8:

[1388] The AI ​​system uses image recognition algorithms to detect specific elements (stamps, expiration dates, etc.).

[1389] Specific operation: Check whether the stamp position and expiration date are correct. Analyze other detection elements in the same way.

[1390] Step 9:

[1391] The server checks the consistency of the application documents and confirmation documents based on the extracted text information.

[1392] Specific operations: For example, compare the "applicant name" written on the application form with the "name" on the verification document to check for any discrepancies.

[1393] Step 10:

[1394] The server receives the results of the AI ​​system and generates feedback.

[1395] Specific operation: Set a status of "pass" or "fail" for each check item and summarize the results.

[1396] Step 11:

[1397] The server notifies the user of the result of the determination.

[1398] Specific operation: Notify the user of the results in real time via a mobile application.

[1399] Step 12:

[1400] The user receives a notification and, if necessary, re-photographs and submits the corrected document.

[1401] Specific operation: If an error occurs, the user re-photographs the corrected document and sends it to the server again.

[1402] Step 13:

[1403] The server uses an emotion engine to analyze the user's emotions.

[1404] Specific operation: Analyzes the user's input speed and frequency of content revisions to determine whether the user is feeling stressed or anxious.

[1405] Step 14:

[1406] The server adjusts the feedback content based on the results analyzed by the emotion engine.

[1407] Specific operation: For example, if it is determined that the user is in a stressful state, a friendly follow-up message is generated and sent.

[1408] Step 15:

[1409] If all conditions are met, the server stores the application and confirmation documents in a database.

[1410] What it does: Securely stores the final data in a database and sets a flag to start the next process.

[1411] Step 16:

[1412] The server sends a notification to the user that the procedure is complete.

[1413] Specific operation: The application sends a notification to the user that the procedure has been completed.

[1414] Step 17:

[1415] The user receives a final notification and confirms that the transaction is complete.

[1416] What happens: The user checks the in-app notification to confirm the transaction is complete.

[1417] Example 2

[1418] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1419] Previous document review processes were often manual, which was time-consuming, labor-intensive, and prone to errors. Furthermore, there was little technology available to reduce the stress and anxiety users felt when submitting documents. Therefore, there was a need for a system that could review documents efficiently and accurately. Furthermore, there was a need to improve the user experience by analyzing user emotions and providing appropriate feedback.

[1420] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1421] In this invention, the server includes means for temporarily storing image data and analyzing it using AI technology, means for extracting text information from the image data and detecting specific elements (such as a seal or expiration date) using AI technology, and means for detecting the user's emotional state using emotion analysis technology, generating appropriate feedback, and sending it to the user. This not only automates the document review process efficiently and accurately, but also makes it possible to provide feedback that takes the user's emotions into consideration.

[1422] A "terminal" is a device that is operated by a user to photograph a document and generate image data.

[1423] A "camera" is hardware with a photographing function that is installed on a terminal.

[1424] A "server" is a computing system that receives, stores, analyzes, and generates feedback from image data.

[1425] "Image data" is a digital image file of a document captured by the device's camera.

[1426] "AI technology" refers to artificial intelligence technology that analyzes image data using optical character recognition technology and image recognition algorithms.

[1427] "Text information" is character information extracted from image data using AI technology.

[1428] "Specific elements" refer to specific information such as a seal or expiration date on a document.

[1429] "Consistency" is a criterion for evaluating whether the information in the application documents and the confirmation documents match.

[1430] "Validity" is a criterion for assessing whether a document is currently valid.

[1431] "Emotion analysis technology" is a technology that analyzes the content and speed of a user's input and infers the user's emotional state.

[1432] "Feedback" refers to the response or message that the server provides to the user based on the analysis results.

[1433] "Temporary storage" is a process in which the server temporarily stores the image data received in storage.

[1434] This invention is a system that streamlines the document review process and combines it with an emotion engine that analyzes user emotions. This system uses the device's camera to read application and confirmation documents, generate image data, and send it to a server. An AI system on the server analyzes this image data and automatically determines the integrity and validity of the documents. The emotion engine is also used to analyze the user's emotions when they enter information, and the feedback content is adjusted to improve the user experience.

[1435] Generation and transmission of image data by the terminal

[1436] The terminal is operated by the user, who takes a photo of the application or confirmation documents with a camera. At this time, the terminal displays a preview of the captured image and asks the user to confirm it. Once the user has completed the confirmation, image data is generated, compressed, encrypted, and sent to the server. This process uses the camera function of a smartphone or tablet, and the mobile application that controls it.

[1437] Receipt of image data by the server

[1438] The server receives and temporarily stores image data sent from the device. This data is received via API, temporarily stored in storage, and then passed to the AI ​​system.

[1439] Image data analysis by the server

[1440] The AI ​​system in the server uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from images and detect specific elements (such as seals, expiration dates, etc.) The technologies used include Python, TensorFlow, Tesseract OCR, and OpenCV.

[1441] Determining document integrity

[1442] The server checks the consistency of the application and confirmation documents based on the text information extracted from the AI ​​system. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document. The AI ​​system also automatically determines whether a seal is affixed and whether the expiration date is valid.

[1443] Emotion analysis using an emotion engine

[1444] The server is equipped with an emotion engine that analyzes the user's emotions based on their input and typing speed. It uses natural language processing libraries (e.g., spaCy, NLTK) and machine learning models (e.g., BERT) to detect stress and anxiety when the user submits their application and generates feedback accordingly.

[1445] Generate and send feedback

[1446] The server generates an appropriate feedback message based on the analysis results of the emotion engine and sends it to the user, such as "Did you have difficulty entering the information? We are here to help if you need any assistance."

[1447] Final approval and data storage

[1448] If all conditions are met, the server stores the application and confirmation documents in the database and takes steps to proceed to the next step. The emotion engine also further analyzes the user's feedback and stores it as data to improve the user experience. Finally, a notification of the completion of the process is sent to the user, who then receives the notification and confirms that the process has been completed.

[1449] Examples of specific examples and prompts

[1450] As a concrete example, let's consider a case where a user uses a mobile application on their device to take a picture of the front and back of their driver's license and send the image data to a server. The user uses the camera function to take a clear picture of the driver's license and checks the image on the preview screen within the application. After checking, the image data is compressed, encrypted, and sent to the server.

[1451] Example prompts to input to a generative AI model:

[1452] "Take a photo of the front and back of your driver's license and send the image data to our server."

[1453] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1454] Step 1: Take a photo of the document and preview the image

[1455] The user takes a photo of the application or confirmation document using the device's camera. The user launches the device's mobile application and uses the camera function to take a clear photo of the front of the driver's license. The application then displays the captured image on a preview screen and asks the user to confirm. The user checks the captured image on the preview screen and taps the "Confirm" button to proceed.

[1456] Input: Document image taken by camera

[1457] Output: Image displayed on the preview screen

[1458] Step 2: Generate image data and send it to the server

[1459] The device generates image data based on the image viewed by the user. This image data is compressed, encrypted, and sent to the server. Specifically, the image captured by the device's internal processing is saved in JPEG or PNG format, encrypted with AES, and sent to the server using the HTTPS protocol.

[1460] Input: Image confirmed by user

[1461] Output: Encrypted image data (compressed)

[1462] Step 3: Receiving and temporarily saving image data

[1463] The server receives the image data sent from the device and temporarily stores it. The server decrypts the encrypted image data received via HTTPS communication and temporarily stores it in storage. This temporarily stored data is used in the next analysis step.

[1464] Input: Encrypted image data

[1465] Output: Temporarily saved decrypted image data

[1466] Step 4: Analyzing the image data

[1467] The AI ​​system on the server analyzes the stored image data. It uses OCR (optical character recognition) technology and image recognition algorithms to extract text information from the image data and detect specific elements (such as seals and expiration dates). Specifically, it runs a Python script to extract text information using TensorFlow and the Tesseract OCR library, and performs image recognition using OpenCV.

[1468] Input: Temporarily saved image data

[1469] Output: Extracted text information (name, expiration date, whether or not it has a seal, etc.)

[1470] Step 5: Determine consistency and validity

[1471] The server determines the consistency and validity of the application and confirmation documents based on the text information extracted from the AI ​​system. For example, it checks whether the "applicant name" written on the application form matches the "name" on the confirmation document, whether there is a seal, and whether the expiration date is beyond the current date. This process requires connection to a database, and the check is performed using an SQL query.

[1472] Input: Extracted text information

[1473] Output: Consistency and validity determination results

[1474] Step 6: Sentiment analysis and feedback generation

[1475] The server uses sentiment analysis technology to analyze the user's input content and input speed and infer the user's emotional state. Natural language processing libraries (e.g., spaCy, NLTK) and machine learning models (e.g., BERT) are used for sentiment analysis. Based on the results of this analysis, the server generates an appropriate feedback message. For example, if the input speed is extremely slow or there are many corrections, the server can determine that the user is feeling anxious or stressed and generate a message such as, "Was it difficult to input? We are here to help if you need any help."

[1476] Input: User input and speed data

[1477] Output: The generated feedback message

[1478] Step 7: Submit feedback and final approval

[1479] The server sends the generated feedback message to the user. If all conditions are met, the server saves the application and confirmation documents in the database. Finally, it sends a notification of the completion of the procedure to the user's device, and the user receives the notification and confirms that the procedure has been completed.

[1480] Input: Feedback message and final approval information

[1481] Output: Notify user and save to database

[1482] (Application example 2)

[1483] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1484] In the conventional document review process, manual review required a lot of time and effort. Furthermore, there was a lack of mechanisms to reduce the anxiety and stress users felt when submitting documents. A system to address this issue is needed, and it is necessary to develop a system that not only streamlines the review process but also provides feedback that takes users' emotions into account.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1486] In this invention, the server includes: means for scanning various documents with a photographing device of a terminal and generating image data; means for transmitting the image data to a data processing device and analyzing it with an artificial intelligence system; means for the artificial intelligence system to extract text information from the image data and detect specific information (such as a seal or expiration date); means for the data processing device to automatically determine the integrity and validity of the document based on the text information; means for notifying the user of the determination result and receiving feedback; means including an emotion analysis engine that analyzes emotions based on the user's input content and input speed and adjusts the feedback content; and means for installing and operating the system on a smartphone or computer. This makes it possible to streamline the review process and provide appropriate feedback that takes the user's emotions into consideration.

[1487] "Documents" refers to multiple different types of paper or electronic documents, such as contracts, applications, confirmations, etc.

[1488] "Terminal" means a device that can be operated by a user, such as a smartphone, tablet, laptop, or desktop computer.

[1489] The term "photography device" refers to a device for capturing images, such as a camera mounted on a terminal.

[1490] "Image data" refers to data that digitally represents an image captured by an imaging device.

[1491] "Data processing device" refers to a computer system for analyzing and processing acquired image data and text information.

[1492] An "artificial intelligence system" is a system that uses algorithms such as machine learning and deep learning to automatically analyze data and extract and judge specific information.

[1493] "Text information" refers to text data extracted from image data.

[1494] "Specific information" refers to specific elements written on a document, such as a seal or expiration date.

[1495] "Document integrity and validity" refers to determining whether the application details are correct and consistent, and whether the documents are legally valid.

[1496] "Judgment result" refers to information obtained as a result of analysis by an artificial intelligence system and evaluation by a data processing device.

[1497] "Feedback" refers to answers or advice provided to users based on the results of their decisions.

[1498] An "emotion analysis engine" is software that analyzes the speed and content of a user's input and estimates that person's emotions.

[1499] The present invention provides a system for improving the efficiency of the review process of various documents and adjusting the feedback content by analyzing the user's emotions. Hereinafter, a specific embodiment of the present invention will be described.

[1500] System configuration

[1501] The system is broadly composed of the following components:

[1502] 1. Terminal

[1503] 2. Data Processing Device

[1504] 3. Artificial Intelligence Systems

[1505] 4. Sentiment Analysis Engine

[1506] 1. Terminal

[1507] Users use devices such as smartphones, tablets, and laptops. The devices are equipped with a camera, which is used to capture image data of various documents.

[1508] The user uses an application on the device to take a photo of a document. The captured image is previewed on the device and the user is asked to confirm. Once the user confirms, the image data is compressed, encrypted, and sent to the data processing device.

[1509] 2. Data Processing Device

[1510] The data processing device includes a storage device for temporarily storing the received image data. The data processing device functions as a server and requests the artificial intelligence system to analyze the image data.

[1511] 3. Artificial Intelligence Systems

[1512] The artificial intelligence system uses TensorFlow, an open-source framework, to build a machine learning model. The system uses OCR technology to extract text from images and detect specific information (such as seals and expiration dates).

[1513] As a specific example, an image of a driver's license can be analyzed to extract information such as the name, date of birth, and expiration date. This can be done with high accuracy using a pre-trained model.

[1514] 4. Sentiment Analysis Engine

[1515] The sentiment analysis engine uses natural language processing tools such as TextBlob to analyze the content and speed of the user's input to detect stress or anxiety, allowing the system to generate appropriate feedback for the user.

[1516] For example, consider the following prompt:

[1517] "I've been a little concerned about the service here lately."

[1518] The sentiment analysis engine analyzes this input, and if the emotion is strongly negative, it generates a friendly message such as, "Was it difficult to enter? We are here to help if you need any help."

[1519] These elements work together to improve the efficiency of the review process and the user experience as a whole system.

[1520] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1521] Step 1:

[1522] The user takes an image of a document on the device. The input is the image captured through the device's camera function, and the output is the captured document image data. After taking the image, the user can check the image on the preview screen.

[1523] Step 2:

[1524] The terminal compresses and encrypts the image data of the document that has been confirmed by the user and sends it to the server. The input is the confirmed image data, and the output is the compressed and encrypted image data. This ensures that the data is securely transferred to the server.

[1525] Step 3:

[1526] The server receives the image data sent from the terminal and temporarily stores it in storage. The input is compressed and encrypted image data, and the output is data stored in the server's storage.

[1527] Step 4:

[1528] The server requests the AI ​​system to analyze the stored image data. The input is the image data stored in the storage, and the output is the data sent to the AI ​​system.

[1529] Step 5:

[1530] The AI ​​system extracts text information from image data and detects specific information (such as a seal or expiration date). The input is image data, and the output is the extracted text information and specific information. The AI ​​system uses the TensorFlow framework.

[1531] Step 6:

[1532] The server determines the integrity and validity of the document based on the text information received from the AI ​​system. The input is the text information extracted by the AI ​​system, and the output is the judgment result. The server performs this automatically.

[1533] Step 7:

[1534] The server notifies the user of the decision result and receives feedback. The input is the decision result, and the output is the notification to the user and the feedback from the user.

[1535] Step 8:

[1536] The device analyzes the user's input content and input speed using a sentiment analysis engine. The input is the user's feedback content, and the output is the sentiment analysis result. The sentiment analysis engine uses TextBlob.

[1537] Step 9:

[1538] The server adjusts the feedback content based on the emotion analysis results and provides it to the user. The input is the emotion analysis results, and the output is the adjusted feedback. This improves user satisfaction.

[1539] This series of processes streamlines the document review process while providing appropriate feedback that takes users' emotions into consideration.

[1540] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1541] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1542] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1543] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1544] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1545] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1546] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1547] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1548] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1549] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1550] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1551] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1552] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1553] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1554] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1555] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1556] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1557] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1558] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1559] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1560] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1561] The following is further disclosed regarding the above embodiment.

[1562] (Claim 1)

[1563] A means for reading various documents with a camera on a terminal and generating image data;

[1564] A means for transmitting the image data to a server and analyzing the image data with an AI system;

[1565] A means for the AI ​​system to extract text information from the image data and detect specific elements (such as a seal or expiration date);

[1566] means for the server to automatically determine the consistency and validity of the document based on the text information;

[1567] means for notifying a user of the result of said determination and receiving feedback;

[1568] A system including:

[1569] (Claim 2)

[1570] 2. The system according to claim 1, wherein the terminal includes means for displaying a preview of the captured image and requesting confirmation from the user.

[1571] (Claim 3)

[1572] 2. The system according to claim 1, wherein the server includes a means for temporarily storing the received image data in a storage.

[1573] "Example 1"

[1574] (Claim 1)

[1575] A means for reading various documents with a photographing device of the terminal and generating image data;

[1576] A means for transmitting the image data to a server via a communication device and analyzing the image data with an AI system;

[1577] The AI ​​system extracts text information from the image data using optical character recognition technology and detects specific elements (such as a stamp of approval or a validity period);

[1578] means for the server to automatically determine the consistency and validity of the document based on the text information;

[1579] means for notifying a user of the determination result via a communication device and receiving feedback;

[1580] A system including:

[1581] (Claim 2)

[1582] 2. The system according to claim 1, wherein the terminal includes means for displaying a preview of the captured image and requesting confirmation from the user.

[1583] (Claim 3)

[1584] 2. The system according to claim 1, wherein the server includes a means for temporarily storing the received image data in a storage.

[1585] "Application Example 1"

[1586] (Claim 1)

[1587] A means for reading various information using a camera of the device and generating image data;

[1588] means for transmitting the image data to a network server and analyzing the image data using an artificial intelligence system;

[1589] means for the artificial intelligence system to extract character information from the image data and detect specific elements (such as a seal or expiration date);

[1590] a means for the server to automatically determine the consistency and validity of the document based on the character information;

[1591] means for notifying a user of the result of said determination and receiving feedback;

[1592] A means for authenticating the use of a membership card, coupon, or point card using the feedback;

[1593] A system including:

[1594] (Claim 2)

[1595] 2. The system according to claim 1, wherein the device includes means for displaying a preview of the captured image and requesting confirmation from the user.

[1596] (Claim 3)

[1597] 2. The system according to claim 1, wherein the server includes a means for temporarily storing the received image data in a storage.

[1598] "Example 2: Combining Emotion Engines"

[1599] (Claim 1)

[1600] A means for taking a photograph of the application document or confirmation document with a camera of the terminal and generating image data;

[1601] means for compressing and encrypting the image data and transmitting it to a server;

[1602] The server temporarily stores the received image data and analyzes it using AI technology;

[1603] The AI ​​technology extracts text information from the image data and detects specific elements (such as a seal or expiration date);

[1604] means for the server to automatically determine the consistency and validity of the document based on the text information;

[1605] means for detecting the emotional state of the user by emotion analysis technology installed in the server, generating appropriate feedback, and transmitting the feedback to the user;

[1606] A system including:

[1607] (Claim 2)

[1608] 2. The system according to claim 1, wherein the terminal includes means for displaying a preview of the captured image and requesting confirmation from the user.

[1609] (Claim 3)

[1610] 2. The system according to claim 1, wherein the server includes a means for temporarily storing the received image data in a storage.

[1611] "Application example 2 when combining emotion engines"

[1612] (Claim 1)

[1613] A means for reading various documents with a photographing device of the terminal and generating image data;

[1614] means for transmitting the image data to a data processing device and analyzing the image data with an artificial intelligence system;

[1615] means for the artificial intelligence system to extract character information from the image data and detect specific information (such as a seal or expiration date);

[1616] means for automatically determining the consistency and validity of the document based on the character information;

[1617] means for notifying a user of the result of said determination and receiving feedback;

[1618] a means including an emotion analysis engine for analyzing emotions based on the input content and input speed of a user and adjusting feedback content;

[1619] A means for installing and operating the system on a smartphone or computer;

[1620] A system including:

[1621] (Claim 2)

[1622] 2. The system according to claim 1, wherein the terminal includes means for displaying a preview of the captured image and requesting confirmation from the user.

[1623] (Claim 3)

[1624] 2. The system according to claim 1, wherein the data processing device includes means for temporarily storing the received image data in a storage device. [Explanation of symbols]

[1625] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for reading various documents with a camera on a terminal and generating image data; A means for transmitting the image data to a server and analyzing the image data with an AI system; means for extracting text information from the image data and detecting specific elements; means for the server to automatically determine the consistency and validity of the document based on the text information; means for notifying a user of the result of said determination and receiving feedback; A system including:

2. 2. The system according to claim 1, wherein the terminal includes means for displaying a preview of the captured image and requesting confirmation from the user.

3. 2. The system according to claim 1, wherein the server includes a means for temporarily storing the received image data in a storage.

Images