system

Abstract

Provide a system. A means for accepting upload of an application form; A means for analyzing the received application file and detecting the position of the seal; A means for extracting image data of the detected stamped portion; A means for determining the type of corporate seal from the extracted seal image; A means for converting character information of the seal impression from the seal image into text data; A means for registering the determined information in a database; A means to automatically check the stamping of application forms based on the registered information, a means for notifying the user of the review result; A system that includes a means for accumulating judgment and review results as learning data.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, the type of corporate seal and the characters stamped on application forms are checked visually, which requires a great deal of time and effort. Furthermore, the type of seal and rules vary depending on the business or service, increasing the man-hours required for review. Under these circumstances, efficient and accurate review is difficult, and human error is likely to occur. [Means for solving the problem]

[0005] The present invention is a system that includes a means for accepting the upload of an application form, a means for analyzing the received application form file and detecting the position of the stamp, a means for extracting image data of the detected stamp portion, a means for determining the type of corporate seal from the extracted stamp image, a means for converting the character information of the seal impression from the stamp image into text data, a means for registering the determined information in a database, a means for automatically checking the stamp on the application form based on the registered information, a means for notifying the user of the review results, and a means for accumulating the determination results and review results as learning data.

[0006] This eliminates the need for visual inspection and enables efficient and accurate stamp checks. It also contributes to reducing the amount of work required for review and preventing human error. Furthermore, by continuously accumulating learning data using AI, the accuracy of automated review can be improved.

[0007] An "application form" is a document that a customer or user fills in and officially submits in order to negotiate a service or contract.

[0008] The "means for accepting uploads" refers to a means having a function for accepting an action by a user to send an application file from a terminal to the system and starting processing.

[0009] "Means for analyzing the received application form file and detecting the position of the stamp" refers to means for using PDF analysis, image processing, etc. to identify the position of the stamp in the uploaded application form file.

[0010] The "means for extracting image data of the stamped portion" is a means having the function of cutting out the portion based on the stamped position detected from the application file and saving it as independent image data.

[0011] The "means for determining the type of corporate seal" is a means having the function of analyzing the extracted seal image and determining whether it is a round seal, a square seal, or a seal of a person in charge.

[0012] The "means for converting the character information of the seal impression into text data" is a means having a function for extracting character information from the seal image and formatting the information as electronic text.

[0013] The "means for registering the determined information in a database" refers to a means that has the function of recording the detected type of stamp and text information in a database, allowing for later reference and search.

[0014] "Means for automatically checking the stamp on an application form" refers to a means that has the function of automatically checking whether the stamp is genuine by using information registered in a database.

[0015] "Means for notifying the user of the review results" refers to a means that has a notification function for informing the user of the results and status of the stamp check.

[0016] "Means for accumulating judgment results and review results as learning data" refers to a means that has the function of saving judgment results and actual review results in a database or storage system in order to improve the accuracy of the AI ​​model. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a system for automating the check of seals on application forms, and a specific embodiment thereof will be described.

[0039] System Overview

[0040] This system accepts uploaded application forms, automatically determines the type of seal and character information, and uses that information to check the seal. The server handles the main processing, and users use their terminals to upload application form files and check the review results.

[0041] Program processing

[0042] 1. Upload your application form

[0043] The user uploads the application form PDF file to the server using a terminal. The server receives the uploaded file and starts processing, which digitally imports the application form into the system.

[0044] 2. Parse the PDF

[0045] The server analyzes the received PDF application file to detect the location of the stamp. Using a PDF analysis library, it reads the page content of the file and extracts the image data. At this stage, the specific coordinate information of the stamp is obtained.

[0046] 3. Extraction of imprint image

[0047] The server extracts an image of the stamped portion from the detected stamp position and saves it as an independent image file, which is used in subsequent processing.

[0048] 4. Type determination and character recognition

[0049] The server sends the extracted seal image to an image recognition system (e.g., Gemini (registered trademark) AI). The system determines the type of corporate seal (round seal, square seal, or seal of responsible person) and converts the character information of the seal impression into text data. For example, if the seal is a "round seal," the text may be converted to "CEO of ABC Co., Ltd."

[0050] 5. Database registration

[0051] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[0052] 6. Automated Review

[0053] The server automatically checks the seal on the application form based on the information registered in the database. Based on the screening rules, it verifies whether the seal is genuine and records the results in the screening table or log.

[0054] 7. Notification of the review results

[0055] Users can access a dedicated dashboard to check the results and status of their applications, and the server will notify them of the results and request corrections or confirmations as necessary.

[0056] 8. Accumulation of learning data

[0057] The server stores the AI ​​judgment results and human review results as learning data, which is used to retrain the AI ​​model and contribute to improving the accuracy of the entire system.

[0058] Specific examples

[0059] For example, suppose a company submits an application to sign a contract for a new financial service. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database, and an automatic review is performed. The review results are displayed on the user's dashboard and can be confirmed. The results of the review process are accumulated as future learning data.

[0060] This series of processes will enable automatic and accurate checks of application forms for stamps, greatly improving the efficiency of the review process.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] User: Uploads the application form PDF file to the server from the device.

[0064] Operation details: The user selects "Application_Company A.pdf" from a browser or dedicated application and presses the send button. The file is sent to the server via an HTTP request.

[0065] Step 2:

[0066] Server: Receives and stores the uploaded application file.

[0067] Operation details: The server saves the received file in a specific directory, records the file path, and generates an application ID.

[0068] Step 3:

[0069] Server: Analyzes the PDF file and detects the location of the stamp.

[0070] Operation details: The server uses a PDF analysis library (e.g., PyMuPDF or PDFBox) to convert each page of the PDF into image data, then applies a stamp detection algorithm to obtain the coordinate information of the stamp.

[0071] Step 4:

[0072] Server: Extracts image data of the stamped area.

[0073] Operation details: Based on the acquired coordinate information, an image processing library (e.g., OpenCV) is used to extract the seal portion. This is saved as "Seal Impression_CompanyA.png".

[0074] Step 5:

[0075] Server: Sends the stamp image to an image recognition system to determine the type of corporate seal.

[0076] Operation details: The stamp image is sent to an image recognition API (e.g., GeminiAI), and the classification model is used to classify it into one of "round stamp," "square stamp," or "person in charge stamp."

[0077] Step 6:

[0078] Server: Converts the text information of the seal impression from the stamp image using OCR.

[0079] Operation details: Using an OCR library (e.g., Tesseract), extract characters from the stamp image and convert them into text data such as "Seal of the Sales Director of SoftBank Corp."

[0080] Step 7:

[0081] Server: Registers the judgment results and text information in a database.

[0082] Operation details: An INSERT statement is used to add a record containing the application form ID, corporate seal type, and seal text to the database server.

[0083] Step 8:

[0084] Server: Conducts automatic review based on registered information.

[0085] Operation details: According to the screening rules, query the database information to confirm whether the seal is valid. Record the result in the screening table.

[0086] Step 9:

[0087] User: Check the review results.

[0088] How it works: Users access the dashboard to check the review results and status, and manually correct or approve as needed.

[0089] Step 10:

[0090] Server: Accumulates judgment results and human review results as learning data.

[0091] Operation details: The results of the review are saved in a database and accumulated as a dataset for retraining the model. A process to periodically update the AI ​​model is scheduled.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Traditionally, checking the stamps on application forms is often done manually, which is not only time-consuming and labor-intensive, but also carries a high risk of human error. Furthermore, relying on paper media hinders progress in digitizing data, making it difficult to improve work efficiency. It is necessary to solve these problems and automate the stamp check process to improve work efficiency and accuracy.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal portion, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the examination results, means for accumulating the determination and examination results as learning data, means for extracting and saving the image file using the coordinate information of the seal, and means for transmitting the seal image to an image recognition system and receiving the recognition results. This enables automatic and accurate checks of application form seals, significantly improving the efficiency and accuracy of operations.

[0097] An "application form" is a document that contains the information necessary to make an application or contract.

[0098] The "means for accepting uploads" refers to a function for receiving an application file sent by a user to a server using a terminal.

[0099] "Means for analyzing the received application file and detecting the position of the stamp" refers to the function of the server using a general-purpose library for analyzing electronic documents to identify the position of the stamp from the received application file.

[0100] "Means for extracting image data of the detected stamped portion" refers to a function for cutting out an image of the stamped portion from the application form file based on the coordinate information of the identified stamp.

[0101] "Means for determining the type of corporate seal from the extracted seal image" refers to a function that uses image recognition technology to identify the type of seal from the extracted seal image.

[0102] "Means for converting character information of the seal impression from the seal image into text data" refers to a function that uses an image recognition system to convert the text contained in the seal image into character data.

[0103] "Means for registering the determined information in a database" refers to the function of storing the type of seal and character information in a database.

[0104] "Means for automatically checking the stamping of application forms based on registered information" refers to the function in which the system automatically applies screening criteria to check the stamping based on the stamping information stored in the database.

[0105] "Means for notifying the user of the review results" refers to a function that displays and notifies the user of the review results on their dashboard.

[0106] "Means for accumulating judgment results and review results as learning data" refers to the function of saving judgment results and actual review results in a database and accumulating them as learning data in order to improve the accuracy of the AI ​​model.

[0107] "Means for extracting and saving an image file using coordinate information of a stamp" refers to a function that extracts the relevant part from a PDF file based on the location information of the identified stamp and saves it as an image file.

[0108] "Means for sending a stamp image to an image recognition system and receiving the recognition results" refers to the function of sending the extracted stamp image to an image recognition system and receiving the type of recognized stamp and text information.

[0109] The present invention relates to a system for automating the seal check of application forms, and a specific embodiment thereof will be described below. The system is implemented based on a process involving mainly a server, a terminal, and a user.

[0110] System Overview

[0111] This system accepts uploaded application forms, automatically determines the type of seal and character information, and checks the seal based on that information. The server handles the main processing, and users use their terminals to upload application form files and check the review results.

[0112] Hardware and software used

[0113] The server uses the following hardware and software:

[0114] Hardware: High-performance server equipment (e.g., a server equipped with an Intel Xeon processor)

[0115] Software: PDF analysis library (e.g., Apache PDFBox), image recognition system (e.g., GeminiAI), database system (e.g., MySQL)

[0116] Program processing

[0117] Upload your application form

[0118] The user logs in from their device and uploads the application form PDF file to the server. The device sends the file to the server, and the server saves the received file.

[0119] Parsing PDFs

[0120] The server uses the Apache PDFBox library to analyze the received application PDF file and detect the position of the stamp from the content of each page. At this stage, the coordinate information of the stamp is obtained.

[0121] Extraction of stamp image

[0122] The server extracts an image of the relevant part from the application form file based on the coordinate information of the stamp it has acquired, and saves the extracted image as a separate image file.

[0123] Type determination and character recognition

[0124] The server sends the extracted image of the seal to an image recognition system such as GeminiAI. The image recognition system determines the type of seal (e.g., round seal, square seal, seal of responsible person) and converts the characters contained in the seal into text data. For example, it may determine that the seal is a "round seal" and the text is "CEO of ABC Co., Ltd."

[0125] Database registration

[0126] The server registers the determined type of corporate seal and character information in a MySQL database, which stores information such as the application form ID, type of corporate seal, and seal text.

[0127] Automated Review

[0128] The server automatically checks the authenticity of the seal based on the information registered in the database and pre-set review rules, and the review results are recorded in a review table and log.

[0129] Notification of review results

[0130] Users access a dedicated dashboard to view the latest audit results provided by the server, and are notified of any corrections or additional confirmations as needed.

[0131] Accumulation of learning data

[0132] The server accumulates the AI's judgment results and the human review results as learning data, which allows the AI ​​model to be retrained and contributes to improving the accuracy of the entire system.

[0133] Specific examples

[0134] For example, suppose a company submits an application to sign a new financial service contract. The user uploads the application PDF file from their computer to the system. The server receives the PDF file and analyzes it to identify the location of the seal. It extracts the image of the seal and uses an image recognition system to identify that it is a "circle seal" and that the text is "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard and can be confirmed. The results obtained during the review process are accumulated as future learning data.

[0135] This will enable automatic and accurate checks of application forms for stamps, greatly improving the efficiency of the review process.

[0136] Prompt Sentence Examples

[0137] Please explain a system that automatically checks the stamps on application forms submitted by companies for financial services contracts. The system includes uploading PDF files, detecting stamp positions, extracting stamp images, determining type and character recognition, registering in a database, automatic screening, notifying the results of the screening, and accumulating learning data.

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] Step 1: Upload your application

[0140] The user logs in from their device, selects the application form PDF file, and clicks the upload button. This action causes the device to send the file to the server, which then saves the received PDF file in a directory.

[0141] Input: Application form PDF file selected by the user.

[0142] Output: Application file saved in the server's save directory.

[0143] Specific operation: The user clicks the "Select File" button to select a PDF file from the device's file system. Then, by pressing the "Upload" button, the device sends the file to the server via an HTTP request. The server receives the file and saves it in the specified directory.

[0144] Step 2: Parse the PDF

[0145] The server uses a PDF analysis library (e.g., Apache PDFBox) to analyze the received PDF file, extract the image data and text data of each page, and obtain the coordinate information to detect the stamp position.

[0146] Input: Saved application form PDF file.

[0147] Output: Data including the coordinate information of the stamp.

[0148] Specific operation: The server uses the PDFBox library to read the PDF file, scans the elements of each page, identifies areas that may be stamped, obtains their coordinate information, and stores it in a list.

[0149] Step 3: Extract the stamp image

[0150] The server extracts an image of the relevant part from the PDF application file based on the coordinate information of the stamp it has acquired, and saves the extracted image as a separate image file.

[0151] Input: Stamp coordinate information and application form PDF file.

[0152] Output: Image file of the stamped area.

[0153] Specific operation: The server reopens the PDF file, extracts the relevant part based on the specified coordinate information, and then saves the extracted image in a new file in PNG format or other format.

[0154] Step 4: Type determination and character recognition

[0155] The server sends the extracted image of the seal to an image recognition system (e.g., GeminiAI), which determines the type of seal and converts the characters contained in the seal into text data.

[0156] Input: Image file of the stamped area.

[0157] Output: Stamp type and text data.

[0158] Specific operation: The server sends the image file to GeminiAI's API and parses the received JSON response. The response includes the type of seal (e.g., circle) and text data (e.g., "CEO of XYZ Co., Ltd.").

[0159] Step 5: Database Registration

[0160] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[0161] Input: Stamp type and text data.

[0162] Output: Stamp information registered in the database.

[0163] Specific operation: The server uses a database system such as MySQL to store the retrieved information in the corresponding columns, and stores the information using the SQL INSERT statement.

[0164] Step 6: Automated Review

[0165] The server automatically checks the legitimacy of the seal based on the information registered in the database and pre-set review rules.

[0166] Input: Stamp information registered in the database.

[0167] Output: Review results.

[0168] Specific operation: The server evaluates the information in the database using a specific business rule engine. It checks the type of stamp and the degree of match of the stamped text to verify its validity. The results of the verification are recorded in a review table or log.

[0169] Step 7: Notification of review results

[0170] Users can access a dedicated dashboard to check the results of the review, and the server will notify the user of the results and request corrections or confirmations as necessary.

[0171] Input: Review results.

[0172] Output: Review results displayed on the dashboard.

[0173] What happens: When a user accesses the dashboard, the server retrieves and displays the latest audit results. If necessary, an appropriate notification is displayed if corrections or additional actions are required.

[0174] Step 8: Accumulating training data

[0175] The server accumulates the AI's judgment results and the human review results as learning data, which allows the AI ​​model to be retrained and contributes to improving the accuracy of the entire system.

[0176] Input: AI judgment results and human review results.

[0177] Output: Updated training data.

[0178] How it works: The server periodically collects the judgment results and actual review results and stores them in a database. This data is then used to retrain the AI ​​model, contributing to improving its accuracy.

[0179] (Application example 1)

[0180] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0181] Checking the accuracy of stamps on numerous application forms and documents at factories and other worksites takes time and effort. When the type of stamp and text information is checked manually, errors and delays are likely to occur, reducing work efficiency and increasing the risk of human error. Furthermore, there is a lack of rapid feedback on review results and the accumulation of learning data, which hinders progress in improving and streamlining the system.

[0182] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0183] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the screening results as learning data, means for an automated machine on-site to scan documents and upload them to the server, means for displaying the screening results in real time, means for notifying workers of feedback, and means for generating prompt sentences using a generative AI model to facilitate screening. This improves the efficiency of document processing in factories, reducing errors and work time.

[0184] - "Uploading an application" means that a User uses a Terminal to send an application in digital form to the Server.

[0185] "Analyzing the application file" refers to the process in which the server analyzes the file received and understands its contents and structure.

[0186] "Detecting the position of the seal" means identifying the specific position of the seal on the application form.

[0187] "Extracting image data" means separating and making independent the image data of the detected stamped portion.

[0188] "Determining the type of corporate seal" means analyzing the image of the seal to determine whether it is a round seal, a square seal, a seal of a person in charge, etc.

[0189] "Converting the character information of the seal impression into text data" means converting the stamped characters into digital text using optical character recognition technology or the like.

[0190] "Registering information in a database" means saving the determined type of seal and character information in a database.

[0191] "Automatically checks the stamp" means that the program automatically verifies the authenticity of the stamp based on the registered information.

[0192] "Notifying the user of the examination results" means informing the user of the results of the seal check.

[0193] "Storing as learning data" means saving the results of the review and judgment in order to improve the accuracy of the review in the future.

[0194] "Automated machine scanning of documents" means that an automated machine installed on-site converts physical documents into digital images.

[0195] "Uploading to server" means sending scanned digital data to the server.

[0196] "Displaying review results in real time" means instantly displaying the review results on the user's dashboard or the like.

[0197] "Notifying workers of feedback" means informing on-site workers of the review results and any necessary corrections.

[0198] "Generating prompts using a generative AI model" means using an artificial intelligence model to automatically generate messages to facilitate the review process.

[0199] This invention relates to a system that automates the process of checking stamps on application forms at factories. This system comprehensively automates the process from scanning documents to determining the type of stamp and text information, and notifying the results of the inspection in real time.

[0200] The system has several main functions: a means for accepting application form uploads, a means for analyzing received application form files and detecting the position of the seal, a means for extracting image data of the detected seal, a means for determining the type of corporate seal from the extracted seal image, a means for converting the character information of the seal impression from the seal image into text data, a means for registering the determined information in a database, a means for automatically checking the seal on the application form based on the registered information, a means for notifying the user of the screening results, a means for accumulating the determination results and the screening results as learning data, a means for on-site automated machines to scan documents and upload them to a server, a means for displaying the screening results in real time, a means for notifying workers of feedback, and a means for generating prompt messages to facilitate the screening process using a generative AI model.

[0201] Specifically, the server uses the following software and hardware: It uses a PDF analysis library (e.g., PyPDF2) to analyze the application file and identify the location of the seal. It uses an image processing library (e.g., PIL and OpenCV) to extract image data of the seal area, and an OCR library (e.g., pytesseract) to convert the character information of the seal impression into text data. The determined information is stored in a database using an HTTP request library (e.g., requests). It uses a generative AI model to generate prompts to facilitate the review process.

[0202] For example, imagine a factory needs to apply for a manufacturing license. An automated machine scans the physical document and uploads it to a server as a PDF file. The server then analyzes the PDF file, locating the stamp, extracting image data, and capturing text information through OCR. This information is stored in a database for automated review. A generative AI model is used to generate prompts for review. The review results are displayed in real time on a dashboard, providing feedback to workers.

[0203] An example of a prompt might be:

[0204] "A new manufacturing permit has been scanned by the robot. Please use the following information to determine the stamp type and text information on the application:

[0205] Stamp image: [Image URL]

[0206] Review rules: [URL for detailed review rules]

[0207] In this way, the present invention significantly improves the efficiency of the stamp checking process on application forms at the factory site, preventing mistakes and delays.

[0208] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0209] Step 1:

[0210] The user uploads the PDF file of the application form to the server using a terminal. The input is the PDF file of the application form, and the output is a digital file stored on the server. This process imports paper documents into the system as digital data.

[0211] Step 2:

[0212] The server analyzes the received PDF file to detect the location of the stamp. Specifically, it uses a PDF analysis library (e.g. PyPDF2). The input is the PDF file, and the output is the coordinate information (position) of the stamp. This process yields data that identifies where the stamp is located.

[0213] Step 3:

[0214] The server extracts image data from the identified stamp location. It uses an image processing library (e.g., PIL and OpenCV) to extract the image of the stamped area. The input is the stamp coordinate information and the PDF file, and the output is the stamped image. This results in the stamped area being obtained as a separate image file.

[0215] Step 4:

[0216] The server analyzes the extracted seal image using an OCR library (e.g., pytesseract) and converts the character information of the seal impression into text data. The input is the seal image, and the output is text data. Specifically, the OCR engine recognizes the characters in the image and converts them into digital text.

[0217] Step 5:

[0218] The server determines the extracted text data and type of seal and registers it in a database. It uses an HTTP request library (e.g., requests). The input is the type of seal and text data, and the output is a database entry. At this step, the determined information is saved in the system.

[0219] Step 6:

[0220] The server automatically performs a stamp check based on the information stored in the database. It evaluates the data according to the review rules and verifies whether the stamp is genuine. The input is the database entry and the output is the review result. Specifically, the server checks the integrity of the data based on the defined review rules.

[0221] Step 7:

[0222] The server notifies the user of the review results. The review results are displayed in real time on the dashboard for the user to check. The input is the review result data, and the output is a notification on the dashboard. In this step, the user can check the results immediately.

[0223] Step 8:

[0224] The server accumulates the judgment and review results as learning data. This allows the generative AI model to provide more accurate results in future reviews. The input is the judgment and review results, and the output is the learning data. This process improves the accuracy of the system.

[0225] Step 9:

[0226] An automated machine on-site scans the document and uploads it to a server. This step is the process of converting paper documents into digital data. The input is the physical document and the output is digital image data.

[0227] Step 10:

[0228] The server uses the generative AI model to generate prompts to facilitate the review. The input is the AI ​​model and the necessary data, and the output is the prompt. Specifically, the prompt is automatically generated and used during the review.

[0229] Example prompt sentence:

[0230] "A new manufacturing permit has been scanned by the robot. Please use the following information to determine the stamp type and text information on the application:

[0231] Stamp image: [Image URL]

[0232] Review rules: [URL for detailed review rules]

[0233] The above steps automate the process of checking stamps on application forms at the factory, thereby improving efficiency.

[0234] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0235] The present invention relates to a system that automates the process of checking whether an application form has been stamped and that also incorporates an emotion engine that recognizes the user's emotions. A specific example of the system will be described below.

[0236] System Overview

[0237] This system accepts uploaded application forms, automatically determines the type of seal and text information, and also has the ability to recognize the user's emotions. The server handles the main processing, and users use their terminals to upload application files and check the review results and their own emotions.

[0238] Program processing

[0239] 1. Upload your application form

[0240] The user uploads the application form PDF file from their device to the server, which receives the uploaded file and starts processing it, thereby digitally importing the application form into the system.

[0241] 2. Parse the PDF

[0242] The server analyzes the received PDF application file to detect the location of the stamp. It uses a PDF analysis library to read the page content of the file and extract the image data. At this stage, the specific coordinate information of the stamp is obtained.

[0243] 3. Extraction of imprint image

[0244] The server extracts an image of the stamped portion from the detected stamp position and saves it as an independent image file, which is used in subsequent processing.

[0245] 4. Type determination and character recognition

[0246] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The system determines the type of corporate seal (round seal, square seal, or seal of the person in charge) and converts the character information of the seal impression into text data. For example, if the seal is a "round seal," the text may be converted to "CEO of ABC Co., Ltd."

[0247] 5. Database registration

[0248] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[0249] 6. Automated Review

[0250] The server automatically checks the seal on the application form based on the information registered in the database. Based on the screening rules, it verifies whether the seal is genuine and records the results in the screening table or log.

[0251] 7. Notification of the review results

[0252] Users can access a dedicated dashboard to check the results and status of their applications, and the server will notify them of the results and request corrections or confirmations as necessary.

[0253] 8. Accumulation of learning data

[0254] The server stores the AI ​​judgment results and human review results as learning data, which is used to retrain the AI ​​model and contribute to improving the accuracy of the entire system.

[0255] 9. Implementing the Emotion Engine

[0256] The server is equipped with an emotion engine that recognizes the user's emotions. When a user uploads an application form or checks the screening results, the device's camera and microphone are used to capture the user's facial image and voice. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.).

[0257] Specific examples

[0258] For example, suppose a company submits an application to sign a new financial service contract. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[0259] This system not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, providing a more comprehensive user experience.

[0260] The processing flow will be explained below.

[0261] The present invention relates to a system that automates the process of checking whether an application form has been stamped and that also incorporates an emotion engine that recognizes the user's emotions. A specific example of the system will be described below.

[0262] System Overview

[0263] This system accepts uploaded application forms, automatically determines the type of seal and text information, and also has the ability to recognize the user's emotions. The server handles the main processing, and users use their terminals to upload application files and check the review results and their own emotions.

[0264] Program processing

[0265] Step 1:

[0266] User: Uploads the application form PDF file to the server from the device.

[0267] Operation details: The user selects "Application_Company A.pdf" from a browser or dedicated application and presses the send button. The file is sent to the server via an HTTP request.

[0268] Step 2:

[0269] Server: Receives and stores the uploaded application file.

[0270] Operation details: The server saves the received file in a specific directory, records the file path, and generates an application ID.

[0271] Step 3:

[0272] Server: Analyzes the PDF file and detects the location of the stamp.

[0273] Operation details: The server uses a PDF analysis library (e.g., PyMuPDF or PDFBox) to convert each page of the PDF into image data, then applies a stamp detection algorithm to obtain the coordinate information of the stamp.

[0274] Step 4:

[0275] Server: Extracts image data of the stamped area.

[0276] Operation details: Based on the acquired coordinate information, an image processing library (e.g., OpenCV) is used to extract the seal portion. This is saved as "Seal Impression_CompanyA.png".

[0277] Step 5:

[0278] Server: Sends the stamp image to an image recognition system to determine the type of corporate seal.

[0279] Operation details: The stamp image is sent to an image recognition API (e.g., GeminiAI), and the classification model is used to classify it into one of "round stamp," "square stamp," or "person in charge stamp."

[0280] Step 6:

[0281] Server: Converts the text information of the seal impression from the stamp image using OCR.

[0282] Operation details: Using an OCR library (e.g., Tesseract), extract characters from the stamp image and convert them into text data such as "CEO, ABC Co., Ltd."

[0283] Step 7:

[0284] Server: Registers the judgment results and text information in a database.

[0285] Operation details: An INSERT statement is used to add a record containing the application form ID, corporate seal type, and seal text to the database server.

[0286] Step 8:

[0287] Server: Conducts automatic review based on registered information.

[0288] Operation details: According to the screening rules, query the database information to confirm whether the seal is valid. Record the result in the screening table.

[0289] Step 9:

[0290] User: Check the review results.

[0291] How it works: Users access the dashboard to check the review results and status, and manually correct or approve as needed.

[0292] Step 10:

[0293] Server: Accumulates judgment results and human review results as learning data.

[0294] Operation details: The results of the review are saved in a database and accumulated as a dataset for retraining the model. A process to periodically update the AI ​​model is scheduled.

[0295] Step 11:

[0296] Server: Uses an emotion engine to recognize the user's emotions.

[0297] Operation details: The server captures the user's facial image and voice via the device's camera and microphone. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.) in real time.

[0298] Step 12:

[0299] Server: Provides the user with feedback on the recognized emotion information.

[0300] Operation details: The recognized emotional information is visualized and fed back to the user via a dashboard. Users can check their own emotional information and understand their feelings during the review process.

[0301] Specific examples

[0302] For example, suppose a company submits an application to sign a new financial service contract. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[0303] This system not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, providing a more comprehensive user experience.

[0304] Example 2

[0305] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0306] The traditional application form seal checking process is often done manually, which is inefficient and has a high risk of misjudgment. Furthermore, the lack of a feedback function that takes into account the user's emotions results in a poor user experience. To solve these problems, a system that integrates an automated seal checking function with a function that recognizes the user's emotions is needed.

[0307] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0308] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the judgment results and the screening results as learning data, and means for recognizing the user's emotions. This makes it possible to automate the application form seal check process and provide feedback that takes the user's emotions into consideration.

[0309] "Application" means a document that contains the information necessary to carry out a particular transaction.

[0310] A "PDF analysis library" is a software library for analyzing and manipulating PDF files programmatically.

[0311] "Seal" refers to a seal that is stamped on a document to indicate its legitimacy or approval.

[0312] "Image recognition technology" is a technology that allows computers to analyze images and videos and identify objects and characters.

[0313] A "database" is a system for efficiently managing, searching, and storing large amounts of data.

[0314] "Examination rules" are standards and guidelines for determining whether the contents of the application form and the stamp are genuine.

[0315] "User" means any person or entity that uses this system to upload applications and review results.

[0316] "Terminal" means a computing device through which a user accesses the system.

[0317] The "emotion engine" is a system that analyzes the user's facial images and voice data to recognize emotions.

[0318] The present invention relates to a system that automates the process of checking the seals on application forms and recognizes the emotions of users, and will be described below with reference to specific examples.

[0319] The system mainly consists of three components: a server, a terminal, and a user. The server is responsible for the main processing, and the user uses the terminal to upload the application form and check the review results and emotion recognition feedback.

[0320] 1. Upload your application form

[0321] The user uploads the application form PDF file to the server from their device. First, the user logs in to the system's dedicated portal and opens the application form upload page. Then, they click "Select file" and select the application form PDF from local storage. Then, they can press the "Upload" button to send the file to the server.

[0322] 2. Parse the PDF

[0323] The server receives the uploaded PDF file and begins parsing it. The server first saves the file in an inbox, then calls a PDF parsing library (e.g., PyPDF2) to open the file. The server reads each page in turn, extracts the image data, and applies an algorithm to identify the location of the stamp.

[0324] 3. Extraction of imprint image

[0325] The server extracts an image of the stamp based on the coordinate information of the identified stamp. The server uses an image processing library (e.g., OpenCV) to cut out a rectangular area from the specified coordinates and save it as a new image file. This image file is saved in a temporary folder and used for subsequent processing.

[0326] 4. Type determination and character recognition

[0327] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The server creates an HTTP request, attaches the seal image file, and sends it to the AI ​​image recognition API. The API determines the type of corporate seal (round seal, square seal, or seal of responsible person) and converts the character information of the seal impression into text data. For example, the API returns the character information "round seal" and "CEO of ABC Co., Ltd."

[0328] 5. Database registration

[0329] The server registers the determined corporate seal type and character information in a database. The server uses a database connection library (e.g., SQLAlchemy) to store the obtained information in the corresponding table. This table contains information such as the application form ID, corporate seal type, and seal text.

[0330] 6. Automated Review

[0331] The server automatically checks the seal on the application form based on the information registered in the database. The server checks whether the corporate seal matches the application details based on pre-set review rules. The results are registered in the review table and also recorded in a log file.

[0332] 7. Notification of the review results

[0333] The user accesses a dedicated dashboard to check the results of the review. When the user logs in, the server displays the relevant review results on the dashboard. If necessary, the server sends the user an email notifying them of the review results.

[0334] 8. Accumulation of learning data

[0335] The server accumulates the AI's judgment results and the human review results as learning data. The server periodically aggregates this data and uses it to retrain the AI ​​model, thereby improving the accuracy of the entire system.

[0336] 9. Implementing the Emotion Engine

[0337] The server is equipped with an emotion engine that recognizes the user's emotions. When a user uploads an application form or checks the screening results, the device's camera and microphone are used to capture the user's facial image and voice. The server acquires this data and sends it to the emotion engine. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.). The analysis results are stored in a database and used to improve the user experience.

[0338] Specific examples

[0339] For example, consider the case where a company submits an application for a new financial service contract. The user logs in to the system using a terminal and uploads the application. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted and sent to an image recognition system, which determines that it is a "circle stamp" and obtains the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[0340] Prompt Sentence Examples

[0341] "I would like to upload a new financial services application to the system. What are the next steps?"

[0342] 1. How to upload your application form

[0343] 2. How to check the review results

[0344] 3. How to check emotional information

[0345] "

[0346] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0347] Specific processing flow of the program

[0348] Step 1: Upload your application

[0349] Input: Application form PDF file selected by the user on the device.

[0350] Operation:

[0351] Users log into the system's dedicated portal.

[0352] Click the "Select File" button and select the application form PDF file from your local storage.

[0353] Click the "Upload" button to send the file to the server.

[0354] Output: Application form PDF file uploaded to the server.

[0355] Step 2: Parse the PDF

[0356] Input: Application form PDF file saved on the server.

[0357] Operation:

[0358] The server stores the file in the inbox.

[0359] Open the file using a PDF parsing library (e.g. PyPDF2).

[0360] Each page is read in turn, image data is extracted, and an algorithm is applied to identify the location of the stamp.

[0361] Output: Coordinate information of the imprint.

[0362] Step 3: Extract the stamp image

[0363] Input: Stamp coordinate information and application form PDF file.

[0364] Operation:

[0365] The server uses an image processing library (e.g., OpenCV) to extract the stamped portion from the specified coordinates.

[0366] The extracted image data of the stamped portion is saved as a new image file.

[0367] Output: Extracted stamp image file.

[0368] Step 4: Type determination and character recognition

[0369] Input: Extracted stamp image file.

[0370] Operation:

[0371] The server creates an HTTP request, attaches the stamp image file, and sends it to the AI ​​image recognition API (e.g., GeminiAI).

[0372] The AI ​​image recognition API determines the type of corporate seal (round seal, square seal, seal of responsible person) and converts the character information of the seal impression into text data.

[0373] Output: Type of corporate seal and text information of the seal impression.

[0374] Step 5: Database Registration

[0375] Input: The type of corporate seal determined and text information of the seal impression.

[0376] Operation:

[0377] The server uses a database connection library (e.g. SQLAlchemy) to store the retrieved information in the corresponding table.

[0378] The database contains information such as application form ID, type of corporate seal, and stamp text.

[0379] Output: The type of corporate seal registered in the database and text information of the seal impression.

[0380] Step 6: Automated Review

[0381] Input: The type of corporate seal registered in the database and text information of the seal impression.

[0382] Operation:

[0383] The server checks whether the corporate seal matches the application details based on pre-set review rules.

[0384] The results of the review are registered in the review table and also recorded in a log file.

[0385] Output: Review results.

[0386] Step 7: Notification of review results

[0387] Input: Review results.

[0388] Operation:

[0389] Users can access a dedicated dashboard to check the results of their review.

[0390] When a user logs in, the server displays the corresponding review results on the dashboard.

[0391] If necessary, the server will send an email to notify the user of the results of the review.

[0392] Output: Notification of review results to the user.

[0393] Step 8: Accumulating training data

[0394] Input: AI judgment results and human review results.

[0395] Operation:

[0396] The server periodically aggregates this data.

[0397] It is accumulated as learning data to be used for retraining AI models.

[0398] Output: Training data for retraining the AI ​​model.

[0399] Step 9: Implementing the Emotion Engine

[0400] Input: User's facial image and voice data.

[0401] Operation:

[0402] When a user uploads an application form or checks the review results, the device's camera and microphone are used to capture facial images and audio data.

[0403] The server retrieves this data and sends it to the emotion engine.

[0404] The emotion engine analyzes the data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.).

[0405] The analysis results are stored in a database.

[0406] Output: User emotion recognition results.

[0407] (Application example 2)

[0408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0409] Conventional application management systems have difficulty not only verifying the authenticity of stamps but also grasping the emotional state of workers. Furthermore, particularly at work sites such as logistics centers, there is a lack of mechanisms for monitoring workers' psychological states in real time and responding accordingly. This can result in a loss of safety and efficiency. Therefore, there is a need for a system that can automate the stamp checks on application forms, recognize workers' emotions, and manage that information.

[0410] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting upload of an application form, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal portion, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal of the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the judgment results and the screening results as learning data, means for recognizing the user's emotions, means for recording the emotion recognition data in a database, and means for the user to confirm emotions during the application form processing process. This not only enables automatic and accurate seal checks on application forms, but also enables real-time recognition and management of the emotional state of workers.

[0411] The "means for accepting application form uploads" is a function for importing digital files of application forms sent by users into the system.

[0412] The "means for analyzing the received application form file and detecting the position of the stamp" is a function for analyzing the contents of the application form obtained as an electronic file and identifying the location of the stamp.

[0413] The "means for extracting image data of the detected stamped portion" is a function for cutting out the relevant portion from the specified stamped position as an image.

[0414] The "means for determining the type of corporate seal from the extracted seal image" is a function that analyzes the cut-out seal portion and identifies what type of corporate seal it is (round seal, square seal, etc.).

[0415] The "means for converting the character information of the seal impression from the seal image into text data" is a function that converts the character information contained in the seal image into text format using optical character recognition (OCR) technology.

[0416] "Means for registering the determined information in a database" is a function for storing analyzed corporate seal information and character information in a database.

[0417] "Means for automatically checking the stamp on the application form based on registered information" refers to a function in which the system automatically checks whether the stamp on the application form is genuine, using information registered in the database.

[0418] "Means for notifying the user of the review results" is a function that reports the results of the stamp check to the user.

[0419] "Means for accumulating judgment results and review results as learning data" refers to a function that saves the judgment and review results made by the system as learning data and uses them to improve the system in the future.

[0420] The "means for recognizing the user's emotions" is a function that analyzes the user's facial image and voice data to determine the user's emotional state.

[0421] The "means for recording emotion recognition data in a database" is a function for storing the recognized emotion data of a user in a database.

[0422] The "means for users to check their emotions during the application process" is a function that allows users to check their own emotional state during the process of checking the review results and processing status of their application.

[0423] This invention relates to a system that automates the process of checking whether an application form has been stamped and combines it with an emotion engine that recognizes the emotions of workers. Specifically, this invention describes an embodiment of a system that uses smart glasses in a logistics center.

[0424] System Overview

[0425] This system consists of a terminal (specifically, smart glasses) and a server. The user (worker) scans the application form using the smart glasses and sends the image data to the server. The server processes the data, checks the stamp and recognizes emotions.

[0426] Hardware and software used

[0427] Hardware:

[0428] Smart glasses (e.g., Google® Glass®): A device worn by workers to scan application forms.

[0429] Server: A central device that stores data and performs various analytical processes.

[0430] software:

[0431] OpenCV: An image processing library used to detect the stamp position.

[0432] pytesseract: An OCR library used to extract text information from image data.

[0433] EmotionRecognizer: An optional emotion recognition library used to analyze a user's emotional state in real time.

[0434] Flask: A web application framework used to implement system-wide APIs.

[0435] Detailed explanation of functions

[0436] 1. Upload your application form

[0437] The user wears the smart glasses and scans the application form with the camera. The smart glasses then send the captured image data to the server, where it is stored in the server's database.

[0438] 2. Parse the PDF

[0439] The server analyzes the received image data using OpenCV to identify the location of the seal. First, it converts the image to grayscale, then performs threshold processing to highlight the seal area.

[0440] 3. Extraction of imprint image

[0441] The server extracts the identified stamped portion as an image and saves it as a separate image file. This extraction process is also performed using OpenCV.

[0442] 4. Type determination and character recognition

[0443] The server performs OCR processing on the extracted seal image using pytesseract, which converts the type of seal (e.g., a round or square corporate seal) and character information into text data.

[0444] 5. Emotion recognition

[0445] At the same time, the server uses EmotionRecognizer to analyze the user's facial image and voice data in real time to recognize their emotional state, for example, to determine whether the user is in an anxious state.

[0446] Specific examples

[0447] When worker A processes an application for receiving new goods, he scans the application using smart glasses. The server receives this data and simultaneously checks the stamp and recognizes worker A's emotional state. If the stamp is confirmed to be authentic and emotion analysis reveals that worker A is in an anxious state, an alert is sent to the manager.

[0448] Prompt Sentence Examples

[0449] You are a worker in charge of receiving new goods. Using smart glasses, you scan the application form and check the authenticity of the stamp. At the same time, the system also recognizes your emotional state (e.g., anxiety, surprise). The manager sees this and takes the necessary action.

[0450] In this way, the present invention not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, thereby improving work efficiency and safety.

[0451] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0452] Step 1:

[0453] The user scans an application form using smart glasses. The smart glasses capture the image data of the application form and send it to the server. At this time, the camera in the smart glasses captures the entire application form as a high-resolution image file. The input is the actual application form, and the output is the image data on the server.

[0454] Step 2:

[0455] The server analyzes the received image data and detects the location of the seal. The server converts the image to grayscale using OpenCV and performs threshold processing to highlight the presence of the seal. It then performs contour extraction and identifies the location of the seal. The input is the image data of the application form, and the output is the coordinate information of the seal.

[0456] Step 3:

[0457] The server extracts image data of the detected seal area. Based on the identified coordinate information, the seal area is cut out and saved as a separate image file. This image file is used for subsequent OCR processing. The input is image data with the seal coordinate information, and the output is image data of the seal area.

[0458] Step 4:

[0459] The server determines the type of corporate seal from the extracted image of the seal. The server uses pytesseract to perform OCR on this image and determines the type of corporate seal, such as whether it is a round seal or a square seal, from the acquired character information. The server also converts the character information contained in the seal into text data. The input is image data of the seal portion, and the output is the type of seal and text data.

[0460] Step 5:

[0461] The server registers the determined information in a database. The server stores the corporate seal type and text data obtained by OCR in the database and links it to the application form ID. The input is the seal type and text data, and the output is registration in the database.

[0462] Step 6:

[0463] The server automatically checks the seal on the application form based on the registered information. It checks whether the seal is genuine based on the information registered in the database and the screening rules. The results are recorded in the screening table. The input is the database information and screening rules, and the output is the screening results.

[0464] Step 7:

[0465] The server notifies the user of the results of the review. The server notifies the user of the results of the review through the dashboard and prompts corrections if necessary. The input is the review result data, and the output is a notification to the user.

[0466] Step 8:

[0467] The server accumulates the judgment and review results as learning data. This accumulates data to be used for retraining the AI ​​model, contributing to improving the accuracy of the system. The input is the review data, and the output is the accumulated learning data.

[0468] Step 9:

[0469] The server recognizes the user's emotions. The EmotionRecognizer analyzes the user's facial images and voice data collected by the smart glasses' camera and microphone to recognize their emotional state. The input is the user's facial images and voice data, and the output is the user's emotional state data.

[0470] Step 10:

[0471] The server records the emotion recognition data in a database. The recognized emotion data is linked to the application form ID and saved in the database. The input is emotional state data, and the output is a record in the database.

[0472] Step 11:

[0473] The user checks their emotions while processing the application form. They then check the review results and their emotional state on the dashboard and implement the countermeasures suggested by the system. The input is the data displayed on the dashboard, and the output is the user's confirmation and response actions.

[0474] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0475] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0476] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0477] [Second embodiment]

[0478] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0479] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0480] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0481] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0482] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0483] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0484] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0485] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0486] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0487] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0488] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0489] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0490] The present invention relates to a system for automating the check of seals on application forms, and a specific embodiment thereof will be described.

[0491] System Overview

[0492] This system accepts uploaded application forms, automatically determines the type of seal and character information, and uses that information to check the seal. The server handles the main processing, and users use their terminals to upload application form files and check the review results.

[0493] Program processing

[0494] 1. Upload your application form

[0495] The user uploads the application form PDF file to the server using a terminal. The server receives the uploaded file and starts processing, which digitally imports the application form into the system.

[0496] 2. Parse the PDF

[0497] The server analyzes the received PDF application file to detect the location of the stamp. Using a PDF analysis library, it reads the page content of the file and extracts the image data. At this stage, the specific coordinate information of the stamp is obtained.

[0498] 3. Extraction of imprint image

[0499] The server extracts an image of the stamped portion from the detected stamp position and saves it as an independent image file, which is used in subsequent processing.

[0500] 4. Type determination and character recognition

[0501] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The system determines the type of corporate seal (round seal, square seal, or seal of the person in charge) and converts the character information of the seal impression into text data. For example, if the seal is a "round seal," the text may be converted to "CEO of ABC Co., Ltd."

[0502] 5. Database registration

[0503] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[0504] 6. Automated Review

[0505] The server automatically checks the seal on the application form based on the information registered in the database. Based on the screening rules, it verifies whether the seal is genuine and records the results in the screening table or log.

[0506] 7. Notification of the review results

[0507] Users can access a dedicated dashboard to check the results and status of their applications, and the server will notify them of the results and request corrections or confirmations as necessary.

[0508] 8. Accumulation of learning data

[0509] The server stores the AI ​​judgment results and human review results as learning data, which is used to retrain the AI ​​model and contribute to improving the accuracy of the entire system.

[0510] Specific examples

[0511] For example, suppose a company submits an application to sign a contract for a new financial service. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database, and an automatic review is performed. The review results are displayed on the user's dashboard and can be confirmed. The results of the review process are accumulated as future learning data.

[0512] This series of processes will enable automatic and accurate checks of application forms for stamps, greatly improving the efficiency of the review process.

[0513] The processing flow will be explained below.

[0514] Step 1:

[0515] User: Uploads the application form PDF file to the server from the device.

[0516] Operation details: The user selects "Application_Company A.pdf" from a browser or dedicated application and presses the send button. The file is sent to the server via an HTTP request.

[0517] Step 2:

[0518] Server: Receives and stores the uploaded application file.

[0519] Operation details: The server saves the received file in a specific directory, records the file path, and generates an application ID.

[0520] Step 3:

[0521] Server: Analyzes the PDF file and detects the location of the stamp.

[0522] Operation details: The server uses a PDF analysis library (e.g., PyMuPDF or PDFBox) to convert each page of the PDF into image data, then applies a stamp detection algorithm to obtain the coordinate information of the stamp.

[0523] Step 4:

[0524] Server: Extracts image data of the stamped area.

[0525] Operation details: Based on the acquired coordinate information, an image processing library (e.g., OpenCV) is used to extract the seal portion. This is saved as "Seal Impression_CompanyA.png".

[0526] Step 5:

[0527] Server: Sends the stamp image to an image recognition system to determine the type of corporate seal.

[0528] Operation details: The stamp image is sent to an image recognition API (e.g., GeminiAI), and the classification model is used to classify it into one of "round stamp," "square stamp," or "person in charge stamp."

[0529] Step 6:

[0530] Server: Converts the text information of the seal impression from the stamp image using OCR.

[0531] Operation details: Using an OCR library (e.g., Tesseract), extract characters from the stamp image and convert them into text data such as "Seal of the Sales Director of SoftBank Corp."

[0532] Step 7:

[0533] Server: Registers the judgment results and text information in a database.

[0534] Operation details: An INSERT statement is used to add a record containing the application form ID, corporate seal type, and seal text to the database server.

[0535] Step 8:

[0536] Server: Conducts automatic review based on registered information.

[0537] Operation details: According to the screening rules, query the database information to confirm whether the seal is valid. Record the result in the screening table.

[0538] Step 9:

[0539] User: Check the review results.

[0540] How it works: Users access the dashboard to check the review results and status, and manually correct or approve as needed.

[0541] Step 10:

[0542] Server: Accumulates judgment results and human review results as learning data.

[0543] Operation details: The results of the review are saved in a database and accumulated as a dataset for retraining the model. A process to periodically update the AI ​​model is scheduled.

[0544] Example 1

[0545] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0546] Traditionally, checking the stamps on application forms is often done manually, which is not only time-consuming and labor-intensive, but also carries a high risk of human error. Furthermore, relying on paper media hinders progress in digitizing data, making it difficult to improve work efficiency. It is necessary to solve these problems and automate the stamp check process to improve work efficiency and accuracy.

[0547] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0548] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal portion, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the examination results, means for accumulating the determination and examination results as learning data, means for extracting and saving the image file using the coordinate information of the seal, and means for transmitting the seal image to an image recognition system and receiving the recognition results. This enables automatic and accurate checks of application form seals, significantly improving the efficiency and accuracy of operations.

[0549] An "application form" is a document that contains the information necessary to make an application or contract.

[0550] The "means for accepting uploads" refers to a function for receiving an application file sent by a user to a server using a terminal.

[0551] "Means for analyzing the received application file and detecting the position of the stamp" refers to the function of the server using a general-purpose library for analyzing electronic documents to identify the position of the stamp from the received application file.

[0552] "Means for extracting image data of the detected stamped portion" refers to a function for cutting out an image of the stamped portion from the application form file based on the coordinate information of the identified stamp.

[0553] "Means for determining the type of corporate seal from the extracted seal image" refers to a function that uses image recognition technology to identify the type of seal from the extracted seal image.

[0554] "Means for converting character information of the seal impression from the seal image into text data" refers to a function that uses an image recognition system to convert the text contained in the seal image into character data.

[0555] "Means for registering the determined information in a database" refers to the function of storing the type of seal and character information in a database.

[0556] "Means for automatically checking the stamping of application forms based on registered information" refers to the function in which the system automatically applies screening criteria to check the stamping based on the stamping information stored in the database.

[0557] "Means for notifying the user of the review results" refers to a function that displays and notifies the user of the review results on their dashboard.

[0558] "Means for accumulating judgment results and review results as learning data" refers to the function of saving judgment results and actual review results in a database and accumulating them as learning data in order to improve the accuracy of the AI ​​model.

[0559] "Means for extracting and saving an image file using coordinate information of a stamp" refers to a function that extracts the relevant part from a PDF file based on the location information of the identified stamp and saves it as an image file.

[0560] "Means for sending a stamp image to an image recognition system and receiving the recognition results" refers to the function of sending the extracted stamp image to an image recognition system and receiving the type of recognized stamp and text information.

[0561] The present invention relates to a system for automating the seal check of application forms, and a specific embodiment thereof will be described below. The system is implemented based on a process involving mainly a server, a terminal, and a user.

[0562] System Overview

[0563] This system accepts uploaded application forms, automatically determines the type of seal and character information, and checks the seal based on that information. The server handles the main processing, and users use their terminals to upload application form files and check the review results.

[0564] Hardware and software used

[0565] The server uses the following hardware and software:

[0566] Hardware: High-performance server equipment (e.g., a server equipped with an Intel Xeon processor)

[0567] Software: PDF analysis library (e.g., Apache PDFBox), image recognition system (e.g., GeminiAI), database system (e.g., MySQL)

[0568] Program processing

[0569] Upload your application form

[0570] The user logs in from their device and uploads the application form PDF file to the server. The device sends the file to the server, and the server saves the received file.

[0571] Parsing PDFs

[0572] The server uses the Apache PDFBox library to analyze the received application PDF file and detect the position of the stamp from the content of each page. At this stage, the coordinate information of the stamp is obtained.

[0573] Extraction of stamp image

[0574] The server extracts an image of the relevant part from the application form file based on the coordinate information of the stamp it has acquired, and saves the extracted image as a separate image file.

[0575] Type determination and character recognition

[0576] The server sends the extracted image of the seal to an image recognition system such as GeminiAI. The image recognition system determines the type of seal (e.g., round seal, square seal, seal of responsible person) and converts the characters contained in the seal into text data. For example, it may determine that the seal is a "round seal" and the text is "CEO of ABC Co., Ltd."

[0577] Database registration

[0578] The server registers the determined type of corporate seal and character information in a MySQL database, which stores information such as the application form ID, type of corporate seal, and seal text.

[0579] Automated Review

[0580] The server automatically checks the authenticity of the seal based on the information registered in the database and pre-set review rules, and the review results are recorded in a review table and log.

[0581] Notification of review results

[0582] Users access a dedicated dashboard to view the latest audit results provided by the server, and are notified of any corrections or additional confirmations as needed.

[0583] Accumulation of learning data

[0584] The server accumulates the AI's judgment results and the human review results as learning data, which allows the AI ​​model to be retrained and contributes to improving the accuracy of the entire system.

[0585] Specific examples

[0586] For example, suppose a company submits an application to sign a new financial service contract. The user uploads the application PDF file from their computer to the system. The server receives the PDF file and analyzes it to identify the location of the seal. It extracts the image of the seal and uses an image recognition system to identify that it is a "circle seal" and that the text is "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard and can be confirmed. The results obtained during the review process are accumulated as future learning data.

[0587] This will enable automatic and accurate checks of application forms for stamps, greatly improving the efficiency of the review process.

[0588] Prompt Sentence Examples

[0589] Please explain a system that automatically checks the stamps on application forms submitted by companies for financial services contracts. The system includes uploading PDF files, detecting stamp positions, extracting stamp images, determining type and character recognition, registering in a database, automatic screening, notifying the results of the screening, and accumulating learning data.

[0590] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0591] Step 1: Upload your application

[0592] The user logs in from their device, selects the application form PDF file, and clicks the upload button. This action causes the device to send the file to the server, which then saves the received PDF file in a directory.

[0593] Input: Application form PDF file selected by the user.

[0594] Output: Application file saved in the server's save directory.

[0595] Specific operation: The user clicks the "Select File" button to select a PDF file from the device's file system. Then, by pressing the "Upload" button, the device sends the file to the server via an HTTP request. The server receives the file and saves it in the specified directory.

[0596] Step 2: Parse the PDF

[0597] The server uses a PDF analysis library (e.g., Apache PDFBox) to analyze the received PDF file, extract the image data and text data of each page, and obtain the coordinate information to detect the stamp position.

[0598] Input: Saved application form PDF file.

[0599] Output: Data including the coordinate information of the stamp.

[0600] Specific operation: The server uses the PDFBox library to read the PDF file, scans the elements of each page, identifies areas that may be stamped, obtains their coordinate information, and stores it in a list.

[0601] Step 3: Extract the stamp image

[0602] The server extracts an image of the relevant part from the PDF application file based on the coordinate information of the stamp it has acquired, and saves the extracted image as a separate image file.

[0603] Input: Stamp coordinate information and application form PDF file.

[0604] Output: Image file of the stamped area.

[0605] Specific operation: The server reopens the PDF file, extracts the relevant part based on the specified coordinate information, and then saves the extracted image in a new file in PNG format or other format.

[0606] Step 4: Type determination and character recognition

[0607] The server sends the extracted image of the seal to an image recognition system (e.g., GeminiAI), which determines the type of seal and converts the characters contained in the seal into text data.

[0608] Input: Image file of the stamped area.

[0609] Output: Stamp type and text data.

[0610] Specific operation: The server sends the image file to GeminiAI's API and parses the received JSON response. The response includes the type of seal (e.g., circle) and text data (e.g., "CEO of XYZ Co., Ltd.").

[0611] Step 5: Database Registration

[0612] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[0613] Input: Stamp type and text data.

[0614] Output: Stamp information registered in the database.

[0615] Specific operation: The server uses a database system such as MySQL to store the retrieved information in the corresponding columns, and stores the information using the SQL INSERT statement.

[0616] Step 6: Automated Review

[0617] The server automatically checks the legitimacy of the seal based on the information registered in the database and pre-set review rules.

[0618] Input: Stamp information registered in the database.

[0619] Output: Review results.

[0620] Specific operation: The server evaluates the information in the database using a specific business rule engine. It checks the type of stamp and the degree of match of the stamped text to verify its validity. The results of the verification are recorded in a review table or log.

[0621] Step 7: Notification of review results

[0622] Users can access a dedicated dashboard to check the results of the review, and the server will notify the user of the results and request corrections or confirmations as necessary.

[0623] Input: Review results.

[0624] Output: Review results displayed on the dashboard.

[0625] What happens: When a user accesses the dashboard, the server retrieves and displays the latest audit results. If necessary, an appropriate notification is displayed if corrections or additional actions are required.

[0626] Step 8: Accumulating training data

[0627] The server accumulates the AI's judgment results and the human review results as learning data, which allows the AI ​​model to be retrained and contributes to improving the accuracy of the entire system.

[0628] Input: AI judgment results and human review results.

[0629] Output: Updated training data.

[0630] How it works: The server periodically collects the judgment results and actual review results and stores them in a database. This data is then used to retrain the AI ​​model, contributing to improving its accuracy.

[0631] (Application example 1)

[0632] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0633] Checking the accuracy of stamps on numerous application forms and documents at factories and other worksites takes time and effort. When the type of stamp and text information is checked manually, errors and delays are likely to occur, reducing work efficiency and increasing the risk of human error. Furthermore, there is a lack of rapid feedback on review results and the accumulation of learning data, which hinders progress in improving and streamlining the system.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0635] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the screening results as learning data, means for an automated machine on-site to scan documents and upload them to the server, means for displaying the screening results in real time, means for notifying workers of feedback, and means for generating prompt sentences using a generative AI model to facilitate screening. This improves the efficiency of document processing in factories, reducing errors and work time.

[0636] - "Uploading an application" means that a User uses a Terminal to send an application in digital form to the Server.

[0637] "Analyzing the application file" refers to the process in which the server analyzes the file received and understands its contents and structure.

[0638] "Detecting the position of the seal" means identifying the specific position of the seal on the application form.

[0639] "Extracting image data" means separating and making independent the image data of the detected stamped portion.

[0640] "Determining the type of corporate seal" means analyzing the image of the seal to determine whether it is a round seal, a square seal, a seal of a person in charge, etc.

[0641] "Converting the character information of the seal impression into text data" means converting the stamped characters into digital text using optical character recognition technology or the like.

[0642] "Registering information in a database" means saving the determined type of seal and character information in a database.

[0643] "Automatically checks the stamp" means that the program automatically verifies the authenticity of the stamp based on the registered information.

[0644] "Notifying the user of the examination results" means informing the user of the results of the seal check.

[0645] "Storing as learning data" means saving the results of the review and judgment in order to improve the accuracy of the review in the future.

[0646] "Automated machine scanning of documents" means that an automated machine installed on-site converts physical documents into digital images.

[0647] "Uploading to server" means sending scanned digital data to the server.

[0648] "Displaying review results in real time" means instantly displaying the review results on the user's dashboard or the like.

[0649] "Notifying workers of feedback" means informing on-site workers of the review results and any necessary corrections.

[0650] "Generating prompts using a generative AI model" means using an artificial intelligence model to automatically generate messages to facilitate the review process.

[0651] This invention relates to a system that automates the process of checking stamps on application forms at factories. This system comprehensively automates the process from scanning documents to determining the type of stamp and text information, and notifying the results of the inspection in real time.

[0652] The system has several main functions: a means for accepting application form uploads, a means for analyzing received application form files and detecting the position of the seal, a means for extracting image data of the detected seal, a means for determining the type of corporate seal from the extracted seal image, a means for converting the character information of the seal impression from the seal image into text data, a means for registering the determined information in a database, a means for automatically checking the seal on the application form based on the registered information, a means for notifying the user of the screening results, a means for accumulating the determination results and the screening results as learning data, a means for on-site automated machines to scan documents and upload them to a server, a means for displaying the screening results in real time, a means for notifying workers of feedback, and a means for generating prompt messages to facilitate the screening process using a generative AI model.

[0653] Specifically, the server uses the following software and hardware: It uses a PDF analysis library (e.g., PyPDF2) to analyze the application file and identify the location of the seal. It uses an image processing library (e.g., PIL and OpenCV) to extract image data of the seal area, and an OCR library (e.g., pytesseract) to convert the character information of the seal impression into text data. The determined information is stored in a database using an HTTP request library (e.g., requests). It uses a generative AI model to generate prompts to facilitate the review process.

[0654] For example, imagine a factory needs to apply for a manufacturing license. An automated machine scans the physical document and uploads it to a server as a PDF file. The server then analyzes the PDF file, locating the stamp, extracting image data, and capturing text information through OCR. This information is stored in a database for automated review. A generative AI model is used to generate prompts for review. The review results are displayed in real time on a dashboard, providing feedback to workers.

[0655] An example of a prompt might be:

[0656] "A new manufacturing permit has been scanned by the robot. Please use the following information to determine the stamp type and text information on the application:

[0657] Stamp image: [Image URL]

[0658] Review rules: [URL for detailed review rules]

[0659] In this way, the present invention significantly improves the efficiency of the stamp checking process on application forms at the factory site, preventing mistakes and delays.

[0660] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0661] Step 1:

[0662] The user uploads the PDF file of the application form to the server using a terminal. The input is the PDF file of the application form, and the output is a digital file stored on the server. This process imports paper documents into the system as digital data.

[0663] Step 2:

[0664] The server analyzes the received PDF file to detect the location of the stamp. Specifically, it uses a PDF analysis library (e.g. PyPDF2). The input is the PDF file, and the output is the coordinate information (position) of the stamp. This process yields data that identifies where the stamp is located.

[0665] Step 3:

[0666] The server extracts image data from the identified stamp location. It uses an image processing library (e.g., PIL and OpenCV) to extract the image of the stamped area. The input is the stamp coordinate information and the PDF file, and the output is the stamped image. This results in the stamped area being obtained as a separate image file.

[0667] Step 4:

[0668] The server analyzes the extracted seal image using an OCR library (e.g., pytesseract) and converts the character information of the seal impression into text data. The input is the seal image, and the output is text data. Specifically, the OCR engine recognizes the characters in the image and converts them into digital text.

[0669] Step 5:

[0670] The server determines the extracted text data and type of seal and registers it in a database. It uses an HTTP request library (e.g., requests). The input is the type of seal and text data, and the output is a database entry. At this step, the determined information is saved in the system.

[0671] Step 6:

[0672] The server automatically performs a stamp check based on the information stored in the database. It evaluates the data according to the review rules and verifies whether the stamp is genuine. The input is the database entry and the output is the review result. Specifically, the server checks the integrity of the data based on the defined review rules.

[0673] Step 7:

[0674] The server notifies the user of the review results. The review results are displayed in real time on the dashboard for the user to check. The input is the review result data, and the output is a notification on the dashboard. In this step, the user can check the results immediately.

[0675] Step 8:

[0676] The server accumulates the judgment and review results as learning data. This allows the generative AI model to provide more accurate results in future reviews. The input is the judgment and review results, and the output is the learning data. This process improves the accuracy of the system.

[0677] Step 9:

[0678] An automated machine on-site scans the document and uploads it to a server. This step is the process of converting paper documents into digital data. The input is the physical document and the output is digital image data.

[0679] Step 10:

[0680] The server uses the generative AI model to generate prompts to facilitate the review. The input is the AI ​​model and the necessary data, and the output is the prompt. Specifically, the prompt is automatically generated and used during the review.

[0681] Example prompt sentence:

[0682] "A new manufacturing permit has been scanned by the robot. Please use the following information to determine the stamp type and text information on the application:

[0683] Stamp image: [Image URL]

[0684] Review rules: [URL for detailed review rules]

[0685] The above steps automate the process of checking stamps on application forms at the factory, thereby improving efficiency.

[0686] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0687] The present invention relates to a system that automates the process of checking whether an application form has been stamped and that also incorporates an emotion engine that recognizes the user's emotions. A specific example of the system will be described below.

[0688] System Overview

[0689] This system accepts uploaded application forms, automatically determines the type of seal and text information, and also has the ability to recognize the user's emotions. The server handles the main processing, and users use their terminals to upload application files and check the review results and their own emotions.

[0690] Program processing

[0691] 1. Upload your application form

[0692] The user uploads the application form PDF file from their device to the server, which receives the uploaded file and starts processing it, thereby digitally importing the application form into the system.

[0693] 2. Parse the PDF

[0694] The server analyzes the received PDF application file to detect the location of the stamp. It uses a PDF analysis library to read the page content of the file and extract the image data. At this stage, the specific coordinate information of the stamp is obtained.

[0695] 3. Extraction of imprint image

[0696] The server extracts an image of the stamped portion from the detected stamp position and saves it as an independent image file, which is used in subsequent processing.

[0697] 4. Type determination and character recognition

[0698] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The system determines the type of corporate seal (round seal, square seal, or seal of the person in charge) and converts the character information of the seal impression into text data. For example, if the seal is a "round seal," the text may be converted to "CEO of ABC Co., Ltd."

[0699] 5. Database registration

[0700] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[0701] 6. Automated Review

[0702] The server automatically checks the seal on the application form based on the information registered in the database. Based on the screening rules, it verifies whether the seal is genuine and records the results in the screening table or log.

[0703] 7. Notification of the review results

[0704] Users can access a dedicated dashboard to check the results and status of their applications, and the server will notify them of the results and request corrections or confirmations as necessary.

[0705] 8. Accumulation of learning data

[0706] The server stores the AI ​​judgment results and human review results as learning data, which is used to retrain the AI ​​model and contribute to improving the accuracy of the entire system.

[0707] 9. Implementing the Emotion Engine

[0708] The server is equipped with an emotion engine that recognizes the user's emotions. When a user uploads an application form or checks the screening results, the device's camera and microphone are used to capture the user's facial image and voice. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.).

[0709] Specific examples

[0710] For example, suppose a company submits an application to sign a new financial service contract. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[0711] This system not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, providing a more comprehensive user experience.

[0712] The processing flow will be explained below.

[0713] The present invention relates to a system that automates the process of checking whether an application form has been stamped and that also incorporates an emotion engine that recognizes the user's emotions. A specific example of the system will be described below.

[0714] System Overview

[0715] This system accepts uploaded application forms, automatically determines the type of seal and text information, and also has the ability to recognize the user's emotions. The server handles the main processing, and users use their terminals to upload application files and check the review results and their own emotions.

[0716] Program processing

[0717] Step 1:

[0718] User: Uploads the application form PDF file to the server from the device.

[0719] Operation details: The user selects "Application_Company A.pdf" from a browser or dedicated application and presses the send button. The file is sent to the server via an HTTP request.

[0720] Step 2:

[0721] Server: Receives and stores the uploaded application file.

[0722] Operation details: The server saves the received file in a specific directory, records the file path, and generates an application ID.

[0723] Step 3:

[0724] Server: Analyzes the PDF file and detects the location of the stamp.

[0725] Operation details: The server uses a PDF analysis library (e.g., PyMuPDF or PDFBox) to convert each page of the PDF into image data, then applies a stamp detection algorithm to obtain the coordinate information of the stamp.

[0726] Step 4:

[0727] Server: Extracts image data of the stamped area.

[0728] Operation details: Based on the acquired coordinate information, an image processing library (e.g., OpenCV) is used to extract the seal portion. This is saved as "Seal Impression_CompanyA.png".

[0729] Step 5:

[0730] Server: Sends the stamp image to an image recognition system to determine the type of corporate seal.

[0731] Operation details: The stamp image is sent to an image recognition API (e.g., GeminiAI), and the classification model is used to classify it into one of "round stamp," "square stamp," or "person in charge stamp."

[0732] Step 6:

[0733] Server: Converts the text information of the seal impression from the stamp image using OCR.

[0734] Operation details: Using an OCR library (e.g., Tesseract), extract characters from the stamp image and convert them into text data such as "CEO, ABC Co., Ltd."

[0735] Step 7:

[0736] Server: Registers the judgment results and text information in a database.

[0737] Operation details: An INSERT statement is used to add a record containing the application form ID, corporate seal type, and seal text to the database server.

[0738] Step 8:

[0739] Server: Conducts automatic review based on registered information.

[0740] Operation details: According to the screening rules, query the database information to confirm whether the seal is valid. Record the result in the screening table.

[0741] Step 9:

[0742] User: Check the review results.

[0743] How it works: Users access the dashboard to check the review results and status, and manually correct or approve as needed.

[0744] Step 10:

[0745] Server: Accumulates judgment results and human review results as learning data.

[0746] Operation details: The results of the review are saved in a database and accumulated as a dataset for retraining the model. A process to periodically update the AI ​​model is scheduled.

[0747] Step 11:

[0748] Server: Uses an emotion engine to recognize the user's emotions.

[0749] Operation details: The server captures the user's facial image and voice via the device's camera and microphone. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.) in real time.

[0750] Step 12:

[0751] Server: Provides the user with feedback on the recognized emotion information.

[0752] Operation details: The recognized emotional information is visualized and fed back to the user via a dashboard. Users can check their own emotional information and understand their feelings during the review process.

[0753] Specific examples

[0754] For example, suppose a company submits an application to sign a new financial service contract. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[0755] This system not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, providing a more comprehensive user experience.

[0756] Example 2

[0757] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0758] The traditional application form seal checking process is often done manually, which is inefficient and has a high risk of misjudgment. Furthermore, the lack of a feedback function that takes into account the user's emotions results in a poor user experience. To solve these problems, a system that integrates an automated seal checking function with a function that recognizes the user's emotions is needed.

[0759] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0760] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the judgment results and the screening results as learning data, and means for recognizing the user's emotions. This makes it possible to automate the application form seal check process and provide feedback that takes the user's emotions into consideration.

[0761] "Application" means a document that contains the information necessary to carry out a particular transaction.

[0762] A "PDF analysis library" is a software library for analyzing and manipulating PDF files programmatically.

[0763] "Seal" refers to a seal that is stamped on a document to indicate its legitimacy or approval.

[0764] "Image recognition technology" is a technology that allows computers to analyze images and videos and identify objects and characters.

[0765] A "database" is a system for efficiently managing, searching, and storing large amounts of data.

[0766] "Examination rules" are standards and guidelines for determining whether the contents of the application form and the stamp are genuine.

[0767] "User" means any person or entity that uses this system to upload applications and review results.

[0768] "Terminal" means a computing device through which a user accesses the system.

[0769] The "emotion engine" is a system that analyzes the user's facial images and voice data to recognize emotions.

[0770] The present invention relates to a system that automates the process of checking the seals on application forms and recognizes the emotions of users, and will be described below with reference to specific examples.

[0771] The system mainly consists of three components: a server, a terminal, and a user. The server is responsible for the main processing, and the user uses the terminal to upload the application form and check the review results and emotion recognition feedback.

[0772] 1. Upload your application form

[0773] The user uploads the application form PDF file to the server from their device. First, the user logs in to the system's dedicated portal and opens the application form upload page. Then, they click "Select file" and select the application form PDF from local storage. Then, they can press the "Upload" button to send the file to the server.

[0774] 2. Parse the PDF

[0775] The server receives the uploaded PDF file and begins parsing it. The server first saves the file in an inbox, then calls a PDF parsing library (e.g., PyPDF2) to open the file. The server reads each page in turn, extracts the image data, and applies an algorithm to identify the location of the stamp.

[0776] 3. Extraction of imprint image

[0777] The server extracts an image of the stamp based on the coordinate information of the identified stamp. The server uses an image processing library (e.g., OpenCV) to cut out a rectangular area from the specified coordinates and save it as a new image file. This image file is saved in a temporary folder and used for subsequent processing.

[0778] 4. Type determination and character recognition

[0779] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The server creates an HTTP request, attaches the seal image file, and sends it to the AI ​​image recognition API. The API determines the type of corporate seal (round seal, square seal, or seal of responsible person) and converts the character information of the seal impression into text data. For example, the API returns the character information "round seal" and "CEO of ABC Co., Ltd."

[0780] 5. Database registration

[0781] The server registers the determined corporate seal type and character information in a database. The server uses a database connection library (e.g., SQLAlchemy) to store the obtained information in the corresponding table. This table contains information such as the application form ID, corporate seal type, and seal text.

[0782] 6. Automated Review

[0783] The server automatically checks the seal on the application form based on the information registered in the database. The server checks whether the corporate seal matches the application details based on pre-set review rules. The results are registered in the review table and also recorded in a log file.

[0784] 7. Notification of the review results

[0785] The user accesses a dedicated dashboard to check the results of the review. When the user logs in, the server displays the relevant review results on the dashboard. If necessary, the server sends the user an email notifying them of the review results.

[0786] 8. Accumulation of learning data

[0787] The server accumulates the AI's judgment results and the human review results as learning data. The server periodically aggregates this data and uses it to retrain the AI ​​model, thereby improving the accuracy of the entire system.

[0788] 9. Implementing the Emotion Engine

[0789] The server is equipped with an emotion engine that recognizes the user's emotions. When a user uploads an application form or checks the screening results, the device's camera and microphone are used to capture the user's facial image and voice. The server acquires this data and sends it to the emotion engine. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.). The analysis results are stored in a database and used to improve the user experience.

[0790] Specific examples

[0791] For example, consider the case where a company submits an application for a new financial service contract. The user logs in to the system using a terminal and uploads the application. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted and sent to an image recognition system, which determines that it is a "circle stamp" and obtains the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[0792] Prompt Sentence Examples

[0793] "I would like to upload a new financial services application to the system. What are the next steps?"

[0794] 1. How to upload your application form

[0795] 2. How to check the review results

[0796] 3. How to check emotional information

[0797] "

[0798] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0799] Specific processing flow of the program

[0800] Step 1: Upload your application

[0801] Input: Application form PDF file selected by the user on the device.

[0802] Operation:

[0803] Users log into the system's dedicated portal.

[0804] Click the "Select File" button and select the application form PDF file from your local storage.

[0805] Click the "Upload" button to send the file to the server.

[0806] Output: Application form PDF file uploaded to the server.

[0807] Step 2: Parse the PDF

[0808] Input: Application form PDF file saved on the server.

[0809] Operation:

[0810] The server stores the file in the inbox.

[0811] Open the file using a PDF parsing library (e.g. PyPDF2).

[0812] Each page is read in turn, image data is extracted, and an algorithm is applied to identify the location of the stamp.

[0813] Output: Coordinate information of the imprint.

[0814] Step 3: Extract the stamp image

[0815] Input: Stamp coordinate information and application form PDF file.

[0816] Operation:

[0817] The server uses an image processing library (e.g., OpenCV) to extract the stamped portion from the specified coordinates.

[0818] The extracted image data of the stamped portion is saved as a new image file.

[0819] Output: Extracted stamp image file.

[0820] Step 4: Type determination and character recognition

[0821] Input: Extracted stamp image file.

[0822] Operation:

[0823] The server creates an HTTP request, attaches the stamp image file, and sends it to the AI ​​image recognition API (e.g., GeminiAI).

[0824] The AI ​​image recognition API determines the type of corporate seal (round seal, square seal, seal of responsible person) and converts the character information of the seal impression into text data.

[0825] Output: Type of corporate seal and text information of the seal impression.

[0826] Step 5: Database Registration

[0827] Input: The type of corporate seal determined and text information of the seal impression.

[0828] Operation:

[0829] The server uses a database connection library (e.g. SQLAlchemy) to store the retrieved information in the corresponding table.

[0830] The database contains information such as application form ID, type of corporate seal, and stamp text.

[0831] Output: The type of corporate seal registered in the database and text information of the seal impression.

[0832] Step 6: Automated Review

[0833] Input: The type of corporate seal registered in the database and text information of the seal impression.

[0834] Operation:

[0835] The server checks whether the corporate seal matches the application details based on pre-set review rules.

[0836] The results of the review are registered in the review table and also recorded in a log file.

[0837] Output: Review results.

[0838] Step 7: Notification of review results

[0839] Input: Review results.

[0840] Operation:

[0841] Users can access a dedicated dashboard to check the results of their review.

[0842] When a user logs in, the server displays the corresponding review results on the dashboard.

[0843] If necessary, the server will send an email to notify the user of the results of the review.

[0844] Output: Notification of review results to the user.

[0845] Step 8: Accumulating training data

[0846] Input: AI judgment results and human review results.

[0847] Operation:

[0848] The server periodically aggregates this data.

[0849] It is accumulated as learning data to be used for retraining AI models.

[0850] Output: Training data for retraining the AI ​​model.

[0851] Step 9: Implementing the Emotion Engine

[0852] Input: User's facial image and voice data.

[0853] Operation:

[0854] When a user uploads an application form or checks the review results, the device's camera and microphone are used to capture facial images and audio data.

[0855] The server retrieves this data and sends it to the emotion engine.

[0856] The emotion engine analyzes the data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.).

[0857] The analysis results are stored in a database.

[0858] Output: User emotion recognition results.

[0859] (Application example 2)

[0860] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0861] Conventional application management systems have difficulty not only verifying the authenticity of stamps but also grasping the emotional state of workers. Furthermore, particularly at work sites such as logistics centers, there is a lack of mechanisms for monitoring workers' psychological states in real time and responding accordingly. This can result in a loss of safety and efficiency. Therefore, there is a need for a system that can automate the stamp checks on application forms, recognize workers' emotions, and manage that information.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting upload of an application form, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal portion, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal of the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the judgment results and the screening results as learning data, means for recognizing the user's emotions, means for recording the emotion recognition data in a database, and means for the user to confirm emotions during the application form processing process. This not only enables automatic and accurate seal checks on application forms, but also enables real-time recognition and management of the emotional state of workers.

[0863] The "means for accepting application form uploads" is a function for importing digital files of application forms sent by users into the system.

[0864] The "means for analyzing the received application form file and detecting the position of the stamp" is a function for analyzing the contents of the application form obtained as an electronic file and identifying the location of the stamp.

[0865] The "means for extracting image data of the detected stamped portion" is a function for cutting out the relevant portion from the specified stamped position as an image.

[0866] The "means for determining the type of corporate seal from the extracted seal image" is a function that analyzes the cut-out seal portion and identifies what type of corporate seal it is (round seal, square seal, etc.).

[0867] The "means for converting the character information of the seal impression from the seal image into text data" is a function that converts the character information contained in the seal image into text format using optical character recognition (OCR) technology.

[0868] "Means for registering the determined information in a database" is a function for storing analyzed corporate seal information and character information in a database.

[0869] "Means for automatically checking the stamp on the application form based on registered information" refers to a function in which the system automatically checks whether the stamp on the application form is genuine, using information registered in the database.

[0870] "Means for notifying the user of the review results" is a function that reports the results of the stamp check to the user.

[0871] "Means for accumulating judgment results and review results as learning data" refers to a function that saves the judgment and review results made by the system as learning data and uses them to improve the system in the future.

[0872] The "means for recognizing the user's emotions" is a function that analyzes the user's facial image and voice data to determine the user's emotional state.

[0873] The "means for recording emotion recognition data in a database" is a function for storing the recognized emotion data of a user in a database.

[0874] The "means for users to check their emotions during the application process" is a function that allows users to check their own emotional state during the process of checking the review results and processing status of their application.

[0875] This invention relates to a system that automates the process of checking whether an application form has been stamped and combines it with an emotion engine that recognizes the emotions of workers. Specifically, this invention describes an embodiment of a system that uses smart glasses in a logistics center.

[0876] System Overview

[0877] This system consists of a terminal (specifically, smart glasses) and a server. The user (worker) scans the application form using the smart glasses and sends the image data to the server. The server processes the data, checks the stamp and recognizes emotions.

[0878] Hardware and software used

[0879] Hardware:

[0880] Smart glasses (e.g., Google Glass): A device worn by workers to scan application forms.

[0881] Server: A central device that stores data and performs various analytical processes.

[0882] software:

[0883] OpenCV: An image processing library used to detect the stamp position.

[0884] pytesseract: An OCR library used to extract text information from image data.

[0885] EmotionRecognizer: An optional emotion recognition library used to analyze a user's emotional state in real time.

[0886] Flask: A web application framework used to implement system-wide APIs.

[0887] Detailed explanation of functions

[0888] 1. Upload your application form

[0889] The user wears the smart glasses and scans the application form with the camera. The smart glasses then send the captured image data to the server, where it is stored in the server's database.

[0890] 2. Parse the PDF

[0891] The server analyzes the received image data using OpenCV to identify the location of the seal. First, it converts the image to grayscale, then performs threshold processing to highlight the seal area.

[0892] 3. Extraction of imprint image

[0893] The server extracts the identified stamped portion as an image and saves it as a separate image file. This extraction process is also performed using OpenCV.

[0894] 4. Type determination and character recognition

[0895] The server performs OCR processing on the extracted seal image using pytesseract, which converts the type of seal (e.g., a round or square corporate seal) and character information into text data.

[0896] 5. Emotion recognition

[0897] At the same time, the server uses EmotionRecognizer to analyze the user's facial image and voice data in real time to recognize their emotional state, for example, to determine whether the user is in an anxious state.

[0898] Specific examples

[0899] When worker A processes an application for receiving new goods, he scans the application using smart glasses. The server receives this data and simultaneously checks the stamp and recognizes worker A's emotional state. If the stamp is confirmed to be authentic and emotion analysis reveals that worker A is in an anxious state, an alert is sent to the manager.

[0900] Prompt Sentence Examples

[0901] You are a worker in charge of receiving new goods. Using smart glasses, you scan the application form and check the authenticity of the stamp. At the same time, the system also recognizes your emotional state (e.g., anxiety, surprise). The manager sees this and takes the necessary action.

[0902] In this way, the present invention not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, thereby improving work efficiency and safety.

[0903] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0904] Step 1:

[0905] The user scans an application form using smart glasses. The smart glasses capture the image data of the application form and send it to the server. At this time, the camera in the smart glasses captures the entire application form as a high-resolution image file. The input is the actual application form, and the output is the image data on the server.

[0906] Step 2:

[0907] The server analyzes the received image data and detects the location of the seal. The server converts the image to grayscale using OpenCV and performs threshold processing to highlight the presence of the seal. It then performs contour extraction and identifies the location of the seal. The input is the image data of the application form, and the output is the coordinate information of the seal.

[0908] Step 3:

[0909] The server extracts image data of the detected seal area. Based on the identified coordinate information, the seal area is cut out and saved as a separate image file. This image file is used for subsequent OCR processing. The input is image data with the seal coordinate information, and the output is image data of the seal area.

[0910] Step 4:

[0911] The server determines the type of corporate seal from the extracted image of the seal. The server uses pytesseract to perform OCR on this image and determines the type of corporate seal, such as whether it is a round seal or a square seal, from the acquired character information. The server also converts the character information contained in the seal into text data. The input is image data of the seal portion, and the output is the type of seal and text data.

[0912] Step 5:

[0913] The server registers the determined information in a database. The server stores the corporate seal type and text data obtained by OCR in the database and links it to the application form ID. The input is the seal type and text data, and the output is registration in the database.

[0914] Step 6:

[0915] The server automatically checks the seal on the application form based on the registered information. It checks whether the seal is genuine based on the information registered in the database and the screening rules. The results are recorded in the screening table. The input is the database information and screening rules, and the output is the screening results.

[0916] Step 7:

[0917] The server notifies the user of the results of the review. The server notifies the user of the results of the review through the dashboard and prompts corrections if necessary. The input is the review result data, and the output is a notification to the user.

[0918] Step 8:

[0919] The server accumulates the judgment and review results as learning data. This accumulates data to be used for retraining the AI ​​model, contributing to improving the accuracy of the system. The input is the review data, and the output is the accumulated learning data.

[0920] Step 9:

[0921] The server recognizes the user's emotions. The EmotionRecognizer analyzes the user's facial images and voice data collected by the smart glasses' camera and microphone to recognize their emotional state. The input is the user's facial images and voice data, and the output is the user's emotional state data.

[0922] Step 10:

[0923] The server records the emotion recognition data in a database. The recognized emotion data is linked to the application form ID and saved in the database. The input is emotional state data, and the output is a record in the database.

[0924] Step 11:

[0925] The user checks their emotions while processing the application form. They then check the review results and their emotional state on the dashboard and implement the countermeasures suggested by the system. The input is the data displayed on the dashboard, and the output is the user's confirmation and response actions.

[0926] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0927] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0928] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0929] [Third embodiment]

[0930] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0931] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0932] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0933] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0934] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0935] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0936] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0937] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0938] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0939] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0940] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0941] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0942] The present invention relates to a system for automating the check of seals on application forms, and a specific embodiment thereof will be described.

[0943] System Overview

[0944] This system accepts uploaded application forms, automatically determines the type of seal and character information, and uses that information to check the seal. The server handles the main processing, and users use their terminals to upload application form files and check the review results.

[0945] Program processing

[0946] 1. Upload your application form

[0947] The user uploads the application form PDF file to the server using a terminal. The server receives the uploaded file and starts processing, which digitally imports the application form into the system.

[0948] 2. Parse the PDF

[0949] The server analyzes the received PDF application file to detect the location of the stamp. Using a PDF analysis library, it reads the page content of the file and extracts the image data. At this stage, the specific coordinate information of the stamp is obtained.

[0950] 3. Extraction of imprint image

[0951] The server extracts an image of the stamped portion from the detected stamp position and saves it as an independent image file, which is used in subsequent processing.

[0952] 4. Type determination and character recognition

[0953] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The system determines the type of corporate seal (round seal, square seal, or seal of the person in charge) and converts the character information of the seal impression into text data. For example, if the seal is a "round seal," the text may be converted to "CEO of ABC Co., Ltd."

[0954] 5. Database registration

[0955] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[0956] 6. Automated Review

[0957] The server automatically checks the seal on the application form based on the information registered in the database. Based on the screening rules, it verifies whether the seal is genuine and records the results in the screening table or log.

[0958] 7. Notification of the review results

[0959] Users can access a dedicated dashboard to check the results and status of their applications, and the server will notify them of the results and request corrections or confirmations as necessary.

[0960] 8. Accumulation of learning data

[0961] The server stores the AI ​​judgment results and human review results as learning data, which is used to retrain the AI ​​model and contribute to improving the accuracy of the entire system.

[0962] Specific examples

[0963] For example, suppose a company submits an application to sign a contract for a new financial service. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database, and an automatic review is performed. The review results are displayed on the user's dashboard and can be confirmed. The results of the review process are accumulated as future learning data.

[0964] This series of processes will enable automatic and accurate checks of application forms for stamps, greatly improving the efficiency of the review process.

[0965] The processing flow will be explained below.

[0966] Step 1:

[0967] User: Uploads the application form PDF file to the server from the device.

[0968] Operation details: The user selects "Application_Company A.pdf" from a browser or dedicated application and presses the send button. The file is sent to the server via an HTTP request.

[0969] Step 2:

[0970] Server: Receives and stores the uploaded application file.

[0971] Operation details: The server saves the received file in a specific directory, records the file path, and generates an application ID.

[0972] Step 3:

[0973] Server: Analyzes the PDF file and detects the location of the stamp.

[0974] Operation details: The server uses a PDF analysis library (e.g., PyMuPDF or PDFBox) to convert each page of the PDF into image data, then applies a stamp detection algorithm to obtain the coordinate information of the stamp.

[0975] Step 4:

[0976] Server: Extracts image data of the stamped area.

[0977] Operation details: Based on the acquired coordinate information, an image processing library (e.g., OpenCV) is used to extract the seal portion. This is saved as "Seal Impression_CompanyA.png".

[0978] Step 5:

[0979] Server: Sends the stamp image to an image recognition system to determine the type of corporate seal.

[0980] Operation details: The stamp image is sent to an image recognition API (e.g., GeminiAI), and the classification model is used to classify it into one of "round stamp," "square stamp," or "person in charge stamp."

[0981] Step 6:

[0982] Server: Converts the text information of the seal impression from the stamp image using OCR.

[0983] Operation details: Using an OCR library (e.g., Tesseract), extract characters from the stamp image and convert them into text data such as "Seal of the Sales Director of SoftBank Corp."

[0984] Step 7:

[0985] Server: Registers the judgment results and text information in a database.

[0986] Operation details: An INSERT statement is used to add a record containing the application form ID, corporate seal type, and seal text to the database server.

[0987] Step 8:

[0988] Server: Conducts automatic review based on registered information.

[0989] Operation details: According to the screening rules, query the database information to confirm whether the seal is valid. Record the result in the screening table.

[0990] Step 9:

[0991] User: Check the review results.

[0992] How it works: Users access the dashboard to check the review results and status, and manually correct or approve as needed.

[0993] Step 10:

[0994] Server: Accumulates judgment results and human review results as learning data.

[0995] Operation details: The results of the review are saved in a database and accumulated as a dataset for retraining the model. A process to periodically update the AI ​​model is scheduled.

[0996] Example 1

[0997] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0998] Traditionally, checking the stamps on application forms is often done manually, which is not only time-consuming and labor-intensive, but also carries a high risk of human error. Furthermore, relying on paper media hinders progress in digitizing data, making it difficult to improve work efficiency. It is necessary to solve these problems and automate the stamp check process to improve work efficiency and accuracy.

[0999] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1000] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal portion, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the examination results, means for accumulating the determination and examination results as learning data, means for extracting and saving the image file using the coordinate information of the seal, and means for transmitting the seal image to an image recognition system and receiving the recognition results. This enables automatic and accurate checks of application form seals, significantly improving the efficiency and accuracy of operations.

[1001] An "application form" is a document that contains the information necessary to make an application or contract.

[1002] The "means for accepting uploads" refers to a function for receiving an application file sent by a user to a server using a terminal.

[1003] "Means for analyzing the received application file and detecting the position of the stamp" refers to the function of the server using a general-purpose library for analyzing electronic documents to identify the position of the stamp from the received application file.

[1004] "Means for extracting image data of the detected stamped portion" refers to a function for cutting out an image of the stamped portion from the application form file based on the coordinate information of the identified stamp.

[1005] "Means for determining the type of corporate seal from the extracted seal image" refers to a function that uses image recognition technology to identify the type of seal from the extracted seal image.

[1006] "Means for converting character information of the seal impression from the seal image into text data" refers to a function that uses an image recognition system to convert the text contained in the seal image into character data.

[1007] "Means for registering the determined information in a database" refers to the function of storing the type of seal and character information in a database.

[1008] "Means for automatically checking the stamping of application forms based on registered information" refers to the function in which the system automatically applies screening criteria to check the stamping based on the stamping information stored in the database.

[1009] "Means for notifying the user of the review results" refers to a function that displays and notifies the user of the review results on their dashboard.

[1010] "Means for accumulating judgment results and review results as learning data" refers to the function of saving judgment results and actual review results in a database and accumulating them as learning data in order to improve the accuracy of the AI ​​model.

[1011] "Means for extracting and saving an image file using coordinate information of a stamp" refers to a function that extracts the relevant part from a PDF file based on the location information of the identified stamp and saves it as an image file.

[1012] "Means for sending a stamp image to an image recognition system and receiving the recognition results" refers to the function of sending the extracted stamp image to an image recognition system and receiving the type of recognized stamp and text information.

[1013] The present invention relates to a system for automating the seal check of application forms, and a specific embodiment thereof will be described below. The system is implemented based on a process involving mainly a server, a terminal, and a user.

[1014] System Overview

[1015] This system accepts uploaded application forms, automatically determines the type of seal and character information, and checks the seal based on that information. The server handles the main processing, and users use their terminals to upload application form files and check the review results.

[1016] Hardware and software used

[1017] The server uses the following hardware and software:

[1018] Hardware: High-performance server equipment (e.g., a server equipped with an Intel Xeon processor)

[1019] Software: PDF analysis library (e.g., Apache PDFBox), image recognition system (e.g., GeminiAI), database system (e.g., MySQL)

[1020] Program processing

[1021] Upload your application form

[1022] The user logs in from their device and uploads the application form PDF file to the server. The device sends the file to the server, and the server saves the received file.

[1023] Parsing PDFs

[1024] The server uses the Apache PDFBox library to analyze the received application PDF file and detect the position of the stamp from the content of each page. At this stage, the coordinate information of the stamp is obtained.

[1025] Extraction of stamp image

[1026] The server extracts an image of the relevant part from the application form file based on the coordinate information of the stamp it has acquired, and saves the extracted image as a separate image file.

[1027] Type determination and character recognition

[1028] The server sends the extracted image of the seal to an image recognition system such as GeminiAI. The image recognition system determines the type of seal (e.g., round seal, square seal, seal of responsible person) and converts the characters contained in the seal into text data. For example, it may determine that the seal is a "round seal" and the text is "CEO of ABC Co., Ltd."

[1029] Database registration

[1030] The server registers the determined type of corporate seal and character information in a MySQL database, which stores information such as the application form ID, type of corporate seal, and seal text.

[1031] Automated Review

[1032] The server automatically checks the authenticity of the seal based on the information registered in the database and pre-set review rules, and the review results are recorded in a review table and log.

[1033] Notification of review results

[1034] Users access a dedicated dashboard to view the latest audit results provided by the server, and are notified of any corrections or additional confirmations as needed.

[1035] Accumulation of learning data

[1036] The server accumulates the AI's judgment results and the human review results as learning data, which allows the AI ​​model to be retrained and contributes to improving the accuracy of the entire system.

[1037] Specific examples

[1038] For example, suppose a company submits an application to sign a new financial service contract. The user uploads the application PDF file from their computer to the system. The server receives the PDF file and analyzes it to identify the location of the seal. It extracts the image of the seal and uses an image recognition system to identify that it is a "circle seal" and that the text is "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard and can be confirmed. The results obtained during the review process are accumulated as future learning data.

[1039] This will enable automatic and accurate checks of application forms for stamps, greatly improving the efficiency of the review process.

[1040] Prompt Sentence Examples

[1041] Please explain a system that automatically checks the stamps on application forms submitted by companies for financial services contracts. The system includes uploading PDF files, detecting stamp positions, extracting stamp images, determining type and character recognition, registering in a database, automatic screening, notifying the results of the screening, and accumulating learning data.

[1042] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1043] Step 1: Upload your application

[1044] The user logs in from their device, selects the application form PDF file, and clicks the upload button. This action causes the device to send the file to the server, which then saves the received PDF file in a directory.

[1045] Input: Application form PDF file selected by the user.

[1046] Output: Application file saved in the server's save directory.

[1047] Specific operation: The user clicks the "Select File" button to select a PDF file from the device's file system. Then, by pressing the "Upload" button, the device sends the file to the server via an HTTP request. The server receives the file and saves it in the specified directory.

[1048] Step 2: Parse the PDF

[1049] The server uses a PDF analysis library (e.g., Apache PDFBox) to analyze the received PDF file, extract the image data and text data of each page, and obtain the coordinate information to detect the stamp position.

[1050] Input: Saved application form PDF file.

[1051] Output: Data including the coordinate information of the stamp.

[1052] Specific operation: The server uses the PDFBox library to read the PDF file, scans the elements of each page, identifies areas that may be stamped, obtains their coordinate information, and stores it in a list.

[1053] Step 3: Extract the stamp image

[1054] The server extracts an image of the relevant part from the PDF application file based on the coordinate information of the stamp it has acquired, and saves the extracted image as a separate image file.

[1055] Input: Stamp coordinate information and application form PDF file.

[1056] Output: Image file of the stamped area.

[1057] Specific operation: The server reopens the PDF file, extracts the relevant part based on the specified coordinate information, and then saves the extracted image in a new file in PNG format or other format.

[1058] Step 4: Type determination and character recognition

[1059] The server sends the extracted image of the seal to an image recognition system (e.g., GeminiAI), which determines the type of seal and converts the characters contained in the seal into text data.

[1060] Input: Image file of the stamped area.

[1061] Output: Stamp type and text data.

[1062] Specific operation: The server sends the image file to GeminiAI's API and parses the received JSON response. The response includes the type of seal (e.g., circle) and text data (e.g., "CEO of XYZ Co., Ltd.").

[1063] Step 5: Database Registration

[1064] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[1065] Input: Stamp type and text data.

[1066] Output: Stamp information registered in the database.

[1067] Specific operation: The server uses a database system such as MySQL to store the retrieved information in the corresponding columns, and stores the information using the SQL INSERT statement.

[1068] Step 6: Automated Review

[1069] The server automatically checks the legitimacy of the seal based on the information registered in the database and pre-set review rules.

[1070] Input: Stamp information registered in the database.

[1071] Output: Review results.

[1072] Specific operation: The server evaluates the information in the database using a specific business rule engine. It checks the type of stamp and the degree of match of the stamped text to verify its validity. The results of the verification are recorded in a review table or log.

[1073] Step 7: Notification of review results

[1074] Users can access a dedicated dashboard to check the results of the review, and the server will notify the user of the results and request corrections or confirmations as necessary.

[1075] Input: Review results.

[1076] Output: Review results displayed on the dashboard.

[1077] What happens: When a user accesses the dashboard, the server retrieves and displays the latest audit results. If necessary, an appropriate notification is displayed if corrections or additional actions are required.

[1078] Step 8: Accumulating training data

[1079] The server accumulates the AI's judgment results and the human review results as learning data, which allows the AI ​​model to be retrained and contributes to improving the accuracy of the entire system.

[1080] Input: AI judgment results and human review results.

[1081] Output: Updated training data.

[1082] How it works: The server periodically collects the judgment results and actual review results and stores them in a database. This data is then used to retrain the AI ​​model, contributing to improving its accuracy.

[1083] (Application example 1)

[1084] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1085] Checking the accuracy of stamps on numerous application forms and documents at factories and other worksites takes time and effort. When the type of stamp and text information is checked manually, errors and delays are likely to occur, reducing work efficiency and increasing the risk of human error. Furthermore, there is a lack of rapid feedback on review results and the accumulation of learning data, which hinders progress in improving and streamlining the system.

[1086] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1087] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the screening results as learning data, means for an automated machine on-site to scan documents and upload them to the server, means for displaying the screening results in real time, means for notifying workers of feedback, and means for generating prompt sentences using a generative AI model to facilitate screening. This improves the efficiency of document processing in factories, reducing errors and work time.

[1088] - "Uploading an application" means that a User uses a Terminal to send an application in digital form to the Server.

[1089] "Analyzing the application file" refers to the process in which the server analyzes the file received and understands its contents and structure.

[1090] "Detecting the position of the seal" means identifying the specific position of the seal on the application form.

[1091] "Extracting image data" means separating and making independent the image data of the detected stamped portion.

[1092] "Determining the type of corporate seal" means analyzing the image of the seal to determine whether it is a round seal, a square seal, a seal of a person in charge, etc.

[1093] "Converting the character information of the seal impression into text data" means converting the stamped characters into digital text using optical character recognition technology or the like.

[1094] "Registering information in a database" means saving the determined type of seal and character information in a database.

[1095] "Automatically checks the stamp" means that the program automatically verifies the authenticity of the stamp based on the registered information.

[1096] "Notifying the user of the examination results" means informing the user of the results of the seal check.

[1097] "Storing as learning data" means saving the results of the review and judgment in order to improve the accuracy of the review in the future.

[1098] "Automated machine scanning of documents" means that an automated machine installed on-site converts physical documents into digital images.

[1099] "Uploading to server" means sending scanned digital data to the server.

[1100] "Displaying review results in real time" means instantly displaying the review results on the user's dashboard or the like.

[1101] "Notifying workers of feedback" means informing on-site workers of the review results and any necessary corrections.

[1102] "Generating prompts using a generative AI model" means using an artificial intelligence model to automatically generate messages to facilitate the review process.

[1103] This invention relates to a system that automates the process of checking stamps on application forms at factories. This system comprehensively automates the process from scanning documents to determining the type of stamp and text information, and notifying the results of the inspection in real time.

[1104] The system has several main functions: a means for accepting application form uploads, a means for analyzing received application form files and detecting the position of the seal, a means for extracting image data of the detected seal, a means for determining the type of corporate seal from the extracted seal image, a means for converting the character information of the seal impression from the seal image into text data, a means for registering the determined information in a database, a means for automatically checking the seal on the application form based on the registered information, a means for notifying the user of the screening results, a means for accumulating the determination results and the screening results as learning data, a means for on-site automated machines to scan documents and upload them to a server, a means for displaying the screening results in real time, a means for notifying workers of feedback, and a means for generating prompt messages to facilitate the screening process using a generative AI model.

[1105] Specifically, the server uses the following software and hardware: It uses a PDF analysis library (e.g., PyPDF2) to analyze the application file and identify the location of the seal. It uses an image processing library (e.g., PIL and OpenCV) to extract image data of the seal area, and an OCR library (e.g., pytesseract) to convert the character information of the seal impression into text data. The determined information is stored in a database using an HTTP request library (e.g., requests). It uses a generative AI model to generate prompts to facilitate the review process.

[1106] For example, imagine a factory needs to apply for a manufacturing license. An automated machine scans the physical document and uploads it to a server as a PDF file. The server then analyzes the PDF file, locating the stamp, extracting image data, and capturing text information through OCR. This information is stored in a database for automated review. A generative AI model is used to generate prompts for review. The review results are displayed in real time on a dashboard, providing feedback to workers.

[1107] An example of a prompt might be:

[1108] "A new manufacturing permit has been scanned by the robot. Please use the following information to determine the stamp type and text information on the application:

[1109] Stamp image: [Image URL]

[1110] Review rules: [URL for detailed review rules]

[1111] In this way, the present invention significantly improves the efficiency of the stamp checking process on application forms at the factory site, preventing mistakes and delays.

[1112] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1113] Step 1:

[1114] The user uploads the PDF file of the application form to the server using a terminal. The input is the PDF file of the application form, and the output is a digital file stored on the server. This process imports paper documents into the system as digital data.

[1115] Step 2:

[1116] The server analyzes the received PDF file to detect the location of the stamp. Specifically, it uses a PDF analysis library (e.g. PyPDF2). The input is the PDF file, and the output is the coordinate information (position) of the stamp. This process yields data that identifies where the stamp is located.

[1117] Step 3:

[1118] The server extracts image data from the identified stamp location. It uses an image processing library (e.g., PIL and OpenCV) to extract the image of the stamped area. The input is the stamp coordinate information and the PDF file, and the output is the stamped image. This results in the stamped area being obtained as a separate image file.

[1119] Step 4:

[1120] The server analyzes the extracted seal image using an OCR library (e.g., pytesseract) and converts the character information of the seal impression into text data. The input is the seal image, and the output is text data. Specifically, the OCR engine recognizes the characters in the image and converts them into digital text.

[1121] Step 5:

[1122] The server determines the extracted text data and type of seal and registers it in a database. It uses an HTTP request library (e.g., requests). The input is the type of seal and text data, and the output is a database entry. At this step, the determined information is saved in the system.

[1123] Step 6:

[1124] The server automatically performs a stamp check based on the information stored in the database. It evaluates the data according to the review rules and verifies whether the stamp is genuine. The input is the database entry and the output is the review result. Specifically, the server checks the integrity of the data based on the defined review rules.

[1125] Step 7:

[1126] The server notifies the user of the review results. The review results are displayed in real time on the dashboard for the user to check. The input is the review result data, and the output is a notification on the dashboard. In this step, the user can check the results immediately.

[1127] Step 8:

[1128] The server accumulates the judgment and review results as learning data. This allows the generative AI model to provide more accurate results in future reviews. The input is the judgment and review results, and the output is the learning data. This process improves the accuracy of the system.

[1129] Step 9:

[1130] An automated machine on-site scans the document and uploads it to a server. This step is the process of converting paper documents into digital data. The input is the physical document and the output is digital image data.

[1131] Step 10:

[1132] The server uses the generative AI model to generate prompts to facilitate the review. The input is the AI ​​model and the necessary data, and the output is the prompt. Specifically, the prompt is automatically generated and used during the review.

[1133] Example prompt sentence:

[1134] "A new manufacturing permit has been scanned by the robot. Please use the following information to determine the stamp type and text information on the application:

[1135] Stamp image: [Image URL]

[1136] Review rules: [URL for detailed review rules]

[1137] The above steps automate the process of checking stamps on application forms at the factory, thereby improving efficiency.

[1138] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1139] The present invention relates to a system that automates the process of checking whether an application form has been stamped and that also incorporates an emotion engine that recognizes the user's emotions. A specific example of the system will be described below.

[1140] System Overview

[1141] This system accepts uploaded application forms, automatically determines the type of seal and text information, and also has the ability to recognize the user's emotions. The server handles the main processing, and users use their terminals to upload application files and check the review results and their own emotions.

[1142] Program processing

[1143] 1. Upload your application form

[1144] The user uploads the application form PDF file from their device to the server, which receives the uploaded file and starts processing it, thereby digitally importing the application form into the system.

[1145] 2. Parse the PDF

[1146] The server analyzes the received PDF application file to detect the location of the stamp. It uses a PDF analysis library to read the page content of the file and extract the image data. At this stage, the specific coordinate information of the stamp is obtained.

[1147] 3. Extraction of imprint image

[1148] The server extracts an image of the stamped portion from the detected stamp position and saves it as an independent image file, which is used in subsequent processing.

[1149] 4. Type determination and character recognition

[1150] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The system determines the type of corporate seal (round seal, square seal, or seal of the person in charge) and converts the character information of the seal impression into text data. For example, if the seal is a "round seal," the text may be converted to "CEO of ABC Co., Ltd."

[1151] 5. Database registration

[1152] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[1153] 6. Automated Review

[1154] The server automatically checks the seal on the application form based on the information registered in the database. Based on the screening rules, it verifies whether the seal is genuine and records the results in the screening table or log.

[1155] 7. Notification of the review results

[1156] Users can access a dedicated dashboard to check the results and status of their applications, and the server will notify them of the results and request corrections or confirmations as necessary.

[1157] 8. Accumulation of learning data

[1158] The server stores the AI ​​judgment results and human review results as learning data, which is used to retrain the AI ​​model and contribute to improving the accuracy of the entire system.

[1159] 9. Implementing the Emotion Engine

[1160] The server is equipped with an emotion engine that recognizes the user's emotions. When a user uploads an application form or checks the screening results, the device's camera and microphone are used to capture the user's facial image and voice. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.).

[1161] Specific examples

[1162] For example, suppose a company submits an application to sign a new financial service contract. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[1163] This system not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, providing a more comprehensive user experience.

[1164] The processing flow will be explained below.

[1165] The present invention relates to a system that automates the process of checking whether an application form has been stamped and that also incorporates an emotion engine that recognizes the user's emotions. A specific example of the system will be described below.

[1166] System Overview

[1167] This system accepts uploaded application forms, automatically determines the type of seal and text information, and also has the ability to recognize the user's emotions. The server handles the main processing, and users use their terminals to upload application files and check the review results and their own emotions.

[1168] Program processing

[1169] Step 1:

[1170] User: Uploads the application form PDF file to the server from the device.

[1171] Operation details: The user selects "Application_Company A.pdf" from a browser or dedicated application and presses the send button. The file is sent to the server via an HTTP request.

[1172] Step 2:

[1173] Server: Receives and stores the uploaded application file.

[1174] Operation details: The server saves the received file in a specific directory, records the file path, and generates an application ID.

[1175] Step 3:

[1176] Server: Analyzes the PDF file and detects the location of the stamp.

[1177] Operation details: The server uses a PDF analysis library (e.g., PyMuPDF or PDFBox) to convert each page of the PDF into image data, then applies a stamp detection algorithm to obtain the coordinate information of the stamp.

[1178] Step 4:

[1179] Server: Extracts image data of the stamped area.

[1180] Operation details: Based on the acquired coordinate information, an image processing library (e.g., OpenCV) is used to extract the seal portion. This is saved as "Seal Impression_CompanyA.png".

[1181] Step 5:

[1182] Server: Sends the stamp image to an image recognition system to determine the type of corporate seal.

[1183] Operation details: The stamp image is sent to an image recognition API (e.g., GeminiAI), and the classification model is used to classify it into one of "round stamp," "square stamp," or "person in charge stamp."

[1184] Step 6:

[1185] Server: Converts the text information of the seal impression from the stamp image using OCR.

[1186] Operation details: Using an OCR library (e.g., Tesseract), extract characters from the stamp image and convert them into text data such as "CEO, ABC Co., Ltd."

[1187] Step 7:

[1188] Server: Registers the judgment results and text information in a database.

[1189] Operation details: An INSERT statement is used to add a record containing the application form ID, corporate seal type, and seal text to the database server.

[1190] Step 8:

[1191] Server: Conducts automatic review based on registered information.

[1192] Operation details: According to the screening rules, query the database information to confirm whether the seal is valid. Record the result in the screening table.

[1193] Step 9:

[1194] User: Check the review results.

[1195] How it works: Users access the dashboard to check the review results and status, and manually correct or approve as needed.

[1196] Step 10:

[1197] Server: Accumulates judgment results and human review results as learning data.

[1198] Operation details: The results of the review are saved in a database and accumulated as a dataset for retraining the model. A process to periodically update the AI ​​model is scheduled.

[1199] Step 11:

[1200] Server: Uses an emotion engine to recognize the user's emotions.

[1201] Operation details: The server captures the user's facial image and voice via the device's camera and microphone. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.) in real time.

[1202] Step 12:

[1203] Server: Provides the user with feedback on the recognized emotion information.

[1204] Operation details: The recognized emotional information is visualized and fed back to the user via a dashboard. Users can check their own emotional information and understand their feelings during the review process.

[1205] Specific examples

[1206] For example, suppose a company submits an application to sign a new financial service contract. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[1207] This system not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, providing a more comprehensive user experience.

[1208] Example 2

[1209] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1210] The traditional application form seal checking process is often done manually, which is inefficient and has a high risk of misjudgment. Furthermore, the lack of a feedback function that takes into account the user's emotions results in a poor user experience. To solve these problems, a system that integrates an automated seal checking function with a function that recognizes the user's emotions is needed.

[1211] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1212] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the judgment results and the screening results as learning data, and means for recognizing the user's emotions. This makes it possible to automate the application form seal check process and provide feedback that takes the user's emotions into consideration.

[1213] "Application" means a document that contains the information necessary to carry out a particular transaction.

[1214] A "PDF analysis library" is a software library for analyzing and manipulating PDF files programmatically.

[1215] "Seal" refers to a seal that is stamped on a document to indicate its legitimacy or approval.

[1216] "Image recognition technology" is a technology that allows computers to analyze images and videos and identify objects and characters.

[1217] A "database" is a system for efficiently managing, searching, and storing large amounts of data.

[1218] "Examination rules" are standards and guidelines for determining whether the contents of the application form and the stamp are genuine.

[1219] "User" means any person or entity that uses this system to upload applications and review results.

[1220] "Terminal" means a computing device through which a user accesses the system.

[1221] The "emotion engine" is a system that analyzes the user's facial images and voice data to recognize emotions.

[1222] The present invention relates to a system that automates the process of checking the seals on application forms and recognizes the emotions of users, and will be described below with reference to specific examples.

[1223] The system mainly consists of three components: a server, a terminal, and a user. The server is responsible for the main processing, and the user uses the terminal to upload the application form and check the review results and emotion recognition feedback.

[1224] 1. Upload your application form

[1225] The user uploads the application form PDF file to the server from their device. First, the user logs in to the system's dedicated portal and opens the application form upload page. Then, they click "Select file" and select the application form PDF from local storage. Then, they can press the "Upload" button to send the file to the server.

[1226] 2. Parse the PDF

[1227] The server receives the uploaded PDF file and begins parsing it. The server first saves the file in an inbox, then calls a PDF parsing library (e.g., PyPDF2) to open the file. The server reads each page in turn, extracts the image data, and applies an algorithm to identify the location of the stamp.

[1228] 3. Extraction of imprint image

[1229] The server extracts an image of the stamp based on the coordinate information of the identified stamp. The server uses an image processing library (e.g., OpenCV) to cut out a rectangular area from the specified coordinates and save it as a new image file. This image file is saved in a temporary folder and used for subsequent processing.

[1230] 4. Type determination and character recognition

[1231] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The server creates an HTTP request, attaches the seal image file, and sends it to the AI ​​image recognition API. The API determines the type of corporate seal (round seal, square seal, or seal of responsible person) and converts the character information of the seal impression into text data. For example, the API returns the character information "round seal" and "CEO of ABC Co., Ltd."

[1232] 5. Database registration

[1233] The server registers the determined corporate seal type and character information in a database. The server uses a database connection library (e.g., SQLAlchemy) to store the obtained information in the corresponding table. This table contains information such as the application form ID, corporate seal type, and seal text.

[1234] 6. Automated Review

[1235] The server automatically checks the seal on the application form based on the information registered in the database. The server checks whether the corporate seal matches the application details based on pre-set review rules. The results are registered in the review table and also recorded in a log file.

[1236] 7. Notification of the review results

[1237] The user accesses a dedicated dashboard to check the results of the review. When the user logs in, the server displays the relevant review results on the dashboard. If necessary, the server sends the user an email notifying them of the review results.

[1238] 8. Accumulation of learning data

[1239] The server accumulates the AI's judgment results and the human review results as learning data. The server periodically aggregates this data and uses it to retrain the AI ​​model, thereby improving the accuracy of the entire system.

[1240] 9. Implementing the Emotion Engine

[1241] The server is equipped with an emotion engine that recognizes the user's emotions. When a user uploads an application form or checks the screening results, the device's camera and microphone are used to capture the user's facial image and voice. The server acquires this data and sends it to the emotion engine. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.). The analysis results are stored in a database and used to improve the user experience.

[1242] Specific examples

[1243] For example, consider the case where a company submits an application for a new financial service contract. The user logs in to the system using a terminal and uploads the application. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted and sent to an image recognition system, which determines that it is a "circle stamp" and obtains the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[1244] Prompt Sentence Examples

[1245] "I would like to upload a new financial services application to the system. What are the next steps?"

[1246] 1. How to upload your application form

[1247] 2. How to check the review results

[1248] 3. How to check emotional information

[1249] "

[1250] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1251] Specific processing flow of the program

[1252] Step 1: Upload your application

[1253] Input: Application form PDF file selected by the user on the device.

[1254] Operation:

[1255] Users log into the system's dedicated portal.

[1256] Click the "Select File" button and select the application form PDF file from your local storage.

[1257] Click the "Upload" button to send the file to the server.

[1258] Output: Application form PDF file uploaded to the server.

[1259] Step 2: Parse the PDF

[1260] Input: Application form PDF file saved on the server.

[1261] Operation:

[1262] The server stores the file in the inbox.

[1263] Open the file using a PDF parsing library (e.g. PyPDF2).

[1264] Each page is read in turn, image data is extracted, and an algorithm is applied to identify the location of the stamp.

[1265] Output: Coordinate information of the imprint.

[1266] Step 3: Extract the stamp image

[1267] Input: Stamp coordinate information and application form PDF file.

[1268] Operation:

[1269] The server uses an image processing library (e.g., OpenCV) to extract the stamped portion from the specified coordinates.

[1270] The extracted image data of the stamped portion is saved as a new image file.

[1271] Output: Extracted stamp image file.

[1272] Step 4: Type determination and character recognition

[1273] Input: Extracted stamp image file.

[1274] Operation:

[1275] The server creates an HTTP request, attaches the stamp image file, and sends it to the AI ​​image recognition API (e.g., GeminiAI).

[1276] The AI ​​image recognition API determines the type of corporate seal (round seal, square seal, seal of responsible person) and converts the character information of the seal impression into text data.

[1277] Output: Type of corporate seal and text information of the seal impression.

[1278] Step 5: Database Registration

[1279] Input: The type of corporate seal determined and text information of the seal impression.

[1280] Operation:

[1281] The server uses a database connection library (e.g. SQLAlchemy) to store the retrieved information in the corresponding table.

[1282] The database contains information such as application form ID, type of corporate seal, and stamp text.

[1283] Output: The type of corporate seal registered in the database and text information of the seal impression.

[1284] Step 6: Automated Review

[1285] Input: The type of corporate seal registered in the database and text information of the seal impression.

[1286] Operation:

[1287] The server checks whether the corporate seal matches the application details based on pre-set review rules.

[1288] The results of the review are registered in the review table and also recorded in a log file.

[1289] Output: Review results.

[1290] Step 7: Notification of review results

[1291] Input: Review results.

[1292] Operation:

[1293] Users can access a dedicated dashboard to check the results of their review.

[1294] When a user logs in, the server displays the corresponding review results on the dashboard.

[1295] If necessary, the server will send an email to notify the user of the results of the review.

[1296] Output: Notification of review results to the user.

[1297] Step 8: Accumulating training data

[1298] Input: AI judgment results and human review results.

[1299] Operation:

[1300] The server periodically aggregates this data.

[1301] It is accumulated as learning data to be used for retraining AI models.

[1302] Output: Training data for retraining the AI ​​model.

[1303] Step 9: Implementing the Emotion Engine

[1304] Input: User's facial image and voice data.

[1305] Operation:

[1306] When a user uploads an application form or checks the review results, the device's camera and microphone are used to capture facial images and audio data.

[1307] The server retrieves this data and sends it to the emotion engine.

[1308] The emotion engine analyzes the data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.).

[1309] The analysis results are stored in a database.

[1310] Output: User emotion recognition results.

[1311] (Application example 2)

[1312] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1313] Conventional application management systems have difficulty not only verifying the authenticity of stamps but also grasping the emotional state of workers. Furthermore, particularly at work sites such as logistics centers, there is a lack of mechanisms for monitoring workers' psychological states in real time and responding accordingly. This can result in a loss of safety and efficiency. Therefore, there is a need for a system that can automate the stamp checks on application forms, recognize workers' emotions, and manage that information.

[1314] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting upload of an application form, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal portion, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal of the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the judgment results and the screening results as learning data, means for recognizing the user's emotions, means for recording the emotion recognition data in a database, and means for the user to confirm emotions during the application form processing process. This not only enables automatic and accurate seal checks on application forms, but also enables real-time recognition and management of the emotional state of workers.

[1315] The "means for accepting application form uploads" is a function for importing digital files of application forms sent by users into the system.

[1316] The "means for analyzing the received application form file and detecting the position of the stamp" is a function for analyzing the contents of the application form obtained as an electronic file and identifying the location of the stamp.

[1317] The "means for extracting image data of the detected stamped portion" is a function for cutting out the relevant portion from the specified stamped position as an image.

[1318] The "means for determining the type of corporate seal from the extracted seal image" is a function that analyzes the cut-out seal portion and identifies what type of corporate seal it is (round seal, square seal, etc.).

[1319] The "means for converting the character information of the seal impression from the seal image into text data" is a function that converts the character information contained in the seal image into text format using optical character recognition (OCR) technology.

[1320] "Means for registering the determined information in a database" is a function for storing analyzed corporate seal information and character information in a database.

[1321] "Means for automatically checking the stamp on the application form based on registered information" refers to a function in which the system automatically checks whether the stamp on the application form is genuine, using information registered in the database.

[1322] "Means for notifying the user of the review results" is a function that reports the results of the stamp check to the user.

[1323] "Means for accumulating judgment results and review results as learning data" refers to a function that saves the judgment and review results made by the system as learning data and uses them to improve the system in the future.

[1324] The "means for recognizing the user's emotions" is a function that analyzes the user's facial image and voice data to determine the user's emotional state.

[1325] The "means for recording emotion recognition data in a database" is a function for storing the recognized emotion data of a user in a database.

[1326] The "means for users to check their emotions during the application process" is a function that allows users to check their own emotional state during the process of checking the review results and processing status of their application.

[1327] This invention relates to a system that automates the process of checking whether an application form has been stamped and combines it with an emotion engine that recognizes the emotions of workers. Specifically, this invention describes an embodiment of a system that uses smart glasses in a logistics center.

[1328] System Overview

[1329] This system consists of a terminal (specifically, smart glasses) and a server. The user (worker) scans the application form using the smart glasses and sends the image data to the server. The server processes the data, checks the stamp and recognizes emotions.

[1330] Hardware and software used

[1331] Hardware:

[1332] Smart glasses (e.g., Google Glass): A device worn by workers to scan application forms.

[1333] Server: A central device that stores data and performs various analytical processes.

[1334] software:

[1335] OpenCV: An image processing library used to detect the stamp position.

[1336] pytesseract: An OCR library used to extract text information from image data.

[1337] EmotionRecognizer: An optional emotion recognition library used to analyze a user's emotional state in real time.

[1338] Flask: A web application framework used to implement system-wide APIs.

[1339] Detailed explanation of functions

[1340] 1. Upload your application form

[1341] The user wears the smart glasses and scans the application form with the camera. The smart glasses then send the captured image data to the server, where it is stored in the server's database.

[1342] 2. Parse the PDF

[1343] The server analyzes the received image data using OpenCV to identify the location of the seal. First, it converts the image to grayscale, then performs threshold processing to highlight the seal area.

[1344] 3. Extraction of imprint image

[1345] The server extracts the identified stamped portion as an image and saves it as a separate image file. This extraction process is also performed using OpenCV.

[1346] 4. Type determination and character recognition

[1347] The server performs OCR processing on the extracted seal image using pytesseract, which converts the type of seal (e.g., a round or square corporate seal) and character information into text data.

[1348] 5. Emotion recognition

[1349] At the same time, the server uses EmotionRecognizer to analyze the user's facial image and voice data in real time to recognize their emotional state, for example, to determine whether the user is in an anxious state.

[1350] Specific examples

[1351] When worker A processes an application for receiving new goods, he scans the application using smart glasses. The server receives this data and simultaneously checks the stamp and recognizes worker A's emotional state. If the stamp is confirmed to be authentic and emotion analysis reveals that worker A is in an anxious state, an alert is sent to the manager.

[1352] Prompt Sentence Examples

[1353] You are a worker in charge of receiving new goods. Using smart glasses, you scan the application form and check the authenticity of the stamp. At the same time, the system also recognizes your emotional state (e.g., anxiety, surprise). The manager sees this and takes the necessary action.

[1354] In this way, the present invention not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, thereby improving work efficiency and safety.

[1355] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1356] Step 1:

[1357] The user scans an application form using smart glasses. The smart glasses capture the image data of the application form and send it to the server. At this time, the camera in the smart glasses captures the entire application form as a high-resolution image file. The input is the actual application form, and the output is the image data on the server.

[1358] Step 2:

[1359] The server analyzes the received image data and detects the location of the seal. The server converts the image to grayscale using OpenCV and performs threshold processing to highlight the presence of the seal. It then performs contour extraction and identifies the location of the seal. The input is the image data of the application form, and the output is the coordinate information of the seal.

[1360] Step 3:

[1361] The server extracts image data of the detected seal area. Based on the identified coordinate information, the seal area is cut out and saved as a separate image file. This image file is used for subsequent OCR processing. The input is image data with the seal coordinate information, and the output is image data of the seal area.

[1362] Step 4:

[1363] The server determines the type of corporate seal from the extracted image of the seal. The server uses pytesseract to perform OCR on this image and determines the type of corporate seal, such as whether it is a round seal or a square seal, from the acquired character information. The server also converts the character information contained in the seal into text data. The input is image data of the seal portion, and the output is the type of seal and text data.

[1364] Step 5:

[1365] The server registers the determined information in a database. The server stores the corporate seal type and text data obtained by OCR in the database and links it to the application form ID. The input is the seal type and text data, and the output is registration in the database.

[1366] Step 6:

[1367] The server automatically checks the seal on the application form based on the registered information. It checks whether the seal is genuine based on the information registered in the database and the screening rules. The results are recorded in the screening table. The input is the database information and screening rules, and the output is the screening results.

[1368] Step 7:

[1369] The server notifies the user of the results of the review. The server notifies the user of the results of the review through the dashboard and prompts corrections if necessary. The input is the review result data, and the output is a notification to the user.

[1370] Step 8:

[1371] The server accumulates the judgment and review results as learning data. This accumulates data to be used for retraining the AI ​​model, contributing to improving the accuracy of the system. The input is the review data, and the output is the accumulated learning data.

[1372] Step 9:

[1373] The server recognizes the user's emotions. The EmotionRecognizer analyzes the user's facial images and voice data collected by the smart glasses' camera and microphone to recognize their emotional state. The input is the user's facial images and voice data, and the output is the user's emotional state data.

[1374] Step 10:

[1375] The server records the emotion recognition data in a database. The recognized emotion data is linked to the application form ID and saved in the database. The input is emotional state data, and the output is a record in the database.

[1376] Step 11:

[1377] The user checks their emotions while processing the application form. They then check the review results and their emotional state on the dashboard and implement the countermeasures suggested by the system. The input is the data displayed on the dashboard, and the output is the user's confirmation and response actions.

[1378] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1379] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1380] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1381] [Fourth embodiment]

[1382] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1383] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1384] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1385] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1386] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1387] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1388] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1389] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1390] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1391] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1392] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1393] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1394] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1395] The present invention relates to a system for automating the check of seals on application forms, and a specific embodiment thereof will be described.

[1396] System Overview

[1397] This system accepts uploaded application forms, automatically determines the type of seal and character information, and uses that information to check the seal. The server handles the main processing, and users use their terminals to upload application form files and check the review results.

[1398] Program processing

[1399] 1. Upload your application form

[1400] The user uploads the application form PDF file to the server using a terminal. The server receives the uploaded file and starts processing, which digitally imports the application form into the system.

[1401] 2. Parse the PDF

[1402] The server analyzes the received PDF application file to detect the location of the stamp. Using a PDF analysis library, it reads the page content of the file and extracts the image data. At this stage, the specific coordinate information of the stamp is obtained.

[1403] 3. Extraction of imprint image

[1404] The server extracts an image of the stamped portion from the detected stamp position and saves it as an independent image file, which is used in subsequent processing.

[1405] 4. Type determination and character recognition

[1406] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The system determines the type of corporate seal (round seal, square seal, or seal of the person in charge) and converts the character information of the seal impression into text data. For example, if the seal is a "round seal," the text may be converted to "CEO of ABC Co., Ltd."

[1407] 5. Database registration

[1408] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[1409] 6. Automated Review

[1410] The server automatically checks the seal on the application form based on the information registered in the database. Based on the screening rules, it verifies whether the seal is genuine and records the results in the screening table or log.

[1411] 7. Notification of the review results

[1412] Users can access a dedicated dashboard to check the results and status of their applications, and the server will notify them of the results and request corrections or confirmations as necessary.

[1413] 8. Accumulation of learning data

[1414] The server stores the AI ​​judgment results and human review results as learning data, which is used to retrain the AI ​​model and contribute to improving the accuracy of the entire system.

[1415] Specific examples

[1416] For example, suppose a company submits an application to sign a contract for a new financial service. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database, and an automatic review is performed. The review results are displayed on the user's dashboard and can be confirmed. The results of the review process are accumulated as future learning data.

[1417] This series of processes will enable automatic and accurate checks of application forms for stamps, greatly improving the efficiency of the review process.

[1418] The processing flow will be explained below.

[1419] Step 1:

[1420] User: Uploads the application form PDF file to the server from the device.

[1421] Operation details: The user selects "Application_Company A.pdf" from a browser or dedicated application and presses the send button. The file is sent to the server via an HTTP request.

[1422] Step 2:

[1423] Server: Receives and stores the uploaded application file.

[1424] Operation details: The server saves the received file in a specific directory, records the file path, and generates an application ID.

[1425] Step 3:

[1426] Server: Analyzes the PDF file and detects the location of the stamp.

[1427] Operation details: The server uses a PDF analysis library (e.g., PyMuPDF or PDFBox) to convert each page of the PDF into image data, then applies a stamp detection algorithm to obtain the coordinate information of the stamp.

[1428] Step 4:

[1429] Server: Extracts image data of the stamped area.

[1430] Operation details: Based on the acquired coordinate information, an image processing library (e.g., OpenCV) is used to extract the seal portion. This is saved as "Seal Impression_CompanyA.png".

[1431] Step 5:

[1432] Server: Sends the stamp image to an image recognition system to determine the type of corporate seal.

[1433] Operation details: The stamp image is sent to an image recognition API (e.g., GeminiAI), and the classification model is used to classify it into one of "round stamp," "square stamp," or "person in charge stamp."

[1434] Step 6:

[1435] Server: Converts the text information of the seal impression from the stamp image using OCR.

[1436] Operation details: Using an OCR library (e.g., Tesseract), extract characters from the stamp image and convert them into text data such as "Seal of the Sales Director of SoftBank Corp."

[1437] Step 7:

[1438] Server: Registers the judgment results and text information in a database.

[1439] Operation details: An INSERT statement is used to add a record containing the application form ID, corporate seal type, and seal text to the database server.

[1440] Step 8:

[1441] Server: Conducts automatic review based on registered information.

[1442] Operation details: According to the screening rules, query the database information to confirm whether the seal is valid. Record the result in the screening table.

[1443] Step 9:

[1444] User: Check the review results.

[1445] How it works: Users access the dashboard to check the review results and status, and manually correct or approve as needed.

[1446] Step 10:

[1447] Server: Accumulates judgment results and human review results as learning data.

[1448] Operation details: The results of the review are saved in a database and accumulated as a dataset for retraining the model. A process to periodically update the AI ​​model is scheduled.

[1449] Example 1

[1450] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1451] Traditionally, checking the stamps on application forms is often done manually, which is not only time-consuming and labor-intensive, but also carries a high risk of human error. Furthermore, relying on paper media hinders progress in digitizing data, making it difficult to improve work efficiency. It is necessary to solve these problems and automate the stamp check process to improve work efficiency and accuracy.

[1452] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1453] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal portion, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the examination results, means for accumulating the determination and examination results as learning data, means for extracting and saving the image file using the coordinate information of the seal, and means for transmitting the seal image to an image recognition system and receiving the recognition results. This enables automatic and accurate checks of application form seals, significantly improving the efficiency and accuracy of operations.

[1454] An "application form" is a document that contains the information necessary to make an application or contract.

[1455] The "means for accepting uploads" refers to a function for receiving an application file sent by a user to a server using a terminal.

[1456] "Means for analyzing the received application file and detecting the position of the stamp" refers to the function of the server using a general-purpose library for analyzing electronic documents to identify the position of the stamp from the received application file.

[1457] "Means for extracting image data of the detected stamped portion" refers to a function for cutting out an image of the stamped portion from the application form file based on the coordinate information of the identified stamp.

[1458] "Means for determining the type of corporate seal from the extracted seal image" refers to a function that uses image recognition technology to identify the type of seal from the extracted seal image.

[1459] "Means for converting character information of the seal impression from the seal image into text data" refers to a function that uses an image recognition system to convert the text contained in the seal image into character data.

[1460] "Means for registering the determined information in a database" refers to the function of storing the type of seal and character information in a database.

[1461] "Means for automatically checking the stamping of application forms based on registered information" refers to the function in which the system automatically applies screening criteria to check the stamping based on the stamping information stored in the database.

[1462] "Means for notifying the user of the review results" refers to a function that displays and notifies the user of the review results on their dashboard.

[1463] "Means for accumulating judgment results and review results as learning data" refers to the function of saving judgment results and actual review results in a database and accumulating them as learning data in order to improve the accuracy of the AI ​​model.

[1464] "Means for extracting and saving an image file using coordinate information of a stamp" refers to a function that extracts the relevant part from a PDF file based on the location information of the identified stamp and saves it as an image file.

[1465] "Means for sending a stamp image to an image recognition system and receiving the recognition results" refers to the function of sending the extracted stamp image to an image recognition system and receiving the type of recognized stamp and text information.

[1466] The present invention relates to a system for automating the seal check of application forms, and a specific embodiment thereof will be described below. The system is implemented based on a process involving mainly a server, a terminal, and a user.

[1467] System Overview

[1468] This system accepts uploaded application forms, automatically determines the type of seal and character information, and checks the seal based on that information. The server handles the main processing, and users use their terminals to upload application form files and check the review results.

[1469] Hardware and software used

[1470] The server uses the following hardware and software:

[1471] Hardware: High-performance server equipment (e.g., a server equipped with an Intel Xeon processor)

[1472] Software: PDF analysis library (e.g., Apache PDFBox), image recognition system (e.g., GeminiAI), database system (e.g., MySQL)

[1473] Program processing

[1474] Upload your application form

[1475] The user logs in from their device and uploads the application form PDF file to the server. The device sends the file to the server, and the server saves the received file.

[1476] Parsing PDFs

[1477] The server uses the Apache PDFBox library to analyze the received application PDF file and detect the position of the stamp from the content of each page. At this stage, the coordinate information of the stamp is obtained.

[1478] Extraction of stamp image

[1479] The server extracts an image of the relevant part from the application form file based on the coordinate information of the stamp it has acquired, and saves the extracted image as a separate image file.

[1480] Type determination and character recognition

[1481] The server sends the extracted image of the seal to an image recognition system such as GeminiAI. The image recognition system determines the type of seal (e.g., round seal, square seal, seal of responsible person) and converts the characters contained in the seal into text data. For example, it may determine that the seal is a "round seal" and the text is "CEO of ABC Co., Ltd."

[1482] Database registration

[1483] The server registers the determined type of corporate seal and character information in a MySQL database, which stores information such as the application form ID, type of corporate seal, and seal text.

[1484] Automated Review

[1485] The server automatically checks the authenticity of the seal based on the information registered in the database and pre-set review rules, and the review results are recorded in a review table and log.

[1486] Notification of review results

[1487] Users access a dedicated dashboard to view the latest audit results provided by the server, and are notified of any corrections or additional confirmations as needed.

[1488] Accumulation of learning data

[1489] The server accumulates the AI's judgment results and the human review results as learning data, which allows the AI ​​model to be retrained and contributes to improving the accuracy of the entire system.

[1490] Specific examples

[1491] For example, suppose a company submits an application to sign a new financial service contract. The user uploads the application PDF file from their computer to the system. The server receives the PDF file and analyzes it to identify the location of the seal. It extracts the image of the seal and uses an image recognition system to identify that it is a "circle seal" and that the text is "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard and can be confirmed. The results obtained during the review process are accumulated as future learning data.

[1492] This will enable automatic and accurate checks of application forms for stamps, greatly improving the efficiency of the review process.

[1493] Prompt Sentence Examples

[1494] Please explain a system that automatically checks the stamps on application forms submitted by companies for financial services contracts. The system includes uploading PDF files, detecting stamp positions, extracting stamp images, determining type and character recognition, registering in a database, automatic screening, notifying the results of the screening, and accumulating learning data.

[1495] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1496] Step 1: Upload your application

[1497] The user logs in from their device, selects the application form PDF file, and clicks the upload button. This action causes the device to send the file to the server, which then saves the received PDF file in a directory.

[1498] Input: Application form PDF file selected by the user.

[1499] Output: Application file saved in the server's save directory.

[1500] Specific operation: The user clicks the "Select File" button to select a PDF file from the device's file system. Then, by pressing the "Upload" button, the device sends the file to the server via an HTTP request. The server receives the file and saves it in the specified directory.

[1501] Step 2: Parse the PDF

[1502] The server uses a PDF analysis library (e.g., Apache PDFBox) to analyze the received PDF file, extract the image data and text data of each page, and obtain the coordinate information to detect the stamp position.

[1503] Input: Saved application form PDF file.

[1504] Output: Data including the coordinate information of the stamp.

[1505] Specific operation: The server uses the PDFBox library to read the PDF file, scans the elements of each page, identifies areas that may be stamped, obtains their coordinate information, and stores it in a list.

[1506] Step 3: Extract the stamp image

[1507] The server extracts an image of the relevant part from the PDF application file based on the coordinate information of the stamp it has acquired, and saves the extracted image as a separate image file.

[1508] Input: Stamp coordinate information and application form PDF file.

[1509] Output: Image file of the stamped area.

[1510] Specific operation: The server reopens the PDF file, extracts the relevant part based on the specified coordinate information, and then saves the extracted image in a new file in PNG format or other format.

[1511] Step 4: Type determination and character recognition

[1512] The server sends the extracted image of the seal to an image recognition system (e.g., GeminiAI), which determines the type of seal and converts the characters contained in the seal into text data.

[1513] Input: Image file of the stamped area.

[1514] Output: Stamp type and text data.

[1515] Specific operation: The server sends the image file to GeminiAI's API and parses the received JSON response. The response includes the type of seal (e.g., circle) and text data (e.g., "CEO of XYZ Co., Ltd.").

[1516] Step 5: Database Registration

[1517] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[1518] Input: Stamp type and text data.

[1519] Output: Stamp information registered in the database.

[1520] Specific operation: The server uses a database system such as MySQL to store the retrieved information in the corresponding columns, and stores the information using the SQL INSERT statement.

[1521] Step 6: Automated Review

[1522] The server automatically checks the legitimacy of the seal based on the information registered in the database and pre-set review rules.

[1523] Input: Stamp information registered in the database.

[1524] Output: Review results.

[1525] Specific operation: The server evaluates the information in the database using a specific business rule engine. It checks the type of stamp and the degree of match of the stamped text to verify its validity. The results of the verification are recorded in a review table or log.

[1526] Step 7: Notification of review results

[1527] Users can access a dedicated dashboard to check the results of the review, and the server will notify the user of the results and request corrections or confirmations as necessary.

[1528] Input: Review results.

[1529] Output: Review results displayed on the dashboard.

[1530] What happens: When a user accesses the dashboard, the server retrieves and displays the latest audit results. If necessary, an appropriate notification is displayed if corrections or additional actions are required.

[1531] Step 8: Accumulating training data

[1532] The server accumulates the AI's judgment results and the human review results as learning data, which allows the AI ​​model to be retrained and contributes to improving the accuracy of the entire system.

[1533] Input: AI judgment results and human review results.

[1534] Output: Updated training data.

[1535] How it works: The server periodically collects the judgment results and actual review results and stores them in a database. This data is then used to retrain the AI ​​model, contributing to improving its accuracy.

[1536] (Application example 1)

[1537] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1538] Checking the accuracy of stamps on numerous application forms and documents at factories and other worksites takes time and effort. When the type of stamp and text information is checked manually, errors and delays are likely to occur, reducing work efficiency and increasing the risk of human error. Furthermore, there is a lack of rapid feedback on review results and the accumulation of learning data, which hinders progress in improving and streamlining the system.

[1539] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1540] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the screening results as learning data, means for an automated machine on-site to scan documents and upload them to the server, means for displaying the screening results in real time, means for notifying workers of feedback, and means for generating prompt sentences using a generative AI model to facilitate screening. This improves the efficiency of document processing in factories, reducing errors and work time.

[1541] - "Uploading an application" means that a User uses a Terminal to send an application in digital form to the Server.

[1542] "Analyzing the application file" refers to the process in which the server analyzes the file received and understands its contents and structure.

[1543] "Detecting the position of the seal" means identifying the specific position of the seal on the application form.

[1544] "Extracting image data" means separating and making independent the image data of the detected stamped portion.

[1545] "Determining the type of corporate seal" means analyzing the image of the seal to determine whether it is a round seal, a square seal, a seal of a person in charge, etc.

[1546] "Converting the character information of the seal impression into text data" means converting the stamped characters into digital text using optical character recognition technology or the like.

[1547] "Registering information in a database" means saving the determined type of seal and character information in a database.

[1548] "Automatically checks the stamp" means that the program automatically verifies the authenticity of the stamp based on the registered information.

[1549] "Notifying the user of the examination results" means informing the user of the results of the seal check.

[1550] "Storing as learning data" means saving the results of the review and judgment in order to improve the accuracy of the review in the future.

[1551] "Automated machine scanning of documents" means that an automated machine installed on-site converts physical documents into digital images.

[1552] "Uploading to server" means sending scanned digital data to the server.

[1553] "Displaying review results in real time" means instantly displaying the review results on the user's dashboard or the like.

[1554] "Notifying workers of feedback" means informing on-site workers of the review results and any necessary corrections.

[1555] "Generating prompts using a generative AI model" means using an artificial intelligence model to automatically generate messages to facilitate the review process.

[1556] This invention relates to a system that automates the process of checking stamps on application forms at factories. This system comprehensively automates the process from scanning documents to determining the type of stamp and text information, and notifying the results of the inspection in real time.

[1557] The system has several main functions: a means for accepting application form uploads, a means for analyzing received application form files and detecting the position of the seal, a means for extracting image data of the detected seal, a means for determining the type of corporate seal from the extracted seal image, a means for converting the character information of the seal impression from the seal image into text data, a means for registering the determined information in a database, a means for automatically checking the seal on the application form based on the registered information, a means for notifying the user of the screening results, a means for accumulating the determination results and the screening results as learning data, a means for on-site automated machines to scan documents and upload them to a server, a means for displaying the screening results in real time, a means for notifying workers of feedback, and a means for generating prompt messages to facilitate the screening process using a generative AI model.

[1558] Specifically, the server uses the following software and hardware: It uses a PDF analysis library (e.g., PyPDF2) to analyze the application file and identify the location of the seal. It uses an image processing library (e.g., PIL and OpenCV) to extract image data of the seal area, and an OCR library (e.g., pytesseract) to convert the character information of the seal impression into text data. The determined information is stored in a database using an HTTP request library (e.g., requests). It uses a generative AI model to generate prompts to facilitate the review process.

[1559] For example, imagine a factory needs to apply for a manufacturing license. An automated machine scans the physical document and uploads it to a server as a PDF file. The server then analyzes the PDF file, locating the stamp, extracting image data, and capturing text information through OCR. This information is stored in a database for automated review. A generative AI model is used to generate prompts for review. The review results are displayed in real time on a dashboard, providing feedback to workers.

[1560] An example of a prompt might be:

[1561] "A new manufacturing permit has been scanned by the robot. Please use the following information to determine the stamp type and text information on the application:

[1562] Stamp image: [Image URL]

[1563] Review rules: [URL for detailed review rules]

[1564] In this way, the present invention significantly improves the efficiency of the stamp checking process on application forms at the factory site, preventing mistakes and delays.

[1565] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1566] Step 1:

[1567] The user uploads the PDF file of the application form to the server using a terminal. The input is the PDF file of the application form, and the output is a digital file stored on the server. This process imports paper documents into the system as digital data.

[1568] Step 2:

[1569] The server analyzes the received PDF file to detect the location of the stamp. Specifically, it uses a PDF analysis library (e.g. PyPDF2). The input is the PDF file, and the output is the coordinate information (position) of the stamp. This process yields data that identifies where the stamp is located.

[1570] Step 3:

[1571] The server extracts image data from the identified stamp location. It uses an image processing library (e.g., PIL and OpenCV) to extract the image of the stamped area. The input is the stamp coordinate information and the PDF file, and the output is the stamped image. This results in the stamped area being obtained as a separate image file.

[1572] Step 4:

[1573] The server analyzes the extracted seal image using an OCR library (e.g., pytesseract) and converts the character information of the seal impression into text data. The input is the seal image, and the output is text data. Specifically, the OCR engine recognizes the characters in the image and converts them into digital text.

[1574] Step 5:

[1575] The server determines the extracted text data and type of seal and registers it in a database. It uses an HTTP request library (e.g., requests). The input is the type of seal and text data, and the output is a database entry. At this step, the determined information is saved in the system.

[1576] Step 6:

[1577] The server automatically performs a stamp check based on the information stored in the database. It evaluates the data according to the review rules and verifies whether the stamp is genuine. The input is the database entry and the output is the review result. Specifically, the server checks the integrity of the data based on the defined review rules.

[1578] Step 7:

[1579] The server notifies the user of the review results. The review results are displayed in real time on the dashboard for the user to check. The input is the review result data, and the output is a notification on the dashboard. In this step, the user can check the results immediately.

[1580] Step 8:

[1581] The server accumulates the judgment and review results as learning data. This allows the generative AI model to provide more accurate results in future reviews. The input is the judgment and review results, and the output is the learning data. This process improves the accuracy of the system.

[1582] Step 9:

[1583] An automated machine on-site scans the document and uploads it to a server. This step is the process of converting paper documents into digital data. The input is the physical document and the output is digital image data.

[1584] Step 10:

[1585] The server uses the generative AI model to generate prompts to facilitate the review. The input is the AI ​​model and the necessary data, and the output is the prompt. Specifically, the prompt is automatically generated and used during the review.

[1586] Example prompt sentence:

[1587] "A new manufacturing permit has been scanned by the robot. Please use the following information to determine the stamp type and text information on the application:

[1588] Stamp image: [Image URL]

[1589] Review rules: [URL for detailed review rules]

[1590] The above steps automate the process of checking stamps on application forms at the factory, thereby improving efficiency.

[1591] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1592] The present invention relates to a system that automates the process of checking whether an application form has been stamped and that also incorporates an emotion engine that recognizes the user's emotions. A specific example of the system will be described below.

[1593] System Overview

[1594] This system accepts uploaded application forms, automatically determines the type of seal and text information, and also has the ability to recognize the user's emotions. The server handles the main processing, and users use their terminals to upload application files and check the review results and their own emotions.

[1595] Program processing

[1596] 1. Upload your application form

[1597] The user uploads the application form PDF file from their device to the server, which receives the uploaded file and starts processing it, thereby digitally importing the application form into the system.

[1598] 2. Parse the PDF

[1599] The server analyzes the received PDF application file to detect the location of the stamp. It uses a PDF analysis library to read the page content of the file and extract the image data. At this stage, the specific coordinate information of the stamp is obtained.

[1600] 3. Extraction of imprint image

[1601] The server extracts an image of the stamped portion from the detected stamp position and saves it as an independent image file, which is used in subsequent processing.

[1602] 4. Type determination and character recognition

[1603] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The system determines the type of corporate seal (round seal, square seal, or seal of the person in charge) and converts the character information of the seal impression into text data. For example, if the seal is a "round seal," the text may be converted to "CEO of ABC Co., Ltd."

[1604] 5. Database registration

[1605] The server registers the determined type of corporate seal and character information in a database, which stores information such as the application form ID, type of corporate seal, and seal text.

[1606] 6. Automated Review

[1607] The server automatically checks the seal on the application form based on the information registered in the database. Based on the screening rules, it verifies whether the seal is genuine and records the results in the screening table or log.

[1608] 7. Notification of the review results

[1609] Users can access a dedicated dashboard to check the results and status of their applications, and the server will notify them of the results and request corrections or confirmations as necessary.

[1610] 8. Accumulation of learning data

[1611] The server stores the AI ​​judgment results and human review results as learning data, which is used to retrain the AI ​​model and contribute to improving the accuracy of the entire system.

[1612] 9. Implementing the Emotion Engine

[1613] The server is equipped with an emotion engine that recognizes the user's emotions. When a user uploads an application form or checks the screening results, the device's camera and microphone are used to capture the user's facial image and voice. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.).

[1614] Specific examples

[1615] For example, suppose a company submits an application to sign a new financial service contract. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[1616] This system not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, providing a more comprehensive user experience.

[1617] The processing flow will be explained below.

[1618] The present invention relates to a system that automates the process of checking whether an application form has been stamped and that also incorporates an emotion engine that recognizes the user's emotions. A specific example of the system will be described below.

[1619] System Overview

[1620] This system accepts uploaded application forms, automatically determines the type of seal and text information, and also has the ability to recognize the user's emotions. The server handles the main processing, and users use their terminals to upload application files and check the review results and their own emotions.

[1621] Program processing

[1622] Step 1:

[1623] User: Uploads the application form PDF file to the server from the device.

[1624] Operation details: The user selects "Application_Company A.pdf" from a browser or dedicated application and presses the send button. The file is sent to the server via an HTTP request.

[1625] Step 2:

[1626] Server: Receives and stores the uploaded application file.

[1627] Operation details: The server saves the received file in a specific directory, records the file path, and generates an application ID.

[1628] Step 3:

[1629] Server: Analyzes the PDF file and detects the location of the stamp.

[1630] Operation details: The server uses a PDF analysis library (e.g., PyMuPDF or PDFBox) to convert each page of the PDF into image data, then applies a stamp detection algorithm to obtain the coordinate information of the stamp.

[1631] Step 4:

[1632] Server: Extracts image data of the stamped area.

[1633] Operation details: Based on the acquired coordinate information, an image processing library (e.g., OpenCV) is used to extract the seal portion. This is saved as "Seal Impression_CompanyA.png".

[1634] Step 5:

[1635] Server: Sends the stamp image to an image recognition system to determine the type of corporate seal.

[1636] Operation details: The stamp image is sent to an image recognition API (e.g., GeminiAI), and the classification model is used to classify it into one of "round stamp," "square stamp," or "person in charge stamp."

[1637] Step 6:

[1638] Server: Converts the text information of the seal impression from the stamp image using OCR.

[1639] Operation details: Using an OCR library (e.g., Tesseract), extract characters from the stamp image and convert them into text data such as "CEO, ABC Co., Ltd."

[1640] Step 7:

[1641] Server: Registers the judgment results and text information in a database.

[1642] Operation details: An INSERT statement is used to add a record containing the application form ID, corporate seal type, and seal text to the database server.

[1643] Step 8:

[1644] Server: Conducts automatic review based on registered information.

[1645] Operation details: According to the screening rules, query the database information to confirm whether the seal is valid. Record the result in the screening table.

[1646] Step 9:

[1647] User: Check the review results.

[1648] How it works: Users access the dashboard to check the review results and status, and manually correct or approve as needed.

[1649] Step 10:

[1650] Server: Accumulates judgment results and human review results as learning data.

[1651] Operation details: The results of the review are saved in a database and accumulated as a dataset for retraining the model. A process to periodically update the AI ​​model is scheduled.

[1652] Step 11:

[1653] Server: Uses an emotion engine to recognize the user's emotions.

[1654] Operation details: The server captures the user's facial image and voice via the device's camera and microphone. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.) in real time.

[1655] Step 12:

[1656] Server: Provides the user with feedback on the recognized emotion information.

[1657] Operation details: The recognized emotional information is visualized and fed back to the user via a dashboard. Users can check their own emotional information and understand their feelings during the review process.

[1658] Specific examples

[1659] For example, suppose a company submits an application to sign a new financial service contract. The user uses a terminal to upload the PDF file of the application to the system. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted, and an image recognition system is used to determine that it is a "circle stamp" and to obtain the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[1660] This system not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, providing a more comprehensive user experience.

[1661] Example 2

[1662] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1663] The traditional application form seal checking process is often done manually, which is inefficient and has a high risk of misjudgment. Furthermore, the lack of a feedback function that takes into account the user's emotions results in a poor user experience. To solve these problems, a system that integrates an automated seal checking function with a function that recognizes the user's emotions is needed.

[1664] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1665] In this invention, the server includes means for accepting application form uploads, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal on the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the judgment results and the screening results as learning data, and means for recognizing the user's emotions. This makes it possible to automate the application form seal check process and provide feedback that takes the user's emotions into consideration.

[1666] "Application" means a document that contains the information necessary to carry out a particular transaction.

[1667] A "PDF analysis library" is a software library for analyzing and manipulating PDF files programmatically.

[1668] "Seal" refers to a seal that is stamped on a document to indicate its legitimacy or approval.

[1669] "Image recognition technology" is a technology that allows computers to analyze images and videos and identify objects and characters.

[1670] A "database" is a system for efficiently managing, searching, and storing large amounts of data.

[1671] "Examination rules" are standards and guidelines for determining whether the contents of the application form and the stamp are genuine.

[1672] "User" means any person or entity that uses this system to upload applications and review results.

[1673] "Terminal" means a computing device through which a user accesses the system.

[1674] The "emotion engine" is a system that analyzes the user's facial images and voice data to recognize emotions.

[1675] The present invention relates to a system that automates the process of checking the seals on application forms and recognizes the emotions of users, and will be described below with reference to specific examples.

[1676] The system mainly consists of three components: a server, a terminal, and a user. The server is responsible for the main processing, and the user uses the terminal to upload the application form and check the review results and emotion recognition feedback.

[1677] 1. Upload your application form

[1678] The user uploads the application form PDF file to the server from their device. First, the user logs in to the system's dedicated portal and opens the application form upload page. Then, they click "Select file" and select the application form PDF from local storage. Then, they can press the "Upload" button to send the file to the server.

[1679] 2. Parse the PDF

[1680] The server receives the uploaded PDF file and begins parsing it. The server first saves the file in an inbox, then calls a PDF parsing library (e.g., PyPDF2) to open the file. The server reads each page in turn, extracts the image data, and applies an algorithm to identify the location of the stamp.

[1681] 3. Extraction of imprint image

[1682] The server extracts an image of the stamp based on the coordinate information of the identified stamp. The server uses an image processing library (e.g., OpenCV) to cut out a rectangular area from the specified coordinates and save it as a new image file. This image file is saved in a temporary folder and used for subsequent processing.

[1683] 4. Type determination and character recognition

[1684] The server sends the extracted seal image to an image recognition system (e.g., GeminiAI). The server creates an HTTP request, attaches the seal image file, and sends it to the AI ​​image recognition API. The API determines the type of corporate seal (round seal, square seal, or seal of responsible person) and converts the character information of the seal impression into text data. For example, the API returns the character information "round seal" and "CEO of ABC Co., Ltd."

[1685] 5. Database registration

[1686] The server registers the determined corporate seal type and character information in a database. The server uses a database connection library (e.g., SQLAlchemy) to store the obtained information in the corresponding table. This table contains information such as the application form ID, corporate seal type, and seal text.

[1687] 6. Automated Review

[1688] The server automatically checks the seal on the application form based on the information registered in the database. The server checks whether the corporate seal matches the application details based on pre-set review rules. The results are registered in the review table and also recorded in a log file.

[1689] 7. Notification of the review results

[1690] The user accesses a dedicated dashboard to check the results of the review. When the user logs in, the server displays the relevant review results on the dashboard. If necessary, the server sends the user an email notifying them of the review results.

[1691] 8. Accumulation of learning data

[1692] The server accumulates the AI's judgment results and the human review results as learning data. The server periodically aggregates this data and uses it to retrain the AI ​​model, thereby improving the accuracy of the entire system.

[1693] 9. Implementing the Emotion Engine

[1694] The server is equipped with an emotion engine that recognizes the user's emotions. When a user uploads an application form or checks the screening results, the device's camera and microphone are used to capture the user's facial image and voice. The server acquires this data and sends it to the emotion engine. The emotion engine analyzes this data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.). The analysis results are stored in a database and used to improve the user experience.

[1695] Specific examples

[1696] For example, consider the case where a company submits an application for a new financial service contract. The user logs in to the system using a terminal and uploads the application. The server receives the PDF file, analyzes it, and identifies the stamp position. The image of the stamp is extracted and sent to an image recognition system, which determines that it is a "circle stamp" and obtains the text information "Director of XYZ Co., Ltd." This information is registered in a database and an automatic review is performed. The review results are displayed on the user's dashboard for confirmation. Furthermore, the user's emotions as they review the review results are recognized in real time, and this information is also recorded in the database.

[1697] Prompt Sentence Examples

[1698] "I would like to upload a new financial services application to the system. What are the next steps?"

[1699] 1. How to upload your application form

[1700] 2. How to check the review results

[1701] 3. How to check emotional information

[1702] "

[1703] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1704] Specific processing flow of the program

[1705] Step 1: Upload your application

[1706] Input: Application form PDF file selected by the user on the device.

[1707] Operation:

[1708] Users log into the system's dedicated portal.

[1709] Click the "Select File" button and select the application form PDF file from your local storage.

[1710] Click the "Upload" button to send the file to the server.

[1711] Output: Application form PDF file uploaded to the server.

[1712] Step 2: Parse the PDF

[1713] Input: Application form PDF file saved on the server.

[1714] Operation:

[1715] The server stores the file in the inbox.

[1716] Open the file using a PDF parsing library (e.g. PyPDF2).

[1717] Each page is read in turn, image data is extracted, and an algorithm is applied to identify the location of the stamp.

[1718] Output: Coordinate information of the imprint.

[1719] Step 3: Extract the stamp image

[1720] Input: Stamp coordinate information and application form PDF file.

[1721] Operation:

[1722] The server uses an image processing library (e.g., OpenCV) to extract the stamped portion from the specified coordinates.

[1723] The extracted image data of the stamped portion is saved as a new image file.

[1724] Output: Extracted stamp image file.

[1725] Step 4: Type determination and character recognition

[1726] Input: Extracted stamp image file.

[1727] Operation:

[1728] The server creates an HTTP request, attaches the stamp image file, and sends it to the AI ​​image recognition API (e.g., GeminiAI).

[1729] The AI ​​image recognition API determines the type of corporate seal (round seal, square seal, seal of responsible person) and converts the character information of the seal impression into text data.

[1730] Output: Type of corporate seal and text information of the seal impression.

[1731] Step 5: Database Registration

[1732] Input: The type of corporate seal determined and text information of the seal impression.

[1733] Operation:

[1734] The server uses a database connection library (e.g. SQLAlchemy) to store the retrieved information in the corresponding table.

[1735] The database contains information such as application form ID, type of corporate seal, and stamp text.

[1736] Output: The type of corporate seal registered in the database and text information of the seal impression.

[1737] Step 6: Automated Review

[1738] Input: The type of corporate seal registered in the database and text information of the seal impression.

[1739] Operation:

[1740] The server checks whether the corporate seal matches the application details based on pre-set review rules.

[1741] The results of the review are registered in the review table and also recorded in a log file.

[1742] Output: Review results.

[1743] Step 7: Notification of review results

[1744] Input: Review results.

[1745] Operation:

[1746] Users can access a dedicated dashboard to check the results of their review.

[1747] When a user logs in, the server displays the corresponding review results on the dashboard.

[1748] If necessary, the server will send an email to notify the user of the results of the review.

[1749] Output: Notification of review results to the user.

[1750] Step 8: Accumulating training data

[1751] Input: AI judgment results and human review results.

[1752] Operation:

[1753] The server periodically aggregates this data.

[1754] It is accumulated as learning data to be used for retraining AI models.

[1755] Output: Training data for retraining the AI ​​model.

[1756] Step 9: Implementing the Emotion Engine

[1757] Input: User's facial image and voice data.

[1758] Operation:

[1759] When a user uploads an application form or checks the review results, the device's camera and microphone are used to capture facial images and audio data.

[1760] The server retrieves this data and sends it to the emotion engine.

[1761] The emotion engine analyzes the data and recognizes the user's emotions (e.g., joy, surprise, anxiety, etc.).

[1762] The analysis results are stored in a database.

[1763] Output: User emotion recognition results.

[1764] (Application example 2)

[1765] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1766] Conventional application management systems have difficulty not only verifying the authenticity of stamps but also grasping the emotional state of workers. Furthermore, particularly at work sites such as logistics centers, there is a lack of mechanisms for monitoring workers' psychological states in real time and responding accordingly. This can result in a loss of safety and efficiency. Therefore, there is a need for a system that can automate the stamp checks on application forms, recognize workers' emotions, and manage that information.

[1767] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting upload of an application form, means for analyzing the received application form file to detect the position of the seal, means for extracting image data of the detected seal portion, means for determining the type of corporate seal from the extracted seal image, means for converting character information of the seal impression from the seal image into text data, means for registering the determined information in a database, means for automatically checking the seal of the application form based on the registered information, means for notifying the user of the screening results, means for accumulating the judgment results and the screening results as learning data, means for recognizing the user's emotions, means for recording the emotion recognition data in a database, and means for the user to confirm emotions during the application form processing process. This not only enables automatic and accurate seal checks on application forms, but also enables real-time recognition and management of the emotional state of workers.

[1768] The "means for accepting application form uploads" is a function for importing digital files of application forms sent by users into the system.

[1769] The "means for analyzing the received application form file and detecting the position of the stamp" is a function for analyzing the contents of the application form obtained as an electronic file and identifying the location of the stamp.

[1770] The "means for extracting image data of the detected stamped portion" is a function for cutting out the relevant portion from the specified stamped position as an image.

[1771] The "means for determining the type of corporate seal from the extracted seal image" is a function that analyzes the cut-out seal portion and identifies what type of corporate seal it is (round seal, square seal, etc.).

[1772] The "means for converting the character information of the seal impression from the seal image into text data" is a function that converts the character information contained in the seal image into text format using optical character recognition (OCR) technology.

[1773] "Means for registering the determined information in a database" is a function for storing analyzed corporate seal information and character information in a database.

[1774] "Means for automatically checking the stamp on the application form based on registered information" refers to a function in which the system automatically checks whether the stamp on the application form is genuine, using information registered in the database.

[1775] "Means for notifying the user of the review results" is a function that reports the results of the stamp check to the user.

[1776] "Means for accumulating judgment results and review results as learning data" refers to a function that saves the judgment and review results made by the system as learning data and uses them to improve the system in the future.

[1777] The "means for recognizing the user's emotions" is a function that analyzes the user's facial image and voice data to determine the user's emotional state.

[1778] The "means for recording emotion recognition data in a database" is a function for storing the recognized emotion data of a user in a database.

[1779] The "means for users to check their emotions during the application process" is a function that allows users to check their own emotional state during the process of checking the review results and processing status of their application.

[1780] This invention relates to a system that automates the process of checking whether an application form has been stamped and combines it with an emotion engine that recognizes the emotions of workers. Specifically, this invention describes an embodiment of a system that uses smart glasses in a logistics center.

[1781] System Overview

[1782] This system consists of a terminal (specifically, smart glasses) and a server. The user (worker) scans the application form using the smart glasses and sends the image data to the server. The server processes the data, checks the stamp and recognizes emotions.

[1783] Hardware and software used

[1784] Hardware:

[1785] Smart glasses (e.g., Google Glass): A device worn by workers to scan application forms.

[1786] Server: A central device that stores data and performs various analytical processes.

[1787] software:

[1788] OpenCV: An image processing library used to detect the stamp position.

[1789] pytesseract: An OCR library used to extract text information from image data.

[1790] EmotionRecognizer: An optional emotion recognition library used to analyze a user's emotional state in real time.

[1791] Flask: A web application framework used to implement system-wide APIs.

[1792] Detailed explanation of functions

[1793] 1. Upload your application form

[1794] The user wears the smart glasses and scans the application form with the camera. The smart glasses then send the captured image data to the server, where it is stored in the server's database.

[1795] 2. Parse the PDF

[1796] The server analyzes the received image data using OpenCV to identify the location of the seal. First, it converts the image to grayscale, then performs threshold processing to highlight the seal area.

[1797] 3. Extraction of imprint image

[1798] The server extracts the identified stamped portion as an image and saves it as a separate image file. This extraction process is also performed using OpenCV.

[1799] 4. Type determination and character recognition

[1800] The server performs OCR processing on the extracted seal image using pytesseract, which converts the type of seal (e.g., a round or square corporate seal) and character information into text data.

[1801] 5. Emotion recognition

[1802] At the same time, the server uses EmotionRecognizer to analyze the user's facial image and voice data in real time to recognize their emotional state, for example, to determine whether the user is in an anxious state.

[1803] Specific examples

[1804] When worker A processes an application for receiving new goods, he scans the application using smart glasses. The server receives this data and simultaneously checks the stamp and recognizes worker A's emotional state. If the stamp is confirmed to be authentic and emotion analysis reveals that worker A is in an anxious state, an alert is sent to the manager.

[1805] Prompt Sentence Examples

[1806] You are a worker in charge of receiving new goods. Using smart glasses, you scan the application form and check the authenticity of the stamp. At the same time, the system also recognizes your emotional state (e.g., anxiety, surprise). The manager sees this and takes the necessary action.

[1807] In this way, the present invention not only automatically and accurately checks the stamping of application forms, but also collects user emotional information, thereby improving work efficiency and safety.

[1808] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1809] Step 1:

[1810] The user scans an application form using smart glasses. The smart glasses capture the image data of the application form and send it to the server. At this time, the camera in the smart glasses captures the entire application form as a high-resolution image file. The input is the actual application form, and the output is the image data on the server.

[1811] Step 2:

[1812] The server analyzes the received image data and detects the location of the seal. The server converts the image to grayscale using OpenCV and performs threshold processing to highlight the presence of the seal. It then performs contour extraction and identifies the location of the seal. The input is the image data of the application form, and the output is the coordinate information of the seal.

[1813] Step 3:

[1814] The server extracts image data of the detected seal area. Based on the identified coordinate information, the seal area is cut out and saved as a separate image file. This image file is used for subsequent OCR processing. The input is image data with the seal coordinate information, and the output is image data of the seal area.

[1815] Step 4:

[1816] The server determines the type of corporate seal from the extracted image of the seal. The server uses pytesseract to perform OCR on this image and determines the type of corporate seal, such as whether it is a round seal or a square seal, from the acquired character information. The server also converts the character information contained in the seal into text data. The input is image data of the seal portion, and the output is the type of seal and text data.

[1817] Step 5:

[1818] The server registers the determined information in a database. The server stores the corporate seal type and text data obtained by OCR in the database and links it to the application form ID. The input is the seal type and text data, and the output is registration in the database.

[1819] Step 6:

[1820] The server automatically checks the seal on the application form based on the registered information. It checks whether the seal is genuine based on the information registered in the database and the screening rules. The results are recorded in the screening table. The input is the database information and screening rules, and the output is the screening results.

[1821] Step 7:

[1822] The server notifies the user of the results of the review. The server notifies the user of the results of the review through the dashboard and prompts corrections if necessary. The input is the review result data, and the output is a notification to the user.

[1823] Step 8:

[1824] The server accumulates the judgment and review results as learning data. This accumulates data to be used for retraining the AI ​​model, contributing to improving the accuracy of the system. The input is the review data, and the output is the accumulated learning data.

[1825] Step 9:

[1826] The server recognizes the user's emotions. The EmotionRecognizer analyzes the user's facial images and voice data collected by the smart glasses' camera and microphone to recognize their emotional state. The input is the user's facial images and voice data, and the output is the user's emotional state data. 【182...

Claims

1. a means for accepting application forms for upload; A means for analyzing the received application form file and detecting the position of the seal; A means for extracting image data of the detected stamped portion; A means for determining the type of corporate seal from the extracted seal image; A means for converting character information of the seal impression from the seal image into text data; A means for registering the determined information in a database; A means to automatically check the stamping of application forms based on the registered information, a means for notifying the user of the review result; A system that includes a means for accumulating judgment and review results as learning data.

2. 2. The system according to claim 1, wherein the means for analyzing the received application file and detecting the position of the seal uses a PDF analysis library.

3. 2. The system according to claim 1, wherein the means for determining the type of corporate seal from the extracted seal image uses image recognition technology.

Images