system

Abstract

Provide a system. A method for extracting text from image data, means for translating the extracted text into a target language; A means of fine-tuning based on manga-specific context, A means for inserting translated text into images; A means for completing the background accompanying the insertion; A means for displaying the translation results and allowing the user to make corrections; means for storing and providing the final modified image to the user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When translating manga into multiple languages, traditional manual translation and image editing processes have been extremely time-consuming and labor-intensive. It is also difficult to provide high-quality translations that take into account the unique context and style of manga. Furthermore, it is also difficult to maintain accuracy and consistency with the background when inserting translated text into images. By solving these challenges, we aim to achieve efficient, high-quality multilingual translation. [Means for solving the problem]

[0005] The present invention provides a system that includes means for extracting text from image data of manga, means for translating the extracted text into a target language, means for fine-tuning based on the context specific to manga, means for inserting the translated text into an image, means for completing the background accompanying the insertion, means for displaying the translation result and allowing the user to make corrections, and means for saving the final image after correction and providing it to the user. This system makes it possible to improve the efficiency of manga multilingual translation work and obtain high-quality translation results.

[0006] "Image data" refers to files of visual information stored in digital format, including, for example, comic book pages or illustrations.

[0007] "Means for extracting text" refers to a technique or device that analyzes the textual information contained within image data and converts it into digital text format.

[0008] "Target Language" refers to the language into which the original text is translated.

[0009] "Translation means" refers to a technique or device that converts text expressed in one language into another language.

[0010] "Fine-tuning means" refers to a technology or device that optimizes a translation model based on a specific genre or context to improve its accuracy.

[0011] "Means for inserting translated text into an image" refers to a technique or device that incorporates translated text into the original image and places it in the appropriate position.

[0012] "Means for background interpolation" refers to a technique or device that corrects background inconsistencies that occur when inserting text, maintaining a natural appearance.

[0013] "Means for displaying translation results and allowing the user to make corrections" refers to an interface or function that presents the translated content and edited images to the user and allows the user to make corrections or check them.

[0014] The "means for saving the final corrected image and providing it to the user" refers to a technology or device that saves the final image data that has been checked and corrected by the user and allows the user to download it. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a system for efficiently translating manga into multiple languages, which has the ability to extract text from image data, translate it, generate an image containing the translated text, and allow the user to make any necessary corrections.

[0037] 1. Upload your data

[0038] Users upload manga manuscript data (image files or PDFs) through the system's front end, which then sends the manga pages to the server.

[0039] Example: User clicks upload button in browser, selects file and submits.

[0040] 2. Image Preprocessing

[0041] The server preprocesses the received manga data. First, it checks the image resolution and resizes it if it is inappropriate. If it is a PDF file, it splits each page into a separate image file.

[0042] Example: The server checks the image resolution and resizes it to the optimal resolution if necessary.

[0043] 3. Text Extraction

[0044] The server uses OCR (Optical Character Recognition) technology to extract the text from each image file, which is then stored in a database.

[0045] Example: The server runs an OCR engine to identify the location and content of text balloons from an image.

[0046] 4. Initial Translation

[0047] The server uses generative AI to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[0048] Example: The server translates the extracted text "Hello, how are you?" into English, producing "Hello, how are you?"

[0049] 5. Fine Tuning

[0050] The server uses fine-tuned models tailored to specific manga genres and styles to refine the initial translation results, which are also stored in the database.

[0051] Example: The server inputs the initial translation "This is the hero's sword!" into a fine-tuning model, which generates the more contextually appropriate "This is the courage sword!"

[0052] 6. Image Recognition and Editing

[0053] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[0054] Example: The server replaces the original text "Run quickly!" with "Run quickly!" and adjusts the position and size of the balloon.

[0055] 7. Background Completion

[0056] The server compensates for inconsistencies in the background caused by inserting translated text, and automatically corrects the background so that it does not look unnatural when the text balloon is changed.

[0057] Example: The server regenerates the background of the balloon and inserts the translated text.

[0058] 8. User confirmation and correction

[0059] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[0060] Example: A user corrects the displayed translation "Run quickly!" to "Hurry up!"

[0061] 9. Saving and downloading final data

[0062] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[0063] Example: User completes verification and downloads the corrected image.

[0064] This system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] The user uploads the manga manuscript data (image files or PDF files) through the system's front end. When the user clicks the file selection button in the browser, selects a file from the local disk, and presses the upload button, the file is sent to the server.

[0068] Step 2:

[0069] The server preprocesses the manga data received by checking the image resolution and format, resizing and converting the image if necessary, and splitting each page into an individual image file if it is a PDF file.

[0070] Step 3:

[0071] The server uses OCR (Optical Character Recognition) technology to extract text from the preprocessed image files, specifically by identifying the location and content of text balloons on each page, and stores the extracted text in a database.

[0072] Step 4:

[0073] The server uses the generative AI to initially translate the extracted text into the target language, and the generated initial translation results are also stored in the database.

[0074] Step 5:

[0075] The server refines the initial translation using a fine-tuned model that is specialized for a specific manga genre and style. Specifically, the model takes the initial translation as input and generates a translation that is appropriate for the context and style, which is then stored in a database.

[0076] Step 6:

[0077] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted to insert the translated text.

[0078] Step 7:

[0079] The server automatically detects and corrects any unnatural background caused by changes to the text balloon, ensuring that the background remains natural.

[0080] Step 8:

[0081] The device displays the edited image on the user interface, and the user is provided with a UI to review the translation results and image edits, and manually make corrections if necessary.

[0082] Step 9:

[0083] After the user has completed the confirmation and correction, they issue an instruction to save the final image data, and the server saves the final image and generates a download link.

[0084] Step 10:

[0085] The terminal displays the generated download link to the user, allowing the user to download the final translation data with the corrections reflected.

[0086] This series of processes allows users to efficiently translate manga into multiple languages ​​with high quality.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] Previous multilingual manga translation systems faced challenges in the entire process, from text extraction to translation and image insertion, including translation quality, background completion, and the inability for users to easily make corrections. Furthermore, the lack of fine-tuning specialized for specific categories or styles made it difficult to translate appropriately for the context. As a result, problems arose, such as low translation quality and a heavy burden on users to make corrections.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning specialized for a particular category or style, means for inserting the translated text into an image, means for completing the background accompanying the text insertion, means for displaying the translation result and allowing the user to make corrections, and means for saving the final image after corrections and providing it to the user. This enables high-quality multilingual translation and reduces the burden of corrections on the user.

[0092] "Image data" means digital data that stores visual information, including image files in JPEG and PNG formats and PDF files.

[0093] "Text extraction" is the process of recognizing characters and sentences from image data and obtaining their content as digital text.

[0094] The "target language" is the language into which you are translating, and the language into which you want to convert and display the original text.

[0095] "Fine tuning" is the process of making adjustments to improve accuracy, focusing on a particular category or style.

[0096] "Translated text" means text that has been converted from a source language into a target language.

[0097] "Image insertion" is the process of incorporating translated text into the original image data, including adjusting the position and size of the text.

[0098] "Background completion" is the process of correcting background inconsistencies that occur when inserting text, making it look natural.

[0099] "Translation result" is the text after translation and fine-tuning.

[0100] "Correction" refers to any change a user makes to the translation or image editing results provided by the system.

[0101] "Final image" refers to the final image data provided to the user after translation and corrections are complete.

[0102] The present invention relates to a system for efficiently translating manga into multiple languages, and the system has many functions for text extraction, translation, image generation, and user correction. Specific embodiments for implementing the invention are described below.

[0103] Hardware and Software Used

[0104] server

[0105] The server is implemented in a hardware environment with high-performance computing capabilities (e.g., a virtual server provided by a cloud provider). The server includes the following software components:

[0106] OCR engine: For optical character recognition, it uses, for example, Google® Cloud Vision API.

[0107] Generative AI model: For translation into the target language and fine-tuning, use e.g. OpenAI® GPT-4®.

[0108] Image processing libraries: For example, OpenCV or PyMuPDF are used to check image resolution and resize, split PDFs, analyze text balloon positions, etc.

[0109] Terminal

[0110] A terminal is a device (such as a personal computer or tablet) that allows a user to interact with the system. The terminal implements the system's front end and provides the interface that the user operates.

[0111] User

[0112] The user is the end user who operates the system, and is responsible for providing the manga manuscript data and checking and correcting the results of the translation and corrections.

[0113] Data processing and calculation

[0114] 1. Text Extraction

[0115] The server uses an OCR engine to extract text from the image data, for example, using the Google Cloud Vision API, to identify text balloons in the image and store their locations and contents in a database.

[0116] 2. Initial Translation

[0117] The server translates the extracted text into the target language using a generative AI model (e.g., OpenAI GPT-4). This translation result is again stored in the database. An example of a prompt sentence is a prompt that translates "Hello, how are you?" into "Hello, how are you?"

[0118] 3. Fine Tuning

[0119] The server uses fine-tuning models that are specialized for specific manga categories and styles to refine the initial translation results, for example, refining the hero's line "This is the hero's sword!" to "This is the courage sword!"

[0120] 4. Image Recognition and Editing

[0121] The server uses image recognition technology to insert the translated text into the image appropriately, specifically by adjusting the position and size of the text balloon using OpenCV.

[0122] 5. Background Completion

[0123] The server uses a technique to regenerate the background portion to compensate for the inconsistency of the background caused by the text insertion.

[0124] 6. User confirmation and correction

[0125] The terminal displays the translation results and the status of the image editing to the user. The user can then modify the translation and image editing status as needed, and the modifications are immediately sent to the server.

[0126] Specific examples

[0127] For example, a user may upload a comic page and provide an image containing the text "Hello, how are you?" The server receives this image data, uses an OCR engine to extract the text from the text balloon, and stores the results in a database.

[0128] The server then uses a generative AI model to translate the extracted text into English, for example, "Hello, how are you?" is translated to "Hello, how are you?", and then further refines the translation results using a fine-tuning model suited to the manga category.

[0129] The server places this translated text in the appropriate text balloon and fills in any inconsistencies in the background. Finally, the device displays the edited results to the user, who can then modify the translation as needed. The modified results are then sent to the server and stored in a database.

[0130] This system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections.

[0131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0132] Step 1:

[0133] Uploading data

[0134] Users upload manga manuscript data through the system's front end, providing image files (JPEG, PNG, etc.) or PDF files as input.

[0135] The server processes the received file using an HTTP POST request, and as output, the received manuscript data is saved on the server.

[0136] Step 2:

[0137] Image preprocessing

[0138] The server checks the image resolution of the received manga data and resizes it if it is inappropriate, and if it is a PDF file, splits each page into a separate image file.

[0139] It uses saved original data (image or PDF) as input, and generates an image file resized to the appropriate resolution or split into individual image files as output.

[0140] Specific operations include image resizing using the OpenCV library and splitting the PDF using PyMuPDF.

[0141] Step 3:

[0142] Text Extraction

[0143] The server uses an OCR engine (e.g., Google Cloud Vision API) to extract text from each image file, which is then stored in a database.

[0144] As input, it uses preprocessed image files, and as output, it stores extracted text information and its location data in a database.

[0145] Specific operations include sending an image file to an OCR engine and storing the returned text information in a database.

[0146] Step 4:

[0147] Initial translation

[0148] The server uses a generative AI model (e.g., OpenAI GPT-4) to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[0149] As input, it uses extracted text stored in a database, and as output, the translation results into the target language are stored in a database.

[0150] Specifically, the generative AI model is provided with the extracted text and uses the prompt sentence to obtain the translation result, for example, translating "Hello, how are you?" into "Hello, how are you?"

[0151] Step 5:

[0152] Fine Tuning

[0153] The server uses fine-tuning models specialized for specific manga categories and styles to refine the initial translation results, which are also stored in the database.

[0154] As input, it uses the initial translation results, and as output, the fine-tuned translation results are stored in a database.

[0155] Specifically, it provides the initial translation text to the fine-tuning model, which then generates a more natural and context-appropriate translation result, for example, refining "This is the hero's sword!" to "This is the courage sword!".

[0156] Step 6:

[0157] Image Recognition and Editing

[0158] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[0159] It takes the fine-tuned translation results and the original image file as input, and generates a new image file with the translated text inserted as output.

[0160] Specific actions include using OpenCV to reparse the position and size of the text balloon and insert text, e.g., replacing "Run quickly!" with "Run quickly!".

[0161] Step 7:

[0162] Background Complement

[0163] The server compensates for inconsistencies in the background caused by inserting translated text, and automatically corrects the background so that it does not look unnatural when the text balloon is changed.

[0164] As input, it takes the image file after the text has been inserted, and as output, it generates a new image file with the background filled in with inconsistencies.

[0165] Specifically, it uses generative AI technology and image correction algorithms to recreate the background around the balloon in a natural shape.

[0166] Step 8:

[0167] User confirmation and correction

[0168] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[0169] As input, it takes an image file with the translated text inserted, and as output, it generates the final image data with the user's modifications reflected.

[0170] Specifically, the terminal provides an interface for editing, creating an environment where users can edit directly. The edited data is immediately reflected on the server.

[0171] Step 9:

[0172] Save and download the final data

[0173] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[0174] It takes the image file with the user's modifications as input, and saves the final image data and generates a download link as output.

[0175] Specifically, the modified image file is saved in the database and a URL that can be accessed by the user is created.

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] Multilingual distribution of manga requires efficient translation into different languages. Conventional methods require manual translation and editing, which is extremely time-consuming and laborious. Furthermore, there are often inconsistencies in the background of images when translating text, and the environment for easy user corrections is often inadequate. This makes it difficult for users to smoothly enjoy manga in multiple languages.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning based on context specific to manga, means for inserting the translated text into the image, means for completing the background associated with the insertion, means for displaying the translation result and allowing the user to make corrections, means for saving the final image after corrections and providing it to the user, and means for providing a terminal application for viewing manga in multiple languages. This allows manga text to be translated efficiently into multiple languages, enabling users to easily make corrections and view the text.

[0181] "Image data" is an electronic representation of a comic page or frame.

[0182] "Text" refers to character information contained in image data.

[0183] "Target language" refers to the language into which the text is translated.

[0184] "Fine tuning" refers to adjusting the model to improve the accuracy of machine translation based on the specific context of manga.

[0185] "Background completion" is the process of correcting background inconsistencies that occur when inserting translated text into an image.

[0186] "User" refers to a person who uses the system to translate and view manga in multiple languages.

[0187] "Device application" refers to software that runs on mobile devices such as smartphones and tablets.

[0188] "Extracted text" refers to character information obtained from image data using OCR technology.

[0189] A "text balloon" refers to a speech bubble used to write lines or narration within a manga.

[0190] The "final image" is the final image data after the text has been translated and completed and the user's corrections have been reflected.

[0191] The present invention relates to a system for efficiently translating manga into multiple languages. The system extracts text from image data, translates it, generates an image containing the translated text, and allows the user to make any necessary corrections.

[0192] First, the user uploads the manga image data (image file or PDF) via their device. This image data is then sent to the server. The server preprocesses the received manga data, inspects the image resolution, and resizes it if it is inappropriate. In the case of a PDF file, the server also splits each page into a separate image file.

[0193] The server then uses OCR technology to extract text from each image file, identify the location and content of the text balloons, and store the extracted text in a database. Specifically, this can be done using an OCR engine such as Pytesseract.

[0194] The server then uses a generative AI model to translate the extracted text into the target language. This initial translation result is also stored in the database. For example, the extracted text "Hello, how are you?" can be translated into English to produce "Hello, how are you?"

[0195] Additionally, the server uses fine-tuned models tailored to specific manga genres and styles to refine the initial translation results, resulting in more context-appropriate translations. For example, it takes the initial translation "This is the hero's sword!" and generates the more context-appropriate "This is the courage sword!".

[0196] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image. The size and position of the text balloon are also automatically adjusted. This compensates for inconsistencies in the background and eliminates the awkwardness that comes with inserting the translated text. This allows the original text "Run quickly!" to be replaced with "Run quickly!" and the position and size of the balloon to be adjusted.

[0197] The user can then check the translation results displayed on the device and make corrections as necessary. These corrections are also sent to the server and reflected in the final image data. For example, the user can change the displayed translation "Run quickly!" to "Hurry up!"

[0198] Finally, the server saves the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download. The user can click on this link to download the high-quality multilingual translated manga.

[0199] The implementation of this system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections. An example of a translation and prompt using a generative AI model is "Please translate the following Japanese text into English. Text: 'Hello, how are you?'"

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1:

[0202] Users upload manga image data (image files or PDFs) to the system from their devices. Specifically, they click the upload button in their browser or application, select the file, and send it to the server.

[0203] Input: Comic image data file (PNG, JPEG, PDF, etc.) selected by the user.

[0204] Output: Image data of the manga sent to the server.

[0205] Step 2:

[0206] The server preprocesses the image data it receives, checking the image resolution and resizing it if it is inappropriate, and, in the case of PDF files, splitting each page into a separate image file.

[0207] Input: Image data received by the server.

[0208] Output: Image data resized to optimal resolution and PDF pages split into individual image files.

[0209] What it does: Uses libraries such as OpenCV to check image resolution and resize.

[0210] Step 3:

[0211] The server uses OCR technology to extract text from each image file, and the extracted text and its location are stored in a database.

[0212] Input: The resized image data.

[0213] Output: Extracted text data and text location information.

[0214] What it does: Uses an OCR engine such as Pytesseract to extract the location and content of text balloons from an image file.

[0215] Step 4:

[0216] The server uses a generative AI model to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[0217] Input: Extracted text data.

[0218] Output: The translated text data.

[0219] What it does: Uses a translation engine (e.g., Google Translate API) to translate text data into the target language.

[0220] Step 5:

[0221] The server uses fine-tuned models that are specialized for specific manga genres and styles to refine the initial translation results, which are then stored in a database.

[0222] Input: Text data of the initial translation result.

[0223] Output: Fine-tuned translated text data.

[0224] How it works: Generates context-appropriate translations using a generative AI model specialized for the manga genre.

[0225] Step 6:

[0226] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[0227] Input: Fine-tuned translation text data, image data.

[0228] Output: Image data with inserted text.

[0229] What it does: Uses OpenCV and PIL libraries to insert text into an image and adjust the position and size of the text balloon.

[0230] Step 7:

[0231] The server compensates for inconsistencies in the background caused by the insertion of translated text.

[0232] Input: Image data with inserted text.

[0233] Output: Image data with background interpolated.

[0234] How it works: It uses image editing algorithms to naturally complement the background of the text insert.

[0235] Step 8:

[0236] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[0237] Input: The translated and completed image data.

[0238] Output: Image data with user modifications.

[0239] What it does: Displays the image through the device's interface and allows the user to make corrections. The user enters the corrections and submits them.

[0240] Step 9:

[0241] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[0242] Input: Image data with user modifications.

[0243] Output: Final image data provided via a download link.

[0244] Behavior: Saves the image data and sends a notification to the user providing a download link.

[0245] This allows users to quickly and efficiently translate manga into multiple languages. An example of a translation and prompt using a generative AI model is: "Please translate the following Japanese text into English. Text: 'Hello, how are you?'"

[0246] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0247] This invention relates to a system that combines an emotion engine with an efficient system for translating manga into multiple languages, thereby reflecting user emotional feedback and improving the accuracy of translation and editing. This system extracts text from image data, translates it into the target language, and provides an environment where users can check and correct the results, while analyzing user emotions in real time, enabling more user-friendly operation.

[0248] 1. Upload your data

[0249] The user uploads the manga manuscript data (image file or PDF file) from the system's front end. The file is selected and sent to the server.

[0250] Example: A user clicks the file chooser button in their browser to select and upload a file from their local disk.

[0251] 2. Image Preprocessing

[0252] The server pre-processes the uploaded manga data, inspecting the image resolution and format and resizing or formatting it as needed. In the case of PDF files, each page is split into a separate image file.

[0253] Example: The server checks the image resolution and resizes it if it is inappropriate.

[0254] 3. Text Extraction

[0255] The server uses OCR (Optical Character Recognition) technology to extract text from the preprocessed image files, which is then stored in a database.

[0256] Example: The server identifies the location and content of text balloons on each page and stores them in a database.

[0257] 4. Initial Translation

[0258] The server uses generative AI to perform an initial translation of the extracted text into the target language, and the translation results are stored in a database.

[0259] Example: The server translates the text "Hello, how are you?" to "Hello, how are you?"

[0260] 5. Fine Tuning

[0261] The server uses fine-tuning models specific to a particular genre or style to refine the initial translation results, which are then stored in a database.

[0262] Example: The server refines "This is the hero's sword!" to "This is the courage sword!"

[0263] 6. Image Recognition and Editing

[0264] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted to insert the text.

[0265] Example: The server properly places "Run quickly!" into "Run quickly!"

[0266] 7. Background Completion

[0267] The server will interpolate the background as needed to correct any inconsistencies in the background caused by changes to the text balloon, thus maintaining a natural look.

[0268] Example: The server fills in missing parts of the background and recreates a natural background.

[0269] 8. User Emotion Recognition and Feedback

[0270] The device displays the edited image to the user and uses an emotion engine to analyze the user's emotions in real time based on their facial expressions and voice. Based on the analysis results, the device collects user feedback on the translation results and the edited image.

[0271] Example: The device uses the user's camera and microphone to analyze their reactions and use an emotion engine to collect positive or negative feedback.

[0272] 9. Applying Feedback

[0273] The server has the ability to automatically correct the translated text and edits based on the collected user emotional feedback. If the feedback is mostly positive, the server proceeds to the next step.

[0274] Example: The server responds to negative feedback by correcting the translation and redisplaying it.

[0275] 10. Modifying and Saving

[0276] The user makes final confirmations and corrections while taking into account the emotional feedback. After corrections are complete, the server stores the final translation and editing results in a database and provides them to the user.

[0277] Example: The user completes the review, and if the result is satisfactory, completes the corrections and clicks the save button.

[0278] 11. Data Provision

[0279] The terminal displays the generated download link to the user, allowing the user to download the final data.

[0280] Example: The device displays a download link and the user downloads the modified results.

[0281] This system enables high-quality multilingual translation of manga that incorporates user emotional feedback, improving the user experience.

[0282] The processing flow will be explained below.

[0283] Step 1:

[0284] The user uploads the manga manuscript data (image file or PDF file) from the system's front end. The user clicks the file selection button in the browser, selects the file from the local disk, and presses the upload button, which sends the file to the server.

[0285] Step 2:

[0286] The server preprocesses the manga data received. Specifically, it checks the image resolution and format, resizes and converts the format as needed, and splits each page into a separate image file if it is a PDF file.

[0287] Step 3:

[0288] The server extracts text from the preprocessed image files using OCR (Optical Character Recognition) technology, identifies the location and content of text balloons, and stores the extracted text in a database.

[0289] Step 4:

[0290] The server uses generative AI to perform an initial translation of the extracted text into the target language, and the translation results are stored in a database.

[0291] Step 5:

[0292] The server uses fine-tuning models specialized for specific manga genres and styles to improve the initial translation results, which are then stored in a database.

[0293] Step 6:

[0294] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted and the text is inserted.

[0295] Step 7:

[0296] The server performs background interpolation, correcting inconsistencies in the background caused by text balloon changes to maintain a natural look.

[0297] Step 8:

[0298] The device displays the edited image to the user, and uses an emotion engine to analyze the user's emotions in real time from their facial expressions and voice. Based on the analysis results, the device collects user feedback on the translation results and the edited image.

[0299] Step 9:

[0300] The server automatically corrects the translated text and edits based on the user's emotional feedback collected. If the feedback is positive, the system proceeds to the next step, but if the feedback is negative, the system makes appropriate corrections.

[0301] Step 10:

[0302] The user then makes final confirmations and corrections while taking into account the emotional feedback. After corrections are complete, the server saves the final translation and edited results in a database.

[0303] Step 11:

[0304] The terminal displays the generated download link to the user, allowing the user to download the final data. The user uses this link to obtain the final translation result.

[0305] Example 2

[0306] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0307] Conventional systems for multilingual translation have issues with translation accuracy and the complexity of editing operations. They also face the problem of not being able to properly reflect user emotions and feedback, resulting in a poor user experience. The present invention aims to solve these issues by providing an efficient system that provides high-quality translation results and reflects user emotional feedback.

[0308] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data uploading means, a means for preprocessing uploaded image data, a means for extracting text from image data using OCR technology, a means for translating the extracted text into a target language using a generative AI model, a means for fine-tuning based on genre and style, a means for inserting the translated text into an image, a means for completing the background associated with the insertion, a means for displaying the edited image to the user and collecting user emotional feedback using a sentiment analysis engine, a means for correcting the translation result and editing result based on the collected emotional feedback, and a means for saving the final corrected image and providing it to the user. This makes it possible to efficiently perform high-quality translations with a user-friendly interface and reflect the user's emotional feedback.

[0309] "Data upload" refers to a user sending manga manuscript data (image file or PDF file) to the server from the system's front end.

[0310] "Preprocessing" refers to the process of checking the resolution and format of uploaded image data and resizing or format conversion as necessary.

[0311] "OCR technology" refers to optical character recognition technology that extracts text from image data.

[0312] "Generative AI Model" refers to the artificial intelligence model used to translate extracted text into a target language.

[0313] "Fine-tuning" refers to the process of improving the initial translation results based on a specific genre or style.

[0314] "Inserting translated text" refers to the process of placing translated text at a specified location within the original image.

[0315] "Background completion" refers to the process of correcting background inconsistencies that occur during the process of inserting translated text, resulting in a natural look.

[0316] An "emotion analysis engine" refers to technology that displays edited images to the user and analyzes emotions in real time from the user's facial expressions and voice.

[0317] "Emotional feedback" refers to feedback information on translation results and edited images that is collected based on the results of user emotion analysis.

[0318] "Saving the final image" refers to the process of saving the final translation and editing results after corrections in a database and providing them to the user.

[0319] The present invention provides a system for efficiently translating manga into multiple languages. The system aims to improve the accuracy of translation and editing by incorporating user emotional feedback. Specific embodiments are described below.

[0320] First, the user uploads the manga manuscript data (image files or PDF files) using the system's front-end interface. To do this, the user clicks the file selection button in the browser, selects the required file from the local disk, and presses the upload button.

[0321] The uploaded data is then sent to the server, which begins pre-processing the data. At this stage, the server inspects the image resolution and format and resizes or converts it if necessary. For example, in the case of a PDF file, it splits each page into a separate image file.

[0322] After preprocessing is complete, the server uses OCR technology (e.g., Tesseract or Google Cloud Vision API) to extract text from the image data. The extracted text is stored in a database. For example, the server identifies the location and content of text balloons on each page and stores this information in the database.

[0323] The extracted text is then translated into the target language by a generative AI model (e.g., GPT-3 (registered trademark)) on the server. This translation result is also stored in a database. For example, the server translates the text "Hello, how are you?" into "Hello, how are you?" An example of a prompt sentence used in this process is "Please translate this text 'Hello, how are you?' into English."

[0324] The server then uses fine-tuning models based on specific genres and styles to refine the initial translation. These refined translations are also stored in the database. For example, the server might refine "This is the hero's sword!" to "This is the courage sword!"

[0325] After obtaining the translation result, the server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text in the appropriate position. During this process, the position and size of the balloon are automatically adjusted and the translated text is inserted. For example, the server will properly place the text "Run quickly!" into "Run quickly!"

[0326] Additionally, the server performs background interpolation as needed to correct any inconsistencies in the background caused by changes to the text balloon, ensuring that the overall image maintains a natural look. For example, missing parts of the background are interpolated to recreate a natural background.

[0327] The device then displays the edited image to the user and uses an emotion engine (e.g., Affectiva) to analyze the user's facial expressions and voice in real time to determine their emotions. Based on the analysis results, the device collects user feedback on the translation results and edited image. For example, the device can use the emotion engine to analyze data collected through the user's camera and microphone to collect positive or negative feedback.

[0328] The collected user emotional feedback is used by the server to automatically correct the translated text and edited results. If there is a lot of positive feedback, the server proceeds to the next step, but if there is negative feedback, the translation is corrected and redisplayed.

[0329] Finally, the user makes final confirmation and corrections while taking into account the emotional feedback. After corrections are complete, the server saves the final translation and editing results in a database and provides them to the user. For example, once the user has completed their confirmation and is satisfied with the results, they can click the "Save" button.

[0330] Finally, the terminal displays the generated download link to the user, allowing the user to download the final data. For example, the terminal displays the download link, and the user clicks the link to download the final data.

[0331] Through the above steps, the present invention realizes a system that can improve the accuracy of multilingual translation and enhance the user experience.

[0332] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0333] Step 1: Upload your data

[0334] Users use the front-end interface in their browser to select and upload manga manuscript data (image files or PDF files) from their local disk, which then sends the manga data to the server.

[0335] Input: Manga manuscript data (image file or PDF file)

[0336] Output: Manuscript data sent to the server

[0337] What happens: The user clicks the file selection button to select a file, then clicks the "Upload" button to send the file to the server, and a progress bar appears to show the upload progress.

[0338] Step 2: Image preprocessing

[0339] The server checks the resolution and format of the uploaded manuscript and performs resizing and formatting as needed. In the case of PDF files, each page is split into a separate image file.

[0340] Input: Manuscript data sent to the server

[0341] Output: Preprocessed image data

[0342] Specific operation: The server checks the resolution and resizes it if it is low. In the case of PDF files, each page is split into an image file. The processing result is recorded in the log.

[0343] Step 3: Text Extraction

[0344] The server uses OCR technology to extract text from the preprocessed image files and stores it in a database.

[0345] Input: Preprocessed image data

[0346] Output: Extracted text data

[0347] Specific operation: The server uses OCR technology (e.g., Tesseract or Google Cloud Vision API) to extract text from images, saves the extracted results in a database, and displays a preview in the log.

[0348] Step 4: Initial translation

[0349] The server uses the generative AI model to perform an initial translation of the extracted text into the target language, and stores the translation results in a database.

[0350] Input: Extracted text data

[0351] Output: Initial translated text data

[0352] Specific operation: The server translates the Japanese text "Hello, how are you?" into English "Hello, how are you?". The prompt used and the translation result are recorded in the database and in the log.

[0353] Step 5: Fine Tuning

[0354] The server refines the initial translation results with fine-tuning models based on specific genres and styles, and stores the refined translation results in a database.

[0355] Input: Initial translated text data

[0356] Output: Improved translation text data

[0357] What happens: The server refines the translation of "This is the hero's sword!" to "This is the courage sword!". The fine-tuning results are saved in the database and logged.

[0358] Step 6: Image Recognition and Editing

[0359] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text in the appropriate location. The balloon's position and size are automatically adjusted to insert the translated text.

[0360] Input: Improved translation text data

[0361] Output: Image data neatly placed in a text balloon

[0362] What it does: The server detects the position and size of the text balloon in the image, places the translated text "Run quickly!" in the appropriate position, and generates an image. At this time, it also fine-tunes the size of the text balloon.

[0363] Step 7: Filling in the background

[0364] The server performs background interpolation processing as necessary to correct background inconsistencies resulting from changes to the text balloon.

[0365] Input: Image data placed in a text balloon

[0366] Output: Final image data with background interpolated

[0367] What it does: The server fills in the missing parts of the background and recreates a natural background. It logs the results of the filling process.

[0368] Step 8: User Emotion Recognition and Feedback

[0369] The device displays the edited image to the user and uses an emotion engine to analyze the user's emotions in real time from their facial expressions and voice, collecting emotional feedback from the user.

[0370] Input: Final image data with background interpolated

[0371] Output: User's emotional feedback data

[0372] Specific operation: The device collects the user's facial expressions and voice through the camera and microphone, analyzes them with an emotion engine, and displays and collects positive or negative feedback.

[0373] Step 9: Applying Feedback

[0374] The server automatically corrects the translated text and edits based on the collected user emotional feedback. If there is a lot of positive feedback, the server proceeds as is.

[0375] Input: User emotional feedback data

[0376] Output: Corrected translation text and edits

[0377] Specific behavior: The server corrects the translation in response to negative feedback and displays it again to the user. The corrected translation result is recorded in the log.

[0378] Step 10: Modify and save

[0379] The user then makes final confirmations and corrections while taking into account the emotional feedback. After the corrections are complete, the server stores the final translation and editing results in a database and provides them to the user.

[0380] Input: Final image data with positive feedback applied

[0381] Output: Final image data after user confirmation

[0382] Specific behavior: The user checks the data and if satisfied, clicks the "Save" button. A message indicating that the data has been saved is displayed.

[0383] Step 11: Provide data

[0384] The terminal displays the generated download link to the user, allowing the user to download the final data.

[0385] Input: Final saved image data

[0386] Output: A download link that users can access

[0387] Specific behavior: The device displays a download link, the user clicks the link to download the final data, and a confirmation message is displayed when the download is complete.

[0388] (Application example 2)

[0389] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0390] In multilingual manga translation, simply translating text into another language can result in the loss of context and nuance, leaving users dissatisfied. Manually correcting translation results can also lead to a poor user experience. Furthermore, improper adjustment of the position and size of text balloons can result in visual inconsistencies. To resolve these issues and provide high-quality translation results, a system is needed that incorporates user emotional feedback to improve translation accuracy.

[0391] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0392] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning based on context specific to manga, means for inserting the translated text into the image, means for completing the background accompanying the insertion, means for displaying the translation result and allowing the user to make corrections, means for analyzing the user's emotions in real time and reflecting the feedback, and means for saving the final image after correction and providing it to the user. This makes it possible to provide high-quality translation results that reflect the user's emotional feedback, improving the user experience.

[0393] "Image data" refers to files that digitally represent the visual information contained in manga pages and frames.

[0394] "Text extraction" refers to the process of identifying textual information from image data and extracting it as digital text.

[0395] "Target Language" means the language into which the Source Text is translated.

[0396] "Fine tuning" refers to the process of fine-tuning the initial translation result to suit a specific context or genre, improving the accuracy and quality of the translation.

[0397] "Insertion" refers to the process of placing translated text in a predetermined location in image data.

[0398] "Background completion" refers to the process of correcting missing or inconsistent backgrounds caused by text insertion, maintaining a natural appearance.

[0399] "Correction" refers to the act of the user checking the translation results and the content of the inserted text and making changes as necessary.

[0400] "Real-time emotion analysis" refers to the process of instantly determining the user's current emotional state based on their facial expressions and voice, and obtaining the results as data.

[0401] "Reflecting feedback" refers to the process of improving translation results and the overall operation of the system based on analyzed user emotional data.

[0402] "Final Image" refers to image data that has had all corrections and enhancements completed and is ready to be provided to the user.

[0403] "Preservation" refers to the process of digitally recording the final, corrected and enhanced image and storing it for reuse as needed.

[0404] This invention is a multilingual manga translation system that enables more accurate translation by incorporating user emotional feedback. The details of this system are described below.

[0405] Hardware and software used

[0406] The system uses a smartphone, a head-mounted display (HMD), a camera, and a microphone as hardware, and utilizes OCR technology (e.g., Tesseract), a generative AI model (e.g., OpenAI API, DeepL API), an emotion analysis engine (e.g., Microsoft® Azure® Emotion API), and a database (e.g., MySQL®, PostgreSQL) as software.

[0407] System Overview

[0408] 1. Uploading image data

[0409] Users upload manga image data (e.g., image files or PDF files) to the system via their smartphone or HMD, which gives the system a starting point for the translation process.

[0410] 2. Image Preprocessing

[0411] The server checks the uploaded image data and performs resizing and format conversion for resolution and format, and in the case of PDF files, splits each page into a separate image file.

[0412] 3. Text Extraction

[0413] The server uses OCR technology (e.g., Tesseract) to extract text from the preprocessed image data, and the extracted text is stored in a database.

[0414] 4. Initial Translation

[0415] The server uses a generative AI model (e.g., OpenAI's API) to translate the extracted text into the target language, and the translation results are stored in a database.

[0416] 5. Fine Tuning

[0417] The server uses fine-tuning models specific to specific contexts and genres to improve the quality of the initial translation results, which are then stored in a database.

[0418] 6. Image Recognition and Editing

[0419] The server uses image recognition technology to detect the position and size of the text balloon and position the translated text appropriately, interpolating the background if necessary to maintain a natural look.

[0420] 7. Obtaining Emotional Feedback

[0421] The device uses a camera and microphone to analyze the user's real-time emotional data, such as facial expressions and voice while viewing the translation results, to obtain positive or negative feedback.

[0422] 8. Applying Feedback

[0423] The server automatically improves translation results and edited images based on the user's emotional feedback. If there is a lot of positive feedback, the translation is left as is, but if there is a lot of negative feedback, the translation is retranslated or revised.

[0424] Examples and prompts

[0425] For example, if a user inputs the phrase "This hero's sword defeats all enemies," it would be translated using a generative AI model as follows:

[0426] Translate this to English: This hero's sword will defeat all enemies

[0427] Based on this prompt, it would be translated as "This hero's sword defeats all enemies."

[0428] Process Overview

[0429] This system is characterized by a process that analyzes user sentiment in real time and automatically improves translation results based on that feedback. The combination of hardware and software makes it highly user-friendly and capable of providing high-quality multilingual translation.

[0430] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0431] Step 1:

[0432] A user logs in to the system and uploads manga image data (e.g., image files or PDF files). Specifically, the user selects image data using a smartphone or HMD and uploads it through the system interface. The input data is an image file, and the output is an image file stored on the server.

[0433] Step 2:

[0434] The server preprocesses the uploaded image data. Specifically, it checks the image resolution and format, resizes it, and converts the format. In the case of PDF files, it also splits each page into a separate image file. The input data is the uploaded image data, and the output is the preprocessed image file.

[0435] Step 3:

[0436] The server uses OCR technology (e.g., Tesseract) to extract text from the preprocessed image files. Specifically, the OCR engine analyzes characters in the image and extracts them as digital text. The input data is the preprocessed image file, and the output is the extracted text data.

[0437] Step 4:

[0438] The server uses a generative AI model (e.g., OpenAI's API) to translate the extracted text into the target language. Specifically, it sends a prompt to the generative AI model and obtains the translation result. The input data is the extracted text data, and the output is the initial translation result.

[0439] Step 5:

[0440] The server refines the initial translation using a fine-tuning model specific to a particular context or genre. Specifically, the translation is fed into a pre-trained model, which then performs context-based corrections. The input data is the initial translation, and the output is the fine-tuned translation.

[0441] Step 6:

[0442] The server uses image recognition technology to detect the position and size of the text balloon and place the translated text in the appropriate location. Specifically, an image analysis algorithm identifies the text balloon and inserts the new text in its place. The input data is the fine-tuned translation result and image data, and the output is the image with the inserted text.

[0443] Step 7:

[0444] The server performs background interpolation as needed to fix background inconsistencies caused by text balloon changes. Specifically, image interpolation techniques recreate the background to maintain a natural look. The input data is an image with text inserted, and the output is an image with the background interpolated.

[0445] Step 8:

[0446] The device displays the edited image to the user and uses an emotion engine to analyze emotions in real time from the user's facial expressions and voice. Specifically, the device collects the user's reactions through a camera and microphone, and the emotion analysis engine analyzes the data. The input data is the user's facial expressions and voice, and the output is analyzed emotional data.

[0447] Step 9:

[0448] The server automatically corrects the translated text and edited results based on the collected user emotional feedback. Specifically, if there is a lot of positive feedback, it proceeds to the next step, but if there is a lot of negative feedback, it retranslates or corrects the text. The input data is emotional data, and the output is the corrected translated text and edited results.

[0449] Step 10:

[0450] The user checks the correction results and makes final corrections. Specifically, the user checks the final image displayed on the device and manually makes corrections as necessary. The input data is the corrected translated text or edited results, and the output is the final translation result or edited image.

[0451] Step 11:

[0452] The server stores the final translation and editing results in a database and provides them to users. Specifically, it generates and provides a link that allows users to download the final images stored in the database. The input data is the final translation results and edited images, and the output is a download link that users can access.

[0453] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0454] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0455] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0456] [Second embodiment]

[0457] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0458] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0459] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0460] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0461] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0462] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0463] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0464] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0465] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0466] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0467] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0468] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0469] The present invention relates to a system for efficiently translating manga into multiple languages, which has the ability to extract text from image data, translate it, generate an image containing the translated text, and allow the user to make any necessary corrections.

[0470] 1. Upload your data

[0471] Users upload manga manuscript data (image files or PDFs) through the system's front end, which then sends the manga pages to the server.

[0472] Example: User clicks upload button in browser, selects file and submits.

[0473] 2. Image Preprocessing

[0474] The server preprocesses the received manga data. First, it checks the image resolution and resizes it if it is inappropriate. If it is a PDF file, it splits each page into a separate image file.

[0475] Example: The server checks the image resolution and resizes it to the optimal resolution if necessary.

[0476] 3. Text Extraction

[0477] The server uses OCR (Optical Character Recognition) technology to extract the text from each image file, which is then stored in a database.

[0478] Example: The server runs an OCR engine to identify the location and content of text balloons from an image.

[0479] 4. Initial Translation

[0480] The server uses generative AI to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[0481] Example: The server translates the extracted text "Hello, how are you?" into English, producing "Hello, how are you?"

[0482] 5. Fine Tuning

[0483] The server uses fine-tuned models tailored to specific manga genres and styles to refine the initial translation results, which are also stored in the database.

[0484] Example: The server inputs the initial translation "This is the hero's sword!" into a fine-tuning model, which generates the more contextually appropriate "This is the courage sword!"

[0485] 6. Image Recognition and Editing

[0486] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[0487] Example: The server replaces the original text "Run quickly!" with "Run quickly!" and adjusts the position and size of the balloon.

[0488] 7. Background Completion

[0489] The server compensates for inconsistencies in the background caused by inserting translated text, and automatically corrects the background so that it does not look unnatural when the text balloon is changed.

[0490] Example: The server regenerates the background of the balloon and inserts the translated text.

[0491] 8. User confirmation and correction

[0492] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[0493] Example: A user corrects the displayed translation "Run quickly!" to "Hurry up!"

[0494] 9. Saving and downloading final data

[0495] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[0496] Example: User completes verification and downloads the corrected image.

[0497] This system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections.

[0498] The processing flow will be explained below.

[0499] Step 1:

[0500] The user uploads the manga manuscript data (image files or PDF files) through the system's front end. When the user clicks the file selection button in the browser, selects a file from the local disk, and presses the upload button, the file is sent to the server.

[0501] Step 2:

[0502] The server preprocesses the manga data received by checking the image resolution and format, resizing and converting the image if necessary, and splitting each page into an individual image file if it is a PDF file.

[0503] Step 3:

[0504] The server uses OCR (Optical Character Recognition) technology to extract text from the preprocessed image files, specifically by identifying the location and content of text balloons on each page, and stores the extracted text in a database.

[0505] Step 4:

[0506] The server uses the generative AI to initially translate the extracted text into the target language, and the generated initial translation results are also stored in the database.

[0507] Step 5:

[0508] The server refines the initial translation using a fine-tuned model that is specialized for a specific manga genre and style. Specifically, the model takes the initial translation as input and generates a translation that is appropriate for the context and style, which is then stored in a database.

[0509] Step 6:

[0510] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted to insert the translated text.

[0511] Step 7:

[0512] The server automatically detects and corrects any unnatural background caused by changes to the text balloon, ensuring that the background remains natural.

[0513] Step 8:

[0514] The device displays the edited image on the user interface, and the user is provided with a UI to review the translation results and image edits, and manually make corrections if necessary.

[0515] Step 9:

[0516] After the user has completed the confirmation and correction, they issue an instruction to save the final image data, and the server saves the final image and generates a download link.

[0517] Step 10:

[0518] The terminal displays the generated download link to the user, allowing the user to download the final translation data with the corrections reflected.

[0519] This series of processes allows users to efficiently translate manga into multiple languages ​​with high quality.

[0520] Example 1

[0521] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0522] Previous multilingual manga translation systems faced challenges in the entire process, from text extraction to translation and image insertion, including translation quality, background completion, and the inability for users to easily make corrections. Furthermore, the lack of fine-tuning specialized for specific categories or styles made it difficult to translate appropriately for the context. As a result, problems arose, such as low translation quality and a heavy burden on users to make corrections.

[0523] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0524] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning specialized for a particular category or style, means for inserting the translated text into an image, means for completing the background accompanying the text insertion, means for displaying the translation result and allowing the user to make corrections, and means for saving the final image after corrections and providing it to the user. This enables high-quality multilingual translation and reduces the burden of corrections on the user.

[0525] "Image data" means digital data that stores visual information, including image files in JPEG and PNG formats and PDF files.

[0526] "Text extraction" is the process of recognizing characters and sentences from image data and obtaining their content as digital text.

[0527] The "target language" is the language into which you are translating, and the language into which you want to convert and display the original text.

[0528] "Fine tuning" is the process of making adjustments to improve accuracy, focusing on a particular category or style.

[0529] "Translated text" means text that has been converted from a source language into a target language.

[0530] "Image insertion" is the process of incorporating translated text into the original image data, including adjusting the position and size of the text.

[0531] "Background completion" is the process of correcting background inconsistencies that occur when inserting text, making it look natural.

[0532] "Translation result" is the text after translation and fine-tuning.

[0533] "Correction" refers to any change a user makes to the translation or image editing results provided by the system.

[0534] "Final image" refers to the final image data provided to the user after translation and corrections are complete.

[0535] The present invention relates to a system for efficiently translating manga into multiple languages, and the system has many functions for text extraction, translation, image generation, and user correction. Specific embodiments for implementing the invention are described below.

[0536] Hardware and Software Used

[0537] server

[0538] The server is implemented in a hardware environment with high-performance computing capabilities (e.g., a virtual server provided by a cloud provider). The server includes the following software components:

[0539] OCR engine: Uses, for example, Google Cloud Vision API to perform optical character recognition.

[0540] Generative AI models: for example, using OpenAI GPT-4 for translation into the target language and fine-tuning.

[0541] Image processing libraries: For example, OpenCV or PyMuPDF are used to check image resolution and resize, split PDFs, analyze text balloon positions, etc.

[0542] Terminal

[0543] A terminal is a device (such as a personal computer or tablet) that allows a user to interact with the system. The terminal implements the system's front end and provides the interface that the user operates.

[0544] User

[0545] The user is the end user who operates the system, and is responsible for providing the manga manuscript data and checking and correcting the results of the translation and corrections.

[0546] Data processing and calculation

[0547] 1. Text Extraction

[0548] The server uses an OCR engine to extract text from the image data, for example, using the Google Cloud Vision API, to identify text balloons in the image and store their locations and contents in a database.

[0549] 2. Initial Translation

[0550] The server translates the extracted text into the target language using a generative AI model (e.g., OpenAI GPT-4). This translation result is again stored in the database. An example of a prompt sentence is a prompt that translates "Hello, how are you?" into "Hello, how are you?"

[0551] 3. Fine Tuning

[0552] The server uses fine-tuning models that are specialized for specific manga categories and styles to refine the initial translation results, for example, refining the hero's line "This is the hero's sword!" to "This is the courage sword!"

[0553] 4. Image Recognition and Editing

[0554] The server uses image recognition technology to insert the translated text into the image appropriately, specifically by adjusting the position and size of the text balloon using OpenCV.

[0555] 5. Background Completion

[0556] The server uses a technique to regenerate the background portion to compensate for the inconsistency of the background caused by the text insertion.

[0557] 6. User confirmation and correction

[0558] The terminal displays the translation results and the status of the image editing to the user. The user can then modify the translation and image editing status as needed, and the modifications are immediately sent to the server.

[0559] Specific examples

[0560] For example, a user may upload a comic page and provide an image containing the text "Hello, how are you?" The server receives this image data, uses an OCR engine to extract the text from the text balloon, and stores the results in a database.

[0561] The server then uses a generative AI model to translate the extracted text into English, for example, "Hello, how are you?" is translated to "Hello, how are you?", and then further refines the translation results using a fine-tuning model suited to the manga category.

[0562] The server places this translated text in the appropriate text balloon and fills in any inconsistencies in the background. Finally, the device displays the edited results to the user, who can then modify the translation as needed. The modified results are then sent to the server and stored in a database.

[0563] This system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections.

[0564] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0565] Step 1:

[0566] Uploading data

[0567] Users upload manga manuscript data through the system's front end, providing image files (JPEG, PNG, etc.) or PDF files as input.

[0568] The server processes the received file using an HTTP POST request, and as output, the received manuscript data is saved on the server.

[0569] Step 2:

[0570] Image preprocessing

[0571] The server checks the image resolution of the received manga data and resizes it if it is inappropriate, and if it is a PDF file, splits each page into a separate image file.

[0572] It uses saved original data (image or PDF) as input, and generates an image file resized to the appropriate resolution or split into individual image files as output.

[0573] Specific operations include image resizing using the OpenCV library and splitting the PDF using PyMuPDF.

[0574] Step 3:

[0575] Text Extraction

[0576] The server uses an OCR engine (e.g., Google Cloud Vision API) to extract text from each image file, which is then stored in a database.

[0577] As input, it uses preprocessed image files, and as output, it stores extracted text information and its location data in a database.

[0578] Specific operations include sending an image file to an OCR engine and storing the returned text information in a database.

[0579] Step 4:

[0580] Initial translation

[0581] The server uses a generative AI model (e.g., OpenAI GPT-4) to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[0582] As input, it uses extracted text stored in a database, and as output, the translation results into the target language are stored in a database.

[0583] Specifically, the generative AI model is provided with the extracted text and uses the prompt sentence to obtain the translation result, for example, translating "Hello, how are you?" into "Hello, how are you?"

[0584] Step 5:

[0585] Fine Tuning

[0586] The server uses fine-tuning models specialized for specific manga categories and styles to refine the initial translation results, which are also stored in the database.

[0587] As input, it uses the initial translation results, and as output, the fine-tuned translation results are stored in a database.

[0588] Specifically, it provides the initial translation text to the fine-tuning model, which then generates a more natural and context-appropriate translation result, for example, refining "This is the hero's sword!" to "This is the courage sword!".

[0589] Step 6:

[0590] Image Recognition and Editing

[0591] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[0592] It takes the fine-tuned translation results and the original image file as input, and generates a new image file with the translated text inserted as output.

[0593] Specific actions include using OpenCV to reparse the position and size of the text balloon and insert text, e.g., replacing "Run quickly!" with "Run quickly!".

[0594] Step 7:

[0595] Background Complement

[0596] The server compensates for inconsistencies in the background caused by inserting translated text, and automatically corrects the background so that it does not look unnatural when the text balloon is changed.

[0597] As input, it takes the image file after the text has been inserted, and as output, it generates a new image file with the background filled in with inconsistencies.

[0598] Specifically, it uses generative AI technology and image correction algorithms to recreate the background around the balloon in a natural shape.

[0599] Step 8:

[0600] User confirmation and correction

[0601] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[0602] As input, it takes an image file with the translated text inserted, and as output, it generates the final image data with the user's modifications reflected.

[0603] Specifically, the terminal provides an interface for editing, creating an environment where users can edit directly. The edited data is immediately reflected on the server.

[0604] Step 9:

[0605] Save and download the final data

[0606] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[0607] It takes the image file with the user's modifications as input, and saves the final image data and generates a download link as output.

[0608] Specifically, the modified image file is saved in the database and a URL that can be accessed by the user is created.

[0609] (Application example 1)

[0610] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0611] Multilingual distribution of manga requires efficient translation into different languages. Conventional methods require manual translation and editing, which is extremely time-consuming and laborious. Furthermore, there are often inconsistencies in the background of images when translating text, and the environment for easy user corrections is often inadequate. This makes it difficult for users to smoothly enjoy manga in multiple languages.

[0612] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0613] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning based on context specific to manga, means for inserting the translated text into the image, means for completing the background associated with the insertion, means for displaying the translation result and allowing the user to make corrections, means for saving the final image after corrections and providing it to the user, and means for providing a terminal application for viewing manga in multiple languages. This allows manga text to be translated efficiently into multiple languages, enabling users to easily make corrections and view the text.

[0614] "Image data" is an electronic representation of a comic page or frame.

[0615] "Text" refers to character information contained in image data.

[0616] "Target language" refers to the language into which the text is translated.

[0617] "Fine tuning" refers to adjusting the model to improve the accuracy of machine translation based on the specific context of manga.

[0618] "Background completion" is the process of correcting background inconsistencies that occur when inserting translated text into an image.

[0619] "User" refers to a person who uses the system to translate and view manga in multiple languages.

[0620] "Device application" refers to software that runs on mobile devices such as smartphones and tablets.

[0621] "Extracted text" refers to character information obtained from image data using OCR technology.

[0622] A "text balloon" refers to a speech bubble used to write lines or narration within a manga.

[0623] The "final image" is the final image data after the text has been translated and completed and the user's corrections have been reflected.

[0624] The present invention relates to a system for efficiently translating manga into multiple languages. The system extracts text from image data, translates it, generates an image containing the translated text, and allows the user to make any necessary corrections.

[0625] First, the user uploads the manga image data (image file or PDF) via their device. This image data is then sent to the server. The server preprocesses the received manga data, inspects the image resolution, and resizes it if it is inappropriate. In the case of a PDF file, the server also splits each page into a separate image file.

[0626] The server then uses OCR technology to extract text from each image file, identify the location and content of the text balloons, and store the extracted text in a database. Specifically, this can be done using an OCR engine such as Pytesseract.

[0627] The server then uses a generative AI model to translate the extracted text into the target language. This initial translation result is also stored in the database. For example, the extracted text "Hello, how are you?" can be translated into English to produce "Hello, how are you?"

[0628] Additionally, the server uses fine-tuned models tailored to specific manga genres and styles to refine the initial translation results, resulting in more context-appropriate translations. For example, it takes the initial translation "This is the hero's sword!" and generates the more context-appropriate "This is the courage sword!".

[0629] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image. The size and position of the text balloon are also automatically adjusted. This compensates for inconsistencies in the background and eliminates the awkwardness that comes with inserting the translated text. This allows the original text "Run quickly!" to be replaced with "Run quickly!" and the position and size of the balloon to be adjusted.

[0630] The user can then check the translation results displayed on the device and make corrections as necessary. These corrections are also sent to the server and reflected in the final image data. For example, the user can change the displayed translation "Run quickly!" to "Hurry up!"

[0631] Finally, the server saves the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download. The user can click on this link to download the high-quality multilingual translated manga.

[0632] The implementation of this system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections. An example of a translation and prompt using a generative AI model is "Please translate the following Japanese text into English. Text: 'Hello, how are you?'"

[0633] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0634] Step 1:

[0635] Users upload manga image data (image files or PDFs) to the system from their devices. Specifically, they click the upload button in their browser or application, select the file, and send it to the server.

[0636] Input: Comic image data file (PNG, JPEG, PDF, etc.) selected by the user.

[0637] Output: Image data of the manga sent to the server.

[0638] Step 2:

[0639] The server preprocesses the image data it receives, checking the image resolution and resizing it if it is inappropriate, and, in the case of PDF files, splitting each page into a separate image file.

[0640] Input: Image data received by the server.

[0641] Output: Image data resized to optimal resolution and PDF pages split into individual image files.

[0642] What it does: Uses libraries such as OpenCV to check image resolution and resize.

[0643] Step 3:

[0644] The server uses OCR technology to extract text from each image file, and the extracted text and its location are stored in a database.

[0645] Input: The resized image data.

[0646] Output: Extracted text data and text location information.

[0647] What it does: Uses an OCR engine such as Pytesseract to extract the location and content of text balloons from an image file.

[0648] Step 4:

[0649] The server uses a generative AI model to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[0650] Input: Extracted text data.

[0651] Output: The translated text data.

[0652] What it does: Uses a translation engine (e.g., Google Translate API) to translate text data into the target language.

[0653] Step 5:

[0654] The server uses fine-tuned models that are specialized for specific manga genres and styles to refine the initial translation results, which are then stored in a database.

[0655] Input: Text data of the initial translation result.

[0656] Output: Fine-tuned translated text data.

[0657] How it works: Generates context-appropriate translations using a generative AI model specialized for the manga genre.

[0658] Step 6:

[0659] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[0660] Input: Fine-tuned translation text data, image data.

[0661] Output: Image data with inserted text.

[0662] What it does: Uses OpenCV and PIL libraries to insert text into an image and adjust the position and size of the text balloon.

[0663] Step 7:

[0664] The server compensates for inconsistencies in the background caused by the insertion of translated text.

[0665] Input: Image data with inserted text.

[0666] Output: Image data with background interpolated.

[0667] How it works: It uses image editing algorithms to naturally complement the background of the text insert.

[0668] Step 8:

[0669] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[0670] Input: The translated and completed image data.

[0671] Output: Image data with user modifications.

[0672] What it does: Displays the image through the device's interface and allows the user to make corrections. The user enters the corrections and submits them.

[0673] Step 9:

[0674] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[0675] Input: Image data with user modifications.

[0676] Output: Final image data provided via a download link.

[0677] Behavior: Saves the image data and sends a notification to the user providing a download link.

[0678] This allows users to quickly and efficiently translate manga into multiple languages. An example of a translation and prompt using a generative AI model is: "Please translate the following Japanese text into English. Text: 'Hello, how are you?'"

[0679] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0680] This invention relates to a system that combines an emotion engine with an efficient system for translating manga into multiple languages, thereby reflecting user emotional feedback and improving the accuracy of translation and editing. This system extracts text from image data, translates it into the target language, and provides an environment where users can check and correct the results, while analyzing user emotions in real time, enabling more user-friendly operation.

[0681] 1. Upload your data

[0682] The user uploads the manga manuscript data (image file or PDF file) from the system's front end. The file is selected and sent to the server.

[0683] Example: A user clicks the file chooser button in their browser to select and upload a file from their local disk.

[0684] 2. Image Preprocessing

[0685] The server pre-processes the uploaded manga data, inspecting the image resolution and format and resizing or formatting it as needed. In the case of PDF files, each page is split into a separate image file.

[0686] Example: The server checks the image resolution and resizes it if it is inappropriate.

[0687] 3. Text Extraction

[0688] The server uses OCR (Optical Character Recognition) technology to extract text from the preprocessed image files, which is then stored in a database.

[0689] Example: The server identifies the location and content of text balloons on each page and stores them in a database.

[0690] 4. Initial Translation

[0691] The server uses generative AI to perform an initial translation of the extracted text into the target language, and the translation results are stored in a database.

[0692] Example: The server translates the text "Hello, how are you?" to "Hello, how are you?"

[0693] 5. Fine Tuning

[0694] The server uses fine-tuning models specific to a particular genre or style to refine the initial translation results, which are then stored in a database.

[0695] Example: The server refines "This is the hero's sword!" to "This is the courage sword!"

[0696] 6. Image Recognition and Editing

[0697] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted to insert the text.

[0698] Example: The server properly places "Run quickly!" into "Run quickly!"

[0699] 7. Background Completion

[0700] The server will interpolate the background as needed to correct any inconsistencies in the background caused by changes to the text balloon, thus maintaining a natural look.

[0701] Example: The server fills in missing parts of the background and recreates a natural background.

[0702] 8. User Emotion Recognition and Feedback

[0703] The device displays the edited image to the user and uses an emotion engine to analyze the user's emotions in real time based on their facial expressions and voice. Based on the analysis results, the device collects user feedback on the translation results and the edited image.

[0704] Example: The device uses the user's camera and microphone to analyze their reactions and use an emotion engine to collect positive or negative feedback.

[0705] 9. Applying Feedback

[0706] The server has the ability to automatically correct the translated text and edits based on the collected user emotional feedback. If the feedback is mostly positive, the server proceeds to the next step.

[0707] Example: The server responds to negative feedback by correcting the translation and redisplaying it.

[0708] 10. Modifying and Saving

[0709] The user makes final confirmations and corrections while taking into account the emotional feedback. After corrections are complete, the server stores the final translation and editing results in a database and provides them to the user.

[0710] Example: The user completes the review, and if the result is satisfactory, completes the corrections and clicks the save button.

[0711] 11. Data Provision

[0712] The terminal displays the generated download link to the user, allowing the user to download the final data.

[0713] Example: The device displays a download link and the user downloads the modified results.

[0714] This system enables high-quality multilingual translation of manga that incorporates user emotional feedback, improving the user experience.

[0715] The processing flow will be explained below.

[0716] Step 1:

[0717] The user uploads the manga manuscript data (image file or PDF file) from the system's front end. The user clicks the file selection button in the browser, selects the file from the local disk, and presses the upload button, which sends the file to the server.

[0718] Step 2:

[0719] The server preprocesses the manga data received. Specifically, it checks the image resolution and format, resizes and converts the format as needed, and splits each page into a separate image file if it is a PDF file.

[0720] Step 3:

[0721] The server extracts text from the preprocessed image files using OCR (Optical Character Recognition) technology, identifies the location and content of text balloons, and stores the extracted text in a database.

[0722] Step 4:

[0723] The server uses generative AI to perform an initial translation of the extracted text into the target language, and the translation results are stored in a database.

[0724] Step 5:

[0725] The server uses fine-tuning models specialized for specific manga genres and styles to improve the initial translation results, which are then stored in a database.

[0726] Step 6:

[0727] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted and the text is inserted.

[0728] Step 7:

[0729] The server performs background interpolation, correcting inconsistencies in the background caused by text balloon changes to maintain a natural look.

[0730] Step 8:

[0731] The device displays the edited image to the user, and uses an emotion engine to analyze the user's emotions in real time from their facial expressions and voice. Based on the analysis results, the device collects user feedback on the translation results and the edited image.

[0732] Step 9:

[0733] The server automatically corrects the translated text and edits based on the user's emotional feedback collected. If the feedback is positive, the system proceeds to the next step, but if the feedback is negative, the system makes appropriate corrections.

[0734] Step 10:

[0735] The user then makes final confirmations and corrections while taking into account the emotional feedback. After corrections are complete, the server saves the final translation and edited results in a database.

[0736] Step 11:

[0737] The terminal displays the generated download link to the user, allowing the user to download the final data. The user uses this link to obtain the final translation result.

[0738] Example 2

[0739] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0740] Conventional systems for multilingual translation have issues with translation accuracy and the complexity of editing operations. They also face the problem of not being able to properly reflect user emotions and feedback, resulting in a poor user experience. The present invention aims to solve these issues by providing an efficient system that provides high-quality translation results and reflects user emotional feedback.

[0741] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data uploading means, a means for preprocessing uploaded image data, a means for extracting text from image data using OCR technology, a means for translating the extracted text into a target language using a generative AI model, a means for fine-tuning based on genre and style, a means for inserting the translated text into an image, a means for completing the background associated with the insertion, a means for displaying the edited image to the user and collecting user emotional feedback using a sentiment analysis engine, a means for correcting the translation result and editing result based on the collected emotional feedback, and a means for saving the final corrected image and providing it to the user. This makes it possible to efficiently perform high-quality translations with a user-friendly interface and reflect the user's emotional feedback.

[0742] "Data upload" refers to a user sending manga manuscript data (image file or PDF file) to the server from the system's front end.

[0743] "Preprocessing" refers to the process of checking the resolution and format of uploaded image data and resizing or format conversion as necessary.

[0744] "OCR technology" refers to optical character recognition technology that extracts text from image data.

[0745] "Generative AI Model" refers to the artificial intelligence model used to translate extracted text into a target language.

[0746] "Fine-tuning" refers to the process of improving the initial translation results based on a specific genre or style.

[0747] "Inserting translated text" refers to the process of placing translated text at a specified location within the original image.

[0748] "Background completion" refers to the process of correcting background inconsistencies that occur during the process of inserting translated text, resulting in a natural look.

[0749] An "emotion analysis engine" refers to technology that displays edited images to the user and analyzes emotions in real time from the user's facial expressions and voice.

[0750] "Emotional feedback" refers to feedback information on translation results and edited images that is collected based on the results of user emotion analysis.

[0751] "Saving the final image" refers to the process of saving the final translation and editing results after corrections in a database and providing them to the user.

[0752] The present invention provides a system for efficiently translating manga into multiple languages. The system aims to improve the accuracy of translation and editing by incorporating user emotional feedback. Specific embodiments are described below.

[0753] First, the user uploads the manga manuscript data (image files or PDF files) using the system's front-end interface. To do this, the user clicks the file selection button in the browser, selects the required file from the local disk, and presses the upload button.

[0754] The uploaded data is then sent to the server, which begins pre-processing the data. At this stage, the server inspects the image resolution and format and resizes or converts it if necessary. For example, in the case of a PDF file, it splits each page into a separate image file.

[0755] After preprocessing is complete, the server uses OCR technology (e.g., Tesseract or Google Cloud Vision API) to extract text from the image data. The extracted text is stored in a database. For example, the server identifies the location and content of text balloons on each page and stores this information in the database.

[0756] The extracted text is then translated into the target language by a generative AI model (e.g., GPT-3) on the server. This translation result is also stored in a database. For example, the server translates the text "Hello, how are you?" into "Hello, how are you?". An example of a prompt used in this process is "Please translate this text 'Hello, how are you?' into English."

[0757] The server then uses fine-tuning models based on specific genres and styles to refine the initial translation. These refined translations are also stored in the database. For example, the server might refine "This is the hero's sword!" to "This is the courage sword!"

[0758] After obtaining the translation result, the server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text in the appropriate position. During this process, the position and size of the balloon are automatically adjusted and the translated text is inserted. For example, the server will properly place the text "Run quickly!" into "Run quickly!"

[0759] Additionally, the server performs background interpolation as needed to correct any inconsistencies in the background caused by changes to the text balloon, ensuring that the overall image maintains a natural look. For example, missing parts of the background are interpolated to recreate a natural background.

[0760] The device then displays the edited image to the user and uses an emotion engine (e.g., Affectiva) to analyze the user's facial expressions and voice in real time to determine their emotions. Based on the analysis results, the device collects user feedback on the translation results and edited image. For example, the device can use the emotion engine to analyze data collected through the user's camera and microphone to collect positive or negative feedback.

[0761] The collected user emotional feedback is used by the server to automatically correct the translated text and edited results. If there is a lot of positive feedback, the server proceeds to the next step, but if there is negative feedback, the translation is corrected and redisplayed.

[0762] Finally, the user makes final confirmation and corrections while taking into account the emotional feedback. After corrections are complete, the server saves the final translation and editing results in a database and provides them to the user. For example, once the user has completed their confirmation and is satisfied with the results, they can click the "Save" button.

[0763] Finally, the terminal displays the generated download link to the user, allowing the user to download the final data. For example, the terminal displays the download link, and the user clicks the link to download the final data.

[0764] Through the above steps, the present invention realizes a system that can improve the accuracy of multilingual translation and enhance the user experience.

[0765] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0766] Step 1: Upload your data

[0767] Users use the front-end interface in their browser to select and upload manga manuscript data (image files or PDF files) from their local disk, which then sends the manga data to the server.

[0768] Input: Manga manuscript data (image file or PDF file)

[0769] Output: Manuscript data sent to the server

[0770] What happens: The user clicks the file selection button to select a file, then clicks the "Upload" button to send the file to the server, and a progress bar appears to show the upload progress.

[0771] Step 2: Image preprocessing

[0772] The server checks the resolution and format of the uploaded manuscript and performs resizing and formatting as needed. In the case of PDF files, each page is split into a separate image file.

[0773] Input: Manuscript data sent to the server

[0774] Output: Preprocessed image data

[0775] Specific operation: The server checks the resolution and resizes it if it is low. In the case of PDF files, each page is split into an image file. The processing result is recorded in the log.

[0776] Step 3: Text Extraction

[0777] The server uses OCR technology to extract text from the preprocessed image files and stores it in a database.

[0778] Input: Preprocessed image data

[0779] Output: Extracted text data

[0780] Specific operation: The server uses OCR technology (e.g., Tesseract or Google Cloud Vision API) to extract text from images, saves the extracted results in a database, and displays a preview in the log.

[0781] Step 4: Initial translation

[0782] The server uses the generative AI model to perform an initial translation of the extracted text into the target language, and stores the translation results in a database.

[0783] Input: Extracted text data

[0784] Output: Initial translated text data

[0785] Specific operation: The server translates the Japanese text "Hello, how are you?" into English "Hello, how are you?". The prompt used and the translation result are recorded in the database and in the log.

[0786] Step 5: Fine Tuning

[0787] The server refines the initial translation results with fine-tuning models based on specific genres and styles, and stores the refined translation results in a database.

[0788] Input: Initial translated text data

[0789] Output: Improved translation text data

[0790] What happens: The server refines the translation of "This is the hero's sword!" to "This is the courage sword!". The fine-tuning results are saved in the database and logged.

[0791] Step 6: Image Recognition and Editing

[0792] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text in the appropriate location. The balloon's position and size are automatically adjusted to insert the translated text.

[0793] Input: Improved translation text data

[0794] Output: Image data neatly placed in a text balloon

[0795] What it does: The server detects the position and size of the text balloon in the image, places the translated text "Run quickly!" in the appropriate position, and generates an image. At this time, it also fine-tunes the size of the text balloon.

[0796] Step 7: Filling in the background

[0797] The server performs background interpolation processing as necessary to correct background inconsistencies resulting from changes to the text balloon.

[0798] Input: Image data placed in a text balloon

[0799] Output: Final image data with background interpolated

[0800] What it does: The server fills in the missing parts of the background and recreates a natural background. It logs the results of the filling process.

[0801] Step 8: User Emotion Recognition and Feedback

[0802] The device displays the edited image to the user and uses an emotion engine to analyze the user's emotions in real time from their facial expressions and voice, collecting emotional feedback from the user.

[0803] Input: Final image data with background interpolated

[0804] Output: User's emotional feedback data

[0805] Specific operation: The device collects the user's facial expressions and voice through the camera and microphone, analyzes them with an emotion engine, and displays and collects positive or negative feedback.

[0806] Step 9: Applying Feedback

[0807] The server automatically corrects the translated text and edits based on the collected user emotional feedback. If there is a lot of positive feedback, the server proceeds as is.

[0808] Input: User emotional feedback data

[0809] Output: Corrected translation text and edits

[0810] Specific behavior: The server corrects the translation in response to negative feedback and displays it again to the user. The corrected translation result is recorded in the log.

[0811] Step 10: Modify and save

[0812] The user then makes final confirmations and corrections while taking into account the emotional feedback. After the corrections are complete, the server stores the final translation and editing results in a database and provides them to the user.

[0813] Input: Final image data with positive feedback applied

[0814] Output: Final image data after user confirmation

[0815] Specific behavior: The user checks the data and if satisfied, clicks the "Save" button. A message indicating that the data has been saved is displayed.

[0816] Step 11: Provide data

[0817] The terminal displays the generated download link to the user, allowing the user to download the final data.

[0818] Input: Final saved image data

[0819] Output: A download link that users can access

[0820] Specific behavior: The device displays a download link, the user clicks the link to download the final data, and a confirmation message is displayed when the download is complete.

[0821] (Application example 2)

[0822] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0823] In multilingual manga translation, simply translating text into another language can result in the loss of context and nuance, leaving users dissatisfied. Manually correcting translation results can also lead to a poor user experience. Furthermore, improper adjustment of the position and size of text balloons can result in visual inconsistencies. To resolve these issues and provide high-quality translation results, a system is needed that incorporates user emotional feedback to improve translation accuracy.

[0824] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0825] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning based on context specific to manga, means for inserting the translated text into the image, means for completing the background accompanying the insertion, means for displaying the translation result and allowing the user to make corrections, means for analyzing the user's emotions in real time and reflecting the feedback, and means for saving the final image after correction and providing it to the user. This makes it possible to provide high-quality translation results that reflect the user's emotional feedback, improving the user experience.

[0826] "Image data" refers to files that digitally represent the visual information contained in manga pages and frames.

[0827] "Text extraction" refers to the process of identifying textual information from image data and extracting it as digital text.

[0828] "Target Language" means the language into which the Source Text is translated.

[0829] "Fine tuning" refers to the process of fine-tuning the initial translation result to suit a specific context or genre, improving the accuracy and quality of the translation.

[0830] "Insertion" refers to the process of placing translated text in a predetermined location in image data.

[0831] "Background completion" refers to the process of correcting missing or inconsistent backgrounds caused by text insertion, maintaining a natural appearance.

[0832] "Correction" refers to the act of the user checking the translation results and the content of the inserted text and making changes as necessary.

[0833] "Real-time emotion analysis" refers to the process of instantly determining the user's current emotional state based on their facial expressions and voice, and obtaining the results as data.

[0834] "Reflecting feedback" refers to the process of improving translation results and the overall operation of the system based on analyzed user emotional data.

[0835] "Final Image" refers to image data that has had all corrections and enhancements completed and is ready to be provided to the user.

[0836] "Preservation" refers to the process of digitally recording the final, corrected and enhanced image and storing it for reuse as needed.

[0837] This invention is a multilingual manga translation system that enables more accurate translation by incorporating user emotional feedback. The details of this system are described below.

[0838] Hardware and software used

[0839] The system uses a smartphone, a head-mounted display (HMD), a camera, and a microphone as hardware, and utilizes OCR technology (e.g., Tesseract), a generative AI model (e.g., OpenAI API, DeepL API), an emotion analysis engine (e.g., Microsoft Azure Emotion API), and a database (e.g., MySQL, PostgreSQL) as software.

[0840] System Overview

[0841] 1. Uploading image data

[0842] Users upload manga image data (e.g., image files or PDF files) to the system via their smartphone or HMD, which gives the system a starting point for the translation process.

[0843] 2. Image Preprocessing

[0844] The server checks the uploaded image data and performs resizing and format conversion for resolution and format, and in the case of PDF files, splits each page into a separate image file.

[0845] 3. Text Extraction

[0846] The server uses OCR technology (e.g., Tesseract) to extract text from the preprocessed image data, and the extracted text is stored in a database.

[0847] 4. Initial Translation

[0848] The server uses a generative AI model (e.g., OpenAI's API) to translate the extracted text into the target language, and the translation results are stored in a database.

[0849] 5. Fine Tuning

[0850] The server uses fine-tuning models specific to specific contexts and genres to improve the quality of the initial translation results, which are then stored in a database.

[0851] 6. Image Recognition and Editing

[0852] The server uses image recognition technology to detect the position and size of the text balloon and position the translated text appropriately, interpolating the background if necessary to maintain a natural look.

[0853] 7. Obtaining Emotional Feedback

[0854] The device uses a camera and microphone to analyze the user's real-time emotional data, such as facial expressions and voice while viewing the translation results, to obtain positive or negative feedback.

[0855] 8. Applying Feedback

[0856] The server automatically improves translation results and edited images based on the user's emotional feedback. If there is a lot of positive feedback, the translation is left as is, but if there is a lot of negative feedback, the translation is retranslated or revised.

[0857] Examples and prompts

[0858] For example, if a user inputs the phrase "This hero's sword defeats all enemies," it would be translated using a generative AI model as follows:

[0859] Translate this to English: This hero's sword will defeat all enemies

[0860] Based on this prompt, it would be translated as "This hero's sword defeats all enemies."

[0861] Process Overview

[0862] This system is characterized by a process that analyzes user sentiment in real time and automatically improves translation results based on that feedback. The combination of hardware and software makes it highly user-friendly and capable of providing high-quality multilingual translation.

[0863] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0864] Step 1:

[0865] A user logs in to the system and uploads manga image data (e.g., image files or PDF files). Specifically, the user selects image data using a smartphone or HMD and uploads it through the system interface. The input data is an image file, and the output is an image file stored on the server.

[0866] Step 2:

[0867] The server preprocesses the uploaded image data. Specifically, it checks the image resolution and format, resizes it, and converts the format. In the case of PDF files, it also splits each page into a separate image file. The input data is the uploaded image data, and the output is the preprocessed image file.

[0868] Step 3:

[0869] The server uses OCR technology (e.g., Tesseract) to extract text from the preprocessed image files. Specifically, the OCR engine analyzes characters in the image and extracts them as digital text. The input data is the preprocessed image file, and the output is the extracted text data.

[0870] Step 4:

[0871] The server uses a generative AI model (e.g., OpenAI's API) to translate the extracted text into the target language. Specifically, it sends a prompt to the generative AI model and obtains the translation result. The input data is the extracted text data, and the output is the initial translation result.

[0872] Step 5:

[0873] The server refines the initial translation using a fine-tuning model specific to a particular context or genre. Specifically, the translation is fed into a pre-trained model, which then performs context-based corrections. The input data is the initial translation, and the output is the fine-tuned translation.

[0874] Step 6:

[0875] The server uses image recognition technology to detect the position and size of the text balloon and place the translated text in the appropriate location. Specifically, an image analysis algorithm identifies the text balloon and inserts the new text in its place. The input data is the fine-tuned translation result and image data, and the output is the image with the inserted text.

[0876] Step 7:

[0877] The server performs background interpolation as needed to fix background inconsistencies caused by text balloon changes. Specifically, image interpolation techniques recreate the background to maintain a natural look. The input data is an image with text inserted, and the output is an image with the background interpolated.

[0878] Step 8:

[0879] The device displays the edited image to the user and uses an emotion engine to analyze emotions in real time from the user's facial expressions and voice. Specifically, the device collects the user's reactions through a camera and microphone, and the emotion analysis engine analyzes the data. The input data is the user's facial expressions and voice, and the output is analyzed emotional data.

[0880] Step 9:

[0881] The server automatically corrects the translated text and edited results based on the collected user emotional feedback. Specifically, if there is a lot of positive feedback, it proceeds to the next step, but if there is a lot of negative feedback, it retranslates or corrects the text. The input data is emotional data, and the output is the corrected translated text and edited results.

[0882] Step 10:

[0883] The user checks the correction results and makes final corrections. Specifically, the user checks the final image displayed on the device and manually makes corrections as necessary. The input data is the corrected translated text or edited results, and the output is the final translation result or edited image.

[0884] Step 11:

[0885] The server stores the final translation and editing results in a database and provides them to users. Specifically, it generates and provides a link that allows users to download the final images stored in the database. The input data is the final translation results and edited images, and the output is a download link that users can access.

[0886] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0887] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0888] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0889] [Third embodiment]

[0890] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0891] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0892] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0893] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0894] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0895] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0896] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0897] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0898] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0899] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0900] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0901] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0902] The present invention relates to a system for efficiently translating manga into multiple languages, which has the ability to extract text from image data, translate it, generate an image containing the translated text, and allow the user to make any necessary corrections.

[0903] 1. Upload your data

[0904] Users upload manga manuscript data (image files or PDFs) through the system's front end, which then sends the manga pages to the server.

[0905] Example: User clicks upload button in browser, selects file and submits.

[0906] 2. Image Preprocessing

[0907] The server preprocesses the received manga data. First, it checks the image resolution and resizes it if it is inappropriate. If it is a PDF file, it splits each page into a separate image file.

[0908] Example: The server checks the image resolution and resizes it to the optimal resolution if necessary.

[0909] 3. Text Extraction

[0910] The server uses OCR (Optical Character Recognition) technology to extract the text from each image file, which is then stored in a database.

[0911] Example: The server runs an OCR engine to identify the location and content of text balloons from an image.

[0912] 4. Initial Translation

[0913] The server uses generative AI to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[0914] Example: The server translates the extracted text "Hello, how are you?" into English, producing "Hello, how are you?"

[0915] 5. Fine Tuning

[0916] The server uses fine-tuned models tailored to specific manga genres and styles to refine the initial translation results, which are also stored in the database.

[0917] Example: The server inputs the initial translation "This is the hero's sword!" into a fine-tuning model, which generates the more contextually appropriate "This is the courage sword!"

[0918] 6. Image Recognition and Editing

[0919] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[0920] Example: The server replaces the original text "Run quickly!" with "Run quickly!" and adjusts the position and size of the balloon.

[0921] 7. Background Completion

[0922] The server compensates for inconsistencies in the background caused by inserting translated text, and automatically corrects the background so that it does not look unnatural when the text balloon is changed.

[0923] Example: The server regenerates the background of the balloon and inserts the translated text.

[0924] 8. User confirmation and correction

[0925] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[0926] Example: A user corrects the displayed translation "Run quickly!" to "Hurry up!"

[0927] 9. Saving and downloading final data

[0928] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[0929] Example: User completes verification and downloads the corrected image.

[0930] This system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections.

[0931] The processing flow will be explained below.

[0932] Step 1:

[0933] The user uploads the manga manuscript data (image files or PDF files) through the system's front end. When the user clicks the file selection button in the browser, selects a file from the local disk, and presses the upload button, the file is sent to the server.

[0934] Step 2:

[0935] The server preprocesses the manga data received by checking the image resolution and format, resizing and converting the image if necessary, and splitting each page into an individual image file if it is a PDF file.

[0936] Step 3:

[0937] The server uses OCR (Optical Character Recognition) technology to extract text from the preprocessed image files, specifically by identifying the location and content of text balloons on each page, and stores the extracted text in a database.

[0938] Step 4:

[0939] The server uses the generative AI to initially translate the extracted text into the target language, and the generated initial translation results are also stored in the database.

[0940] Step 5:

[0941] The server refines the initial translation using a fine-tuned model that is specialized for a specific manga genre and style. Specifically, the model takes the initial translation as input and generates a translation that is appropriate for the context and style, which is then stored in a database.

[0942] Step 6:

[0943] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted to insert the translated text.

[0944] Step 7:

[0945] The server automatically detects and corrects any unnatural background caused by changes to the text balloon, ensuring that the background remains natural.

[0946] Step 8:

[0947] The device displays the edited image on the user interface, and the user is provided with a UI to review the translation results and image edits, and manually make corrections if necessary.

[0948] Step 9:

[0949] After the user has completed the confirmation and correction, they issue an instruction to save the final image data, and the server saves the final image and generates a download link.

[0950] Step 10:

[0951] The terminal displays the generated download link to the user, allowing the user to download the final translation data with the corrections reflected.

[0952] This series of processes allows users to efficiently translate manga into multiple languages ​​with high quality.

[0953] Example 1

[0954] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0955] Previous multilingual manga translation systems faced challenges in the entire process, from text extraction to translation and image insertion, including translation quality, background completion, and the inability for users to easily make corrections. Furthermore, the lack of fine-tuning specialized for specific categories or styles made it difficult to translate appropriately for the context. As a result, problems arose, such as low translation quality and a heavy burden on users to make corrections.

[0956] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0957] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning specialized for a particular category or style, means for inserting the translated text into an image, means for completing the background accompanying the text insertion, means for displaying the translation result and allowing the user to make corrections, and means for saving the final image after corrections and providing it to the user. This enables high-quality multilingual translation and reduces the burden of corrections on the user.

[0958] "Image data" means digital data that stores visual information, including image files in JPEG and PNG formats and PDF files.

[0959] "Text extraction" is the process of recognizing characters and sentences from image data and obtaining their content as digital text.

[0960] The "target language" is the language into which you are translating, and the language into which you want to convert and display the original text.

[0961] "Fine tuning" is the process of making adjustments to improve accuracy, focusing on a particular category or style.

[0962] "Translated text" means text that has been converted from a source language into a target language.

[0963] "Image insertion" is the process of incorporating translated text into the original image data, including adjusting the position and size of the text.

[0964] "Background completion" is the process of correcting background inconsistencies that occur when inserting text, making it look natural.

[0965] "Translation result" is the text after translation and fine-tuning.

[0966] "Correction" refers to any change a user makes to the translation or image editing results provided by the system.

[0967] "Final image" refers to the final image data provided to the user after translation and corrections are complete.

[0968] The present invention relates to a system for efficiently translating manga into multiple languages, and the system has many functions for text extraction, translation, image generation, and user correction. Specific embodiments for implementing the invention are described below.

[0969] Hardware and Software Used

[0970] server

[0971] The server is implemented in a hardware environment with high-performance computing capabilities (e.g., a virtual server provided by a cloud provider). The server includes the following software components:

[0972] OCR engine: Uses, for example, Google Cloud Vision API to perform optical character recognition.

[0973] Generative AI models: for example, using OpenAI GPT-4 for translation into the target language and fine-tuning.

[0974] Image processing libraries: For example, OpenCV or PyMuPDF are used to check image resolution and resize, split PDFs, analyze text balloon positions, etc.

[0975] Terminal

[0976] A terminal is a device (such as a personal computer or tablet) that allows a user to interact with the system. The terminal implements the system's front end and provides the interface that the user operates.

[0977] User

[0978] The user is the end user who operates the system, and is responsible for providing the manga manuscript data and checking and correcting the results of the translation and corrections.

[0979] Data processing and calculation

[0980] 1. Text Extraction

[0981] The server uses an OCR engine to extract text from the image data, for example, using the Google Cloud Vision API, to identify text balloons in the image and store their locations and contents in a database.

[0982] 2. Initial Translation

[0983] The server translates the extracted text into the target language using a generative AI model (e.g., OpenAI GPT-4). This translation result is again stored in the database. An example of a prompt sentence is a prompt that translates "Hello, how are you?" into "Hello, how are you?"

[0984] 3. Fine Tuning

[0985] The server uses fine-tuning models that are specialized for specific manga categories and styles to refine the initial translation results, for example, refining the hero's line "This is the hero's sword!" to "This is the courage sword!"

[0986] 4. Image Recognition and Editing

[0987] The server uses image recognition technology to insert the translated text into the image appropriately, specifically by adjusting the position and size of the text balloon using OpenCV.

[0988] 5. Background Completion

[0989] The server uses a technique to regenerate the background portion to compensate for the inconsistency of the background caused by the text insertion.

[0990] 6. User confirmation and correction

[0991] The terminal displays the translation results and the status of the image editing to the user. The user can then modify the translation and image editing status as needed, and the modifications are immediately sent to the server.

[0992] Specific examples

[0993] For example, a user may upload a comic page and provide an image containing the text "Hello, how are you?" The server receives this image data, uses an OCR engine to extract the text from the text balloon, and stores the results in a database.

[0994] The server then uses a generative AI model to translate the extracted text into English, for example, "Hello, how are you?" is translated to "Hello, how are you?", and then further refines the translation results using a fine-tuning model suited to the manga category.

[0995] The server places this translated text in the appropriate text balloon and fills in any inconsistencies in the background. Finally, the device displays the edited results to the user, who can then modify the translation as needed. The modified results are then sent to the server and stored in a database.

[0996] This system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections.

[0997] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0998] Step 1:

[0999] Uploading data

[1000] Users upload manga manuscript data through the system's front end, providing image files (JPEG, PNG, etc.) or PDF files as input.

[1001] The server processes the received file using an HTTP POST request, and as output, the received manuscript data is saved on the server.

[1002] Step 2:

[1003] Image preprocessing

[1004] The server checks the image resolution of the received manga data and resizes it if it is inappropriate, and if it is a PDF file, splits each page into a separate image file.

[1005] It uses saved original data (image or PDF) as input, and generates an image file resized to the appropriate resolution or split into individual image files as output.

[1006] Specific operations include image resizing using the OpenCV library and splitting the PDF using PyMuPDF.

[1007] Step 3:

[1008] Text Extraction

[1009] The server uses an OCR engine (e.g., Google Cloud Vision API) to extract text from each image file, which is then stored in a database.

[1010] As input, it uses preprocessed image files, and as output, it stores extracted text information and its location data in a database.

[1011] Specific operations include sending an image file to an OCR engine and storing the returned text information in a database.

[1012] Step 4:

[1013] Initial translation

[1014] The server uses a generative AI model (e.g., OpenAI GPT-4) to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[1015] As input, it uses extracted text stored in a database, and as output, the translation results into the target language are stored in a database.

[1016] Specifically, the generative AI model is provided with the extracted text and uses the prompt sentence to obtain the translation result, for example, translating "Hello, how are you?" into "Hello, how are you?"

[1017] Step 5:

[1018] Fine Tuning

[1019] The server uses fine-tuning models specialized for specific manga categories and styles to refine the initial translation results, which are also stored in the database.

[1020] As input, it uses the initial translation results, and as output, the fine-tuned translation results are stored in a database.

[1021] Specifically, it provides the initial translation text to the fine-tuning model, which then generates a more natural and context-appropriate translation result, for example, refining "This is the hero's sword!" to "This is the courage sword!".

[1022] Step 6:

[1023] Image Recognition and Editing

[1024] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[1025] It takes the fine-tuned translation results and the original image file as input, and generates a new image file with the translated text inserted as output.

[1026] Specific actions include using OpenCV to reparse the position and size of the text balloon and insert text, e.g., replacing "Run quickly!" with "Run quickly!".

[1027] Step 7:

[1028] Background Complement

[1029] The server compensates for inconsistencies in the background caused by inserting translated text, and automatically corrects the background so that it does not look unnatural when the text balloon is changed.

[1030] As input, it takes the image file after the text has been inserted, and as output, it generates a new image file with the background filled in with inconsistencies.

[1031] Specifically, it uses generative AI technology and image correction algorithms to recreate the background around the balloon in a natural shape.

[1032] Step 8:

[1033] User confirmation and correction

[1034] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[1035] As input, it takes an image file with the translated text inserted, and as output, it generates the final image data with the user's modifications reflected.

[1036] Specifically, the terminal provides an interface for editing, creating an environment where users can edit directly. The edited data is immediately reflected on the server.

[1037] Step 9:

[1038] Save and download the final data

[1039] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[1040] It takes the image file with the user's modifications as input, and saves the final image data and generates a download link as output.

[1041] Specifically, the modified image file is saved in the database and a URL that can be accessed by the user is created.

[1042] (Application example 1)

[1043] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1044] Multilingual distribution of manga requires efficient translation into different languages. Conventional methods require manual translation and editing, which is extremely time-consuming and laborious. Furthermore, there are often inconsistencies in the background of images when translating text, and the environment for easy user corrections is often inadequate. This makes it difficult for users to smoothly enjoy manga in multiple languages.

[1045] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1046] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning based on context specific to manga, means for inserting the translated text into the image, means for completing the background associated with the insertion, means for displaying the translation result and allowing the user to make corrections, means for saving the final image after corrections and providing it to the user, and means for providing a terminal application for viewing manga in multiple languages. This allows manga text to be translated efficiently into multiple languages, enabling users to easily make corrections and view the text.

[1047] "Image data" is an electronic representation of a comic page or frame.

[1048] "Text" refers to character information contained in image data.

[1049] "Target language" refers to the language into which the text is translated.

[1050] "Fine tuning" refers to adjusting the model to improve the accuracy of machine translation based on the specific context of manga.

[1051] "Background completion" is the process of correcting background inconsistencies that occur when inserting translated text into an image.

[1052] "User" refers to a person who uses the system to translate and view manga in multiple languages.

[1053] "Device application" refers to software that runs on mobile devices such as smartphones and tablets.

[1054] "Extracted text" refers to character information obtained from image data using OCR technology.

[1055] A "text balloon" refers to a speech bubble used to write lines or narration within a manga.

[1056] The "final image" is the final image data after the text has been translated and completed and the user's corrections have been reflected.

[1057] The present invention relates to a system for efficiently translating manga into multiple languages. The system extracts text from image data, translates it, generates an image containing the translated text, and allows the user to make any necessary corrections.

[1058] First, the user uploads the manga image data (image file or PDF) via their device. This image data is then sent to the server. The server preprocesses the received manga data, inspects the image resolution, and resizes it if it is inappropriate. In the case of a PDF file, the server also splits each page into a separate image file.

[1059] The server then uses OCR technology to extract text from each image file, identify the location and content of the text balloons, and store the extracted text in a database. Specifically, this can be done using an OCR engine such as Pytesseract.

[1060] The server then uses a generative AI model to translate the extracted text into the target language. This initial translation result is also stored in the database. For example, the extracted text "Hello, how are you?" can be translated into English to produce "Hello, how are you?"

[1061] Additionally, the server uses fine-tuned models tailored to specific manga genres and styles to refine the initial translation results, resulting in more context-appropriate translations. For example, it takes the initial translation "This is the hero's sword!" and generates the more context-appropriate "This is the courage sword!".

[1062] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image. The size and position of the text balloon are also automatically adjusted. This compensates for inconsistencies in the background and eliminates the awkwardness that comes with inserting the translated text. This allows the original text "Run quickly!" to be replaced with "Run quickly!" and the position and size of the balloon to be adjusted.

[1063] The user can then check the translation results displayed on the device and make corrections as necessary. These corrections are also sent to the server and reflected in the final image data. For example, the user can change the displayed translation "Run quickly!" to "Hurry up!"

[1064] Finally, the server saves the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download. The user can click on this link to download the high-quality multilingual translated manga.

[1065] The implementation of this system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections. An example of a translation and prompt using a generative AI model is "Please translate the following Japanese text into English. Text: 'Hello, how are you?'"

[1066] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1067] Step 1:

[1068] Users upload manga image data (image files or PDFs) to the system from their devices. Specifically, they click the upload button in their browser or application, select the file, and send it to the server.

[1069] Input: Comic image data file (PNG, JPEG, PDF, etc.) selected by the user.

[1070] Output: Image data of the manga sent to the server.

[1071] Step 2:

[1072] The server preprocesses the image data it receives, checking the image resolution and resizing it if it is inappropriate, and, in the case of PDF files, splitting each page into a separate image file.

[1073] Input: Image data received by the server.

[1074] Output: Image data resized to optimal resolution and PDF pages split into individual image files.

[1075] What it does: Uses libraries such as OpenCV to check image resolution and resize.

[1076] Step 3:

[1077] The server uses OCR technology to extract text from each image file, and the extracted text and its location are stored in a database.

[1078] Input: The resized image data.

[1079] Output: Extracted text data and text location information.

[1080] What it does: Uses an OCR engine such as Pytesseract to extract the location and content of text balloons from an image file.

[1081] Step 4:

[1082] The server uses a generative AI model to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[1083] Input: Extracted text data.

[1084] Output: The translated text data.

[1085] What it does: Uses a translation engine (e.g., Google Translate API) to translate text data into the target language.

[1086] Step 5:

[1087] The server uses fine-tuned models that are specialized for specific manga genres and styles to refine the initial translation results, which are then stored in a database.

[1088] Input: Text data of the initial translation result.

[1089] Output: Fine-tuned translated text data.

[1090] How it works: Generates context-appropriate translations using a generative AI model specialized for the manga genre.

[1091] Step 6:

[1092] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[1093] Input: Fine-tuned translation text data, image data.

[1094] Output: Image data with inserted text.

[1095] What it does: Uses OpenCV and PIL libraries to insert text into an image and adjust the position and size of the text balloon.

[1096] Step 7:

[1097] The server compensates for inconsistencies in the background caused by the insertion of translated text.

[1098] Input: Image data with inserted text.

[1099] Output: Image data with background interpolated.

[1100] How it works: It uses image editing algorithms to naturally complement the background of the text insert.

[1101] Step 8:

[1102] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[1103] Input: The translated and completed image data.

[1104] Output: Image data with user modifications.

[1105] What it does: Displays the image through the device's interface and allows the user to make corrections. The user enters the corrections and submits them.

[1106] Step 9:

[1107] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[1108] Input: Image data with user modifications.

[1109] Output: Final image data provided via a download link.

[1110] Behavior: Saves the image data and sends a notification to the user providing a download link.

[1111] This allows users to quickly and efficiently translate manga into multiple languages. An example of a translation and prompt using a generative AI model is: "Please translate the following Japanese text into English. Text: 'Hello, how are you?'"

[1112] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1113] This invention relates to a system that combines an emotion engine with an efficient system for translating manga into multiple languages, thereby reflecting user emotional feedback and improving the accuracy of translation and editing. This system extracts text from image data, translates it into the target language, and provides an environment where users can check and correct the results, while analyzing user emotions in real time, enabling more user-friendly operation.

[1114] 1. Upload your data

[1115] The user uploads the manga manuscript data (image file or PDF file) from the system's front end. The file is selected and sent to the server.

[1116] Example: A user clicks the file chooser button in their browser to select and upload a file from their local disk.

[1117] 2. Image Preprocessing

[1118] The server pre-processes the uploaded manga data, inspecting the image resolution and format and resizing or formatting it as needed. In the case of PDF files, each page is split into a separate image file.

[1119] Example: The server checks the image resolution and resizes it if it is inappropriate.

[1120] 3. Text Extraction

[1121] The server uses OCR (Optical Character Recognition) technology to extract text from the preprocessed image files, which is then stored in a database.

[1122] Example: The server identifies the location and content of text balloons on each page and stores them in a database.

[1123] 4. Initial Translation

[1124] The server uses generative AI to perform an initial translation of the extracted text into the target language, and the translation results are stored in a database.

[1125] Example: The server translates the text "Hello, how are you?" to "Hello, how are you?"

[1126] 5. Fine Tuning

[1127] The server uses fine-tuning models specific to a particular genre or style to refine the initial translation results, which are then stored in a database.

[1128] Example: The server refines "This is the hero's sword!" to "This is the courage sword!"

[1129] 6. Image Recognition and Editing

[1130] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted to insert the text.

[1131] Example: The server properly places "Run quickly!" into "Run quickly!"

[1132] 7. Background Completion

[1133] The server will interpolate the background as needed to correct any inconsistencies in the background caused by changes to the text balloon, thus maintaining a natural look.

[1134] Example: The server fills in missing parts of the background and recreates a natural background.

[1135] 8. User Emotion Recognition and Feedback

[1136] The device displays the edited image to the user and uses an emotion engine to analyze the user's emotions in real time based on their facial expressions and voice. Based on the analysis results, the device collects user feedback on the translation results and the edited image.

[1137] Example: The device uses the user's camera and microphone to analyze their reactions and use an emotion engine to collect positive or negative feedback.

[1138] 9. Applying Feedback

[1139] The server has the ability to automatically correct the translated text and edits based on the collected user emotional feedback. If the feedback is mostly positive, the server proceeds to the next step.

[1140] Example: The server responds to negative feedback by correcting the translation and redisplaying it.

[1141] 10. Modifying and Saving

[1142] The user makes final confirmations and corrections while taking into account the emotional feedback. After corrections are complete, the server stores the final translation and editing results in a database and provides them to the user.

[1143] Example: The user completes the review, and if the result is satisfactory, completes the corrections and clicks the save button.

[1144] 11. Data Provision

[1145] The terminal displays the generated download link to the user, allowing the user to download the final data.

[1146] Example: The device displays a download link and the user downloads the modified results.

[1147] This system enables high-quality multilingual translation of manga that incorporates user emotional feedback, improving the user experience.

[1148] The processing flow will be explained below.

[1149] Step 1:

[1150] The user uploads the manga manuscript data (image file or PDF file) from the system's front end. The user clicks the file selection button in the browser, selects the file from the local disk, and presses the upload button, which sends the file to the server.

[1151] Step 2:

[1152] The server preprocesses the manga data received. Specifically, it checks the image resolution and format, resizes and converts the format as needed, and splits each page into a separate image file if it is a PDF file.

[1153] Step 3:

[1154] The server extracts text from the preprocessed image files using OCR (Optical Character Recognition) technology, identifies the location and content of text balloons, and stores the extracted text in a database.

[1155] Step 4:

[1156] The server uses generative AI to perform an initial translation of the extracted text into the target language, and the translation results are stored in a database.

[1157] Step 5:

[1158] The server uses fine-tuning models specialized for specific manga genres and styles to improve the initial translation results, which are then stored in a database.

[1159] Step 6:

[1160] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted and the text is inserted.

[1161] Step 7:

[1162] The server performs background interpolation, correcting inconsistencies in the background caused by text balloon changes to maintain a natural look.

[1163] Step 8:

[1164] The device displays the edited image to the user, and uses an emotion engine to analyze the user's emotions in real time from their facial expressions and voice. Based on the analysis results, the device collects user feedback on the translation results and the edited image.

[1165] Step 9:

[1166] The server automatically corrects the translated text and edits based on the user's emotional feedback collected. If the feedback is positive, the system proceeds to the next step, but if the feedback is negative, the system makes appropriate corrections.

[1167] Step 10:

[1168] The user then makes final confirmations and corrections while taking into account the emotional feedback. After corrections are complete, the server saves the final translation and edited results in a database.

[1169] Step 11:

[1170] The terminal displays the generated download link to the user, allowing the user to download the final data. The user uses this link to obtain the final translation result.

[1171] Example 2

[1172] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1173] Conventional systems for multilingual translation have issues with translation accuracy and the complexity of editing operations. They also face the problem of not being able to properly reflect user emotions and feedback, resulting in a poor user experience. The present invention aims to solve these issues by providing an efficient system that provides high-quality translation results and reflects user emotional feedback.

[1174] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data uploading means, a means for preprocessing uploaded image data, a means for extracting text from image data using OCR technology, a means for translating the extracted text into a target language using a generative AI model, a means for fine-tuning based on genre and style, a means for inserting the translated text into an image, a means for completing the background associated with the insertion, a means for displaying the edited image to the user and collecting user emotional feedback using a sentiment analysis engine, a means for correcting the translation result and editing result based on the collected emotional feedback, and a means for saving the final corrected image and providing it to the user. This makes it possible to efficiently perform high-quality translations with a user-friendly interface and reflect the user's emotional feedback.

[1175] "Data upload" refers to a user sending manga manuscript data (image file or PDF file) to the server from the system's front end.

[1176] "Preprocessing" refers to the process of checking the resolution and format of uploaded image data and resizing or format conversion as necessary.

[1177] "OCR technology" refers to optical character recognition technology that extracts text from image data.

[1178] "Generative AI Model" refers to the artificial intelligence model used to translate extracted text into a target language.

[1179] "Fine-tuning" refers to the process of improving the initial translation results based on a specific genre or style.

[1180] "Inserting translated text" refers to the process of placing translated text at a specified location within the original image.

[1181] "Background completion" refers to the process of correcting background inconsistencies that occur during the process of inserting translated text, resulting in a natural look.

[1182] An "emotion analysis engine" refers to technology that displays edited images to the user and analyzes emotions in real time from the user's facial expressions and voice.

[1183] "Emotional feedback" refers to feedback information on translation results and edited images that is collected based on the results of user emotion analysis.

[1184] "Saving the final image" refers to the process of saving the final translation and editing results after corrections in a database and providing them to the user.

[1185] The present invention provides a system for efficiently translating manga into multiple languages. The system aims to improve the accuracy of translation and editing by incorporating user emotional feedback. Specific embodiments are described below.

[1186] First, the user uploads the manga manuscript data (image files or PDF files) using the system's front-end interface. To do this, the user clicks the file selection button in the browser, selects the required file from the local disk, and presses the upload button.

[1187] The uploaded data is then sent to the server, which begins pre-processing the data. At this stage, the server inspects the image resolution and format and resizes or converts it if necessary. For example, in the case of a PDF file, it splits each page into a separate image file.

[1188] After preprocessing is complete, the server uses OCR technology (e.g., Tesseract or Google Cloud Vision API) to extract text from the image data. The extracted text is stored in a database. For example, the server identifies the location and content of text balloons on each page and stores this information in the database.

[1189] The extracted text is then translated into the target language by a generative AI model (e.g., GPT-3) on the server. This translation result is also stored in a database. For example, the server translates the text "Hello, how are you?" into "Hello, how are you?". An example of a prompt used in this process is "Please translate this text 'Hello, how are you?' into English."

[1190] The server then uses fine-tuning models based on specific genres and styles to refine the initial translation. These refined translations are also stored in the database. For example, the server might refine "This is the hero's sword!" to "This is the courage sword!"

[1191] After obtaining the translation result, the server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text in the appropriate position. During this process, the position and size of the balloon are automatically adjusted and the translated text is inserted. For example, the server will properly place the text "Run quickly!" into "Run quickly!"

[1192] Additionally, the server performs background interpolation as needed to correct any inconsistencies in the background caused by changes to the text balloon, ensuring that the overall image maintains a natural look. For example, missing parts of the background are interpolated to recreate a natural background.

[1193] The device then displays the edited image to the user and uses an emotion engine (e.g., Affectiva) to analyze the user's facial expressions and voice in real time to determine their emotions. Based on the analysis results, the device collects user feedback on the translation results and edited image. For example, the device can use the emotion engine to analyze data collected through the user's camera and microphone to collect positive or negative feedback.

[1194] The collected user emotional feedback is used by the server to automatically correct the translated text and edited results. If there is a lot of positive feedback, the server proceeds to the next step, but if there is negative feedback, the translation is corrected and redisplayed.

[1195] Finally, the user makes final confirmation and corrections while taking into account the emotional feedback. After corrections are complete, the server saves the final translation and editing results in a database and provides them to the user. For example, once the user has completed their confirmation and is satisfied with the results, they can click the "Save" button.

[1196] Finally, the terminal displays the generated download link to the user, allowing the user to download the final data. For example, the terminal displays the download link, and the user clicks the link to download the final data.

[1197] Through the above steps, the present invention realizes a system that can improve the accuracy of multilingual translation and enhance the user experience.

[1198] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1199] Step 1: Upload your data

[1200] Users use the front-end interface in their browser to select and upload manga manuscript data (image files or PDF files) from their local disk, which then sends the manga data to the server.

[1201] Input: Manga manuscript data (image file or PDF file)

[1202] Output: Manuscript data sent to the server

[1203] What happens: The user clicks the file selection button to select a file, then clicks the "Upload" button to send the file to the server, and a progress bar appears to show the upload progress.

[1204] Step 2: Image preprocessing

[1205] The server checks the resolution and format of the uploaded manuscript and performs resizing and formatting as needed. In the case of PDF files, each page is split into a separate image file.

[1206] Input: Manuscript data sent to the server

[1207] Output: Preprocessed image data

[1208] Specific operation: The server checks the resolution and resizes it if it is low. In the case of PDF files, each page is split into an image file. The processing result is recorded in the log.

[1209] Step 3: Text Extraction

[1210] The server uses OCR technology to extract text from the preprocessed image files and stores it in a database.

[1211] Input: Preprocessed image data

[1212] Output: Extracted text data

[1213] Specific operation: The server uses OCR technology (e.g., Tesseract or Google Cloud Vision API) to extract text from images, saves the extracted results in a database, and displays a preview in the log.

[1214] Step 4: Initial translation

[1215] The server uses the generative AI model to perform an initial translation of the extracted text into the target language, and stores the translation results in a database.

[1216] Input: Extracted text data

[1217] Output: Initial translated text data

[1218] Specific operation: The server translates the Japanese text "Hello, how are you?" into English "Hello, how are you?". The prompt used and the translation result are recorded in the database and in the log.

[1219] Step 5: Fine Tuning

[1220] The server refines the initial translation results with fine-tuning models based on specific genres and styles, and stores the refined translation results in a database.

[1221] Input: Initial translated text data

[1222] Output: Improved translation text data

[1223] What happens: The server refines the translation of "This is the hero's sword!" to "This is the courage sword!". The fine-tuning results are saved in the database and logged.

[1224] Step 6: Image Recognition and Editing

[1225] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text in the appropriate location. The balloon's position and size are automatically adjusted to insert the translated text.

[1226] Input: Improved translation text data

[1227] Output: Image data neatly placed in a text balloon

[1228] What it does: The server detects the position and size of the text balloon in the image, places the translated text "Run quickly!" in the appropriate position, and generates an image. At this time, it also fine-tunes the size of the text balloon.

[1229] Step 7: Filling in the background

[1230] The server performs background interpolation processing as necessary to correct background inconsistencies resulting from changes to the text balloon.

[1231] Input: Image data placed in a text balloon

[1232] Output: Final image data with background interpolated

[1233] What it does: The server fills in the missing parts of the background and recreates a natural background. It logs the results of the filling process.

[1234] Step 8: User Emotion Recognition and Feedback

[1235] The device displays the edited image to the user and uses an emotion engine to analyze the user's emotions in real time from their facial expressions and voice, collecting emotional feedback from the user.

[1236] Input: Final image data with background interpolated

[1237] Output: User's emotional feedback data

[1238] Specific operation: The device collects the user's facial expressions and voice through the camera and microphone, analyzes them with an emotion engine, and displays and collects positive or negative feedback.

[1239] Step 9: Applying Feedback

[1240] The server automatically corrects the translated text and edits based on the collected user emotional feedback. If there is a lot of positive feedback, the server proceeds as is.

[1241] Input: User emotional feedback data

[1242] Output: Corrected translation text and edits

[1243] Specific behavior: The server corrects the translation in response to negative feedback and displays it again to the user. The corrected translation result is recorded in the log.

[1244] Step 10: Modify and save

[1245] The user then makes final confirmations and corrections while taking into account the emotional feedback. After the corrections are complete, the server stores the final translation and editing results in a database and provides them to the user.

[1246] Input: Final image data with positive feedback applied

[1247] Output: Final image data after user confirmation

[1248] Specific behavior: The user checks the data and if satisfied, clicks the "Save" button. A message indicating that the data has been saved is displayed.

[1249] Step 11: Provide data

[1250] The terminal displays the generated download link to the user, allowing the user to download the final data.

[1251] Input: Final saved image data

[1252] Output: A download link that users can access

[1253] Specific behavior: The device displays a download link, the user clicks the link to download the final data, and a confirmation message is displayed when the download is complete.

[1254] (Application example 2)

[1255] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1256] In multilingual manga translation, simply translating text into another language can result in the loss of context and nuance, leaving users dissatisfied. Manually correcting translation results can also lead to a poor user experience. Furthermore, improper adjustment of the position and size of text balloons can result in visual inconsistencies. To resolve these issues and provide high-quality translation results, a system is needed that incorporates user emotional feedback to improve translation accuracy.

[1257] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1258] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning based on context specific to manga, means for inserting the translated text into the image, means for completing the background accompanying the insertion, means for displaying the translation result and allowing the user to make corrections, means for analyzing the user's emotions in real time and reflecting the feedback, and means for saving the final image after correction and providing it to the user. This makes it possible to provide high-quality translation results that reflect the user's emotional feedback, improving the user experience.

[1259] "Image data" refers to files that digitally represent the visual information contained in manga pages and frames.

[1260] "Text extraction" refers to the process of identifying textual information from image data and extracting it as digital text.

[1261] "Target Language" means the language into which the Source Text is translated.

[1262] "Fine tuning" refers to the process of fine-tuning the initial translation result to suit a specific context or genre, improving the accuracy and quality of the translation.

[1263] "Insertion" refers to the process of placing translated text in a predetermined location in image data.

[1264] "Background completion" refers to the process of correcting missing or inconsistent backgrounds caused by text insertion, maintaining a natural appearance.

[1265] "Correction" refers to the act of the user checking the translation results and the content of the inserted text and making changes as necessary.

[1266] "Real-time emotion analysis" refers to the process of instantly determining the user's current emotional state based on their facial expressions and voice, and obtaining the results as data.

[1267] "Reflecting feedback" refers to the process of improving translation results and the overall operation of the system based on analyzed user emotional data.

[1268] "Final Image" refers to image data that has had all corrections and enhancements completed and is ready to be provided to the user.

[1269] "Preservation" refers to the process of digitally recording the final, corrected and enhanced image and storing it for reuse as needed.

[1270] This invention is a multilingual manga translation system that enables more accurate translation by incorporating user emotional feedback. The details of this system are described below.

[1271] Hardware and software used

[1272] The system uses a smartphone, a head-mounted display (HMD), a camera, and a microphone as hardware, and utilizes OCR technology (e.g., Tesseract), a generative AI model (e.g., OpenAI API, DeepL API), an emotion analysis engine (e.g., Microsoft Azure Emotion API), and a database (e.g., MySQL, PostgreSQL) as software.

[1273] System Overview

[1274] 1. Uploading image data

[1275] Users upload manga image data (e.g., image files or PDF files) to the system via their smartphone or HMD, which gives the system a starting point for the translation process.

[1276] 2. Image Preprocessing

[1277] The server checks the uploaded image data and performs resizing and format conversion for resolution and format, and in the case of PDF files, splits each page into a separate image file.

[1278] 3. Text Extraction

[1279] The server uses OCR technology (e.g., Tesseract) to extract text from the preprocessed image data, and the extracted text is stored in a database.

[1280] 4. Initial Translation

[1281] The server uses a generative AI model (e.g., OpenAI's API) to translate the extracted text into the target language, and the translation results are stored in a database.

[1282] 5. Fine Tuning

[1283] The server uses fine-tuning models specific to specific contexts and genres to improve the quality of the initial translation results, which are then stored in a database.

[1284] 6. Image Recognition and Editing

[1285] The server uses image recognition technology to detect the position and size of the text balloon and position the translated text appropriately, interpolating the background if necessary to maintain a natural look.

[1286] 7. Obtaining Emotional Feedback

[1287] The device uses a camera and microphone to analyze the user's real-time emotional data, such as facial expressions and voice while viewing the translation results, to obtain positive or negative feedback.

[1288] 8. Applying Feedback

[1289] The server automatically improves translation results and edited images based on the user's emotional feedback. If there is a lot of positive feedback, the translation is left as is, but if there is a lot of negative feedback, the translation is retranslated or revised.

[1290] Examples and prompts

[1291] For example, if a user inputs the phrase "This hero's sword defeats all enemies," it would be translated using a generative AI model as follows:

[1292] Translate this to English: This hero's sword will defeat all enemies

[1293] Based on this prompt, it would be translated as "This hero's sword defeats all enemies."

[1294] Process Overview

[1295] This system is characterized by a process that analyzes user sentiment in real time and automatically improves translation results based on that feedback. The combination of hardware and software makes it highly user-friendly and capable of providing high-quality multilingual translation.

[1296] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1297] Step 1:

[1298] A user logs in to the system and uploads manga image data (e.g., image files or PDF files). Specifically, the user selects image data using a smartphone or HMD and uploads it through the system interface. The input data is an image file, and the output is an image file stored on the server.

[1299] Step 2:

[1300] The server preprocesses the uploaded image data. Specifically, it checks the image resolution and format, resizes it, and converts the format. In the case of PDF files, it also splits each page into a separate image file. The input data is the uploaded image data, and the output is the preprocessed image file.

[1301] Step 3:

[1302] The server uses OCR technology (e.g., Tesseract) to extract text from the preprocessed image files. Specifically, the OCR engine analyzes characters in the image and extracts them as digital text. The input data is the preprocessed image file, and the output is the extracted text data.

[1303] Step 4:

[1304] The server uses a generative AI model (e.g., OpenAI's API) to translate the extracted text into the target language. Specifically, it sends a prompt to the generative AI model and obtains the translation result. The input data is the extracted text data, and the output is the initial translation result.

[1305] Step 5:

[1306] The server refines the initial translation using a fine-tuning model specific to a particular context or genre. Specifically, the translation is fed into a pre-trained model, which then performs context-based corrections. The input data is the initial translation, and the output is the fine-tuned translation.

[1307] Step 6:

[1308] The server uses image recognition technology to detect the position and size of the text balloon and place the translated text in the appropriate location. Specifically, an image analysis algorithm identifies the text balloon and inserts the new text in its place. The input data is the fine-tuned translation result and image data, and the output is the image with the inserted text.

[1309] Step 7:

[1310] The server performs background interpolation as needed to fix background inconsistencies caused by text balloon changes. Specifically, image interpolation techniques recreate the background to maintain a natural look. The input data is an image with text inserted, and the output is an image with the background interpolated.

[1311] Step 8:

[1312] The device displays the edited image to the user and uses an emotion engine to analyze emotions in real time from the user's facial expressions and voice. Specifically, the device collects the user's reactions through a camera and microphone, and the emotion analysis engine analyzes the data. The input data is the user's facial expressions and voice, and the output is analyzed emotional data.

[1313] Step 9:

[1314] The server automatically corrects the translated text and edited results based on the collected user emotional feedback. Specifically, if there is a lot of positive feedback, it proceeds to the next step, but if there is a lot of negative feedback, it retranslates or corrects the text. The input data is emotional data, and the output is the corrected translated text and edited results.

[1315] Step 10:

[1316] The user checks the correction results and makes final corrections. Specifically, the user checks the final image displayed on the device and manually makes corrections as necessary. The input data is the corrected translated text or edited results, and the output is the final translation result or edited image.

[1317] Step 11:

[1318] The server stores the final translation and editing results in a database and provides them to users. Specifically, it generates and provides a link that allows users to download the final images stored in the database. The input data is the final translation results and edited images, and the output is a download link that users can access.

[1319] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1320] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1321] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1322] [Fourth embodiment]

[1323] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1324] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1325] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1326] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1327] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1328] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1329] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1330] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1331] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1332] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1333] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1334] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1335] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1336] The present invention relates to a system for efficiently translating manga into multiple languages, which has the ability to extract text from image data, translate it, generate an image containing the translated text, and allow the user to make any necessary corrections.

[1337] 1. Upload your data

[1338] Users upload manga manuscript data (image files or PDFs) through the system's front end, which then sends the manga pages to the server.

[1339] Example: User clicks upload button in browser, selects file and submits.

[1340] 2. Image Preprocessing

[1341] The server preprocesses the received manga data. First, it checks the image resolution and resizes it if it is inappropriate. If it is a PDF file, it splits each page into a separate image file.

[1342] Example: The server checks the image resolution and resizes it to the optimal resolution if necessary.

[1343] 3. Text Extraction

[1344] The server uses OCR (Optical Character Recognition) technology to extract the text from each image file, which is then stored in a database.

[1345] Example: The server runs an OCR engine to identify the location and content of text balloons from an image.

[1346] 4. Initial Translation

[1347] The server uses generative AI to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[1348] Example: The server translates the extracted text "Hello, how are you?" into English, producing "Hello, how are you?"

[1349] 5. Fine Tuning

[1350] The server uses fine-tuned models tailored to specific manga genres and styles to refine the initial translation results, which are also stored in the database.

[1351] Example: The server inputs the initial translation "This is the hero's sword!" into a fine-tuning model, which generates the more contextually appropriate "This is the courage sword!"

[1352] 6. Image Recognition and Editing

[1353] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[1354] Example: The server replaces the original text "Run quickly!" with "Run quickly!" and adjusts the position and size of the balloon.

[1355] 7. Background Completion

[1356] The server compensates for inconsistencies in the background caused by inserting translated text, and automatically corrects the background so that it does not look unnatural when the text balloon is changed.

[1357] Example: The server regenerates the background of the balloon and inserts the translated text.

[1358] 8. User confirmation and correction

[1359] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[1360] Example: A user corrects the displayed translation "Run quickly!" to "Hurry up!"

[1361] 9. Saving and downloading final data

[1362] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[1363] Example: User completes verification and downloads the corrected image.

[1364] This system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections.

[1365] The processing flow will be explained below.

[1366] Step 1:

[1367] The user uploads the manga manuscript data (image files or PDF files) through the system's front end. When the user clicks the file selection button in the browser, selects a file from the local disk, and presses the upload button, the file is sent to the server.

[1368] Step 2:

[1369] The server preprocesses the manga data received by checking the image resolution and format, resizing and converting the image if necessary, and splitting each page into an individual image file if it is a PDF file.

[1370] Step 3:

[1371] The server uses OCR (Optical Character Recognition) technology to extract text from the preprocessed image files, specifically by identifying the location and content of text balloons on each page, and stores the extracted text in a database.

[1372] Step 4:

[1373] The server uses the generative AI to initially translate the extracted text into the target language, and the generated initial translation results are also stored in the database.

[1374] Step 5:

[1375] The server refines the initial translation using a fine-tuned model that is specialized for a specific manga genre and style. Specifically, the model takes the initial translation as input and generates a translation that is appropriate for the context and style, which is then stored in a database.

[1376] Step 6:

[1377] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted to insert the translated text.

[1378] Step 7:

[1379] The server automatically detects and corrects any unnatural background caused by changes to the text balloon, ensuring that the background remains natural.

[1380] Step 8:

[1381] The device displays the edited image on the user interface, and the user is provided with a UI to review the translation results and image edits, and manually make corrections if necessary.

[1382] Step 9:

[1383] After the user has completed the confirmation and correction, they issue an instruction to save the final image data, and the server saves the final image and generates a download link.

[1384] Step 10:

[1385] The terminal displays the generated download link to the user, allowing the user to download the final translation data with the corrections reflected.

[1386] This series of processes allows users to efficiently translate manga into multiple languages ​​with high quality.

[1387] Example 1

[1388] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1389] Previous multilingual manga translation systems faced challenges in the entire process, from text extraction to translation and image insertion, including translation quality, background completion, and the inability for users to easily make corrections. Furthermore, the lack of fine-tuning specialized for specific categories or styles made it difficult to translate appropriately for the context. As a result, problems arose, such as low translation quality and a heavy burden on users to make corrections.

[1390] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1391] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning specialized for a particular category or style, means for inserting the translated text into an image, means for completing the background accompanying the text insertion, means for displaying the translation result and allowing the user to make corrections, and means for saving the final image after corrections and providing it to the user. This enables high-quality multilingual translation and reduces the burden of corrections on the user.

[1392] "Image data" means digital data that stores visual information, including image files in JPEG and PNG formats and PDF files.

[1393] "Text extraction" is the process of recognizing characters and sentences from image data and obtaining their content as digital text.

[1394] The "target language" is the language into which you are translating, and the language into which you want to convert and display the original text.

[1395] "Fine tuning" is the process of making adjustments to improve accuracy, focusing on a particular category or style.

[1396] "Translated text" means text that has been converted from a source language into a target language.

[1397] "Image insertion" is the process of incorporating translated text into the original image data, including adjusting the position and size of the text.

[1398] "Background completion" is the process of correcting background inconsistencies that occur when inserting text, making it look natural.

[1399] "Translation result" is the text after translation and fine-tuning.

[1400] "Correction" refers to any change a user makes to the translation or image editing results provided by the system.

[1401] "Final image" refers to the final image data provided to the user after translation and corrections are complete.

[1402] The present invention relates to a system for efficiently translating manga into multiple languages, and the system has many functions for text extraction, translation, image generation, and user correction. Specific embodiments for implementing the invention are described below.

[1403] Hardware and Software Used

[1404] server

[1405] The server is implemented in a hardware environment with high-performance computing capabilities (e.g., a virtual server provided by a cloud provider). The server includes the following software components:

[1406] OCR engine: Uses, for example, Google Cloud Vision API to perform optical character recognition.

[1407] Generative AI models: for example, using OpenAI GPT-4 for translation into the target language and fine-tuning.

[1408] Image processing libraries: For example, OpenCV or PyMuPDF are used to check image resolution and resize, split PDFs, analyze text balloon positions, etc.

[1409] Terminal

[1410] A terminal is a device (such as a personal computer or tablet) that allows a user to interact with the system. The terminal implements the system's front end and provides the interface that the user operates.

[1411] User

[1412] The user is the end user who operates the system, and is responsible for providing the manga manuscript data and checking and correcting the results of the translation and corrections.

[1413] Data processing and calculation

[1414] 1. Text Extraction

[1415] The server uses an OCR engine to extract text from the image data, for example, using the Google Cloud Vision API, to identify text balloons in the image and store their locations and contents in a database.

[1416] 2. Initial Translation

[1417] The server translates the extracted text into the target language using a generative AI model (e.g., OpenAI GPT-4). This translation result is again stored in the database. An example of a prompt sentence is a prompt that translates "Hello, how are you?" into "Hello, how are you?"

[1418] 3. Fine Tuning

[1419] The server uses fine-tuning models that are specialized for specific manga categories and styles to refine the initial translation results, for example, refining the hero's line "This is the hero's sword!" to "This is the courage sword!"

[1420] 4. Image Recognition and Editing

[1421] The server uses image recognition technology to insert the translated text into the image appropriately, specifically by adjusting the position and size of the text balloon using OpenCV.

[1422] 5. Background Completion

[1423] The server uses a technique to regenerate the background portion to compensate for the inconsistency of the background caused by the text insertion.

[1424] 6. User confirmation and correction

[1425] The terminal displays the translation results and the status of the image editing to the user. The user can then modify the translation and image editing status as needed, and the modifications are immediately sent to the server.

[1426] Specific examples

[1427] For example, a user may upload a comic page and provide an image containing the text "Hello, how are you?" The server receives this image data, uses an OCR engine to extract the text from the text balloon, and stores the results in a database.

[1428] The server then uses a generative AI model to translate the extracted text into English, for example, "Hello, how are you?" is translated to "Hello, how are you?", and then further refines the translation results using a fine-tuning model suited to the manga category.

[1429] The server places this translated text in the appropriate text balloon and fills in any inconsistencies in the background. Finally, the device displays the edited results to the user, who can then modify the translation as needed. The modified results are then sent to the server and stored in a database.

[1430] This system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections.

[1431] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1432] Step 1:

[1433] Uploading data

[1434] Users upload manga manuscript data through the system's front end, providing image files (JPEG, PNG, etc.) or PDF files as input.

[1435] The server processes the received file using an HTTP POST request, and as output, the received manuscript data is saved on the server.

[1436] Step 2:

[1437] Image preprocessing

[1438] The server checks the image resolution of the received manga data and resizes it if it is inappropriate, and if it is a PDF file, splits each page into a separate image file.

[1439] It uses saved original data (image or PDF) as input, and generates an image file resized to the appropriate resolution or split into individual image files as output.

[1440] Specific operations include image resizing using the OpenCV library and splitting the PDF using PyMuPDF.

[1441] Step 3:

[1442] Text Extraction

[1443] The server uses an OCR engine (e.g., Google Cloud Vision API) to extract text from each image file, which is then stored in a database.

[1444] As input, it uses preprocessed image files, and as output, it stores extracted text information and its location data in a database.

[1445] Specific operations include sending an image file to an OCR engine and storing the returned text information in a database.

[1446] Step 4:

[1447] Initial translation

[1448] The server uses a generative AI model (e.g., OpenAI GPT-4) to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[1449] As input, it uses extracted text stored in a database, and as output, the translation results into the target language are stored in a database.

[1450] Specifically, the generative AI model is provided with the extracted text and uses the prompt sentence to obtain the translation result, for example, translating "Hello, how are you?" into "Hello, how are you?"

[1451] Step 5:

[1452] Fine Tuning

[1453] The server uses fine-tuning models specialized for specific manga categories and styles to refine the initial translation results, which are also stored in the database.

[1454] As input, it uses the initial translation results, and as output, the fine-tuned translation results are stored in a database.

[1455] Specifically, it provides the initial translation text to the fine-tuning model, which then generates a more natural and context-appropriate translation result, for example, refining "This is the hero's sword!" to "This is the courage sword!".

[1456] Step 6:

[1457] Image Recognition and Editing

[1458] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[1459] It takes the fine-tuned translation results and the original image file as input, and generates a new image file with the translated text inserted as output.

[1460] Specific actions include using OpenCV to reparse the position and size of the text balloon and insert text, e.g., replacing "Run quickly!" with "Run quickly!".

[1461] Step 7:

[1462] Background Complement

[1463] The server compensates for inconsistencies in the background caused by inserting translated text, and automatically corrects the background so that it does not look unnatural when the text balloon is changed.

[1464] As input, it takes the image file after the text has been inserted, and as output, it generates a new image file with the background filled in with inconsistencies.

[1465] Specifically, it uses generative AI technology and image correction algorithms to recreate the background around the balloon in a natural shape.

[1466] Step 8:

[1467] User confirmation and correction

[1468] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[1469] As input, it takes an image file with the translated text inserted, and as output, it generates the final image data with the user's modifications reflected.

[1470] Specifically, the terminal provides an interface for editing, creating an environment where users can edit directly. The edited data is immediately reflected on the server.

[1471] Step 9:

[1472] Save and download the final data

[1473] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[1474] It takes the image file with the user's modifications as input, and saves the final image data and generates a download link as output.

[1475] Specifically, the modified image file is saved in the database and a URL that can be accessed by the user is created.

[1476] (Application example 1)

[1477] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1478] Multilingual distribution of manga requires efficient translation into different languages. Conventional methods require manual translation and editing, which is extremely time-consuming and laborious. Furthermore, there are often inconsistencies in the background of images when translating text, and the environment for easy user corrections is often inadequate. This makes it difficult for users to smoothly enjoy manga in multiple languages.

[1479] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1480] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning based on context specific to manga, means for inserting the translated text into the image, means for completing the background associated with the insertion, means for displaying the translation result and allowing the user to make corrections, means for saving the final image after corrections and providing it to the user, and means for providing a terminal application for viewing manga in multiple languages. This allows manga text to be translated efficiently into multiple languages, enabling users to easily make corrections and view the text.

[1481] "Image data" is an electronic representation of a comic page or frame.

[1482] "Text" refers to character information contained in image data.

[1483] "Target language" refers to the language into which the text is translated.

[1484] "Fine tuning" refers to adjusting the model to improve the accuracy of machine translation based on the specific context of manga.

[1485] "Background completion" is the process of correcting background inconsistencies that occur when inserting translated text into an image.

[1486] "User" refers to a person who uses the system to translate and view manga in multiple languages.

[1487] "Device application" refers to software that runs on mobile devices such as smartphones and tablets.

[1488] "Extracted text" refers to character information obtained from image data using OCR technology.

[1489] A "text balloon" refers to a speech bubble used to write lines or narration within a manga.

[1490] The "final image" is the final image data after the text has been translated and completed and the user's corrections have been reflected.

[1491] The present invention relates to a system for efficiently translating manga into multiple languages. The system extracts text from image data, translates it, generates an image containing the translated text, and allows the user to make any necessary corrections.

[1492] First, the user uploads the manga image data (image file or PDF) via their device. This image data is then sent to the server. The server preprocesses the received manga data, inspects the image resolution, and resizes it if it is inappropriate. In the case of a PDF file, the server also splits each page into a separate image file.

[1493] The server then uses OCR technology to extract text from each image file, identify the location and content of the text balloons, and store the extracted text in a database. Specifically, this can be done using an OCR engine such as Pytesseract.

[1494] The server then uses a generative AI model to translate the extracted text into the target language. This initial translation result is also stored in the database. For example, the extracted text "Hello, how are you?" can be translated into English to produce "Hello, how are you?"

[1495] Additionally, the server uses fine-tuned models tailored to specific manga genres and styles to refine the initial translation results, resulting in more context-appropriate translations. For example, it takes the initial translation "This is the hero's sword!" and generates the more context-appropriate "This is the courage sword!".

[1496] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image. The size and position of the text balloon are also automatically adjusted. This compensates for inconsistencies in the background and eliminates the awkwardness that comes with inserting the translated text. This allows the original text "Run quickly!" to be replaced with "Run quickly!" and the position and size of the balloon to be adjusted.

[1497] The user can then check the translation results displayed on the device and make corrections as necessary. These corrections are also sent to the server and reflected in the final image data. For example, the user can change the displayed translation "Run quickly!" to "Hurry up!"

[1498] Finally, the server saves the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download. The user can click on this link to download the high-quality multilingual translated manga.

[1499] The implementation of this system allows users to quickly and highly accurately translate manga into multiple languages, providing an environment in which they can make any necessary corrections. An example of a translation and prompt using a generative AI model is "Please translate the following Japanese text into English. Text: 'Hello, how are you?'"

[1500] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1501] Step 1:

[1502] Users upload manga image data (image files or PDFs) to the system from their devices. Specifically, they click the upload button in their browser or application, select the file, and send it to the server.

[1503] Input: Comic image data file (PNG, JPEG, PDF, etc.) selected by the user.

[1504] Output: Image data of the manga sent to the server.

[1505] Step 2:

[1506] The server preprocesses the image data it receives, checking the image resolution and resizing it if it is inappropriate, and, in the case of PDF files, splitting each page into a separate image file.

[1507] Input: Image data received by the server.

[1508] Output: Image data resized to optimal resolution and PDF pages split into individual image files.

[1509] What it does: Uses libraries such as OpenCV to check image resolution and resize.

[1510] Step 3:

[1511] The server uses OCR technology to extract text from each image file, and the extracted text and its location are stored in a database.

[1512] Input: The resized image data.

[1513] Output: Extracted text data and text location information.

[1514] What it does: Uses an OCR engine such as Pytesseract to extract the location and content of text balloons from an image file.

[1515] Step 4:

[1516] The server uses a generative AI model to translate the extracted text into the target language, and this initial translation result is also stored in the database.

[1517] Input: Extracted text data.

[1518] Output: The translated text data.

[1519] What it does: Uses a translation engine (e.g., Google Translate API) to translate text data into the target language.

[1520] Step 5:

[1521] The server uses fine-tuned models that are specialized for specific manga genres and styles to refine the initial translation results, which are then stored in a database.

[1522] Input: Text data of the initial translation result.

[1523] Output: Fine-tuned translated text data.

[1524] How it works: Generates context-appropriate translations using a generative AI model specialized for the manga genre.

[1525] Step 6:

[1526] The server uses image recognition technology to reanalyze the position and content of the text balloon and insert the translated text into the image, automatically adjusting the size and position of the text balloon.

[1527] Input: Fine-tuned translation text data, image data.

[1528] Output: Image data with inserted text.

[1529] What it does: Uses OpenCV and PIL libraries to insert text into an image and adjust the position and size of the text balloon.

[1530] Step 7:

[1531] The server compensates for inconsistencies in the background caused by the insertion of translated text.

[1532] Input: Image data with inserted text.

[1533] Output: Image data with background interpolated.

[1534] How it works: It uses image editing algorithms to naturally complement the background of the text insert.

[1535] Step 8:

[1536] The terminal displays the edited image to the user, who can then check the translation results and edits and make corrections as necessary. These corrections are also sent to the server.

[1537] Input: The translated and completed image data.

[1538] Output: Image data with user modifications.

[1539] What it does: Displays the image through the device's interface and allows the user to make corrections. The user enters the corrections and submits them.

[1540] Step 9:

[1541] The server stores the final image data that reflects the user's confirmation and corrections, and provides a link for the user to download it.

[1542] Input: Image data with user modifications.

[1543] Output: Final image data provided via a download link.

[1544] Behavior: Saves the image data and sends a notification to the user providing a download link.

[1545] This allows users to quickly and efficiently translate manga into multiple languages. An example of a translation and prompt using a generative AI model is: "Please translate the following Japanese text into English. Text: 'Hello, how are you?'"

[1546] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1547] This invention relates to a system that combines an emotion engine with an efficient system for translating manga into multiple languages, thereby reflecting user emotional feedback and improving the accuracy of translation and editing. This system extracts text from image data, translates it into the target language, and provides an environment where users can check and correct the results, while analyzing user emotions in real time, enabling more user-friendly operation.

[1548] 1. Upload your data

[1549] The user uploads the manga manuscript data (image file or PDF file) from the system's front end. The file is selected and sent to the server.

[1550] Example: A user clicks the file chooser button in their browser to select and upload a file from their local disk.

[1551] 2. Image Preprocessing

[1552] The server pre-processes the uploaded manga data, inspecting the image resolution and format and resizing or formatting it as needed. In the case of PDF files, each page is split into a separate image file.

[1553] Example: The server checks the image resolution and resizes it if it is inappropriate.

[1554] 3. Text Extraction

[1555] The server uses OCR (Optical Character Recognition) technology to extract text from the preprocessed image files, which is then stored in a database.

[1556] Example: The server identifies the location and content of text balloons on each page and stores them in a database.

[1557] 4. Initial Translation

[1558] The server uses generative AI to perform an initial translation of the extracted text into the target language, and the translation results are stored in a database.

[1559] Example: The server translates the text "Hello, how are you?" to "Hello, how are you?"

[1560] 5. Fine Tuning

[1561] The server uses fine-tuning models specific to a particular genre or style to refine the initial translation results, which are then stored in a database.

[1562] Example: The server refines "This is the hero's sword!" to "This is the courage sword!"

[1563] 6. Image Recognition and Editing

[1564] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted to insert the text.

[1565] Example: The server properly places "Run quickly!" into "Run quickly!"

[1566] 7. Background Completion

[1567] The server will interpolate the background as needed to correct any inconsistencies in the background caused by changes to the text balloon, thus maintaining a natural look.

[1568] Example: The server fills in missing parts of the background and recreates a natural background.

[1569] 8. User Emotion Recognition and Feedback

[1570] The device displays the edited image to the user and uses an emotion engine to analyze the user's emotions in real time based on their facial expressions and voice. Based on the analysis results, the device collects user feedback on the translation results and the edited image.

[1571] Example: The device uses the user's camera and microphone to analyze their reactions and use an emotion engine to collect positive or negative feedback.

[1572] 9. Applying Feedback

[1573] The server has the ability to automatically correct the translated text and edits based on the collected user emotional feedback. If the feedback is mostly positive, the server proceeds to the next step.

[1574] Example: The server responds to negative feedback by correcting the translation and redisplaying it.

[1575] 10. Modifying and Saving

[1576] The user makes final confirmations and corrections while taking into account the emotional feedback. After corrections are complete, the server stores the final translation and editing results in a database and provides them to the user.

[1577] Example: The user completes the review, and if the result is satisfactory, completes the corrections and clicks the save button.

[1578] 11. Data Provision

[1579] The terminal displays the generated download link to the user, allowing the user to download the final data.

[1580] Example: The device displays a download link and the user downloads the modified results.

[1581] This system enables high-quality multilingual translation of manga that incorporates user emotional feedback, improving the user experience.

[1582] The processing flow will be explained below.

[1583] Step 1:

[1584] The user uploads the manga manuscript data (image file or PDF file) from the system's front end. The user clicks the file selection button in the browser, selects the file from the local disk, and presses the upload button, which sends the file to the server.

[1585] Step 2:

[1586] The server preprocesses the manga data received. Specifically, it checks the image resolution and format, resizes and converts the format as needed, and splits each page into a separate image file if it is a PDF file.

[1587] Step 3:

[1588] The server extracts text from the preprocessed image files using OCR (Optical Character Recognition) technology, identifies the location and content of text balloons, and stores the extracted text in a database.

[1589] Step 4:

[1590] The server uses generative AI to perform an initial translation of the extracted text into the target language, and the translation results are stored in a database.

[1591] Step 5:

[1592] The server uses fine-tuning models specialized for specific manga genres and styles to improve the initial translation results, which are then stored in a database.

[1593] Step 6:

[1594] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text appropriately. The balloon's position and size are automatically adjusted and the text is inserted.

[1595] Step 7:

[1596] The server performs background interpolation, correcting inconsistencies in the background caused by text balloon changes to maintain a natural look.

[1597] Step 8:

[1598] The device displays the edited image to the user, and uses an emotion engine to analyze the user's emotions in real time from their facial expressions and voice. Based on the analysis results, the device collects user feedback on the translation results and the edited image.

[1599] Step 9:

[1600] The server automatically corrects the translated text and edits based on the user's emotional feedback collected. If the feedback is positive, the system proceeds to the next step, but if the feedback is negative, the system makes appropriate corrections.

[1601] Step 10:

[1602] The user then makes final confirmations and corrections while taking into account the emotional feedback. After corrections are complete, the server saves the final translation and edited results in a database.

[1603] Step 11:

[1604] The terminal displays the generated download link to the user, allowing the user to download the final data. The user uses this link to obtain the final translation result.

[1605] Example 2

[1606] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1607] Conventional systems for multilingual translation have issues with translation accuracy and the complexity of editing operations. They also face the problem of not being able to properly reflect user emotions and feedback, resulting in a poor user experience. The present invention aims to solve these issues by providing an efficient system that provides high-quality translation results and reflects user emotional feedback.

[1608] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data uploading means, a means for preprocessing uploaded image data, a means for extracting text from image data using OCR technology, a means for translating the extracted text into a target language using a generative AI model, a means for fine-tuning based on genre and style, a means for inserting the translated text into an image, a means for completing the background associated with the insertion, a means for displaying the edited image to the user and collecting user emotional feedback using a sentiment analysis engine, a means for correcting the translation result and editing result based on the collected emotional feedback, and a means for saving the final corrected image and providing it to the user. This makes it possible to efficiently perform high-quality translations with a user-friendly interface and reflect the user's emotional feedback.

[1609] "Data upload" refers to a user sending manga manuscript data (image file or PDF file) to the server from the system's front end.

[1610] "Preprocessing" refers to the process of checking the resolution and format of uploaded image data and resizing or format conversion as necessary.

[1611] "OCR technology" refers to optical character recognition technology that extracts text from image data.

[1612] "Generative AI Model" refers to the artificial intelligence model used to translate extracted text into a target language.

[1613] "Fine-tuning" refers to the process of improving the initial translation results based on a specific genre or style.

[1614] "Inserting translated text" refers to the process of placing translated text at a specified location within the original image.

[1615] "Background completion" refers to the process of correcting background inconsistencies that occur during the process of inserting translated text, resulting in a natural look.

[1616] An "emotion analysis engine" refers to technology that displays edited images to the user and analyzes emotions in real time from the user's facial expressions and voice.

[1617] "Emotional feedback" refers to feedback information on translation results and edited images that is collected based on the results of user emotion analysis.

[1618] "Saving the final image" refers to the process of saving the final translation and editing results after corrections in a database and providing them to the user.

[1619] The present invention provides a system for efficiently translating manga into multiple languages. The system aims to improve the accuracy of translation and editing by incorporating user emotional feedback. Specific embodiments are described below.

[1620] First, the user uploads the manga manuscript data (image files or PDF files) using the system's front-end interface. To do this, the user clicks the file selection button in the browser, selects the required file from the local disk, and presses the upload button.

[1621] The uploaded data is then sent to the server, which begins pre-processing the data. At this stage, the server inspects the image resolution and format and resizes or converts it if necessary. For example, in the case of a PDF file, it splits each page into a separate image file.

[1622] After preprocessing is complete, the server uses OCR technology (e.g., Tesseract or Google Cloud Vision API) to extract text from the image data. The extracted text is stored in a database. For example, the server identifies the location and content of text balloons on each page and stores this information in the database.

[1623] The extracted text is then translated into the target language by a generative AI model (e.g., GPT-3) on the server. This translation result is also stored in a database. For example, the server translates the text "Hello, how are you?" into "Hello, how are you?". An example of a prompt used in this process is "Please translate this text 'Hello, how are you?' into English."

[1624] The server then uses fine-tuning models based on specific genres and styles to refine the initial translation. These refined translations are also stored in the database. For example, the server might refine "This is the hero's sword!" to "This is the courage sword!"

[1625] After obtaining the translation result, the server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text in the appropriate position. During this process, the position and size of the balloon are automatically adjusted and the translated text is inserted. For example, the server will properly place the text "Run quickly!" into "Run quickly!"

[1626] Additionally, the server performs background interpolation as needed to correct any inconsistencies in the background caused by changes to the text balloon, ensuring that the overall image maintains a natural look. For example, missing parts of the background are interpolated to recreate a natural background.

[1627] The device then displays the edited image to the user and uses an emotion engine (e.g., Affectiva) to analyze the user's facial expressions and voice in real time to determine their emotions. Based on the analysis results, the device collects user feedback on the translation results and edited image. For example, the device can use the emotion engine to analyze data collected through the user's camera and microphone to collect positive or negative feedback.

[1628] The collected user emotional feedback is used by the server to automatically correct the translated text and edited results. If there is a lot of positive feedback, the server proceeds to the next step, but if there is negative feedback, the translation is corrected and redisplayed.

[1629] Finally, the user makes final confirmation and corrections while taking into account the emotional feedback. After corrections are complete, the server saves the final translation and editing results in a database and provides them to the user. For example, once the user has completed their confirmation and is satisfied with the results, they can click the "Save" button.

[1630] Finally, the terminal displays the generated download link to the user, allowing the user to download the final data. For example, the terminal displays the download link, and the user clicks the link to download the final data.

[1631] Through the above steps, the present invention realizes a system that can improve the accuracy of multilingual translation and enhance the user experience.

[1632] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1633] Step 1: Upload your data

[1634] Users use the front-end interface in their browser to select and upload manga manuscript data (image files or PDF files) from their local disk, which then sends the manga data to the server.

[1635] Input: Manga manuscript data (image file or PDF file)

[1636] Output: Manuscript data sent to the server

[1637] What happens: The user clicks the file selection button to select a file, then clicks the "Upload" button to send the file to the server, and a progress bar appears to show the upload progress.

[1638] Step 2: Image preprocessing

[1639] The server checks the resolution and format of the uploaded manuscript and performs resizing and formatting as needed. In the case of PDF files, each page is split into a separate image file.

[1640] Input: Manuscript data sent to the server

[1641] Output: Preprocessed image data

[1642] Specific operation: The server checks the resolution and resizes it if it is low. In the case of PDF files, each page is split into an image file. The processing result is recorded in the log.

[1643] Step 3: Text Extraction

[1644] The server uses OCR technology to extract text from the preprocessed image files and stores it in a database.

[1645] Input: Preprocessed image data

[1646] Output: Extracted text data

[1647] Specific operation: The server uses OCR technology (e.g., Tesseract or Google Cloud Vision API) to extract text from images, saves the extracted results in a database, and displays a preview in the log.

[1648] Step 4: Initial translation

[1649] The server uses the generative AI model to perform an initial translation of the extracted text into the target language, and stores the translation results in a database.

[1650] Input: Extracted text data

[1651] Output: Initial translated text data

[1652] Specific operation: The server translates the Japanese text "Hello, how are you?" into English "Hello, how are you?". The prompt used and the translation result are recorded in the database and in the log.

[1653] Step 5: Fine Tuning

[1654] The server refines the initial translation results with fine-tuning models based on specific genres and styles, and stores the refined translation results in a database.

[1655] Input: Initial translated text data

[1656] Output: Improved translation text data

[1657] What happens: The server refines the translation of "This is the hero's sword!" to "This is the courage sword!". The fine-tuning results are saved in the database and logged.

[1658] Step 6: Image Recognition and Editing

[1659] The server uses image recognition technology to detect the exact position and size of the text balloon and place the translated text in the appropriate location. The balloon's position and size are automatically adjusted to insert the translated text.

[1660] Input: Improved translation text data

[1661] Output: Image data neatly placed in a text balloon

[1662] What it does: The server detects the position and size of the text balloon in the image, places the translated text "Run quickly!" in the appropriate position, and generates an image. At this time, it also fine-tunes the size of the text balloon.

[1663] Step 7: Filling in the background

[1664] The server performs background interpolation processing as necessary to correct background inconsistencies resulting from changes to the text balloon.

[1665] Input: Image data placed in a text balloon

[1666] Output: Final image data with background interpolated

[1667] What it does: The server fills in the missing parts of the background and recreates a natural background. It logs the results of the filling process.

[1668] Step 8: User Emotion Recognition and Feedback

[1669] The device displays the edited image to the user and uses an emotion engine to analyze the user's emotions in real time from their facial expressions and voice, collecting emotional feedback from the user.

[1670] Input: Final image data with background interpolated

[1671] Output: User's emotional feedback data

[1672] Specific operation: The device collects the user's facial expressions and voice through the camera and microphone, analyzes them with an emotion engine, and displays and collects positive or negative feedback.

[1673] Step 9: Applying Feedback

[1674] The server automatically corrects the translated text and edits based on the collected user emotional feedback. If there is a lot of positive feedback, the server proceeds as is.

[1675] Input: User emotional feedback data

[1676] Output: Corrected translation text and edits

[1677] Specific behavior: The server corrects the translation in response to negative feedback and displays it again to the user. The corrected translation result is recorded in the log.

[1678] Step 10: Modify and save

[1679] The user then makes final confirmations and corrections while taking into account the emotional feedback. After the corrections are complete, the server stores the final translation and editing results in a database and provides them to the user.

[1680] Input: Final image data with positive feedback applied

[1681] Output: Final image data after user confirmation

[1682] Specific behavior: The user checks the data and if satisfied, clicks the "Save" button. A message indicating that the data has been saved is displayed.

[1683] Step 11: Provide data

[1684] The terminal displays the generated download link to the user, allowing the user to download the final data.

[1685] Input: Final saved image data

[1686] Output: A download link that users can access

[1687] Specific behavior: The device displays a download link, the user clicks the link to download the final data, and a confirmation message is displayed when the download is complete.

[1688] (Application example 2)

[1689] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1690] In multilingual manga translation, simply translating text into another language can result in the loss of context and nuance, leaving users dissatisfied. Manually correcting translation results can also lead to a poor user experience. Furthermore, improper adjustment of the position and size of text balloons can result in visual inconsistencies. To resolve these issues and provide high-quality translation results, a system is needed that incorporates user emotional feedback to improve translation accuracy.

[1691] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1692] In this invention, the server includes means for extracting text from image data, means for translating the extracted text into a target language, means for fine-tuning based on context specific to manga, means for inserting the translated text into the image, means for completing the background accompanying the insertion, means for displaying the translation result and allowing the user to make corrections, means for analyzing the user's emotions in real time and reflecting the feedback, and means for saving the final image after correction and providing it to the user. This makes it possible to provide high-quality translation results that reflect the user's emotional feedback, improving the user experience.

[1693] "Image data" refers to files that digitally represent the visual information contained in manga pages and frames.

[1694] "Text extraction" refers to the process of identifying textual information from image data and extracting it as digital text.

[1695] "Target Language" means the language into which the Source Text is translated.

[1696] "Fine tuning" refers to the process of fine-tuning the initial translation result to suit a specific context or genre, improving the accuracy and quality of the translation.

[1697] "Insertion" refers to the process of placing translated text in a predetermined location in image data.

[1698] "Background completion" refers to the process of correcting missing or inconsistent backgrounds caused by text insertion, maintaining a natural appearance.

[1699] "Correction" refers to the act of the user checking the translation results and the content of the inserted text and making changes as necessary.

[1700] "Real-time emotion analysis" refers to the process of instantly determining the user's current emotional state based on their facial expressions and voice, and obtaining the results as data.

[1701] "Reflecting feedback" refers to the process of improving translation results and the overall operation of the system based on analyzed user emotional data.

[1702] "Final Image" refers to image data that has had all corrections and enhancements completed and is ready to be provided to the user.

[1703] "Preservation" refers to the process of digitally recording the final, corrected and enhanced image and storing it for reuse as needed.

[1704] This invention is a multilingual manga translation system that enables more accurate translation by incorporating user emotional feedback. The details of this system are described below.

[1705] Hardware and software used

[1706] The system uses a smartphone, a head-mounted display (HMD), a camera, and a microphone as hardware, and utilizes OCR technology (e.g., Tesseract), a generative AI model (e.g., OpenAI API, DeepL API), an emotion analysis engine (e.g., Microsoft Azure Emotion API), and a database (e.g., MySQL, PostgreSQL) as software.

[1707] System Overview

[1708] 1. Uploading image data

[1709] Users upload manga image data (e.g., image files or PDF files) to the system via their smartphone or HMD, which gives the system a starting point for the translation process.

[1710] 2. Image Preprocessing

[1711] The server checks the uploaded image data and performs resizing and format conversion for resolution and format, and in the case of PDF files, splits each page into a separate image file.

[1712] 3. Text Extraction

[1713] The server uses OCR technology (e.g., Tesseract) to extract text from the preprocessed image data, and the extracted text is stored in a database.

[1714] 4. Initial Translation

[1715] The server uses a generative AI model (e.g., OpenAI's API) to translate the extracted text into the target language, and the translation results are stored in a database.

[1716] 5. Fine Tuning

[1717] The server uses fine-tuning models specific to specific contexts and genres to improve the quality of the initial translation results, which are then stored in a database.

[1718] 6. Image Recognition and Editing

[1719] The server uses image recognition technology to detect the position and size of the text balloon and position the translated text appropriately, interpolating the background if necessary to maintain a natural look.

[1720] 7. Obtaining Emotional Feedback

[1721] The device uses a camera and microphone to analyze the user's real-time emotional data, such as facial expressions and voice while viewing the translation results, to obtain positive or negative feedback.

[1722] 8. Applying Feedback

[1723] The server automatically improves translation results and edited images based on the user's emotional feedback. If there is a lot of positive feedback, the translation is left as is, but if there is a lot of negative feedback, the translation is retranslated or revised.

[1724] Examples and prompts

[1725] For example, if a user inputs the phrase "This hero's sword defeats all enemies," it would be translated using a generative AI model as follows:

[1726] Translate this to English: This hero's sword will defeat all enemies

[1727] Based on this prompt, it would be translated as "This hero's sword defeats all enemies."

[1728] Process Overview

[1729] This system is characterized by a process that analyzes user sentiment in real time and automatically improves translation results based on that feedback. The combination of hardware and software makes it highly user-friendly and capable of providing high-quality multilingual translation.

[1730] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1731] Step 1:

[1732] A user logs in to the system and uploads manga image data (e.g., image files or PDF files). Specifically, the user selects image data using a smartphone or HMD and uploads it through the system interface. The input data is an image file, and the output is an image file stored on the server.

[1733] Step 2:

[1734] The server preprocesses the uploaded image data. Specifically, it checks the image resolution and format, resizes it, and converts the format. In the case of PDF files, it also splits each page into a separate image file. The input data is the uploaded image data, and the output is the preprocessed image file.

[1735] Step 3:

[1736] The server uses OCR technology (e.g., Tesseract) to extract text from the preprocessed image files. Specifically, the OCR engine analyzes characters in the image and extracts them as digital text. The input data is the preprocessed image file, and the output is the extracted text data.

[1737] Step 4:

[1738] The server uses a generative AI model (e.g., OpenAI's API) to translate the extracted text into the target language. Specifically, it sends a prompt to the generative AI model and obtains the translation result. The input data is the extracted text data, and the output is the initial translation result.

[1739] Step 5:

[1740] The server refines the initial translation using a fine-tuning model specific to a particular context or genre. Specifically, the translation is fed into a pre-trained model, which then performs context-based corrections. The input data is the initial translation, and the output is the fine-tuned translation.

[1741] Step 6:

[1742] The server uses image recognition technology to detect the position and size of the text balloon and place the translated text in the appropriate location. Specifically, an image analysis algorithm identifies the text balloon and inserts the new text in its place. The input data is the fine-tuned translation result and image data, and the output is the image with the inserted text.

[1743] Step 7:

[1744] The server performs background interpolation as needed to fix background inconsistencies caused by text balloon changes. Specifically, image interpolation techniques recreate the background to maintain a natural look. The input data is an image with text inserted, and the output is an image with the background interpolated.

[1745] Step 8:

[1746] The device displays the edited image to the user and uses an emotion engine to analyze emotions in real time from the user's facial expressions and voice. Specifically, the device collects the user's reactions through a camera and microphone, and the emotion analysis engine analyzes the data. The input data is the user's facial expressions and voice, and the output is analyzed emotional data.

[1747] Step 9:

[1748] The server automatically corrects the translated text and edited results based on the collected user emotional feedback. Specifically, if there is a lot of positive feedback, it proceeds to the next step, but if there is a lot of negative feedback, it retranslates or corrects the text. The input data is emotional data, and the output is the corrected translated text and edited results.

[1749] Step 10:

[1750] The user checks the correction results and makes final corrections. Specifically, the user checks the final image displayed on the device and manually makes corrections as necessary. The input data is the corrected translated text or edited results, and the output is the final translation result or edited image.

[1751] Step 11:

[1752] The server stores the final translation and editing results in a database and provides them to users. Specifically, it generates and provides a link that allows users to download the final images stored in the database. The input data is the final translation results and edited images, and the output is a download link that users can access.

[1753] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1754] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1755] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1756] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1757] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1758] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1759] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1760] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1761] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1762] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1763] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1764] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1765] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1766] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1767] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1768] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1769] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1770] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1771] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1772] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1773] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1774] The following is further disclosed regarding the above embodiment.

[1775] (Claim 1)

[1776] means for extracting text from image data;

[1777] means for translating the extracted text into a target language;

[1778] A means of fine-tuning based on manga-specific context,

[1779] A means for inserting translated text into images;

[1780] A means for completing the background accompanying the insertion;

[1781] A means for displaying the translation results and allowing the user to make corrections;

[1782] means for storing and providing the final modified image to the user;

[1783] A system including:

[1784] (Claim 2)

[1785] 10. The system of claim 1, further comprising means for locating the extracted text and adjusting the position and size of the text balloon as needed.

[1786] (Claim 3)

[1787] 10. The system of claim 1, further comprising means for extracting text from image data using OCR technology.

[1788] "Example 1"

[1789] (Claim 1)

[1790] means for extracting text from image data;

[1791] means for translating the extracted text into a target language;

[1792] A means of fine-tuning specific categories and styles,

[1793] A means for inserting translated text into images;

[1794] A means for completing the background accompanying the insertion of text;

[1795] A means for displaying the translation results and allowing the user to make corrections;

[1796] means for storing and providing the final modified image to the user;

[1797] A system including:

[1798] (Claim 2)

[1799] 10. The system of claim 1, further comprising means for locating the extracted text and adjusting the position and size of the text balloon as needed.

[1800] (Claim 3)

[1801] 10. The system of claim 1, further comprising means for extracting text from image data using optical character recognition techniques.

[1802] "Application Example 1"

[1803] (Claim 1)

[1804] means for extracting text from image data;

[1805] means for translating the extracted text into a target language;

[1806] A means of fine-tuning based on manga-specific context,

[1807] A means for inserting translated text into images;

[1808] A means for completing the background accompanying the insertion;

[1809] A means for displaying the translation results and allowing the user to make corrections;

[1810] means for storing and providing the final modified image to the user;

[1811] A means for providing a terminal application for viewing manga in multiple languages;

[1812] A system including:

[1813] (Claim 2)

[1814] 10. The system of claim 1, further comprising means for locating the extracted text and adjusting the position and size of the text balloon as needed.

[1815] (Claim 3)

[1816] 10. The system of claim 1, further comprising means for extracting text from image data using OCR technology.

[1817] "Example 2: Combining Emotion Engines"

[1818] (Claim 1)

[1819] A data upload means;

[1820] means for pre-processing the uploaded image data;

[1821] A means for extracting text from image data using OCR technology;

[1822] a means for translating the extracted text into a target language using a generative AI model;

[1823] A means of fine-tuning based on genre and style,

[1824] a means for inserting the translated text into the image;

[1825] A means for completing the background accompanying the insertion;

[1826] means for displaying the edited image to a user and collecting user emotional feedback using a sentiment analysis engine;

[1827] A means to correct translation and editing results based on collected emotional feedback, and

[1828] means for storing and providing the final modified image to the user;

[1829] A system including:

[1830] (Claim 2)

[1831] 10. The system of claim 1, further comprising means for locating the extracted text and adjusting the position and size of the text balloon as needed.

[1832] (Claim 3)

[1833] 10. The system of claim 1, further comprising means for translating the extracted text into a target language using a generative AI model.

[1834] "Application example 2 when combining emotion engines"

[1835] (Claim 1)

[1836] means for extracting text from image data;

[1837] means for translating the extracted text into a target language;

[1838] A means of fine-tuning based on manga-specific context,

[1839] A means for inserting translated text into images;

[1840] A means for completing the background accompanying the insertion;

[1841] A means for displaying the translation results and allowing the user to make corrections;

[1842] A means to analyze user emotions in real time and reflect their feedback,

[1843] means for storing and providing the final modified image to the user;

[1844] A system including:

[1845] (Claim 2)

[1846] 10. The system of claim 1, further comprising means for locating the extracted text and adjusting the position and size of the text balloon as needed.

[1847] (Claim 3)

[1848] 10. The system of claim 1, further comprising means for extracting text from image data using OCR technology. [Explanation of symbols]

[1849] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for extracting text from image data; means for translating the extracted text into a target language; A means of fine-tuning based on manga-specific context, A means for inserting translated text into images; A means for completing the background accompanying the insertion; A means for displaying the translation results and allowing the user to make corrections; means for storing and providing the final modified image to the user; A system including:

2. 10. The system of claim 1, further comprising means for locating the extracted text and adjusting the position and size of the text balloon as needed.

3. 10. The system of claim 1, further comprising means for extracting text from image data using OCR technology.

Images