system

Abstract

Provide a system. The method includes: a means for a user to upload a screenshot image of an application; a means for the server to store the uploaded screenshot image; means for the server to analyze the stored screenshot images and recognize text and GUI elements within the images; means for the server to infer an application screen layout based on the recognized text and GUI elements; means for the server to generate source code based on the inferred layout; a means by which the server provides the generated source code to the user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the development of mobile and web applications has progressed rapidly, and the need for reverse engineering of existing applications has also increased. Conventional reverse engineering methods require a great deal of manual work to recreate specific application screens and functions, which is time-consuming and labor-intensive. As a result, rapid prototyping and redevelopment are difficult to achieve. Furthermore, the design information of application screens is often difficult to see from the outside, making it difficult to obtain the design information. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system for automatically generating source code for an application based on screenshots uploaded by a user. The system includes the following means:

[0006] A way for users to upload screenshots of their applications

[0007] A means for the server to store uploaded screenshot images

[0008] A means for the server to analyze the saved screenshot images and recognize text and GUI elements within the images

[0009] A means for the server to infer the layout of an application screen based on recognized text and GUI elements

[0010] A means for the server to generate source code based on the inferred layout

[0011] A means by which the server provides the generated source code to the user

[0012] This allows users to quickly obtain reverse-engineered application code simply by uploading screenshots, significantly reducing the amount of work required for manual reproduction. Furthermore, by pre-processing screenshot images through grayscale conversion, recognition accuracy can be improved, and by extracting text from images using OCR technology, more accurate and detailed information can be obtained, improving the accuracy of reproducing application screen design information.

[0013] "User" refers to the user or operator who uploads screenshots to the system and receives the results.

[0014] An "application" is a type of software that runs on a computer or mobile device and is a program that provides specific functions or services.

[0015] A "screenshot" refers to capturing and saving an entire or part of an application screen in image format.

[0016] A "server" refers to a computer system that provides various services and data to computers and devices via a network.

[0017] "Saving" means the act of writing and storing data such as screenshot images in a file or database.

[0018] "Analysis" refers to the process of examining images or data in detail and processing them to understand their structure and content.

[0019] "Text" refers to textual information displayed within an image and used as part of a user interface.

[0020] "GUI elements" refers to various elements that make up the graphical user interface of an application, such as interactive components such as buttons and input fields.

[0021] "Layout" refers to layout information that indicates how each element is arranged within an application screen.

[0022] "Inference" refers to the act of deriving new information or results based on recognized information by utilizing existing data and knowledge.

[0023] "Source code" refers to code that describes the programs necessary for an application to run.

[0024] "Providing" means the act of passing generated data or information to a user.

[0025] "Preprocessing" refers to preparatory processing carried out before the main processing is carried out, and mainly refers to data formatting and noise removal.

[0026] "Grayscale conversion" refers to the process of converting a color image into an image consisting only of shades of black and white.

[0027] "OCR technology" is an abbreviation for optical character recognition technology, which refers to the technology of analyzing characters on an image and converting them into digital text. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0029] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0030] First, the terms used in the following description will be explained.

[0031] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0032] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0033] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0034] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0035] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0036] [First embodiment]

[0037] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0038] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0039] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0040] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0041] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0042] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0043] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0044] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0045] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0046] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0047] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0048] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] This invention is a system that reverse engineers applications based on screenshots and automatically generates corresponding source code, enabling rapid prototyping and redevelopment.

[0050] User-uploaded screenshots

[0051] A user takes a screenshot of an application and accesses the system's upload interface, where the user selects the screenshot and clicks the upload button, which causes the device to send the screenshot image to the server.

[0052] Saving a screenshot

[0053] The server saves the received screenshot. The server saves the uploaded image in a specific directory and sets the file name in an easy-to-manage format. This saving process makes it possible to reference the image data required for subsequent image analysis processing.

[0054] Image Processing and Element Recognition

[0055] The server analyzes the saved screenshot. First, the server performs grayscale conversion as a preprocessing of the image. Then, it uses OCR technology to extract text information from the image. At the same time, the server applies image processing algorithms to detect GUI elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[0056] Layout and Functionality Guessing

[0057] The server predicts the application's screen layout based on the text and GUI element information it recognizes. The server analyzes how each element is arranged on the screen and generates layout information. For example, it predicts the overall screen structure based on the position of buttons and the placement of text fields.

[0058] Code generation

[0059] The server automatically generates the corresponding source code based on the inferred layout information. The generated code is in accordance with the selected programming language and is used to reproduce the inferred layout and functionality. For example, it includes code to configure the screen layout using a GUI library and code to display the extracted text.

[0060] Providing reverse engineering results

[0061] The server provides the generated source code to the user. The server packages the generated code in JSON format or similar and provides a download link to the user. The user can use this link to download the generated source code and view or edit it.

[0062] Specific examples

[0063] For example, a user can upload a screenshot of a specific mobile application to the system. The image might contain multiple buttons and text fields. The server analyzes the image and identifies the positions and labels of the buttons and text fields. The server then uses this information to generate Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[0064] This system allows users to quickly obtain source code that precisely reproduces the screen layout of the target application, significantly reducing the painstaking manual redevelopment work required in the past. Furthermore, OCR technology and grayscale conversion preprocessing provide high recognition accuracy, resulting in more accurate reverse engineering results.

[0065] The above is an embodiment of the present invention.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The user takes a screenshot of the application and selects it by accessing the system's upload interface.

[0069] Step 2:

[0070] After the user selects a screenshot, they click the upload button, which causes the device to send the selected screenshot image to the server in an HTTP POST request.

[0071] Step 3:

[0072] The server saves the received screenshot image. The server saves the image in a specific directory and sets a file name for easy management.

[0073] Step 4:

[0074] The server loads the saved screenshot image. As a preprocessing step, the server first converts the screenshot to a grayscale image, which makes subsequent image analysis easier.

[0075] Step 5:

[0076] The server uses OCR technology to extract text information from the grayscale image and stores the extracted text information as digital data.

[0077] Step 6:

[0078] The server uses image processing algorithms to recognize GUI elements on the screenshot, specifically identifying interactive components such as buttons, input fields, and labels.

[0079] Step 7:

[0080] The server predicts the application screen layout based on the recognized text and GUI element information, and determines the overall layout structure based on the position and type of each element.

[0081] Step 8:

[0082] The server automatically generates the corresponding source code based on the estimated layout information. The server uses a GUI library in the selected programming language (e.g., Python) to generate code that reproduces the screen layout.

[0083] Step 9:

[0084] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link.

[0085] Step 10:

[0086] The user clicks on the link provided by the server to download the generated source code, which the user can then view and edit as needed.

[0087] The above is the specific processing flow of the program.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] Modern software development requires rapid redevelopment and prototyping of existing applications. However, manual coding is time-consuming and labor-intensive, especially when it comes to recreating GUI screen layouts. Furthermore, systems that use accurate OCR technology to recognize text and interface elements and generate code based on that recognition are still rare. This continues to require significant developer time.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes means for a user to upload screenshot images of a software application, means for the server to save the uploaded screenshot images, means for the server to analyze the saved screenshot images and recognize text and interface elements in the images, means for the server to infer a layout of the application screen based on the recognized text and interface elements, means for the server to generate program code based on the inferred layout, and means for the server to provide the generated program code to the user. This allows the user to quickly obtain highly accurate source code for redevelopment, significantly reducing the manual workload.

[0093] "User" means any person or entity that uses the System to upload screenshots of an Application and obtain the generated source code.

[0094] "Software application" refers generally to a program that runs on a computer for use by a user.

[0095] A "screenshot image" is image data that captures the screen of a software application.

[0096] "Uploading Method" refers to the function or process by which a user can send a screenshot image to the system.

[0097] "Server" refers to the computer system or device used to process user requests and store, analyze, and generate code from screenshots.

[0098] "Storage means" refers to a function or process that stores screenshot images in a specific location and makes them accessible as needed.

[0099] "Analysis Method" refers to the technology or algorithms used to analyze the saved screenshot image and recognize text and interface elements within the image.

[0100] "Text" refers to the written information extracted from an image through OCR technology.

[0101] "Interface elements" refers to GUI components such as buttons and text fields that exist within an image.

[0102] "Inference means" refers to a process or function for determining the layout of an application screen based on recognized text and interface elements.

[0103] "Program Code" means the set of source code that describes the operation of a software application.

[0104] "Providing means" refers to the functions and processes for transmitting or making the generated program code available for download to users.

[0105] "Tone conversion" refers to an image processing technique for converting the color information of a screenshot image to grayscale.

[0106] "Optical character recognition technology" refers to technology for extracting characters in an image as digital text.

[0107] MODE FOR CARRYING OUT THE INVENTION

[0108] This invention is a system that reverse engineers applications based on screenshots and automatically generates corresponding source code, enabling rapid prototyping and redevelopment.

[0109] User-uploaded screenshots

[0110] The user first takes a screenshot of the software application, then accesses the system's upload interface, selects the screenshot file, and clicks the upload button, which causes the device to send the screenshot image to the server.

[0111] Saving a screenshot

[0112] The server saves screenshots received from users. Specifically, the server saves the uploaded screenshot images in a specific directory and sets the file name in an easy-to-manage format. This saving process makes it possible to reference the image data required for subsequent image analysis processing.

[0113] Image Processing and Element Recognition

[0114] The server analyzes the saved screenshot. First, it performs grayscale conversion as image preprocessing. Then, it uses OCR technology (e.g., Tesseract OCR) to extract text information from the image. At the same time, the server applies an image processing algorithm (e.g., OpenCV) to detect interface elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[0115] Layout and Functionality Guessing

[0116] The server infers the application's screen layout based on the recognized text and interface element information. The server analyzes how each element is arranged on the screen and generates layout information. For example, it infers the overall screen structure based on the position of buttons and the arrangement of text fields.

[0117] Code generation

[0118] The server automatically generates the corresponding program code based on the inferred layout information. The generated code is based on the selected programming language (e.g., Python) and is used to reproduce the inferred layout and functionality. For example, it includes code to configure the screen layout using Python's Tkinter library and code to display the extracted text.

[0119] Providing reverse engineering results

[0120] The server provides the generated program code to the user. The server packages the generated code in JSON format or similar and provides the user with a download link. The user can use this link to download the generated source code and view or edit it.

[0121] Specific examples

[0122] For example, a user uploads a screenshot of a specific screen in a mobile application to the system. The image contains multiple buttons and text fields. The server analyzes the image and identifies the positions and labels of the buttons and text fields. Based on the identified information, the server then generates Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[0123] An example of a prompt to input to a generative AI model is, "Based on the screenshot below, please recognize GUI elements and generate source code using Python's Tkinter library."

[0124] This allows users to quickly obtain source code that precisely reproduces the screen layout of the target application, significantly reducing the effort required for manual redevelopment. Furthermore, OCR technology and grayscale conversion preprocessing enable high recognition accuracy, resulting in more accurate reverse engineering results. The above is an embodiment of the present invention.

[0125] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0126] Step 1:

[0127] A user takes a screenshot of the application, accesses the upload interface, selects the screenshot file, and clicks the upload button, which causes the device to send the screenshot image to the server.

[0128] Input: A screenshot file selected by the user.

[0129] Output: Screenshot data sent to the server

[0130] Specific operation: The device creates an HTTP request, attaches the screenshot image data, and sends it to the server.

[0131] Step 2:

[0132] The server receives the screenshot upload request and saves the screenshot image in a specific directory, renaming the file to a unique, easily manageable format (e.g., "screenshot_20231015_123456.png").

[0133] Input: Screenshot data sent from the device

[0134] Output: Screenshot files saved in a specific directory

[0135] Specific operation: The server saves the received screenshot data in the appropriate directory and sets a unique file name.

[0136] Step 3:

[0137] The server reads the saved screenshot and performs grayscale conversion as a preprocessing of the image, thereby highlighting the detailed information in the image.

[0138] Input: Saved screenshot file

[0139] Output: Grayscale converted image data

[0140] Specific operation: The server uses the OpenCV library to convert the screenshot image to grayscale.

[0141] Step 4:

[0142] The server uses OCR technology (e.g., Tesseract OCR) to extract text information from the grayscale image. At the same time, the server uses image processing algorithms to detect interface elements (buttons, text fields, etc.) in the image.

[0143] Input: Grayscale converted image data

[0144] Output: Text information and data about the location and type of interface elements

[0145] What it does: The server uses Tesseract OCR to extract text from images and OpenCV algorithms to detect interface elements.

[0146] Step 5:

[0147] The server analyzes the recognized text and interface element information to predict the layout of the application screen, thereby determining the overall screen structure and generating layout information.

[0148] Input: Text information and data about the location and type of interface elements

[0149] Output: Application screen layout information (e.g., JSON format)

[0150] Specific operation: The server analyzes the position and type of each element, generates layout information, and saves it in JSON format.

[0151] Step 6:

[0152] The server automatically generates the corresponding program code based on the inferred layout information. The code is generated in the selected programming language (e.g., Python), and includes GUI layout code using the Tkinter library.

[0153] Input: Application screen layout information

[0154] Output: Generated program code

[0155] Specific operation: The server uses a generative AI model to automatically generate program code based on layout information.

[0156] Step 7:

[0157] The server packages the generated program code in JSON format or similar and provides a download link to the user, who can then use the link to download the generated source code and view or edit it.

[0158] Input: Generated program code

[0159] Output: Download link

[0160] Specific operation: The server packages the generated code and provides the user with a download link via email or other means.

[0161] (Application example 1)

[0162] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0163] Designing and redesigning machine control panels in a factory takes a great deal of time and effort. When each machine has a different control panel, it is difficult to reuse the same design drawings, and in order to improve factory productivity, it is necessary to generate the source code for the control panel quickly and accurately. Current technology mainly involves manual design and program porting, which requires specialized knowledge and time. This poses the problem of reduced factory efficiency and increased costs.

[0164] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0165] In this invention, the server includes: means for a user to upload screenshot images of an application; means for the server to save the uploaded screenshot images; means for the server to analyze the saved screenshot images and recognize text and GUI elements in the images; means for the server to infer an application screen layout based on the recognized text and GUI elements; means for the server to generate source code based on the inferred layout; means for the server to provide the generated source code to the user; means for reverse engineering screenshots of machine operation panels in a factory; means for automatically generating source code for a machine operation program based on screenshot images of a display device to be used; and means for performing image analysis and code generation via a cloud server. This enables rapid and accurate design and redesign of machine operation panels in a factory.

[0166] "User" means any person or institution that uses the System to upload screenshots of an Application.

[0167] A "screenshot image of an application" refers to a still image of an application screen, and is data for visualizing the contents of that screen.

[0168] The "server" is a computer system that stores the uploaded screenshot images, analyzes them, and ultimately generates the source code.

[0169] The "storing means" refers to the process and technology by which the server stores the received screenshot images in a specific directory.

[0170] The "analyzing means" is a technology for recognizing and extracting text and GUI elements from the saved screenshot image.

[0171] "Text" refers to the text information displayed in the screenshot image.

[0172] "GUI elements" refer to components of a user interface, such as buttons and text fields.

[0173] "Guessing means" is a technique for predicting the layout of an application screen based on recognized text and GUI elements.

[0174] "Source code" is a document written in a particular programming language that describes the behavior of a program.

[0175] The "machine operation panel in a factory" is an interface including a display device and an input device for operating a machine installed in a factory.

[0176] "Reverse engineering" refers to the technique of analyzing existing products or code and recreating their structure and function in order to understand the operation and design of the device or software.

[0177] The "display device used" is a visual output device such as a display or monitor included in a machine control panel in a factory.

[0178] A "cloud server" is a server infrastructure that can be accessed remotely over the internet and provides the computational resources to perform image analysis and code generation.

[0179] This invention is a system that performs reverse engineering based on screenshots of machine operation panels in factories and automatically generates corresponding source code, enabling rapid prototyping and redesign, thereby improving factory production efficiency.

[0180] User-uploaded screenshots

[0181] Users can take screenshots of machine control panels in the factory using a smartphone or head-mounted display, and then access the system's upload interface to upload the screenshot, which causes the device to send the screenshot image to the cloud server.

[0182] Saving a screenshot

[0183] The cloud server saves the received screenshots. The server saves the uploaded images in a specific directory and names the files in an easy-to-manage format. This saving process allows the image data required for subsequent image analysis to be referenced.

[0184] Image Processing and Element Recognition

[0185] The cloud server analyzes the stored screenshots and extracts information from the images using the following techniques:

[0186] Converting an image to grayscale using Pillow.

[0187] Pytesseract is used to perform OCR processing and extract text information from the image.

[0188] At the same time, it applies image processing algorithms to detect GUI elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[0189] Layout and Functionality Guessing

[0190] The cloud server predicts the screen layout of the machine operation panel based on the recognized text and GUI element information. The server analyzes how each element is arranged on the screen and generates layout information. For example, it predicts the overall screen structure based on the position of buttons and the arrangement of text fields.

[0191] Generate source code

[0192] The cloud server automatically generates the corresponding source code based on the inferred layout information. The generated code is written in the selected programming language to reproduce the inferred layout and functionality. For example, it generates Python code using the Tkinter library to reproduce the screen according to the recognized layout.

[0193] Providing reverse engineering results

[0194] The cloud server provides the generated source code to the user. The server packages the generated code in JSON format or similar and provides a download link to the user. The user can use this link to download the generated source code and view or edit it.

[0195] Specific examples

[0196] For example, a user can upload a screenshot of an operation panel to a cloud server. The image contains several buttons and text fields. The cloud server analyzes the image and identifies the positions and labels of the buttons and text fields. The server then uses this information to generate Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[0197] Prompt Sentence Examples

[0198] Analyze the panel image below, extract the position and label of each GUI element, and generate a program to reproduce it using Tkinter. Image link: [Link URL]

[0199] The system enables the rapid and accurate design and redesign of machine control panels in factories, significantly reducing the traditional manual and laborious redevelopment work. Furthermore, OCR technology and grayscale conversion preprocessing provide high recognition accuracy and more accurate reverse engineering results.

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1: User uploads screenshot

[0202] The user takes a screenshot of a machine control panel in the factory, accesses the system's upload interface via a smartphone or head-mounted display, and clicks the upload button to send the screenshot to the cloud server.

[0203] Input: Screenshot of the machine control panel

[0204] Output: Screenshot image is sent to the cloud server

[0205] Step 2: Save the screenshot

[0206] The cloud server saves the received screenshot images. The server stores the uploaded images in a specific directory and names the files in an easy-to-manage format.

[0207] Input: Screenshot image sent by the user

[0208] Output: Saved screenshot image

[0209] Step 3: Image preprocessing

[0210] The cloud server preprocesses the saved screenshot images and converts them to grayscale using the Pillow library.

[0211] Input: Saved screenshot image

[0212] Output: Image converted to grayscale

[0213] Step 4: Parsing Text and GUI Elements

[0214] The server uses OCR technology (Pytesseract) to extract text information from the grayscale image, and simultaneously applies image processing algorithms to detect GUI elements (buttons, text fields, etc.) in the image.

[0215] Input: Image converted to grayscale

[0216] Output: Extracted text information and the location and type of GUI elements

[0217] Step 5: Inferring the screen layout

[0218] Based on the text and GUI element information recognized by the cloud server, the screen layout of the machine operation panel is analyzed and layout information is generated.

[0219] Input: Extracted text information and the position and type of GUI elements

[0220] Output: Inferred screen layout information

[0221] Step 6: Generate source code

[0222] The cloud server automatically generates the corresponding source code based on the inferred layout information. Specifically, it generates Python code using the Tkinter library to reproduce the screen according to the recognized layout.

[0223] Input: Inferred screen layout information

[0224] Output: Generated source code

[0225] Step 7: Provide generated source code

[0226] The cloud server packages the generated source code in JSON format or similar and provides the user with a download link, which the user can use to download, view, and edit the generated source code.

[0227] Input: Generated source code

[0228] Output: A download link provided to the user

[0229] By following the above steps, it is possible to quickly and accurately generate source code for an operation program through reverse engineering of a machine operation panel in a factory.

[0230] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0231] This invention is a system that reverse engineers applications based on screenshots and automatically generates the corresponding source code, and also has the ability to recognize and utilize user emotions, which improves the user experience and enables more feedback-driven interactive application prototyping.

[0232] User-uploaded screenshots

[0233] A user takes a screenshot of an application and accesses the system's upload interface. The user selects a screenshot and clicks the upload button, causing the device to send the selected screenshot image to the server.

[0234] Saving a screenshot

[0235] The server saves the screenshots it receives. The images are saved in a specific directory and given file names for easy management. This saving process allows the image data to be referenced for subsequent image analysis and emotion recognition processing.

[0236] Image Processing and Element Recognition

[0237] The server reads the saved screenshot and converts it into a grayscale image. It uses OCR technology to extract text information from the grayscale image, while simultaneously using image processing algorithms to recognize GUI elements. This identifies the location and type of text and GUI elements on the screen.

[0238] emotion recognition

[0239] The server uses an emotion engine to recognize the user's emotion based on the screenshot. The emotion engine identifies the user's emotional state (e.g., joy, surprise, sadness, etc.) by analyzing facial expressions and gaze in the image.

[0240] Layout and Functionality Guessing

[0241] The server predicts the application screen layout based on the recognized text and GUI element information and emotional information. The server determines the optimal layout by taking into account the position and type of each element as well as the user's emotional state.

[0242] Code generation

[0243] The server automatically generates the corresponding source code based on the estimated layout information and the user's emotional information. The server uses a selected programming language (e.g., Python) to generate user interface code that reflects the emotional information. For example, if the user is surprised, the code could display a warning message.

[0244] Providing reverse engineering results

[0245] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link, and the user clicks the link to download the generated source code. The downloaded code can then be viewed and edited.

[0246] Specific examples

[0247] For example, a user uploads a screenshot of a specific mobile application. The server recognizes that the image contains multiple buttons and text fields and uses an emotion engine to detect that the user is surprised. Based on this information, the server generates Python code that includes the screen layout and feedback corresponding to the surprise. This code uses the Tkinter library to position the buttons and text fields appropriately and displays a message reflecting the surprise on the screen.

[0248] This system allows users to quickly obtain source code that precisely replicates the screen layout of the target application, and even enables interactive responses based on user emotions within the system. This not only significantly reduces the amount of manual redevelopment work and user experience tracking required, but also enables more adaptive and user-centric application development through emotion-based feedback.

[0249] The above is an embodiment of the present invention.

[0250] The processing flow will be explained below.

[0251] Step 1:

[0252] The user takes a screenshot of the application. The user accesses the system's upload interface, selects the screenshot, and clicks the upload button.

[0253] Step 2:

[0254] The device sends the selected screenshot image to the server in the form of an HTTP POST request.

[0255] Step 3:

[0256] The screenshot image received by the server is saved in a specific directory. When saving, the file name is set in an easy-to-manage format.

[0257] Step 4:

[0258] The server loads the saved screenshot image and performs grayscale conversion as a preprocessing step to simplify image processing.

[0259] Step 5:

[0260] The server extracts text information from the grayscale image using OCR technology and stores the extracted text information as digital data.

[0261] Step 6:

[0262] The server uses image processing algorithms to recognize GUI elements (buttons, input fields, etc.) on the screenshot and identify the location and type of each element.

[0263] Step 7:

[0264] The server uses an emotion engine to recognize the user's emotions from the screenshots, for example, by analyzing the user's facial expressions and gaze to identify emotions such as joy, surprise, and sadness.

[0265] Step 8:

[0266] The server predicts the application screen layout based on the text and GUI element information recognized by the server, as well as emotional information. The optimal layout is determined taking into account the user's emotional state.

[0267] Step 9:

[0268] The server automatically generates the corresponding source code based on the inferred layout information and the user's emotion information. The code is generated using a GUI library in the selected programming language (e.g., Python), and includes emotion-based feedback as needed.

[0269] Step 10:

[0270] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link.

[0271] Step 11:

[0272] The user clicks the link sent by the server to download the generated source code, which the user can then view and edit as needed.

[0273] The above is the specific processing flow of a system that combines an emotion engine.

[0274] Example 2

[0275] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0276] Conventional systems require a great deal of time and effort for users to reverse engineer application screens and generate source code. Furthermore, they are unable to provide interactive feedback that takes into account the user's emotional state, limiting the user experience. Therefore, there is a need for a system that can efficiently generate application prototypes while improving the user experience.

[0277] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for a user to upload screenshot images of an application; means for the server to save the uploaded screenshot images; means for the server to analyze the saved screenshot images and recognize text and user interface elements in the images; means for the server to analyze facial expressions and gaze to recognize the user's emotions; means for the server to infer the layout of the application screen based on the recognized text, user interface elements, and emotional information; means for the server to generate source code based on the inferred layout and the user's emotional information; and means for the server to provide the generated source code to the user. This allows the user to quickly obtain source code that reproduces the screen layout of the target application and further receive interactive feedback based on emotions.

[0278] "User" refers to the person who uploads screenshots of an application and requests the generation of source code.

[0279] "Server" refers to the computer system that stores and analyzes screenshots uploaded by users and generates and provides source code.

[0280] A "screenshot image" is an image of the application screen display saved as is.

[0281] "Uploading means" refers to the interface and protocol that allows users to send screenshot images to the server.

[0282] "Storage means" refers to a function that allows the server to store uploaded screenshot images in a specific directory so that they can be used for later analysis.

[0283] "Analysis means" refers to a technique by which the server processes the screenshot image and identifies textual information and user interface elements within the image.

[0284] "User interface elements" refer to interactive components such as buttons and text fields within an application screen.

[0285] "Emotion recognition means" refers to technology that allows the server to analyze the user's facial expressions and gaze using screenshot images to identify their emotional state.

[0286] "Inference means" refers to a technology that infers the layout and functionality of an application screen based on recognized text, user interface elements, and emotion information.

[0287] "Source code generation means" refers to a technology that automatically generates source code in an appropriate programming language based on the estimated layout and user emotional information.

[0288] "Means of provision" refers to the mechanism for providing the generated source code to users in a downloadable format.

[0289] This invention is a system that reverse engineers applications based on screenshots uploaded by users and automatically generates corresponding source code. The system also includes a function to recognize users' emotions and provide interactive feedback based on the emotion information.

[0290] The user first takes a screenshot of the application and accesses the system's upload interface. The terminal (user's device) selects the screenshot and clicks the upload button. This action causes the terminal to send the selected screenshot image to the server. The upload is performed using a web form and an HTTP POST request.

[0291] The server saves the received screenshots in a specific directory, with file names that can be uniquely identified using a timestamp or UUID. The saved screenshots are used for later analysis.

[0292] The server reads the saved screenshot and converts it to a grayscale image using OpenCV, then uses the Tesseract OCR engine to extract text information from the grayscale image, while also using OpenCV to recognize the location and type of GUI elements (buttons, text fields, etc.).

[0293] The server then uses an emotion recognition engine (e.g., Hume or Affectiva) to recognize the user's emotions based on the screenshot, analyzing facial expressions and gaze to identify emotional states (e.g., happiness, surprise, sadness, etc.).

[0294] The server predicts the layout of the application screen based on the recognized text information, GUI element information, and emotion information. It determines the optimal layout by taking into account the user's emotional state in addition to the location and type.

[0295] Based on the inferred layout information and user emotion information, the server automatically generates the corresponding source code. Specifically, it generates user interface code using Python and the Tkinter library. For example, if the user is surprised, it generates code to display a message reflecting surprise.

[0296] Finally, the server packages the generated source code as a JSON or text file and generates a link to provide to the user, who can click on this link to download the generated source code for viewing or editing.

[0297] Specific examples

[0298] For example, a user uploads a screenshot of a specific mobile application. The server recognizes that the image contains several buttons and text fields. If the emotion engine detects that the user is surprised, the server uses this information to generate Python code that includes the screen layout and feedback corresponding to the surprise. This code uses the Tkinter library to position the buttons and text fields appropriately and displays a message reflecting the surprise on the screen.

[0299] Prompt Sentence Examples

[0300] "I uploaded a screenshot of a user looking surprised. The screen contains several buttons and text fields. Based on this, please generate Python code that displays feedback that reflects the user's surprise."

[0301] This system helps users quickly obtain source code that reproduces the screen layout of the target application and create an interface that includes emotion-based interactive feedback. Not only does it significantly reduce the traditional manual redevelopment work and user experience feedback tracking, but emotion-based feedback also enables more adaptive and user-centric application development.

[0302] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0303] Step 1:

[0304] The user takes a screenshot of the application and accesses the upload interface. The terminal sends the screenshot image selected by the user to the server. The input data is the screenshot image selected by the user, and the output data is the image file sent to the server.

[0305] Step 2:

[0306] The server saves the screenshots it receives in a specific directory. A timestamp or UUID is used to make the file name unique when saved. The input data is the screenshot image sent to the server, and the output data is the path to the saved image file. Specifically, the server writes the received image data to the file system.

[0307] Step 3:

[0308] The server reads the saved screenshot and converts it into a grayscale image using OpenCV. The input data is the saved image file, and the output data is a grayscale image. Specifically, the server uses the image processing library OpenCV to convert the color image into grayscale.

[0309] Step 4:

[0310] The server uses the Tesseract OCR engine to extract text information from grayscale images. At the same time, it uses OpenCV to recognize the location and type of GUI elements (buttons, text fields, etc.). The input data is a grayscale image, and the output data is the recognized text data and GUI element information. Specifically, the server performs OCR to convert the characters in the image into text and uses a contour detection algorithm to identify GUI elements.

[0311] Step 5:

[0312] The server uses an emotion recognition engine (e.g., Hume or Affectiva) to recognize the user's emotion based on the screenshot. The input data is the screenshot image, and the output data is the recognized emotion data. Specifically, the server applies face recognition and emotion classification algorithms.

[0313] Step 6:

[0314] The server infers the layout of the application screen based on the recognized text information, GUI element information, and emotion information. The input data is the recognized text information, GUI element information, and emotion information, and the output data is the inferred layout information. Specifically, the server applies a rule-based layout inference algorithm to determine the placement and characteristics of each element.

[0315] Step 7:

[0316] The server automatically generates the corresponding source code based on the inferred layout information and user emotion information. The input data is the inferred layout information and emotion information, and the output data is the generated source code. Specifically, the server generates the user interface source code using Python and Tkinter.

[0317] Step 8:

[0318] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The input data is the generated source code, and the output data is a download link provided to the user. Specifically, the server packages the generated content, creates an HTTP link, and notifies the user.

[0319] (Application example 2)

[0320] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] In recent years, with the advancement of automation in factories, the use of factory robots for inspection and monitoring has been increasing. However, conventional factory robots have limited ability to acquire and analyze information from dashboards and panels, and are unable to provide real-time feedback based on the emotions of human operators. This makes efficient and interactive inspection work difficult, and there is a need for an improved user experience.

[0322] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0323] In this invention, the server includes means for a user to upload screenshot images of an application, means for the server to save the uploaded screenshot images, means for the server to analyze the saved screenshot images and recognize text and GUI elements in the images, means for the server to infer an application screen layout based on the recognized text and GUI elements, means for the server to generate source code based on the inferred layout, means for the server to provide the generated source code to the user, means for the server to recognize user emotions based on the uploaded screenshot images, and means for the server to optimize the user interface based on the recognized emotion information. This enables factory robots to efficiently acquire and analyze information from dashboards and panels and provide interactive feedback to human operators according to their emotions.

[0324] A "screenshot image" is an image file that a user records as a still image of the contents displayed on the screen of a device.

[0325] "Upload" is the act of a user sending data from their device to a server.

[0326] A "server" is a computer system that stores, manages, and processes information on a network.

[0327] "Grayscale conversion" is a process of converting a color image into an image that contains only brightness information.

[0328] "OCR" stands for Optical Character Recognition, a technology that converts characters in an image into digital text.

[0329] "Text and GUI elements" refers to the textual information and graphical user interface components displayed on the screen.

[0330] "Layout" is the arrangement design of how elements on a screen are arranged.

[0331] "Source code" is a human-readable form of code that describes a program.

[0332] "Emotion recognition" is a technology that analyzes a user's facial expressions from images and videos and determines their psychological state.

[0333] "User interface" refers to the screens and operating specifications used to exchange information between the user and the system.

[0334] "Optimization" means adjusting the system's performance and functionality to the best possible state based on certain criteria.

[0335] This invention relates to a system that provides optimized feedback to users through information analysis and emotion recognition of dashboards and panels by an inspection robot in a factory. Specific embodiments of this system are described below.

[0336] The system begins with the robot taking images of each piece of equipment and dashboard in the factory, and then uploading the screenshots from the user's device. These images are then transferred to the server via the Internet and uploaded. The server saves the uploaded screenshots in a specific directory and assigns them file names for easy management.

[0337] The server then analyzes the saved screenshot image. This analysis process first converts the colorful image to grayscale, then uses OCR technology to extract text information from the image. At the same time, it uses image processing algorithms to recognize GUI elements. This step uses OpenCV (an open-source computer vision library) and pytesseract (an OCR library).

[0338] The server then uses FER (Emotion Recognition Library) to analyze the user's facial expressions in the screenshot and recognize the user's emotional state. For example, if the user is surprised, that information will be included in the analysis results.

[0339] Based on this information (the position and type of text and GUI elements, and the user's emotional state), the server infers the layout of the application screen and generates source code based on that. The generated source code is written in a programming language such as Python and creates an interface optimized for the user's emotions. For example, if surprise is detected, a warning message will be displayed on the screen.

[0340] Finally, the server packages the generated source code as a JSON or text file and provides a download link for users to view and edit their own generated code.

[0341] For example, a robot can take a photo of a dashboard in a factory and upload the image to a server. The server then analyzes the image, extracts text information using OCR technology, and recognizes the operator's emotions using an emotion recognition library. If the robot detects that the operator is surprised, the generated source code will include a warning message.

[0342] An example prompt might be, "Extract text information from a scanned screenshot of a factory dashboard, recognize emotions, and provide feedback. Please enter the path to the screenshot image."

[0343] The above is a specific embodiment for carrying out the present invention.

[0344] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0345] Step 1:

[0346] The user takes a screenshot of the application and uploads it to the server using their device.

[0347] Specific operation: A robot takes a picture of a dashboard in a factory and sends the image file to a server via a terminal. The input is a screenshot image file, and the output is an image file saved on the server.

[0348] Step 2:

[0349] The server stores the uploaded screenshot image.

[0350] Specific operation: The server saves the received image files in a specific directory and formats the file names in an easy-to-manage format. The input is the uploaded image file, and the output is an image file that has been saved and is available for subsequent processing.

[0351] Step 3:

[0352] The server converts the saved screenshot image to grayscale and uses OCR technology to extract the text in the image.

[0353] Specific operation: The server uses OpenCV to convert the image to grayscale, and then uses pytesseract to identify text parts in the image and extract the text information. The input is the saved grayscale image file, and the output is the extracted text information.

[0354] Step 4:

[0355] The server uses image processing algorithms to recognize GUI elements.

[0356] Specific operation: The server analyzes GUI components such as buttons and text fields in the image and identifies their positions and types. The input is a grayscale converted image file, and the output is the position and type information of the recognized GUI elements.

[0357] Step 5:

[0358] The server uses an emotion recognition library (such as FER) to analyze the user's facial expressions contained in the screenshot image and recognize the user's emotional state.

[0359] How it works: The server uses an emotion recognition library to analyze facial expressions in uploaded images and determine emotions such as surprise or joy. The input is an image file containing facial expressions, and the output is the recognized emotional state information.

[0360] Step 6:

[0361] The server predicts the layout of the application screen based on the recognized text information and the position information, type information, and emotional state of GUI elements.

[0362] Specific operation: The server integrates the collected information and applies an algorithm to design the optimal screen layout. The input is text information, GUI element information, and emotion information, and the output is the inferred application screen layout.

[0363] Step 7:

[0364] The server automatically generates source code based on the inferred layout and emotion information.

[0365] How it works: The server uses a programming language such as Python to generate source code that includes emotion-based user interface optimization. The input is the inferred layout and emotion information, and the output is the generated source code.

[0366] Step 8:

[0367] The server packages the generated source code and provides a link for users to download it.

[0368] Specific operation: The server compiles the source code as a JSON or text file and generates a link that users can use to download it. The input is the generated source code, and the output is the download link.

[0369] An example prompt is, "Extract text information from scanned screenshots of a factory dashboard, recognize emotions, and provide feedback. Please enter the path to the screenshot image."

[0370] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0371] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0372] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0373] [Second embodiment]

[0374] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0375] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0376] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0377] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0378] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0379] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0380] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0381] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0382] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0383] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0384] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0385] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0386] This invention is a system that reverse engineers applications based on screenshots and automatically generates corresponding source code, enabling rapid prototyping and redevelopment.

[0387] User-uploaded screenshots

[0388] A user takes a screenshot of an application and accesses the system's upload interface, where the user selects the screenshot and clicks the upload button, which causes the device to send the screenshot image to the server.

[0389] Saving a screenshot

[0390] The server saves the received screenshot. The server saves the uploaded image in a specific directory and sets the file name in an easy-to-manage format. This saving process makes it possible to reference the image data required for subsequent image analysis processing.

[0391] Image Processing and Element Recognition

[0392] The server analyzes the saved screenshot. First, the server performs grayscale conversion as a preprocessing of the image. Then, it uses OCR technology to extract text information from the image. At the same time, the server applies image processing algorithms to detect GUI elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[0393] Layout and Functionality Guessing

[0394] The server predicts the application's screen layout based on the text and GUI element information it recognizes. The server analyzes how each element is arranged on the screen and generates layout information. For example, it predicts the overall screen structure based on the position of buttons and the placement of text fields.

[0395] Code generation

[0396] The server automatically generates the corresponding source code based on the inferred layout information. The generated code is in accordance with the selected programming language and is used to reproduce the inferred layout and functionality. For example, it includes code to configure the screen layout using a GUI library and code to display the extracted text.

[0397] Providing reverse engineering results

[0398] The server provides the generated source code to the user. The server packages the generated code in JSON format or similar and provides a download link to the user. The user can use this link to download the generated source code and view or edit it.

[0399] Specific examples

[0400] For example, a user can upload a screenshot of a specific mobile application to the system. The image might contain multiple buttons and text fields. The server analyzes the image and identifies the positions and labels of the buttons and text fields. The server then uses this information to generate Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[0401] This system allows users to quickly obtain source code that precisely reproduces the screen layout of the target application, significantly reducing the painstaking manual redevelopment work required in the past. Furthermore, OCR technology and grayscale conversion preprocessing provide high recognition accuracy, resulting in more accurate reverse engineering results.

[0402] The above is an embodiment of the present invention.

[0403] The processing flow will be explained below.

[0404] Step 1:

[0405] The user takes a screenshot of the application and selects it by accessing the system's upload interface.

[0406] Step 2:

[0407] After the user selects a screenshot, they click the upload button, which causes the device to send the selected screenshot image to the server in an HTTP POST request.

[0408] Step 3:

[0409] The server saves the received screenshot image. The server saves the image in a specific directory and sets a file name for easy management.

[0410] Step 4:

[0411] The server loads the saved screenshot image. As a preprocessing step, the server first converts the screenshot to a grayscale image, which makes subsequent image analysis easier.

[0412] Step 5:

[0413] The server uses OCR technology to extract text information from the grayscale image and stores the extracted text information as digital data.

[0414] Step 6:

[0415] The server uses image processing algorithms to recognize GUI elements on the screenshot, specifically identifying interactive components such as buttons, input fields, and labels.

[0416] Step 7:

[0417] The server predicts the application screen layout based on the recognized text and GUI element information, and determines the overall layout structure based on the position and type of each element.

[0418] Step 8:

[0419] The server automatically generates the corresponding source code based on the estimated layout information. The server uses a GUI library in the selected programming language (e.g., Python) to generate code that reproduces the screen layout.

[0420] Step 9:

[0421] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link.

[0422] Step 10:

[0423] The user clicks on the link provided by the server to download the generated source code, which the user can then view and edit as needed.

[0424] The above is the specific processing flow of the program.

[0425] Example 1

[0426] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0427] Modern software development requires rapid redevelopment and prototyping of existing applications. However, manual coding is time-consuming and labor-intensive, especially when it comes to recreating GUI screen layouts. Furthermore, systems that use accurate OCR technology to recognize text and interface elements and generate code based on that recognition are still rare. This continues to require significant developer time.

[0428] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0429] In this invention, the server includes means for a user to upload screenshot images of a software application, means for the server to save the uploaded screenshot images, means for the server to analyze the saved screenshot images and recognize text and interface elements in the images, means for the server to infer a layout of the application screen based on the recognized text and interface elements, means for the server to generate program code based on the inferred layout, and means for the server to provide the generated program code to the user. This allows the user to quickly obtain highly accurate source code for redevelopment, significantly reducing the manual workload.

[0430] "User" means any person or entity that uses the System to upload screenshots of an Application and obtain the generated source code.

[0431] "Software application" refers generally to a program that runs on a computer for use by a user.

[0432] A "screenshot image" is image data that captures the screen of a software application.

[0433] "Uploading Method" refers to the function or process by which a user can send a screenshot image to the system.

[0434] "Server" refers to the computer system or device used to process user requests and store, analyze, and generate code from screenshots.

[0435] "Storage means" refers to a function or process that stores screenshot images in a specific location and makes them accessible as needed.

[0436] "Analysis Method" refers to the technology or algorithms used to analyze the saved screenshot image and recognize text and interface elements within the image.

[0437] "Text" refers to the written information extracted from an image through OCR technology.

[0438] "Interface elements" refers to GUI components such as buttons and text fields that exist within an image.

[0439] "Inference means" refers to a process or function for determining the layout of an application screen based on recognized text and interface elements.

[0440] "Program Code" means the set of source code that describes the operation of a software application.

[0441] "Providing means" refers to the functions and processes for transmitting or making the generated program code available for download to users.

[0442] "Tone conversion" refers to an image processing technique for converting the color information of a screenshot image to grayscale.

[0443] "Optical character recognition technology" refers to technology for extracting characters in an image as digital text.

[0444] MODE FOR CARRYING OUT THE INVENTION

[0445] This invention is a system that reverse engineers applications based on screenshots and automatically generates corresponding source code, enabling rapid prototyping and redevelopment.

[0446] User-uploaded screenshots

[0447] The user first takes a screenshot of the software application, then accesses the system's upload interface, selects the screenshot file, and clicks the upload button, which causes the device to send the screenshot image to the server.

[0448] Saving a screenshot

[0449] The server saves screenshots received from users. Specifically, the server saves the uploaded screenshot images in a specific directory and sets the file name in an easy-to-manage format. This saving process makes it possible to reference the image data required for subsequent image analysis processing.

[0450] Image Processing and Element Recognition

[0451] The server analyzes the saved screenshot. First, it performs grayscale conversion as image preprocessing. Then, it uses OCR technology (e.g., Tesseract OCR) to extract text information from the image. At the same time, the server applies an image processing algorithm (e.g., OpenCV) to detect interface elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[0452] Layout and Functionality Guessing

[0453] The server infers the application's screen layout based on the recognized text and interface element information. The server analyzes how each element is arranged on the screen and generates layout information. For example, it infers the overall screen structure based on the position of buttons and the arrangement of text fields.

[0454] Code generation

[0455] The server automatically generates the corresponding program code based on the inferred layout information. The generated code is based on the selected programming language (e.g., Python) and is used to reproduce the inferred layout and functionality. For example, it includes code to configure the screen layout using Python's Tkinter library and code to display the extracted text.

[0456] Providing reverse engineering results

[0457] The server provides the generated program code to the user. The server packages the generated code in JSON format or similar and provides the user with a download link. The user can use this link to download the generated source code and view or edit it.

[0458] Specific examples

[0459] For example, a user uploads a screenshot of a specific screen in a mobile application to the system. The image contains multiple buttons and text fields. The server analyzes the image and identifies the positions and labels of the buttons and text fields. Based on the identified information, the server then generates Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[0460] An example of a prompt to input to a generative AI model is, "Based on the screenshot below, please recognize GUI elements and generate source code using Python's Tkinter library."

[0461] This allows users to quickly obtain source code that precisely reproduces the screen layout of the target application, significantly reducing the effort required for manual redevelopment. Furthermore, OCR technology and grayscale conversion preprocessing enable high recognition accuracy, resulting in more accurate reverse engineering results. The above is an embodiment of the present invention.

[0462] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0463] Step 1:

[0464] A user takes a screenshot of the application, accesses the upload interface, selects the screenshot file, and clicks the upload button, which causes the device to send the screenshot image to the server.

[0465] Input: A screenshot file selected by the user.

[0466] Output: Screenshot data sent to the server

[0467] Specific operation: The device creates an HTTP request, attaches the screenshot image data, and sends it to the server.

[0468] Step 2:

[0469] The server receives the screenshot upload request and saves the screenshot image in a specific directory, renaming the file to a unique, easily manageable format (e.g., "screenshot_20231015_123456.png").

[0470] Input: Screenshot data sent from the device

[0471] Output: Screenshot files saved in a specific directory

[0472] Specific operation: The server saves the received screenshot data in the appropriate directory and sets a unique file name.

[0473] Step 3:

[0474] The server reads the saved screenshot and performs grayscale conversion as a preprocessing of the image, thereby highlighting the detailed information in the image.

[0475] Input: Saved screenshot file

[0476] Output: Grayscale converted image data

[0477] Specific operation: The server uses the OpenCV library to convert the screenshot image to grayscale.

[0478] Step 4:

[0479] The server uses OCR technology (e.g., Tesseract OCR) to extract text information from the grayscale image. At the same time, the server uses image processing algorithms to detect interface elements (buttons, text fields, etc.) in the image.

[0480] Input: Grayscale converted image data

[0481] Output: Text information and data about the location and type of interface elements

[0482] What it does: The server uses Tesseract OCR to extract text from images and OpenCV algorithms to detect interface elements.

[0483] Step 5:

[0484] The server analyzes the recognized text and interface element information to predict the layout of the application screen, thereby determining the overall screen structure and generating layout information.

[0485] Input: Text information and data about the location and type of interface elements

[0486] Output: Application screen layout information (e.g., JSON format)

[0487] Specific operation: The server analyzes the position and type of each element, generates layout information, and saves it in JSON format.

[0488] Step 6:

[0489] The server automatically generates the corresponding program code based on the inferred layout information. The code is generated in the selected programming language (e.g., Python), and includes GUI layout code using the Tkinter library.

[0490] Input: Application screen layout information

[0491] Output: Generated program code

[0492] Specific operation: The server uses a generative AI model to automatically generate program code based on layout information.

[0493] Step 7:

[0494] The server packages the generated program code in JSON format or similar and provides a download link to the user, who can then use the link to download the generated source code and view or edit it.

[0495] Input: Generated program code

[0496] Output: Download link

[0497] Specific operation: The server packages the generated code and provides the user with a download link via email or other means.

[0498] (Application example 1)

[0499] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0500] Designing and redesigning machine control panels in a factory takes a great deal of time and effort. When each machine has a different control panel, it is difficult to reuse the same design drawings, and in order to improve factory productivity, it is necessary to generate the source code for the control panel quickly and accurately. Current technology mainly involves manual design and program porting, which requires specialized knowledge and time. This poses the problem of reduced factory efficiency and increased costs.

[0501] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0502] In this invention, the server includes: means for a user to upload screenshot images of an application; means for the server to save the uploaded screenshot images; means for the server to analyze the saved screenshot images and recognize text and GUI elements in the images; means for the server to infer an application screen layout based on the recognized text and GUI elements; means for the server to generate source code based on the inferred layout; means for the server to provide the generated source code to the user; means for reverse engineering screenshots of machine operation panels in a factory; means for automatically generating source code for a machine operation program based on screenshot images of a display device to be used; and means for performing image analysis and code generation via a cloud server. This enables rapid and accurate design and redesign of machine operation panels in a factory.

[0503] "User" means any person or institution that uses the System to upload screenshots of an Application.

[0504] A "screenshot image of an application" refers to a still image of an application screen, and is data for visualizing the contents of that screen.

[0505] The "server" is a computer system that stores the uploaded screenshot images, analyzes them, and ultimately generates the source code.

[0506] The "storing means" refers to the process and technology by which the server stores the received screenshot images in a specific directory.

[0507] The "analyzing means" is a technology for recognizing and extracting text and GUI elements from the saved screenshot image.

[0508] "Text" refers to the text information displayed in the screenshot image.

[0509] "GUI elements" refer to components of a user interface, such as buttons and text fields.

[0510] "Guessing means" is a technique for predicting the layout of an application screen based on recognized text and GUI elements.

[0511] "Source code" is a document written in a particular programming language that describes the behavior of a program.

[0512] The "machine operation panel in a factory" is an interface including a display device and an input device for operating a machine installed in a factory.

[0513] "Reverse engineering" refers to the technique of analyzing existing products or code and recreating their structure and function in order to understand the operation and design of the device or software.

[0514] The "display device used" is a visual output device such as a display or monitor included in a machine control panel in a factory.

[0515] A "cloud server" is a server infrastructure that can be accessed remotely over the internet and provides the computational resources to perform image analysis and code generation.

[0516] This invention is a system that performs reverse engineering based on screenshots of machine operation panels in factories and automatically generates corresponding source code, enabling rapid prototyping and redesign, thereby improving factory production efficiency.

[0517] User-uploaded screenshots

[0518] Users can take screenshots of machine control panels in the factory using a smartphone or head-mounted display, and then access the system's upload interface to upload the screenshot, which causes the device to send the screenshot image to the cloud server.

[0519] Saving a screenshot

[0520] The cloud server saves the received screenshots. The server saves the uploaded images in a specific directory and names the files in an easy-to-manage format. This saving process allows the image data required for subsequent image analysis to be referenced.

[0521] Image Processing and Element Recognition

[0522] The cloud server analyzes the stored screenshots and extracts information from the images using the following techniques:

[0523] Converting an image to grayscale using Pillow.

[0524] Pytesseract is used to perform OCR processing and extract text information from the image.

[0525] At the same time, it applies image processing algorithms to detect GUI elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[0526] Layout and Functionality Guessing

[0527] The cloud server predicts the screen layout of the machine operation panel based on the recognized text and GUI element information. The server analyzes how each element is arranged on the screen and generates layout information. For example, it predicts the overall screen structure based on the position of buttons and the arrangement of text fields.

[0528] Generate source code

[0529] The cloud server automatically generates the corresponding source code based on the inferred layout information. The generated code is written in the selected programming language to reproduce the inferred layout and functionality. For example, it generates Python code using the Tkinter library to reproduce the screen according to the recognized layout.

[0530] Providing reverse engineering results

[0531] The cloud server provides the generated source code to the user. The server packages the generated code in JSON format or similar and provides a download link to the user. The user can use this link to download the generated source code and view or edit it.

[0532] Specific examples

[0533] For example, a user can upload a screenshot of an operation panel to a cloud server. The image contains several buttons and text fields. The cloud server analyzes the image and identifies the positions and labels of the buttons and text fields. The server then uses this information to generate Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[0534] Prompt Sentence Examples

[0535] Analyze the panel image below, extract the position and label of each GUI element, and generate a program to reproduce it using Tkinter. Image link: [Link URL]

[0536] The system enables the rapid and accurate design and redesign of machine control panels in factories, significantly reducing the traditional manual and laborious redevelopment work. Furthermore, OCR technology and grayscale conversion preprocessing provide high recognition accuracy and more accurate reverse engineering results.

[0537] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0538] Step 1: User uploads screenshot

[0539] The user takes a screenshot of a machine control panel in the factory, accesses the system's upload interface via a smartphone or head-mounted display, and clicks the upload button to send the screenshot to the cloud server.

[0540] Input: Screenshot of the machine control panel

[0541] Output: Screenshot image is sent to the cloud server

[0542] Step 2: Save the screenshot

[0543] The cloud server saves the received screenshot images. The server stores the uploaded images in a specific directory and names the files in an easy-to-manage format.

[0544] Input: Screenshot image sent by the user

[0545] Output: Saved screenshot image

[0546] Step 3: Image preprocessing

[0547] The cloud server preprocesses the saved screenshot images and converts them to grayscale using the Pillow library.

[0548] Input: Saved screenshot image

[0549] Output: Image converted to grayscale

[0550] Step 4: Parsing Text and GUI Elements

[0551] The server uses OCR technology (Pytesseract) to extract text information from the grayscale image, and simultaneously applies image processing algorithms to detect GUI elements (buttons, text fields, etc.) in the image.

[0552] Input: Image converted to grayscale

[0553] Output: Extracted text information and the location and type of GUI elements

[0554] Step 5: Inferring the screen layout

[0555] Based on the text and GUI element information recognized by the cloud server, the screen layout of the machine operation panel is analyzed and layout information is generated.

[0556] Input: Extracted text information and the position and type of GUI elements

[0557] Output: Inferred screen layout information

[0558] Step 6: Generate source code

[0559] The cloud server automatically generates the corresponding source code based on the inferred layout information. Specifically, it generates Python code using the Tkinter library to reproduce the screen according to the recognized layout.

[0560] Input: Inferred screen layout information

[0561] Output: Generated source code

[0562] Step 7: Provide generated source code

[0563] The cloud server packages the generated source code in JSON format or similar and provides the user with a download link, which the user can use to download, view, and edit the generated source code.

[0564] Input: Generated source code

[0565] Output: A download link provided to the user

[0566] By following the above steps, it is possible to quickly and accurately generate source code for an operation program through reverse engineering of a machine operation panel in a factory.

[0567] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0568] This invention is a system that reverse engineers applications based on screenshots and automatically generates the corresponding source code, and also has the ability to recognize and utilize user emotions, which improves the user experience and enables more feedback-driven interactive application prototyping.

[0569] User-uploaded screenshots

[0570] A user takes a screenshot of an application and accesses the system's upload interface. The user selects a screenshot and clicks the upload button, causing the device to send the selected screenshot image to the server.

[0571] Saving a screenshot

[0572] The server saves the screenshots it receives. The images are saved in a specific directory and given file names for easy management. This saving process allows the image data to be referenced for subsequent image analysis and emotion recognition processing.

[0573] Image Processing and Element Recognition

[0574] The server reads the saved screenshot and converts it into a grayscale image. It uses OCR technology to extract text information from the grayscale image, while simultaneously using image processing algorithms to recognize GUI elements. This identifies the location and type of text and GUI elements on the screen.

[0575] emotion recognition

[0576] The server uses an emotion engine to recognize the user's emotion based on the screenshot. The emotion engine identifies the user's emotional state (e.g., joy, surprise, sadness, etc.) by analyzing facial expressions and gaze in the image.

[0577] Layout and Functionality Guessing

[0578] The server predicts the application screen layout based on the recognized text and GUI element information and emotional information. The server determines the optimal layout by taking into account the position and type of each element as well as the user's emotional state.

[0579] Code generation

[0580] The server automatically generates the corresponding source code based on the estimated layout information and the user's emotional information. The server uses a selected programming language (e.g., Python) to generate user interface code that reflects the emotional information. For example, if the user is surprised, the code could display a warning message.

[0581] Providing reverse engineering results

[0582] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link, and the user clicks the link to download the generated source code. The downloaded code can then be viewed and edited.

[0583] Specific examples

[0584] For example, a user uploads a screenshot of a specific mobile application. The server recognizes that the image contains multiple buttons and text fields and uses an emotion engine to detect that the user is surprised. Based on this information, the server generates Python code that includes the screen layout and feedback corresponding to the surprise. This code uses the Tkinter library to position the buttons and text fields appropriately and displays a message reflecting the surprise on the screen.

[0585] This system allows users to quickly obtain source code that precisely replicates the screen layout of the target application, and even enables interactive responses based on user emotions within the system. This not only significantly reduces the amount of manual redevelopment work and user experience tracking required, but also enables more adaptive and user-centric application development through emotion-based feedback.

[0586] The above is an embodiment of the present invention.

[0587] The processing flow will be explained below.

[0588] Step 1:

[0589] The user takes a screenshot of the application. The user accesses the system's upload interface, selects the screenshot, and clicks the upload button.

[0590] Step 2:

[0591] The device sends the selected screenshot image to the server in the form of an HTTP POST request.

[0592] Step 3:

[0593] The screenshot image received by the server is saved in a specific directory. When saving, the file name is set in an easy-to-manage format.

[0594] Step 4:

[0595] The server loads the saved screenshot image and performs grayscale conversion as a preprocessing step to simplify image processing.

[0596] Step 5:

[0597] The server extracts text information from the grayscale image using OCR technology and stores the extracted text information as digital data.

[0598] Step 6:

[0599] The server uses image processing algorithms to recognize GUI elements (buttons, input fields, etc.) on the screenshot and identify the location and type of each element.

[0600] Step 7:

[0601] The server uses an emotion engine to recognize the user's emotions from the screenshots, for example, by analyzing the user's facial expressions and gaze to identify emotions such as joy, surprise, and sadness.

[0602] Step 8:

[0603] The server predicts the application screen layout based on the text and GUI element information recognized by the server, as well as emotional information. The optimal layout is determined taking into account the user's emotional state.

[0604] Step 9:

[0605] The server automatically generates the corresponding source code based on the inferred layout information and the user's emotion information. The code is generated using a GUI library in the selected programming language (e.g., Python), and includes emotion-based feedback as needed.

[0606] Step 10:

[0607] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link.

[0608] Step 11:

[0609] The user clicks the link sent by the server to download the generated source code, which the user can then view and edit as needed.

[0610] The above is the specific processing flow of a system that combines an emotion engine.

[0611] Example 2

[0612] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0613] Conventional systems require a great deal of time and effort for users to reverse engineer application screens and generate source code. Furthermore, they are unable to provide interactive feedback that takes into account the user's emotional state, limiting the user experience. Therefore, there is a need for a system that can efficiently generate application prototypes while improving the user experience.

[0614] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for a user to upload screenshot images of an application; means for the server to save the uploaded screenshot images; means for the server to analyze the saved screenshot images and recognize text and user interface elements in the images; means for the server to analyze facial expressions and gaze to recognize the user's emotions; means for the server to infer the layout of the application screen based on the recognized text, user interface elements, and emotional information; means for the server to generate source code based on the inferred layout and the user's emotional information; and means for the server to provide the generated source code to the user. This allows the user to quickly obtain source code that reproduces the screen layout of the target application and further receive interactive feedback based on emotions.

[0615] "User" refers to the person who uploads screenshots of an application and requests the generation of source code.

[0616] "Server" refers to the computer system that stores and analyzes screenshots uploaded by users and generates and provides source code.

[0617] A "screenshot image" is an image of the application screen display saved as is.

[0618] "Uploading means" refers to the interface and protocol that allows users to send screenshot images to the server.

[0619] "Storage means" refers to a function that allows the server to store uploaded screenshot images in a specific directory so that they can be used for later analysis.

[0620] "Analysis means" refers to a technique by which the server processes the screenshot image and identifies textual information and user interface elements within the image.

[0621] "User interface elements" refer to interactive components such as buttons and text fields within an application screen.

[0622] "Emotion recognition means" refers to technology that allows the server to analyze the user's facial expressions and gaze using screenshot images to identify their emotional state.

[0623] "Inference means" refers to a technology that infers the layout and functionality of an application screen based on recognized text, user interface elements, and emotion information.

[0624] "Source code generation means" refers to a technology that automatically generates source code in an appropriate programming language based on the estimated layout and user emotional information.

[0625] "Means of provision" refers to the mechanism for providing the generated source code to users in a downloadable format.

[0626] This invention is a system that reverse engineers applications based on screenshots uploaded by users and automatically generates corresponding source code. The system also includes a function to recognize users' emotions and provide interactive feedback based on the emotion information.

[0627] The user first takes a screenshot of the application and accesses the system's upload interface. The terminal (user's device) selects the screenshot and clicks the upload button. This action causes the terminal to send the selected screenshot image to the server. The upload is performed using a web form and an HTTP POST request.

[0628] The server saves the received screenshots in a specific directory, with file names that can be uniquely identified using a timestamp or UUID. The saved screenshots are used for later analysis.

[0629] The server reads the saved screenshot and converts it to a grayscale image using OpenCV, then uses the Tesseract OCR engine to extract text information from the grayscale image, while also using OpenCV to recognize the location and type of GUI elements (buttons, text fields, etc.).

[0630] The server then uses an emotion recognition engine (e.g., Hume or Affectiva) to recognize the user's emotions based on the screenshot, analyzing facial expressions and gaze to identify emotional states (e.g., happiness, surprise, sadness, etc.).

[0631] The server predicts the layout of the application screen based on the recognized text information, GUI element information, and emotion information. It determines the optimal layout by taking into account the user's emotional state in addition to the location and type.

[0632] Based on the inferred layout information and user emotion information, the server automatically generates the corresponding source code. Specifically, it generates user interface code using Python and the Tkinter library. For example, if the user is surprised, it generates code to display a message reflecting surprise.

[0633] Finally, the server packages the generated source code as a JSON or text file and generates a link to provide to the user, who can click on this link to download the generated source code for viewing or editing.

[0634] Specific examples

[0635] For example, a user uploads a screenshot of a specific mobile application. The server recognizes that the image contains several buttons and text fields. If the emotion engine detects that the user is surprised, the server uses this information to generate Python code that includes the screen layout and feedback corresponding to the surprise. This code uses the Tkinter library to position the buttons and text fields appropriately and displays a message reflecting the surprise on the screen.

[0636] Prompt Sentence Examples

[0637] "I uploaded a screenshot of a user looking surprised. The screen contains several buttons and text fields. Based on this, please generate Python code that displays feedback that reflects the user's surprise."

[0638] This system helps users quickly obtain source code that reproduces the screen layout of the target application and create an interface that includes emotion-based interactive feedback. Not only does it significantly reduce the traditional manual redevelopment work and user experience feedback tracking, but emotion-based feedback also enables more adaptive and user-centric application development.

[0639] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0640] Step 1:

[0641] The user takes a screenshot of the application and accesses the upload interface. The terminal sends the screenshot image selected by the user to the server. The input data is the screenshot image selected by the user, and the output data is the image file sent to the server.

[0642] Step 2:

[0643] The server saves the screenshots it receives in a specific directory. A timestamp or UUID is used to make the file name unique when saved. The input data is the screenshot image sent to the server, and the output data is the path to the saved image file. Specifically, the server writes the received image data to the file system.

[0644] Step 3:

[0645] The server reads the saved screenshot and converts it into a grayscale image using OpenCV. The input data is the saved image file, and the output data is a grayscale image. Specifically, the server uses the image processing library OpenCV to convert the color image into grayscale.

[0646] Step 4:

[0647] The server uses the Tesseract OCR engine to extract text information from grayscale images. At the same time, it uses OpenCV to recognize the location and type of GUI elements (buttons, text fields, etc.). The input data is a grayscale image, and the output data is the recognized text data and GUI element information. Specifically, the server performs OCR to convert the characters in the image into text and uses a contour detection algorithm to identify GUI elements.

[0648] Step 5:

[0649] The server uses an emotion recognition engine (e.g., Hume or Affectiva) to recognize the user's emotion based on the screenshot. The input data is the screenshot image, and the output data is the recognized emotion data. Specifically, the server applies face recognition and emotion classification algorithms.

[0650] Step 6:

[0651] The server infers the layout of the application screen based on the recognized text information, GUI element information, and emotion information. The input data is the recognized text information, GUI element information, and emotion information, and the output data is the inferred layout information. Specifically, the server applies a rule-based layout inference algorithm to determine the placement and characteristics of each element.

[0652] Step 7:

[0653] The server automatically generates the corresponding source code based on the inferred layout information and user emotion information. The input data is the inferred layout information and emotion information, and the output data is the generated source code. Specifically, the server generates the user interface source code using Python and Tkinter.

[0654] Step 8:

[0655] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The input data is the generated source code, and the output data is a download link provided to the user. Specifically, the server packages the generated content, creates an HTTP link, and notifies the user.

[0656] (Application example 2)

[0657] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0658] In recent years, with the advancement of automation in factories, the use of factory robots for inspection and monitoring has been increasing. However, conventional factory robots have limited ability to acquire and analyze information from dashboards and panels, and are unable to provide real-time feedback based on the emotions of human operators. This makes efficient and interactive inspection work difficult, and there is a need for an improved user experience.

[0659] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0660] In this invention, the server includes means for a user to upload screenshot images of an application, means for the server to save the uploaded screenshot images, means for the server to analyze the saved screenshot images and recognize text and GUI elements in the images, means for the server to infer an application screen layout based on the recognized text and GUI elements, means for the server to generate source code based on the inferred layout, means for the server to provide the generated source code to the user, means for the server to recognize user emotions based on the uploaded screenshot images, and means for the server to optimize the user interface based on the recognized emotion information. This enables factory robots to efficiently acquire and analyze information from dashboards and panels and provide interactive feedback to human operators according to their emotions.

[0661] A "screenshot image" is an image file that a user records as a still image of the contents displayed on the screen of a device.

[0662] "Upload" is the act of a user sending data from their device to a server.

[0663] A "server" is a computer system that stores, manages, and processes information on a network.

[0664] "Grayscale conversion" is a process of converting a color image into an image that contains only brightness information.

[0665] "OCR" stands for Optical Character Recognition, a technology that converts characters in an image into digital text.

[0666] "Text and GUI elements" refers to the textual information and graphical user interface components displayed on the screen.

[0667] "Layout" is the arrangement design of how elements on a screen are arranged.

[0668] "Source code" is a human-readable form of code that describes a program.

[0669] "Emotion recognition" is a technology that analyzes a user's facial expressions from images and videos and determines their psychological state.

[0670] "User interface" refers to the screens and operating specifications used to exchange information between the user and the system.

[0671] "Optimization" means adjusting the system's performance and functionality to the best possible state based on certain criteria.

[0672] This invention relates to a system that provides optimized feedback to users through information analysis and emotion recognition of dashboards and panels by an inspection robot in a factory. Specific embodiments of this system are described below.

[0673] The system begins with the robot taking images of each piece of equipment and dashboard in the factory, and then uploading the screenshots from the user's device. These images are then transferred to the server via the Internet and uploaded. The server saves the uploaded screenshots in a specific directory and assigns them file names for easy management.

[0674] The server then analyzes the saved screenshot image. This analysis process first converts the colorful image to grayscale, then uses OCR technology to extract text information from the image. At the same time, it uses image processing algorithms to recognize GUI elements. This step uses OpenCV (an open-source computer vision library) and pytesseract (an OCR library).

[0675] The server then uses FER (Emotion Recognition Library) to analyze the user's facial expressions in the screenshot and recognize the user's emotional state. For example, if the user is surprised, that information will be included in the analysis results.

[0676] Based on this information (the position and type of text and GUI elements, and the user's emotional state), the server infers the layout of the application screen and generates source code based on that. The generated source code is written in a programming language such as Python and creates an interface optimized for the user's emotions. For example, if surprise is detected, a warning message will be displayed on the screen.

[0677] Finally, the server packages the generated source code as a JSON or text file and provides a download link for users to view and edit their own generated code.

[0678] For example, a robot can take a photo of a dashboard in a factory and upload the image to a server. The server then analyzes the image, extracts text information using OCR technology, and recognizes the operator's emotions using an emotion recognition library. If the robot detects that the operator is surprised, the generated source code will include a warning message.

[0679] An example prompt might be, "Extract text information from a scanned screenshot of a factory dashboard, recognize emotions, and provide feedback. Please enter the path to the screenshot image."

[0680] The above is a specific embodiment for carrying out the present invention.

[0681] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0682] Step 1:

[0683] The user takes a screenshot of the application and uploads it to the server using their device.

[0684] Specific operation: A robot takes a picture of a dashboard in a factory and sends the image file to a server via a terminal. The input is a screenshot image file, and the output is an image file saved on the server.

[0685] Step 2:

[0686] The server stores the uploaded screenshot image.

[0687] Specific operation: The server saves the received image files in a specific directory and formats the file names in an easy-to-manage format. The input is the uploaded image file, and the output is an image file that has been saved and is available for subsequent processing.

[0688] Step 3:

[0689] The server converts the saved screenshot image to grayscale and uses OCR technology to extract the text in the image.

[0690] Specific operation: The server uses OpenCV to convert the image to grayscale, and then uses pytesseract to identify text parts in the image and extract the text information. The input is the saved grayscale image file, and the output is the extracted text information.

[0691] Step 4:

[0692] The server uses image processing algorithms to recognize GUI elements.

[0693] Specific operation: The server analyzes GUI components such as buttons and text fields in the image and identifies their positions and types. The input is a grayscale converted image file, and the output is the position and type information of the recognized GUI elements.

[0694] Step 5:

[0695] The server uses an emotion recognition library (such as FER) to analyze the user's facial expressions contained in the screenshot image and recognize the user's emotional state.

[0696] How it works: The server uses an emotion recognition library to analyze facial expressions in uploaded images and determine emotions such as surprise or joy. The input is an image file containing facial expressions, and the output is the recognized emotional state information.

[0697] Step 6:

[0698] The server predicts the layout of the application screen based on the recognized text information and the position information, type information, and emotional state of GUI elements.

[0699] Specific operation: The server integrates the collected information and applies an algorithm to design the optimal screen layout. The input is text information, GUI element information, and emotion information, and the output is the inferred application screen layout.

[0700] Step 7:

[0701] The server automatically generates source code based on the inferred layout and emotion information.

[0702] How it works: The server uses a programming language such as Python to generate source code that includes emotion-based user interface optimization. The input is the inferred layout and emotion information, and the output is the generated source code.

[0703] Step 8:

[0704] The server packages the generated source code and provides a link for users to download it.

[0705] Specific operation: The server compiles the source code as a JSON or text file and generates a link that users can use to download it. The input is the generated source code, and the output is the download link.

[0706] An example prompt is, "Extract text information from scanned screenshots of a factory dashboard, recognize emotions, and provide feedback. Please enter the path to the screenshot image."

[0707] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0708] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0709] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0710] [Third embodiment]

[0711] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0712] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0713] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0714] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0715] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0716] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0717] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0718] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0719] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0720] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0721] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0722] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0723] This invention is a system that reverse engineers applications based on screenshots and automatically generates corresponding source code, enabling rapid prototyping and redevelopment.

[0724] User-uploaded screenshots

[0725] A user takes a screenshot of an application and accesses the system's upload interface, where the user selects the screenshot and clicks the upload button, which causes the device to send the screenshot image to the server.

[0726] Saving a screenshot

[0727] The server saves the received screenshot. The server saves the uploaded image in a specific directory and sets the file name in an easy-to-manage format. This saving process makes it possible to reference the image data required for subsequent image analysis processing.

[0728] Image Processing and Element Recognition

[0729] The server analyzes the saved screenshot. First, the server performs grayscale conversion as a preprocessing of the image. Then, it uses OCR technology to extract text information from the image. At the same time, the server applies image processing algorithms to detect GUI elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[0730] Layout and Functionality Guessing

[0731] The server predicts the application's screen layout based on the text and GUI element information it recognizes. The server analyzes how each element is arranged on the screen and generates layout information. For example, it predicts the overall screen structure based on the position of buttons and the placement of text fields.

[0732] Code generation

[0733] The server automatically generates the corresponding source code based on the inferred layout information. The generated code is in accordance with the selected programming language and is used to reproduce the inferred layout and functionality. For example, it includes code to configure the screen layout using a GUI library and code to display the extracted text.

[0734] Providing reverse engineering results

[0735] The server provides the generated source code to the user. The server packages the generated code in JSON format or similar and provides a download link to the user. The user can use this link to download the generated source code and view or edit it.

[0736] Specific examples

[0737] For example, a user can upload a screenshot of a specific mobile application to the system. The image might contain multiple buttons and text fields. The server analyzes the image and identifies the positions and labels of the buttons and text fields. The server then uses this information to generate Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[0738] This system allows users to quickly obtain source code that precisely reproduces the screen layout of the target application, significantly reducing the painstaking manual redevelopment work required in the past. Furthermore, OCR technology and grayscale conversion preprocessing provide high recognition accuracy, resulting in more accurate reverse engineering results.

[0739] The above is an embodiment of the present invention.

[0740] The processing flow will be explained below.

[0741] Step 1:

[0742] The user takes a screenshot of the application and selects it by accessing the system's upload interface.

[0743] Step 2:

[0744] After the user selects a screenshot, they click the upload button, which causes the device to send the selected screenshot image to the server in an HTTP POST request.

[0745] Step 3:

[0746] The server saves the received screenshot image. The server saves the image in a specific directory and sets a file name for easy management.

[0747] Step 4:

[0748] The server loads the saved screenshot image. As a preprocessing step, the server first converts the screenshot to a grayscale image, which makes subsequent image analysis easier.

[0749] Step 5:

[0750] The server uses OCR technology to extract text information from the grayscale image and stores the extracted text information as digital data.

[0751] Step 6:

[0752] The server uses image processing algorithms to recognize GUI elements on the screenshot, specifically identifying interactive components such as buttons, input fields, and labels.

[0753] Step 7:

[0754] The server predicts the application screen layout based on the recognized text and GUI element information, and determines the overall layout structure based on the position and type of each element.

[0755] Step 8:

[0756] The server automatically generates the corresponding source code based on the estimated layout information. The server uses a GUI library in the selected programming language (e.g., Python) to generate code that reproduces the screen layout.

[0757] Step 9:

[0758] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link.

[0759] Step 10:

[0760] The user clicks on the link provided by the server to download the generated source code, which the user can then view and edit as needed.

[0761] The above is the specific processing flow of the program.

[0762] Example 1

[0763] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0764] Modern software development requires rapid redevelopment and prototyping of existing applications. However, manual coding is time-consuming and labor-intensive, especially when it comes to recreating GUI screen layouts. Furthermore, systems that use accurate OCR technology to recognize text and interface elements and generate code based on that recognition are still rare. This continues to require significant developer time.

[0765] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0766] In this invention, the server includes means for a user to upload screenshot images of a software application, means for the server to save the uploaded screenshot images, means for the server to analyze the saved screenshot images and recognize text and interface elements in the images, means for the server to infer a layout of the application screen based on the recognized text and interface elements, means for the server to generate program code based on the inferred layout, and means for the server to provide the generated program code to the user. This allows the user to quickly obtain highly accurate source code for redevelopment, significantly reducing the manual workload.

[0767] "User" means any person or entity that uses the System to upload screenshots of an Application and obtain the generated source code.

[0768] "Software application" refers generally to a program that runs on a computer for use by a user.

[0769] A "screenshot image" is image data that captures the screen of a software application.

[0770] "Uploading Method" refers to the function or process by which a user can send a screenshot image to the system.

[0771] "Server" refers to the computer system or device used to process user requests and store, analyze, and generate code from screenshots.

[0772] "Storage means" refers to a function or process that stores screenshot images in a specific location and makes them accessible as needed.

[0773] "Analysis Method" refers to the technology or algorithms used to analyze the saved screenshot image and recognize text and interface elements within the image.

[0774] "Text" refers to the written information extracted from an image through OCR technology.

[0775] "Interface elements" refers to GUI components such as buttons and text fields that exist within an image.

[0776] "Inference means" refers to a process or function for determining the layout of an application screen based on recognized text and interface elements.

[0777] "Program Code" means the set of source code that describes the operation of a software application.

[0778] "Providing means" refers to the functions and processes for transmitting or making the generated program code available for download to users.

[0779] "Tone conversion" refers to an image processing technique for converting the color information of a screenshot image to grayscale.

[0780] "Optical character recognition technology" refers to technology for extracting characters in an image as digital text.

[0781] MODE FOR CARRYING OUT THE INVENTION

[0782] This invention is a system that reverse engineers applications based on screenshots and automatically generates corresponding source code, enabling rapid prototyping and redevelopment.

[0783] User-uploaded screenshots

[0784] The user first takes a screenshot of the software application, then accesses the system's upload interface, selects the screenshot file, and clicks the upload button, which causes the device to send the screenshot image to the server.

[0785] Saving a screenshot

[0786] The server saves screenshots received from users. Specifically, the server saves the uploaded screenshot images in a specific directory and sets the file name in an easy-to-manage format. This saving process makes it possible to reference the image data required for subsequent image analysis processing.

[0787] Image Processing and Element Recognition

[0788] The server analyzes the saved screenshot. First, it performs grayscale conversion as image preprocessing. Then, it uses OCR technology (e.g., Tesseract OCR) to extract text information from the image. At the same time, the server applies an image processing algorithm (e.g., OpenCV) to detect interface elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[0789] Layout and Functionality Guessing

[0790] The server infers the application's screen layout based on the recognized text and interface element information. The server analyzes how each element is arranged on the screen and generates layout information. For example, it infers the overall screen structure based on the position of buttons and the arrangement of text fields.

[0791] Code generation

[0792] The server automatically generates the corresponding program code based on the inferred layout information. The generated code is based on the selected programming language (e.g., Python) and is used to reproduce the inferred layout and functionality. For example, it includes code to configure the screen layout using Python's Tkinter library and code to display the extracted text.

[0793] Providing reverse engineering results

[0794] The server provides the generated program code to the user. The server packages the generated code in JSON format or similar and provides the user with a download link. The user can use this link to download the generated source code and view or edit it.

[0795] Specific examples

[0796] For example, a user uploads a screenshot of a specific screen in a mobile application to the system. The image contains multiple buttons and text fields. The server analyzes the image and identifies the positions and labels of the buttons and text fields. Based on the identified information, the server then generates Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[0797] An example of a prompt to input to a generative AI model is, "Based on the screenshot below, please recognize GUI elements and generate source code using Python's Tkinter library."

[0798] This allows users to quickly obtain source code that precisely reproduces the screen layout of the target application, significantly reducing the effort required for manual redevelopment. Furthermore, OCR technology and grayscale conversion preprocessing enable high recognition accuracy, resulting in more accurate reverse engineering results. The above is an embodiment of the present invention.

[0799] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0800] Step 1:

[0801] A user takes a screenshot of the application, accesses the upload interface, selects the screenshot file, and clicks the upload button, which causes the device to send the screenshot image to the server.

[0802] Input: A screenshot file selected by the user.

[0803] Output: Screenshot data sent to the server

[0804] Specific operation: The device creates an HTTP request, attaches the screenshot image data, and sends it to the server.

[0805] Step 2:

[0806] The server receives the screenshot upload request and saves the screenshot image in a specific directory, renaming the file to a unique, easily manageable format (e.g., "screenshot_20231015_123456.png").

[0807] Input: Screenshot data sent from the device

[0808] Output: Screenshot files saved in a specific directory

[0809] Specific operation: The server saves the received screenshot data in the appropriate directory and sets a unique file name.

[0810] Step 3:

[0811] The server reads the saved screenshot and performs grayscale conversion as a preprocessing of the image, thereby highlighting the detailed information in the image.

[0812] Input: Saved screenshot file

[0813] Output: Grayscale converted image data

[0814] Specific operation: The server uses the OpenCV library to convert the screenshot image to grayscale.

[0815] Step 4:

[0816] The server uses OCR technology (e.g., Tesseract OCR) to extract text information from the grayscale image. At the same time, the server uses image processing algorithms to detect interface elements (buttons, text fields, etc.) in the image.

[0817] Input: Grayscale converted image data

[0818] Output: Text information and data about the location and type of interface elements

[0819] What it does: The server uses Tesseract OCR to extract text from images and OpenCV algorithms to detect interface elements.

[0820] Step 5:

[0821] The server analyzes the recognized text and interface element information to predict the layout of the application screen, thereby determining the overall screen structure and generating layout information.

[0822] Input: Text information and data about the location and type of interface elements

[0823] Output: Application screen layout information (e.g., JSON format)

[0824] Specific operation: The server analyzes the position and type of each element, generates layout information, and saves it in JSON format.

[0825] Step 6:

[0826] The server automatically generates the corresponding program code based on the inferred layout information. The code is generated in the selected programming language (e.g., Python), and includes GUI layout code using the Tkinter library.

[0827] Input: Application screen layout information

[0828] Output: Generated program code

[0829] Specific operation: The server uses a generative AI model to automatically generate program code based on layout information.

[0830] Step 7:

[0831] The server packages the generated program code in JSON format or similar and provides a download link to the user, who can then use the link to download the generated source code and view or edit it.

[0832] Input: Generated program code

[0833] Output: Download link

[0834] Specific operation: The server packages the generated code and provides the user with a download link via email or other means.

[0835] (Application example 1)

[0836] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0837] Designing and redesigning machine control panels in a factory takes a great deal of time and effort. When each machine has a different control panel, it is difficult to reuse the same design drawings, and in order to improve factory productivity, it is necessary to generate the source code for the control panel quickly and accurately. Current technology mainly involves manual design and program porting, which requires specialized knowledge and time. This poses the problem of reduced factory efficiency and increased costs.

[0838] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0839] In this invention, the server includes: means for a user to upload screenshot images of an application; means for the server to save the uploaded screenshot images; means for the server to analyze the saved screenshot images and recognize text and GUI elements in the images; means for the server to infer an application screen layout based on the recognized text and GUI elements; means for the server to generate source code based on the inferred layout; means for the server to provide the generated source code to the user; means for reverse engineering screenshots of machine operation panels in a factory; means for automatically generating source code for a machine operation program based on screenshot images of a display device to be used; and means for performing image analysis and code generation via a cloud server. This enables rapid and accurate design and redesign of machine operation panels in a factory.

[0840] "User" means any person or institution that uses the System to upload screenshots of an Application.

[0841] A "screenshot image of an application" refers to a still image of an application screen, and is data for visualizing the contents of that screen.

[0842] The "server" is a computer system that stores the uploaded screenshot images, analyzes them, and ultimately generates the source code.

[0843] The "storing means" refers to the process and technology by which the server stores the received screenshot images in a specific directory.

[0844] The "analyzing means" is a technology for recognizing and extracting text and GUI elements from the saved screenshot image.

[0845] "Text" refers to the text information displayed in the screenshot image.

[0846] "GUI elements" refer to components of a user interface, such as buttons and text fields.

[0847] "Guessing means" is a technique for predicting the layout of an application screen based on recognized text and GUI elements.

[0848] "Source code" is a document written in a particular programming language that describes the behavior of a program.

[0849] The "machine operation panel in a factory" is an interface including a display device and an input device for operating a machine installed in a factory.

[0850] "Reverse engineering" refers to the technique of analyzing existing products or code and recreating their structure and function in order to understand the operation and design of the device or software.

[0851] The "display device used" is a visual output device such as a display or monitor included in a machine control panel in a factory.

[0852] A "cloud server" is a server infrastructure that can be accessed remotely over the internet and provides the computational resources to perform image analysis and code generation.

[0853] This invention is a system that performs reverse engineering based on screenshots of machine operation panels in factories and automatically generates corresponding source code, enabling rapid prototyping and redesign, thereby improving factory production efficiency.

[0854] User-uploaded screenshots

[0855] Users can take screenshots of machine control panels in the factory using a smartphone or head-mounted display, and then access the system's upload interface to upload the screenshot, which causes the device to send the screenshot image to the cloud server.

[0856] Saving a screenshot

[0857] The cloud server saves the received screenshots. The server saves the uploaded images in a specific directory and names the files in an easy-to-manage format. This saving process allows the image data required for subsequent image analysis to be referenced.

[0858] Image Processing and Element Recognition

[0859] The cloud server analyzes the stored screenshots and extracts information from the images using the following techniques:

[0860] Converting an image to grayscale using Pillow.

[0861] Pytesseract is used to perform OCR processing and extract text information from the image.

[0862] At the same time, it applies image processing algorithms to detect GUI elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[0863] Layout and Functionality Guessing

[0864] The cloud server predicts the screen layout of the machine operation panel based on the recognized text and GUI element information. The server analyzes how each element is arranged on the screen and generates layout information. For example, it predicts the overall screen structure based on the position of buttons and the arrangement of text fields.

[0865] Generate source code

[0866] The cloud server automatically generates the corresponding source code based on the inferred layout information. The generated code is written in the selected programming language to reproduce the inferred layout and functionality. For example, it generates Python code using the Tkinter library to reproduce the screen according to the recognized layout.

[0867] Providing reverse engineering results

[0868] The cloud server provides the generated source code to the user. The server packages the generated code in JSON format or similar and provides a download link to the user. The user can use this link to download the generated source code and view or edit it.

[0869] Specific examples

[0870] For example, a user can upload a screenshot of an operation panel to a cloud server. The image contains several buttons and text fields. The cloud server analyzes the image and identifies the positions and labels of the buttons and text fields. The server then uses this information to generate Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[0871] Prompt Sentence Examples

[0872] Analyze the panel image below, extract the position and label of each GUI element, and generate a program to reproduce it using Tkinter. Image link: [Link URL]

[0873] The system enables the rapid and accurate design and redesign of machine control panels in factories, significantly reducing the traditional manual and laborious redevelopment work. Furthermore, OCR technology and grayscale conversion preprocessing provide high recognition accuracy and more accurate reverse engineering results.

[0874] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0875] Step 1: User uploads screenshot

[0876] The user takes a screenshot of a machine control panel in the factory, accesses the system's upload interface via a smartphone or head-mounted display, and clicks the upload button to send the screenshot to the cloud server.

[0877] Input: Screenshot of the machine control panel

[0878] Output: Screenshot image is sent to the cloud server

[0879] Step 2: Save the screenshot

[0880] The cloud server saves the received screenshot images. The server stores the uploaded images in a specific directory and names the files in an easy-to-manage format.

[0881] Input: Screenshot image sent by the user

[0882] Output: Saved screenshot image

[0883] Step 3: Image preprocessing

[0884] The cloud server preprocesses the saved screenshot images and converts them to grayscale using the Pillow library.

[0885] Input: Saved screenshot image

[0886] Output: Image converted to grayscale

[0887] Step 4: Parsing Text and GUI Elements

[0888] The server uses OCR technology (Pytesseract) to extract text information from the grayscale image, and simultaneously applies image processing algorithms to detect GUI elements (buttons, text fields, etc.) in the image.

[0889] Input: Image converted to grayscale

[0890] Output: Extracted text information and the location and type of GUI elements

[0891] Step 5: Inferring the screen layout

[0892] Based on the text and GUI element information recognized by the cloud server, the screen layout of the machine operation panel is analyzed and layout information is generated.

[0893] Input: Extracted text information and the position and type of GUI elements

[0894] Output: Inferred screen layout information

[0895] Step 6: Generate source code

[0896] The cloud server automatically generates the corresponding source code based on the inferred layout information. Specifically, it generates Python code using the Tkinter library to reproduce the screen according to the recognized layout.

[0897] Input: Inferred screen layout information

[0898] Output: Generated source code

[0899] Step 7: Provide generated source code

[0900] The cloud server packages the generated source code in JSON format or similar and provides the user with a download link, which the user can use to download, view, and edit the generated source code.

[0901] Input: Generated source code

[0902] Output: A download link provided to the user

[0903] By following the above steps, it is possible to quickly and accurately generate source code for an operation program through reverse engineering of a machine operation panel in a factory.

[0904] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0905] This invention is a system that reverse engineers applications based on screenshots and automatically generates the corresponding source code, and also has the ability to recognize and utilize user emotions, which improves the user experience and enables more feedback-driven interactive application prototyping.

[0906] User-uploaded screenshots

[0907] A user takes a screenshot of an application and accesses the system's upload interface. The user selects a screenshot and clicks the upload button, causing the device to send the selected screenshot image to the server.

[0908] Saving a screenshot

[0909] The server saves the screenshots it receives. The images are saved in a specific directory and given file names for easy management. This saving process allows the image data to be referenced for subsequent image analysis and emotion recognition processing.

[0910] Image Processing and Element Recognition

[0911] The server reads the saved screenshot and converts it into a grayscale image. It uses OCR technology to extract text information from the grayscale image, while simultaneously using image processing algorithms to recognize GUI elements. This identifies the location and type of text and GUI elements on the screen.

[0912] emotion recognition

[0913] The server uses an emotion engine to recognize the user's emotion based on the screenshot. The emotion engine identifies the user's emotional state (e.g., joy, surprise, sadness, etc.) by analyzing facial expressions and gaze in the image.

[0914] Layout and Functionality Guessing

[0915] The server predicts the application screen layout based on the recognized text and GUI element information and emotional information. The server determines the optimal layout by taking into account the position and type of each element as well as the user's emotional state.

[0916] Code generation

[0917] The server automatically generates the corresponding source code based on the estimated layout information and the user's emotional information. The server uses a selected programming language (e.g., Python) to generate user interface code that reflects the emotional information. For example, if the user is surprised, the code could display a warning message.

[0918] Providing reverse engineering results

[0919] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link, and the user clicks the link to download the generated source code. The downloaded code can then be viewed and edited.

[0920] Specific examples

[0921] For example, a user uploads a screenshot of a specific mobile application. The server recognizes that the image contains multiple buttons and text fields and uses an emotion engine to detect that the user is surprised. Based on this information, the server generates Python code that includes the screen layout and feedback corresponding to the surprise. This code uses the Tkinter library to position the buttons and text fields appropriately and displays a message reflecting the surprise on the screen.

[0922] This system allows users to quickly obtain source code that precisely replicates the screen layout of the target application, and even enables interactive responses based on user emotions within the system. This not only significantly reduces the amount of manual redevelopment work and user experience tracking required, but also enables more adaptive and user-centric application development through emotion-based feedback.

[0923] The above is an embodiment of the present invention.

[0924] The processing flow will be explained below.

[0925] Step 1:

[0926] The user takes a screenshot of the application. The user accesses the system's upload interface, selects the screenshot, and clicks the upload button.

[0927] Step 2:

[0928] The device sends the selected screenshot image to the server in the form of an HTTP POST request.

[0929] Step 3:

[0930] The screenshot image received by the server is saved in a specific directory. When saving, the file name is set in an easy-to-manage format.

[0931] Step 4:

[0932] The server loads the saved screenshot image and performs grayscale conversion as a preprocessing step to simplify image processing.

[0933] Step 5:

[0934] The server extracts text information from the grayscale image using OCR technology and stores the extracted text information as digital data.

[0935] Step 6:

[0936] The server uses image processing algorithms to recognize GUI elements (buttons, input fields, etc.) on the screenshot and identify the location and type of each element.

[0937] Step 7:

[0938] The server uses an emotion engine to recognize the user's emotions from the screenshots, for example, by analyzing the user's facial expressions and gaze to identify emotions such as joy, surprise, and sadness.

[0939] Step 8:

[0940] The server predicts the application screen layout based on the text and GUI element information recognized by the server, as well as emotional information. The optimal layout is determined taking into account the user's emotional state.

[0941] Step 9:

[0942] The server automatically generates the corresponding source code based on the inferred layout information and the user's emotion information. The code is generated using a GUI library in the selected programming language (e.g., Python), and includes emotion-based feedback as needed.

[0943] Step 10:

[0944] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link.

[0945] Step 11:

[0946] The user clicks the link sent by the server to download the generated source code, which the user can then view and edit as needed.

[0947] The above is the specific processing flow of a system that combines an emotion engine.

[0948] Example 2

[0949] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0950] Conventional systems require a great deal of time and effort for users to reverse engineer application screens and generate source code. Furthermore, they are unable to provide interactive feedback that takes into account the user's emotional state, limiting the user experience. Therefore, there is a need for a system that can efficiently generate application prototypes while improving the user experience.

[0951] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for a user to upload screenshot images of an application; means for the server to save the uploaded screenshot images; means for the server to analyze the saved screenshot images and recognize text and user interface elements in the images; means for the server to analyze facial expressions and gaze to recognize the user's emotions; means for the server to infer the layout of the application screen based on the recognized text, user interface elements, and emotional information; means for the server to generate source code based on the inferred layout and the user's emotional information; and means for the server to provide the generated source code to the user. This allows the user to quickly obtain source code that reproduces the screen layout of the target application and further receive interactive feedback based on emotions.

[0952] "User" refers to the person who uploads screenshots of an application and requests the generation of source code.

[0953] "Server" refers to the computer system that stores and analyzes screenshots uploaded by users and generates and provides source code.

[0954] A "screenshot image" is an image of the application screen display saved as is.

[0955] "Uploading means" refers to the interface and protocol that allows users to send screenshot images to the server.

[0956] "Storage means" refers to a function that allows the server to store uploaded screenshot images in a specific directory so that they can be used for later analysis.

[0957] "Analysis means" refers to a technique by which the server processes the screenshot image and identifies textual information and user interface elements within the image.

[0958] "User interface elements" refer to interactive components such as buttons and text fields within an application screen.

[0959] "Emotion recognition means" refers to technology that allows the server to analyze the user's facial expressions and gaze using screenshot images to identify their emotional state.

[0960] "Inference means" refers to a technology that infers the layout and functionality of an application screen based on recognized text, user interface elements, and emotion information.

[0961] "Source code generation means" refers to a technology that automatically generates source code in an appropriate programming language based on the estimated layout and user emotional information.

[0962] "Means of provision" refers to the mechanism for providing the generated source code to users in a downloadable format.

[0963] This invention is a system that reverse engineers applications based on screenshots uploaded by users and automatically generates corresponding source code. The system also includes a function to recognize users' emotions and provide interactive feedback based on the emotion information.

[0964] The user first takes a screenshot of the application and accesses the system's upload interface. The terminal (user's device) selects the screenshot and clicks the upload button. This action causes the terminal to send the selected screenshot image to the server. The upload is performed using a web form and an HTTP POST request.

[0965] The server saves the received screenshots in a specific directory, with file names that can be uniquely identified using a timestamp or UUID. The saved screenshots are used for later analysis.

[0966] The server reads the saved screenshot and converts it to a grayscale image using OpenCV, then uses the Tesseract OCR engine to extract text information from the grayscale image, while also using OpenCV to recognize the location and type of GUI elements (buttons, text fields, etc.).

[0967] The server then uses an emotion recognition engine (e.g., Hume or Affectiva) to recognize the user's emotions based on the screenshot, analyzing facial expressions and gaze to identify emotional states (e.g., happiness, surprise, sadness, etc.).

[0968] The server predicts the layout of the application screen based on the recognized text information, GUI element information, and emotion information. It determines the optimal layout by taking into account the user's emotional state in addition to the location and type.

[0969] Based on the inferred layout information and user emotion information, the server automatically generates the corresponding source code. Specifically, it generates user interface code using Python and the Tkinter library. For example, if the user is surprised, it generates code to display a message reflecting surprise.

[0970] Finally, the server packages the generated source code as a JSON or text file and generates a link to provide to the user, who can click on this link to download the generated source code for viewing or editing.

[0971] Specific examples

[0972] For example, a user uploads a screenshot of a specific mobile application. The server recognizes that the image contains several buttons and text fields. If the emotion engine detects that the user is surprised, the server uses this information to generate Python code that includes the screen layout and feedback corresponding to the surprise. This code uses the Tkinter library to position the buttons and text fields appropriately and displays a message reflecting the surprise on the screen.

[0973] Prompt Sentence Examples

[0974] "I uploaded a screenshot of a user looking surprised. The screen contains several buttons and text fields. Based on this, please generate Python code that displays feedback that reflects the user's surprise."

[0975] This system helps users quickly obtain source code that reproduces the screen layout of the target application and create an interface that includes emotion-based interactive feedback. Not only does it significantly reduce the traditional manual redevelopment work and user experience feedback tracking, but emotion-based feedback also enables more adaptive and user-centric application development.

[0976] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0977] Step 1:

[0978] The user takes a screenshot of the application and accesses the upload interface. The terminal sends the screenshot image selected by the user to the server. The input data is the screenshot image selected by the user, and the output data is the image file sent to the server.

[0979] Step 2:

[0980] The server saves the screenshots it receives in a specific directory. A timestamp or UUID is used to make the file name unique when saved. The input data is the screenshot image sent to the server, and the output data is the path to the saved image file. Specifically, the server writes the received image data to the file system.

[0981] Step 3:

[0982] The server reads the saved screenshot and converts it into a grayscale image using OpenCV. The input data is the saved image file, and the output data is a grayscale image. Specifically, the server uses the image processing library OpenCV to convert the color image into grayscale.

[0983] Step 4:

[0984] The server uses the Tesseract OCR engine to extract text information from grayscale images. At the same time, it uses OpenCV to recognize the location and type of GUI elements (buttons, text fields, etc.). The input data is a grayscale image, and the output data is the recognized text data and GUI element information. Specifically, the server performs OCR to convert the characters in the image into text and uses a contour detection algorithm to identify GUI elements.

[0985] Step 5:

[0986] The server uses an emotion recognition engine (e.g., Hume or Affectiva) to recognize the user's emotion based on the screenshot. The input data is the screenshot image, and the output data is the recognized emotion data. Specifically, the server applies face recognition and emotion classification algorithms.

[0987] Step 6:

[0988] The server infers the layout of the application screen based on the recognized text information, GUI element information, and emotion information. The input data is the recognized text information, GUI element information, and emotion information, and the output data is the inferred layout information. Specifically, the server applies a rule-based layout inference algorithm to determine the placement and characteristics of each element.

[0989] Step 7:

[0990] The server automatically generates the corresponding source code based on the inferred layout information and user emotion information. The input data is the inferred layout information and emotion information, and the output data is the generated source code. Specifically, the server generates the user interface source code using Python and Tkinter.

[0991] Step 8:

[0992] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The input data is the generated source code, and the output data is a download link provided to the user. Specifically, the server packages the generated content, creates an HTTP link, and notifies the user.

[0993] (Application example 2)

[0994] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0995] In recent years, with the advancement of automation in factories, the use of factory robots for inspection and monitoring has been increasing. However, conventional factory robots have limited ability to acquire and analyze information from dashboards and panels, and are unable to provide real-time feedback based on the emotions of human operators. This makes efficient and interactive inspection work difficult, and there is a need for an improved user experience.

[0996] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0997] In this invention, the server includes means for a user to upload screenshot images of an application, means for the server to save the uploaded screenshot images, means for the server to analyze the saved screenshot images and recognize text and GUI elements in the images, means for the server to infer an application screen layout based on the recognized text and GUI elements, means for the server to generate source code based on the inferred layout, means for the server to provide the generated source code to the user, means for the server to recognize user emotions based on the uploaded screenshot images, and means for the server to optimize the user interface based on the recognized emotion information. This enables factory robots to efficiently acquire and analyze information from dashboards and panels and provide interactive feedback to human operators according to their emotions.

[0998] A "screenshot image" is an image file that a user records as a still image of the contents displayed on the screen of a device.

[0999] "Upload" is the act of a user sending data from their device to a server.

[1000] A "server" is a computer system that stores, manages, and processes information on a network.

[1001] "Grayscale conversion" is a process of converting a color image into an image that contains only brightness information.

[1002] "OCR" stands for Optical Character Recognition, a technology that converts characters in an image into digital text.

[1003] "Text and GUI elements" refers to the textual information and graphical user interface components displayed on the screen.

[1004] "Layout" is the arrangement design of how elements on a screen are arranged.

[1005] "Source code" is a human-readable form of code that describes a program.

[1006] "Emotion recognition" is a technology that analyzes a user's facial expressions from images and videos and determines their psychological state.

[1007] "User interface" refers to the screens and operating specifications used to exchange information between the user and the system.

[1008] "Optimization" means adjusting the system's performance and functionality to the best possible state based on certain criteria.

[1009] This invention relates to a system that provides optimized feedback to users through information analysis and emotion recognition of dashboards and panels by an inspection robot in a factory. Specific embodiments of this system are described below.

[1010] The system begins with the robot taking images of each piece of equipment and dashboard in the factory, and then uploading the screenshots from the user's device. These images are then transferred to the server via the Internet and uploaded. The server saves the uploaded screenshots in a specific directory and assigns them file names for easy management.

[1011] The server then analyzes the saved screenshot image. This analysis process first converts the colorful image to grayscale, then uses OCR technology to extract text information from the image. At the same time, it uses image processing algorithms to recognize GUI elements. This step uses OpenCV (an open-source computer vision library) and pytesseract (an OCR library).

[1012] The server then uses FER (Emotion Recognition Library) to analyze the user's facial expressions in the screenshot and recognize the user's emotional state. For example, if the user is surprised, that information will be included in the analysis results.

[1013] Based on this information (the position and type of text and GUI elements, and the user's emotional state), the server infers the layout of the application screen and generates source code based on that. The generated source code is written in a programming language such as Python and creates an interface optimized for the user's emotions. For example, if surprise is detected, a warning message will be displayed on the screen.

[1014] Finally, the server packages the generated source code as a JSON or text file and provides a download link for users to view and edit their own generated code.

[1015] For example, a robot can take a photo of a dashboard in a factory and upload the image to a server. The server then analyzes the image, extracts text information using OCR technology, and recognizes the operator's emotions using an emotion recognition library. If the robot detects that the operator is surprised, the generated source code will include a warning message.

[1016] An example prompt might be, "Extract text information from a scanned screenshot of a factory dashboard, recognize emotions, and provide feedback. Please enter the path to the screenshot image."

[1017] The above is a specific embodiment for carrying out the present invention.

[1018] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1019] Step 1:

[1020] The user takes a screenshot of the application and uploads it to the server using their device.

[1021] Specific operation: A robot takes a picture of a dashboard in a factory and sends the image file to a server via a terminal. The input is a screenshot image file, and the output is an image file saved on the server.

[1022] Step 2:

[1023] The server stores the uploaded screenshot image.

[1024] Specific operation: The server saves the received image files in a specific directory and formats the file names in an easy-to-manage format. The input is the uploaded image file, and the output is an image file that has been saved and is available for subsequent processing.

[1025] Step 3:

[1026] The server converts the saved screenshot image to grayscale and uses OCR technology to extract the text in the image.

[1027] Specific operation: The server uses OpenCV to convert the image to grayscale, and then uses pytesseract to identify text parts in the image and extract the text information. The input is the saved grayscale image file, and the output is the extracted text information.

[1028] Step 4:

[1029] The server uses image processing algorithms to recognize GUI elements.

[1030] Specific operation: The server analyzes GUI components such as buttons and text fields in the image and identifies their positions and types. The input is a grayscale converted image file, and the output is the position and type information of the recognized GUI elements.

[1031] Step 5:

[1032] The server uses an emotion recognition library (such as FER) to analyze the user's facial expressions contained in the screenshot image and recognize the user's emotional state.

[1033] How it works: The server uses an emotion recognition library to analyze facial expressions in uploaded images and determine emotions such as surprise or joy. The input is an image file containing facial expressions, and the output is the recognized emotional state information.

[1034] Step 6:

[1035] The server predicts the layout of the application screen based on the recognized text information and the position information, type information, and emotional state of GUI elements.

[1036] Specific operation: The server integrates the collected information and applies an algorithm to design the optimal screen layout. The input is text information, GUI element information, and emotion information, and the output is the inferred application screen layout.

[1037] Step 7:

[1038] The server automatically generates source code based on the inferred layout and emotion information.

[1039] How it works: The server uses a programming language such as Python to generate source code that includes emotion-based user interface optimization. The input is the inferred layout and emotion information, and the output is the generated source code.

[1040] Step 8:

[1041] The server packages the generated source code and provides a link for users to download it.

[1042] Specific operation: The server compiles the source code as a JSON or text file and generates a link that users can use to download it. The input is the generated source code, and the output is the download link.

[1043] An example prompt is, "Extract text information from scanned screenshots of a factory dashboard, recognize emotions, and provide feedback. Please enter the path to the screenshot image."

[1044] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1045] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1046] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1047] [Fourth embodiment]

[1048] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1049] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1050] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1051] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1052] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1053] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1054] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1055] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1056] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1057] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1058] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1059] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1060] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1061] This invention is a system that reverse engineers applications based on screenshots and automatically generates corresponding source code, enabling rapid prototyping and redevelopment.

[1062] User-uploaded screenshots

[1063] A user takes a screenshot of an application and accesses the system's upload interface, where the user selects the screenshot and clicks the upload button, which causes the device to send the screenshot image to the server.

[1064] Saving a screenshot

[1065] The server saves the received screenshot. The server saves the uploaded image in a specific directory and sets the file name in an easy-to-manage format. This saving process makes it possible to reference the image data required for subsequent image analysis processing.

[1066] Image Processing and Element Recognition

[1067] The server analyzes the saved screenshot. First, the server performs grayscale conversion as a preprocessing of the image. Then, it uses OCR technology to extract text information from the image. At the same time, the server applies image processing algorithms to detect GUI elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[1068] Layout and Functionality Guessing

[1069] The server predicts the application's screen layout based on the text and GUI element information it recognizes. The server analyzes how each element is arranged on the screen and generates layout information. For example, it predicts the overall screen structure based on the position of buttons and the placement of text fields.

[1070] Code generation

[1071] The server automatically generates the corresponding source code based on the inferred layout information. The generated code is in accordance with the selected programming language and is used to reproduce the inferred layout and functionality. For example, it includes code to configure the screen layout using a GUI library and code to display the extracted text.

[1072] Providing reverse engineering results

[1073] The server provides the generated source code to the user. The server packages the generated code in JSON format or similar and provides a download link to the user. The user can use this link to download the generated source code and view or edit it.

[1074] Specific examples

[1075] For example, a user can upload a screenshot of a specific mobile application to the system. The image might contain multiple buttons and text fields. The server analyzes the image and identifies the positions and labels of the buttons and text fields. The server then uses this information to generate Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[1076] This system allows users to quickly obtain source code that precisely reproduces the screen layout of the target application, significantly reducing the painstaking manual redevelopment work required in the past. Furthermore, OCR technology and grayscale conversion preprocessing provide high recognition accuracy, resulting in more accurate reverse engineering results.

[1077] The above is an embodiment of the present invention.

[1078] The processing flow will be explained below.

[1079] Step 1:

[1080] The user takes a screenshot of the application and selects it by accessing the system's upload interface.

[1081] Step 2:

[1082] After the user selects a screenshot, they click the upload button, which causes the device to send the selected screenshot image to the server in an HTTP POST request.

[1083] Step 3:

[1084] The server saves the received screenshot image. The server saves the image in a specific directory and sets a file name for easy management.

[1085] Step 4:

[1086] The server loads the saved screenshot image. As a preprocessing step, the server first converts the screenshot to a grayscale image, which makes subsequent image analysis easier.

[1087] Step 5:

[1088] The server uses OCR technology to extract text information from the grayscale image and stores the extracted text information as digital data.

[1089] Step 6:

[1090] The server uses image processing algorithms to recognize GUI elements on the screenshot, specifically identifying interactive components such as buttons, input fields, and labels.

[1091] Step 7:

[1092] The server predicts the application screen layout based on the recognized text and GUI element information, and determines the overall layout structure based on the position and type of each element.

[1093] Step 8:

[1094] The server automatically generates the corresponding source code based on the estimated layout information. The server uses a GUI library in the selected programming language (e.g., Python) to generate code that reproduces the screen layout.

[1095] Step 9:

[1096] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link.

[1097] Step 10:

[1098] The user clicks on the link provided by the server to download the generated source code, which the user can then view and edit as needed.

[1099] The above is the specific processing flow of the program.

[1100] Example 1

[1101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1102] Modern software development requires rapid redevelopment and prototyping of existing applications. However, manual coding is time-consuming and labor-intensive, especially when it comes to recreating GUI screen layouts. Furthermore, systems that use accurate OCR technology to recognize text and interface elements and generate code based on that recognition are still rare. This continues to require significant developer time.

[1103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1104] In this invention, the server includes means for a user to upload screenshot images of a software application, means for the server to save the uploaded screenshot images, means for the server to analyze the saved screenshot images and recognize text and interface elements in the images, means for the server to infer a layout of the application screen based on the recognized text and interface elements, means for the server to generate program code based on the inferred layout, and means for the server to provide the generated program code to the user. This allows the user to quickly obtain highly accurate source code for redevelopment, significantly reducing the manual workload.

[1105] "User" means any person or entity that uses the System to upload screenshots of an Application and obtain the generated source code.

[1106] "Software application" refers generally to a program that runs on a computer for use by a user.

[1107] A "screenshot image" is image data that captures the screen of a software application.

[1108] "Uploading Method" refers to the function or process by which a user can send a screenshot image to the system.

[1109] "Server" refers to the computer system or device used to process user requests and store, analyze, and generate code from screenshots.

[1110] "Storage means" refers to a function or process that stores screenshot images in a specific location and makes them accessible as needed.

[1111] "Analysis Method" refers to the technology or algorithms used to analyze the saved screenshot image and recognize text and interface elements within the image.

[1112] "Text" refers to the written information extracted from an image through OCR technology.

[1113] "Interface elements" refers to GUI components such as buttons and text fields that exist within an image.

[1114] "Inference means" refers to a process or function for determining the layout of an application screen based on recognized text and interface elements.

[1115] "Program Code" means the set of source code that describes the operation of a software application.

[1116] "Providing means" refers to the functions and processes for transmitting or making the generated program code available for download to users.

[1117] "Tone conversion" refers to an image processing technique for converting the color information of a screenshot image to grayscale.

[1118] "Optical character recognition technology" refers to technology for extracting characters in an image as digital text.

[1119] MODE FOR CARRYING OUT THE INVENTION

[1120] This invention is a system that reverse engineers applications based on screenshots and automatically generates corresponding source code, enabling rapid prototyping and redevelopment.

[1121] User-uploaded screenshots

[1122] The user first takes a screenshot of the software application, then accesses the system's upload interface, selects the screenshot file, and clicks the upload button, which causes the device to send the screenshot image to the server.

[1123] Saving a screenshot

[1124] The server saves screenshots received from users. Specifically, the server saves the uploaded screenshot images in a specific directory and sets the file name in an easy-to-manage format. This saving process makes it possible to reference the image data required for subsequent image analysis processing.

[1125] Image Processing and Element Recognition

[1126] The server analyzes the saved screenshot. First, it performs grayscale conversion as image preprocessing. Then, it uses OCR technology (e.g., Tesseract OCR) to extract text information from the image. At the same time, the server applies an image processing algorithm (e.g., OpenCV) to detect interface elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[1127] Layout and Functionality Guessing

[1128] The server infers the application's screen layout based on the recognized text and interface element information. The server analyzes how each element is arranged on the screen and generates layout information. For example, it infers the overall screen structure based on the position of buttons and the arrangement of text fields.

[1129] Code generation

[1130] The server automatically generates the corresponding program code based on the inferred layout information. The generated code is based on the selected programming language (e.g., Python) and is used to reproduce the inferred layout and functionality. For example, it includes code to configure the screen layout using Python's Tkinter library and code to display the extracted text.

[1131] Providing reverse engineering results

[1132] The server provides the generated program code to the user. The server packages the generated code in JSON format or similar and provides the user with a download link. The user can use this link to download the generated source code and view or edit it.

[1133] Specific examples

[1134] For example, a user uploads a screenshot of a specific screen in a mobile application to the system. The image contains multiple buttons and text fields. The server analyzes the image and identifies the positions and labels of the buttons and text fields. Based on the identified information, the server then generates Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[1135] An example of a prompt to input to a generative AI model is, "Based on the screenshot below, please recognize GUI elements and generate source code using Python's Tkinter library."

[1136] This allows users to quickly obtain source code that precisely reproduces the screen layout of the target application, significantly reducing the effort required for manual redevelopment. Furthermore, OCR technology and grayscale conversion preprocessing enable high recognition accuracy, resulting in more accurate reverse engineering results. The above is an embodiment of the present invention.

[1137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1138] Step 1:

[1139] A user takes a screenshot of the application, accesses the upload interface, selects the screenshot file, and clicks the upload button, which causes the device to send the screenshot image to the server.

[1140] Input: A screenshot file selected by the user.

[1141] Output: Screenshot data sent to the server

[1142] Specific operation: The device creates an HTTP request, attaches the screenshot image data, and sends it to the server.

[1143] Step 2:

[1144] The server receives the screenshot upload request and saves the screenshot image in a specific directory, renaming the file to a unique, easily manageable format (e.g., "screenshot_20231015_123456.png").

[1145] Input: Screenshot data sent from the device

[1146] Output: Screenshot files saved in a specific directory

[1147] Specific operation: The server saves the received screenshot data in the appropriate directory and sets a unique file name.

[1148] Step 3:

[1149] The server reads the saved screenshot and performs grayscale conversion as a preprocessing of the image, thereby highlighting the detailed information in the image.

[1150] Input: Saved screenshot file

[1151] Output: Grayscale converted image data

[1152] Specific operation: The server uses the OpenCV library to convert the screenshot image to grayscale.

[1153] Step 4:

[1154] The server uses OCR technology (e.g., Tesseract OCR) to extract text information from the grayscale image. At the same time, the server uses image processing algorithms to detect interface elements (buttons, text fields, etc.) in the image.

[1155] Input: Grayscale converted image data

[1156] Output: Text information and data about the location and type of interface elements

[1157] What it does: The server uses Tesseract OCR to extract text from images and OpenCV algorithms to detect interface elements.

[1158] Step 5:

[1159] The server analyzes the recognized text and interface element information to predict the layout of the application screen, thereby determining the overall screen structure and generating layout information.

[1160] Input: Text information and data about the location and type of interface elements

[1161] Output: Application screen layout information (e.g., JSON format)

[1162] Specific operation: The server analyzes the position and type of each element, generates layout information, and saves it in JSON format.

[1163] Step 6:

[1164] The server automatically generates the corresponding program code based on the inferred layout information. The code is generated in the selected programming language (e.g., Python), and includes GUI layout code using the Tkinter library.

[1165] Input: Application screen layout information

[1166] Output: Generated program code

[1167] Specific operation: The server uses a generative AI model to automatically generate program code based on layout information.

[1168] Step 7:

[1169] The server packages the generated program code in JSON format or similar and provides a download link to the user, who can then use the link to download the generated source code and view or edit it.

[1170] Input: Generated program code

[1171] Output: Download link

[1172] Specific operation: The server packages the generated code and provides the user with a download link via email or other means.

[1173] (Application example 1)

[1174] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1175] Designing and redesigning machine control panels in a factory takes a great deal of time and effort. When each machine has a different control panel, it is difficult to reuse the same design drawings, and in order to improve factory productivity, it is necessary to generate the source code for the control panel quickly and accurately. Current technology mainly involves manual design and program porting, which requires specialized knowledge and time. This poses the problem of reduced factory efficiency and increased costs.

[1176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1177] In this invention, the server includes: means for a user to upload screenshot images of an application; means for the server to save the uploaded screenshot images; means for the server to analyze the saved screenshot images and recognize text and GUI elements in the images; means for the server to infer an application screen layout based on the recognized text and GUI elements; means for the server to generate source code based on the inferred layout; means for the server to provide the generated source code to the user; means for reverse engineering screenshots of machine operation panels in a factory; means for automatically generating source code for a machine operation program based on screenshot images of a display device to be used; and means for performing image analysis and code generation via a cloud server. This enables rapid and accurate design and redesign of machine operation panels in a factory.

[1178] "User" means any person or institution that uses the System to upload screenshots of an Application.

[1179] A "screenshot image of an application" refers to a still image of an application screen, and is data for visualizing the contents of that screen.

[1180] The "server" is a computer system that stores the uploaded screenshot images, analyzes them, and ultimately generates the source code.

[1181] The "storing means" refers to the process and technology by which the server stores the received screenshot images in a specific directory.

[1182] The "analyzing means" is a technology for recognizing and extracting text and GUI elements from the saved screenshot image.

[1183] "Text" refers to the text information displayed in the screenshot image.

[1184] "GUI elements" refer to components of a user interface, such as buttons and text fields.

[1185] "Guessing means" is a technique for predicting the layout of an application screen based on recognized text and GUI elements.

[1186] "Source code" is a document written in a particular programming language that describes the behavior of a program.

[1187] The "machine operation panel in a factory" is an interface including a display device and an input device for operating a machine installed in a factory.

[1188] "Reverse engineering" refers to the technique of analyzing existing products or code and recreating their structure and function in order to understand the operation and design of the device or software.

[1189] The "display device used" is a visual output device such as a display or monitor included in a machine control panel in a factory.

[1190] A "cloud server" is a server infrastructure that can be accessed remotely over the internet and provides the computational resources to perform image analysis and code generation.

[1191] This invention is a system that performs reverse engineering based on screenshots of machine operation panels in factories and automatically generates corresponding source code, enabling rapid prototyping and redesign, thereby improving factory production efficiency.

[1192] User-uploaded screenshots

[1193] Users can take screenshots of machine control panels in the factory using a smartphone or head-mounted display, and then access the system's upload interface to upload the screenshot, which causes the device to send the screenshot image to the cloud server.

[1194] Saving a screenshot

[1195] The cloud server saves the received screenshots. The server saves the uploaded images in a specific directory and names the files in an easy-to-manage format. This saving process allows the image data required for subsequent image analysis to be referenced.

[1196] Image Processing and Element Recognition

[1197] The cloud server analyzes the stored screenshots and extracts information from the images using the following techniques:

[1198] Converting an image to grayscale using Pillow.

[1199] Pytesseract is used to perform OCR processing and extract text information from the image.

[1200] At the same time, it applies image processing algorithms to detect GUI elements (e.g., buttons, text fields, etc.) in the image. This element recognition identifies the location and type of each component on the screen.

[1201] Layout and Functionality Guessing

[1202] The cloud server predicts the screen layout of the machine operation panel based on the recognized text and GUI element information. The server analyzes how each element is arranged on the screen and generates layout information. For example, it predicts the overall screen structure based on the position of buttons and the arrangement of text fields.

[1203] Generate source code

[1204] The cloud server automatically generates the corresponding source code based on the inferred layout information. The generated code is written in the selected programming language to reproduce the inferred layout and functionality. For example, it generates Python code using the Tkinter library to reproduce the screen according to the recognized layout.

[1205] Providing reverse engineering results

[1206] The cloud server provides the generated source code to the user. The server packages the generated code in JSON format or similar and provides a download link to the user. The user can use this link to download the generated source code and view or edit it.

[1207] Specific examples

[1208] For example, a user can upload a screenshot of an operation panel to a cloud server. The image contains several buttons and text fields. The cloud server analyzes the image and identifies the positions and labels of the buttons and text fields. The server then uses this information to generate Python code using the Tkinter library. This code recreates the screen according to the identified layout, placing each button and text field in the appropriate position.

[1209] Prompt Sentence Examples

[1210] Analyze the panel image below, extract the position and label of each GUI element, and generate a program to reproduce it using Tkinter. Image link: [Link URL]

[1211] The system enables the rapid and accurate design and redesign of machine control panels in factories, significantly reducing the traditional manual and laborious redevelopment work. Furthermore, OCR technology and grayscale conversion preprocessing provide high recognition accuracy and more accurate reverse engineering results.

[1212] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1213] Step 1: User uploads screenshot

[1214] The user takes a screenshot of a machine control panel in the factory, accesses the system's upload interface via a smartphone or head-mounted display, and clicks the upload button to send the screenshot to the cloud server.

[1215] Input: Screenshot of the machine control panel

[1216] Output: Screenshot image is sent to the cloud server

[1217] Step 2: Save the screenshot

[1218] The cloud server saves the received screenshot images. The server stores the uploaded images in a specific directory and names the files in an easy-to-manage format.

[1219] Input: Screenshot image sent by the user

[1220] Output: Saved screenshot image

[1221] Step 3: Image preprocessing

[1222] The cloud server preprocesses the saved screenshot images and converts them to grayscale using the Pillow library.

[1223] Input: Saved screenshot image

[1224] Output: Image converted to grayscale

[1225] Step 4: Parsing Text and GUI Elements

[1226] The server uses OCR technology (Pytesseract) to extract text information from the grayscale image, and simultaneously applies image processing algorithms to detect GUI elements (buttons, text fields, etc.) in the image.

[1227] Input: Image converted to grayscale

[1228] Output: Extracted text information and the location and type of GUI elements

[1229] Step 5: Inferring the screen layout

[1230] Based on the text and GUI element information recognized by the cloud server, the screen layout of the machine operation panel is analyzed and layout information is generated.

[1231] Input: Extracted text information and the position and type of GUI elements

[1232] Output: Inferred screen layout information

[1233] Step 6: Generate source code

[1234] The cloud server automatically generates the corresponding source code based on the inferred layout information. Specifically, it generates Python code using the Tkinter library to reproduce the screen according to the recognized layout.

[1235] Input: Inferred screen layout information

[1236] Output: Generated source code

[1237] Step 7: Provide generated source code

[1238] The cloud server packages the generated source code in JSON format or similar and provides the user with a download link, which the user can use to download, view, and edit the generated source code.

[1239] Input: Generated source code

[1240] Output: A download link provided to the user

[1241] By following the above steps, it is possible to quickly and accurately generate source code for an operation program through reverse engineering of a machine operation panel in a factory.

[1242] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1243] This invention is a system that reverse engineers applications based on screenshots and automatically generates the corresponding source code, and also has the ability to recognize and utilize user emotions, which improves the user experience and enables more feedback-driven interactive application prototyping.

[1244] User-uploaded screenshots

[1245] A user takes a screenshot of an application and accesses the system's upload interface. The user selects a screenshot and clicks the upload button, causing the device to send the selected screenshot image to the server.

[1246] Saving a screenshot

[1247] The server saves the screenshots it receives. The images are saved in a specific directory and given file names for easy management. This saving process allows the image data to be referenced for subsequent image analysis and emotion recognition processing.

[1248] Image Processing and Element Recognition

[1249] The server reads the saved screenshot and converts it into a grayscale image. It uses OCR technology to extract text information from the grayscale image, while simultaneously using image processing algorithms to recognize GUI elements. This identifies the location and type of text and GUI elements on the screen.

[1250] emotion recognition

[1251] The server uses an emotion engine to recognize the user's emotion based on the screenshot. The emotion engine identifies the user's emotional state (e.g., joy, surprise, sadness, etc.) by analyzing facial expressions and gaze in the image.

[1252] Layout and Functionality Guessing

[1253] The server predicts the application screen layout based on the recognized text and GUI element information and emotional information. The server determines the optimal layout by taking into account the position and type of each element as well as the user's emotional state.

[1254] Code generation

[1255] The server automatically generates the corresponding source code based on the estimated layout information and the user's emotional information. The server uses a selected programming language (e.g., Python) to generate user interface code that reflects the emotional information. For example, if the user is surprised, the code could display a warning message.

[1256] Providing reverse engineering results

[1257] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link, and the user clicks the link to download the generated source code. The downloaded code can then be viewed and edited.

[1258] Specific examples

[1259] For example, a user uploads a screenshot of a specific mobile application. The server recognizes that the image contains multiple buttons and text fields and uses an emotion engine to detect that the user is surprised. Based on this information, the server generates Python code that includes the screen layout and feedback corresponding to the surprise. This code uses the Tkinter library to position the buttons and text fields appropriately and displays a message reflecting the surprise on the screen.

[1260] This system allows users to quickly obtain source code that precisely replicates the screen layout of the target application, and even enables interactive responses based on user emotions within the system. This not only significantly reduces the amount of manual redevelopment work and user experience tracking required, but also enables more adaptive and user-centric application development through emotion-based feedback.

[1261] The above is an embodiment of the present invention.

[1262] The processing flow will be explained below.

[1263] Step 1:

[1264] The user takes a screenshot of the application. The user accesses the system's upload interface, selects the screenshot, and clicks the upload button.

[1265] Step 2:

[1266] The device sends the selected screenshot image to the server in the form of an HTTP POST request.

[1267] Step 3:

[1268] The screenshot image received by the server is saved in a specific directory. When saving, the file name is set in an easy-to-manage format.

[1269] Step 4:

[1270] The server loads the saved screenshot image and performs grayscale conversion as a preprocessing step to simplify image processing.

[1271] Step 5:

[1272] The server extracts text information from the grayscale image using OCR technology and stores the extracted text information as digital data.

[1273] Step 6:

[1274] The server uses image processing algorithms to recognize GUI elements (buttons, input fields, etc.) on the screenshot and identify the location and type of each element.

[1275] Step 7:

[1276] The server uses an emotion engine to recognize the user's emotions from the screenshots, for example, by analyzing the user's facial expressions and gaze to identify emotions such as joy, surprise, and sadness.

[1277] Step 8:

[1278] The server predicts the application screen layout based on the text and GUI element information recognized by the server, as well as emotional information. The optimal layout is determined taking into account the user's emotional state.

[1279] Step 9:

[1280] The server automatically generates the corresponding source code based on the inferred layout information and the user's emotion information. The code is generated using a GUI library in the selected programming language (e.g., Python), and includes emotion-based feedback as needed.

[1281] Step 10:

[1282] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The server notifies the user of this link.

[1283] Step 11:

[1284] The user clicks the link sent by the server to download the generated source code, which the user can then view and edit as needed.

[1285] The above is the specific processing flow of a system that combines an emotion engine.

[1286] Example 2

[1287] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1288] Conventional systems require a great deal of time and effort for users to reverse engineer application screens and generate source code. Furthermore, they are unable to provide interactive feedback that takes into account the user's emotional state, limiting the user experience. Therefore, there is a need for a system that can efficiently generate application prototypes while improving the user experience.

[1289] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for a user to upload screenshot images of an application; means for the server to save the uploaded screenshot images; means for the server to analyze the saved screenshot images and recognize text and user interface elements in the images; means for the server to analyze facial expressions and gaze to recognize the user's emotions; means for the server to infer the layout of the application screen based on the recognized text, user interface elements, and emotional information; means for the server to generate source code based on the inferred layout and the user's emotional information; and means for the server to provide the generated source code to the user. This allows the user to quickly obtain source code that reproduces the screen layout of the target application and further receive interactive feedback based on emotions.

[1290] "User" refers to the person who uploads screenshots of an application and requests the generation of source code.

[1291] "Server" refers to the computer system that stores and analyzes screenshots uploaded by users and generates and provides source code.

[1292] A "screenshot image" is an image of the application screen display saved as is.

[1293] "Uploading means" refers to the interface and protocol that allows users to send screenshot images to the server.

[1294] "Storage means" refers to a function that allows the server to store uploaded screenshot images in a specific directory so that they can be used for later analysis.

[1295] "Analysis means" refers to a technique by which the server processes the screenshot image and identifies textual information and user interface elements within the image.

[1296] "User interface elements" refer to interactive components such as buttons and text fields within an application screen.

[1297] "Emotion recognition means" refers to technology that allows the server to analyze the user's facial expressions and gaze using screenshot images to identify their emotional state.

[1298] "Inference means" refers to a technology that infers the layout and functionality of an application screen based on recognized text, user interface elements, and emotion information.

[1299] "Source code generation means" refers to a technology that automatically generates source code in an appropriate programming language based on the estimated layout and user emotional information.

[1300] "Means of provision" refers to the mechanism for providing the generated source code to users in a downloadable format.

[1301] This invention is a system that reverse engineers applications based on screenshots uploaded by users and automatically generates corresponding source code. The system also includes a function to recognize users' emotions and provide interactive feedback based on the emotion information.

[1302] The user first takes a screenshot of the application and accesses the system's upload interface. The terminal (user's device) selects the screenshot and clicks the upload button. This action causes the terminal to send the selected screenshot image to the server. The upload is performed using a web form and an HTTP POST request.

[1303] The server saves the received screenshots in a specific directory, with file names that can be uniquely identified using a timestamp or UUID. The saved screenshots are used for later analysis.

[1304] The server reads the saved screenshot and converts it to a grayscale image using OpenCV, then uses the Tesseract OCR engine to extract text information from the grayscale image, while also using OpenCV to recognize the location and type of GUI elements (buttons, text fields, etc.).

[1305] The server then uses an emotion recognition engine (e.g., Hume or Affectiva) to recognize the user's emotions based on the screenshot, analyzing facial expressions and gaze to identify emotional states (e.g., happiness, surprise, sadness, etc.).

[1306] The server predicts the layout of the application screen based on the recognized text information, GUI element information, and emotion information. It determines the optimal layout by taking into account the user's emotional state in addition to the location and type.

[1307] Based on the inferred layout information and user emotion information, the server automatically generates the corresponding source code. Specifically, it generates user interface code using Python and the Tkinter library. For example, if the user is surprised, it generates code to display a message reflecting surprise.

[1308] Finally, the server packages the generated source code as a JSON or text file and generates a link to provide to the user, who can click on this link to download the generated source code for viewing or editing.

[1309] Specific examples

[1310] For example, a user uploads a screenshot of a specific mobile application. The server recognizes that the image contains several buttons and text fields. If the emotion engine detects that the user is surprised, the server uses this information to generate Python code that includes the screen layout and feedback corresponding to the surprise. This code uses the Tkinter library to position the buttons and text fields appropriately and displays a message reflecting the surprise on the screen.

[1311] Prompt Sentence Examples

[1312] "I uploaded a screenshot of a user looking surprised. The screen contains several buttons and text fields. Based on this, please generate Python code that displays feedback that reflects the user's surprise."

[1313] This system helps users quickly obtain source code that reproduces the screen layout of the target application and create an interface that includes emotion-based interactive feedback. Not only does it significantly reduce the traditional manual redevelopment work and user experience feedback tracking, but emotion-based feedback also enables more adaptive and user-centric application development.

[1314] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1315] Step 1:

[1316] The user takes a screenshot of the application and accesses the upload interface. The terminal sends the screenshot image selected by the user to the server. The input data is the screenshot image selected by the user, and the output data is the image file sent to the server.

[1317] Step 2:

[1318] The server saves the screenshots it receives in a specific directory. A timestamp or UUID is used to make the file name unique when saved. The input data is the screenshot image sent to the server, and the output data is the path to the saved image file. Specifically, the server writes the received image data to the file system.

[1319] Step 3:

[1320] The server reads the saved screenshot and converts it into a grayscale image using OpenCV. The input data is the saved image file, and the output data is a grayscale image. Specifically, the server uses the image processing library OpenCV to convert the color image into grayscale.

[1321] Step 4:

[1322] The server uses the Tesseract OCR engine to extract text information from grayscale images. At the same time, it uses OpenCV to recognize the location and type of GUI elements (buttons, text fields, etc.). The input data is a grayscale image, and the output data is the recognized text data and GUI element information. Specifically, the server performs OCR to convert the characters in the image into text and uses a contour detection algorithm to identify GUI elements.

[1323] Step 5:

[1324] The server uses an emotion recognition engine (e.g., Hume or Affectiva) to recognize the user's emotion based on the screenshot. The input data is the screenshot image, and the output data is the recognized emotion data. Specifically, the server applies face recognition and emotion classification algorithms.

[1325] Step 6:

[1326] The server infers the layout of the application screen based on the recognized text information, GUI element information, and emotion information. The input data is the recognized text information, GUI element information, and emotion information, and the output data is the inferred layout information. Specifically, the server applies a rule-based layout inference algorithm to determine the placement and characteristics of each element.

[1327] Step 7:

[1328] The server automatically generates the corresponding source code based on the inferred layout information and user emotion information. The input data is the inferred layout information and emotion information, and the output data is the generated source code. Specifically, the server generates the user interface source code using Python and Tkinter.

[1329] Step 8:

[1330] The server packages the generated source code as a JSON or text file and generates a link to provide to the user. The input data is the generated source code, and the output data is a download link provided to the user. Specifically, the server packages the generated content, creates an HTTP link, and notifies the user.

[1331] (Application example 2)

[1332] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1333] In recent years, with the advancement of automation in factories, the use of factory robots for inspection and monitoring has been increasing. However, conventional factory robots have limited ability to acquire and analyze information from dashboards and panels, and are unable to provide real-time feedback based on the emotions of human operators. This makes efficient and interactive inspection work difficult, and there is a need for an improved user experience.

[1334] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1335] In this invention, the server includes means for a user to upload screenshot images of an application, means for the server to save the uploaded screenshot images, means for the server to analyze the saved screenshot images and recognize text and GUI elements in the images, means for the server to infer an application screen layout based on the recognized text and GUI elements, means for the server to generate source code based on the inferred layout, means for the server to provide the generated source code to the user, means for the server to recognize user emotions based on the uploaded screenshot images, and means for the server to optimize the user interface based on the recognized emotion information. This enables factory robots to efficiently acquire and analyze information from dashboards and panels and provide interactive feedback to human operators according to their emotions.

[1336] A "screenshot image" is an image file that a user records as a still image of the contents displayed on the screen of a device.

[1337] "Upload" is the act of a user sending data from their device to a server.

[1338] A "server" is a computer system that stores, manages, and processes information on a network.

[1339] "Grayscale conversion" is a process of converting a color image into an image that contains only brightness information.

[1340] "OCR" stands for Optical Character Recognition, a technology that converts characters in an image into digital text.

[1341] "Text and GUI elements" refers to the textual information and graphical user interface components displayed on the screen.

[1342] "Layout" is the arrangement design of how elements on a screen are arranged.

[1343] "Source code" is a human-readable form of code that describes a program.

[1344] "Emotion recognition" is a technology that analyzes a user's facial expressions from images and videos and determines their psychological state.

[1345] "User interface" refers to the screens and operating specifications used to exchange information between the user and the system.

[1346] "Optimization" means adjusting the system's performance and functionality to the best possible state based on certain criteria.

[1347] This invention relates to a system that provides optimized feedback to users through information analysis and emotion recognition of dashboards and panels by an inspection robot in a factory. Specific embodiments of this system are described below.

[1348] The system begins with the robot taking images of each piece of equipment and dashboard in the factory, and then uploading the screenshots from the user's device. These images are then transferred to the server via the Internet and uploaded. The server saves the uploaded screenshots in a specific directory and assigns them file names for easy management.

[1349] The server then analyzes the saved screenshot image. This analysis process first converts the colorful image to grayscale, then uses OCR technology to extract text information from the image. At the same time, it uses image processing algorithms to recognize GUI elements. This step uses OpenCV (an open-source computer vision library) and pytesseract (an OCR library).

[1350] The server then uses FER (Emotion Recognition Library) to analyze the user's facial expressions in the screenshot and recognize the user's emotional state. For example, if the user is surprised, that information will be included in the analysis results.

[1351] Based on this information (the position and type of text and GUI elements, and the user's emotional state), the server infers the layout of the application screen and generates source code based on that. The generated source code is written in a programming language such as Python and creates an interface optimized for the user's emotions. For example, if surprise is detected, a warning message will be displayed on the screen.

[1352] Finally, the server packages the generated source code as a JSON or text file and provides a download link for users to view and edit their own generated code.

[1353] For example, a robot can take a photo of a dashboard in a factory and upload the image to a server. The server then analyzes the image, extracts text information using OCR technology, and recognizes the operator's emotions using an emotion recognition library. If the robot detects that the operator is surprised, the generated source code will include a warning message.

[1354] An example prompt might be, "Extract text information from a scanned screenshot of a factory dashboard, recognize emotions, and provide feedback. Please enter the path to the screenshot image."

[1355] The above is a specific embodiment for carrying out the present invention.

[1356] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1357] Step 1:

[1358] The user takes a screenshot of the application and uploads it to the server using their device.

[1359] Specific operation: A robot takes a picture of a dashboard in a factory and sends the image file to a server via a terminal. The input is a screenshot image file, and the output is an image file saved on the server.

[1360] Step 2:

[1361] The server stores the uploaded screenshot image.

[1362] Specific operation: The server saves the received image files in a specific directory and formats the file names in an easy-to-manage format. The input is the uploaded image file, and the output is an image file that has been saved and is available for subsequent processing.

[1363] Step 3:

[1364] The server converts the saved screenshot image to grayscale and uses OCR technology to extract the text in the image.

[1365] Specific operation: The server uses OpenCV to convert the image to grayscale, and then uses pytesseract to identify text parts in the image and extract the text information. The input is the saved grayscale image file, and the output is the extracted text information.

[1366] Step 4:

[1367] The server uses image processing algorithms to recognize GUI elements.

[1368] Specific operation: The server analyzes GUI components such as buttons and text fields in the image and identifies their positions and types. The input is a grayscale converted image file, and the output is the position and type information of the recognized GUI elements.

[1369] Step 5:

[1370] The server uses an emotion recognition library (such as FER) to analyze the user's facial expressions contained in the screenshot image and recognize the user's emotional state.

[1371] How it works: The server uses an emotion recognition library to analyze facial expressions in uploaded images and determine emotions such as surprise or joy. The input is an image file containing facial expressions, and the output is the recognized emotional state information.

[1372] Step 6:

[1373] The server predicts the layout of the application screen based on the recognized text information and the position information, type information, and emotional state of GUI elements.

[1374] Specific operation: The server integrates the collected information and applies an algorithm to design the optimal screen layout. The input is text information, GUI element information, and emotion information, and the output is the inferred application screen layout.

[1375] Step 7:

[1376] The server automatically generates source code based on the inferred layout and emotion information.

[1377] How it works: The server uses a programming language such as Python to generate source code that includes emotion-based user interface optimization. The input is the inferred layout and emotion information, and the output is the generated source code.

[1378] Step 8:

[1379] The server packages the generated source code and provides a link for users to download it.

[1380] Specific operation: The server compiles the source code as a JSON or text file and generates a link that users can use to download it. The input is the generated source code, and the output is the download link.

[1381] An example prompt is, "Extract text information from scanned screenshots of a factory dashboard, recognize emotions, and provide feedback. Please enter the path to the screenshot image."

[1382] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1383] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1384] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1385] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1386] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1387] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1388] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1389] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1390] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1391] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1392] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1393] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1394] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1395] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1396] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1397] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1398] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1399] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1400] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1401] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1402] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1403] The following is further disclosed regarding the above embodiment.

[1404] (Claim 1)

[1405] A means for users to upload screenshots of the application;

[1406] a means for the server to store the uploaded screenshot image;

[1407] means for the server to analyze the stored screenshot images and recognize text and GUI elements within the images;

[1408] means for the server to infer an application screen layout based on the recognized text and GUI elements;

[1409] means for the server to generate source code based on the inferred layout;

[1410] a means by which the server provides the generated source code to the user;

[1411] A system including:

[1412] (Claim 2)

[1413] 2. The system of claim 1, wherein the server includes means for performing grayscale conversion as preprocessing of the screenshot image.

[1414] (Claim 3)

[1415] 10. The system of claim 1, wherein the server includes means for extracting text within the image using OCR technology.

[1416] "Example 1"

[1417] (Claim 1)

[1418] a means for a user to upload screenshot images of the software application;

[1419] a means for the server to store the uploaded screenshot image;

[1420] means for the server to analyze the stored screenshot images and recognize text and interface elements within the images;

[1421] means for the server to infer a layout of the application screen based on the recognized text and interface elements;

[1422] means for the server to generate program code based on the inferred layout;

[1423] a means by which the server provides the generated program code to the user;

[1424] A system including:

[1425] (Claim 2)

[1426] 2. The system according to claim 1, wherein the server includes means for performing grayscale conversion as preprocessing of the screenshot image.

[1427] (Claim 3)

[1428] 10. The system of claim 1, wherein the server includes means for extracting text within the image using optical character recognition techniques.

[1429] "Application Example 1"

[1430] (Claim 1)

[1431] A means for users to upload screenshots of the application;

[1432] a means for the server to store the uploaded screenshot image;

[1433] means for the server to analyze the stored screenshot images and recognize text and GUI elements within the images;

[1434] means for the server to infer an application screen layout based on the recognized text and GUI elements;

[1435] means for the server to generate source code based on the inferred layout;

[1436] a means by which the server provides the generated source code to the user;

[1437] A means for reverse engineering screenshots of machine control panels in a factory;

[1438] means for automatically generating a source code for a machine operation program based on a screenshot image of a display device to be used;

[1439] means for performing image analysis and code generation via a cloud server;

[1440] A system including:

[1441] (Claim 2)

[1442] 2. The system of claim 1, wherein the server includes means for performing grayscale conversion as preprocessing of the screenshot image.

[1443] (Claim 3)

[1444] 10. The system of claim 1, wherein the server includes means for extracting text within the image using OCR technology.

[1445] "Example 2: Combining Emotion Engines"

[1446] (Claim 1)

[1447] A means for users to upload screenshots of the application;

[1448] a means for the server to store the uploaded screenshot image;

[1449] means for the server to analyze the stored screenshot images and recognize text and user interface elements within the images;

[1450] The server analyzes facial expressions and gaze to recognize the user's emotions.

[1451] means for the server to infer a layout of an application screen based on the recognized text, user interface elements, and emotion information;

[1452] a means for the server to generate source code based on the inferred layout and user emotion information;

[1453] a means by which the server provides the generated source code to the user;

[1454] A system including:

[1455] (Claim 2)

[1456] 2. The system of claim 1, wherein the server includes means for performing grayscale conversion as preprocessing of the screenshot image.

[1457] (Claim 3)

[1458] 10. The system of claim 1, wherein the server includes means for extracting text within the image using character recognition techniques.

[1459] "Application example 2 when combining emotion engines"

[1460] (Claim 1)

[1461] A means for users to upload screenshots of the application;

[1462] a means for the server to store the uploaded screenshot image;

[1463] means for the server to analyze the stored screenshot images and recognize text and GUI elements within the images;

[1464] means for the server to infer an application screen layout based on the recognized text and GUI elements;

[1465] means for the server to generate source code based on the inferred layout;

[1466] a means by which the server provides the generated source code to the user;

[1467] a means for the server to recognize the user's emotion based on the uploaded screenshot image;

[1468] The system includes a means for the server to optimize the user interface based on the recognized emotion information.

[1469] (Claim 2)

[1470] 2. The system of claim 1, wherein the server includes means for performing grayscale conversion as preprocessing of the screenshot image.

[1471] (Claim 3)

[1472] 10. The system of claim 1, wherein the server includes means for extracting text within the image using OCR technology. [Explanation of symbols]

[1473] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for users to upload screenshots of the application; a means for the server to store the uploaded screenshot image; means for the server to analyze the stored screenshot images and recognize text and GUI elements within the images; means for the server to infer an application screen layout based on the recognized text and GUI elements; means for the server to generate source code based on the inferred layout; a means by which the server provides the generated source code to the user; A system including:

2. 2. The system of claim 1, wherein the server includes means for performing grayscale conversion as preprocessing of the screenshot image.

3. 10. The system of claim 1, wherein the server includes means for extracting text within the image using OCR technology.

Images