system

Abstract

Provides a system for automatically generating specifications. [Solution] A system is provided that includes a means for uploading a code file, a means for uploading screen information, a means for analyzing the uploaded code file and extracting code characteristics, a means for analyzing the screen information and extracting screen characteristics, a means for automatically generating specifications based on the code file and the screen information, a means for saving the automatically generated specifications, and a means for searching for saved specifications.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In software development, code duplication and multiple implementations of similar functions frequently occur, resulting in reduced development efficiency and increased maintenance difficulty. Furthermore, manually creating specifications for related screens takes time and effort, and can be inaccurate. A system that solves these problems and enables efficient and accurate specification creation is needed. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for uploading code files and screen information to a system, a means for analyzing the uploaded code files to extract code characteristics, a means for analyzing the screen information to extract screen characteristics, a means for automatically generating specifications based on this information, a means for saving the automatically generated specifications, and a means for searching saved specifications. This system allows users to automatically create efficient and accurate specifications simply by uploading code and screen information. This prevents code duplication and the separate implementation of similar functions, improving development efficiency and maintainability.

[0006] "Code file" refers to data in file format that stores program code written by a programmer.

[0007] "Screen information" is information about the user interface (UI) that is displayed as a result of executing a program, and is provided in the form of a screenshot, URL, etc.

[0008] "Upload" refers to the act of a user sending data from a terminal to a server.

[0009] "Analysis" refers to the process a system performs to understand input data and extract its characteristics and structure.

[0010] "Characteristics" means significant elements or attributes of the analyzed data.

[0011] "Extraction" refers to the process of extracting and clarifying specific elements from data.

[0012] "Automatic generation" means a process by which a system independently generates specifications or other documents programmatically.

[0013] "Storage" refers to the act of recording generated data or documents in a storage device and managing them so that they can be accessed later.

[0014] "Search" refers to the act of investigating to find specific information among stored data or documents.

[0015] A "specification" refers to a document that describes in detail the functions, structure, operating procedures, etc. of a program. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention relates to a system for automatically generating specifications by uploading code files and screen information to the system. A specific embodiment of this system will be described below.

[0038] The program used in this system analyzes code files and screen information through multiple steps to generate specifications. Three main parties are involved: the server, the terminal, and the user.

[0039] How to upload code and screen information

[0040] The user uses their own terminal through the system interface to select the code file and associated screen information (e.g., screenshots and URLs) and upload them to the system.

[0041] The terminal receives the file selected by the user and transmits it to the server.

[0042] Means of analyzing information

[0043] The server receives the code file and screen information sent from the terminal.

[0044] The server parses the received code file and extracts characteristics of the code, such as function names, arguments, and return values.

[0045] For example, if you receive the following code:

[0046] JavaScript (registered trademark)

[0047] function add(a, b) {

[0048] return a + b;

[0049] }

[0050] The server analyzes information such as "function name: add," "arguments: a, b," and "return value: a + b."

[0051] The server also analyzes the screen information it receives separately. For example, it uses OCR technology to identify text and UI elements from screenshots and extracts characteristics such as "numeric input form" and "calculation button."

[0052] Automatic generation of specifications

[0053] The server automatically generates specifications based on the results of analyzing the code and screen information.

[0054] For example, the analysis results can be used to generate the following specification:

[0055] Code Specifications

[0056] Function name: add

[0057] Arguments: a, b

[0058] Returns: a + b

[0059] Related screens

[0060] URL: (URL goes here)

[0061] Description: A screen with a form for entering numbers and a "Calculate" button

[0062] Processing flow

[0063] 1. User enters a number into a form

[0064] 2. Press the "Calculate" button

[0065] 3. The add function is called and the result is displayed

[0066] Storage method for specifications

[0067] The server stores the automatically generated specifications in a database.

[0068] When saving, the specification is given an identifiable ID and managed so that it can be easily searched for later.

[0069] How to search for specifications

[0070] A user sends a specification search request from the system interface, for example, a query such as "I want to know the specification of the add function."

[0071] The terminal receives a query from a user and sends it to the server.

[0072] The server searches the stored database for the relevant specifications and provides the results to the user.

[0073] This system allows users to avoid code duplication and multiple implementations of similar functionality, improving development efficiency and maintainability. It also automates the specification creation process, saving significant time and effort. This also improves technical accuracy and the productivity of the entire development team.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] The user selects the code file and related screen information (screenshots or URLs) through the system interface and performs the upload.

[0077] Step 2:

[0078] The terminal receives the code file and screen information selected by the user and transmits the data to the server.

[0079] Step 3:

[0080] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[0081] Step 4:

[0082] The server analyzes the received code file. Specifically, it identifies the function names, arguments, and return values ​​in the code. For example, it extracts the function "add" from the code, its arguments "a, b", and its return value "a + b".

[0083] Step 5:

[0084] The server analyzes the received screen information and extracts text and UI elements (e.g., input forms, buttons) from the screenshot using OCR technology.

[0085] Step 6:

[0086] The server automatically generates a specification based on the analysis results. Specifically, it integrates the code analysis results (function names, arguments, return values) and the screen information analysis results (UI elements) and formats them into a specification format.

[0087] Step 7:

[0088] The server stores the automatically generated specifications in a database, assigning an ID to each specification to make it easy to search for later.

[0089] Step 8:

[0090] A user submits a specification search request using the system's interface (e.g., a query such as "I want to know the specification of the add function").

[0091] Step 9:

[0092] The terminal transmits a search query from the user to the server.

[0093] Step 10:

[0094] The server searches the database for the relevant specifications, finds a match with the query, and finds the corresponding specifications.

[0095] Step 11:

[0096] The server provides the user with the relevant specifications as a search result.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] In existing software development environments, the manual analysis of code files and screen information and the creation of specifications requires a great deal of time and effort. Furthermore, manual analysis and documentation has a high risk of human error, leading to incomplete specifications and reduced development efficiency. To solve these problems, a system is needed that can automatically analyze code files and screen information and generate specifications quickly and accurately.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes means for uploading a code file to an electronic computing device, means for uploading screen information to the electronic computing device, means for analyzing the uploaded code file to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating a specification based on the code file and the screen information, means for saving the automatically generated specification in a database, means for searching the saved specification, means for displaying progress and notifying the user, and means for applying the analysis results to a template to generate a specification. This significantly reduces the time and effort associated with manual analysis and document creation, and enables the generation of accurate and consistent specifications.

[0102] A "code file" is electronic data containing the source code of a program or script.

[0103] "Screen information" refers to data that includes information about the user interface, the layout of the screen, and its elements.

[0104] "Computing device" refers to a device that processes and stores data, such as a computer or server.

[0105] "Analysis" is the process of breaking down input data and extracting its components and characteristics.

[0106] "Characteristics" refers to specific attributes or functional characteristics extracted from code or screen information.

[0107] A "specification" is a document that describes in detail the operation, structure, and functions of a system or program.

[0108] A "database" is a management system for systematically storing large amounts of data and making them efficiently searchable and accessible.

[0109] "Search" refers to the process of locating specific information from stored data.

[0110] "Optical character recognition technology" is a technology that analyzes characters in an image and converts them into text data.

[0111] A "template" is a standard format for generating documents in a consistent format.

[0112] "Progress" is information that indicates the current stage of data analysis and processing.

[0113] The term "user" refers to a person who uses the system of the present invention.

[0114] This invention relates to a system that uploads code files and screen information to a system, automatically generates specifications, and stores and searches them. The following steps and techniques are used to realize this system.

[0115] Upload method:

[0116] A user uses his / her device to select a code file (e.g., add.js) and related screen information (e.g., form_screen.png) from the system interface and upload them to the system. The device receives these files and sends them to the server via an HTTP request. The user interface displays the file sending status to the user.

[0117] Information analysis methods:

[0118] The server receives the code file and screen information sent from the device and temporarily stores it. The server analyzes the received code file and extracts characteristics such as function names, arguments, and return values. For example, if the code is JavaScript code, the server identifies the function name "add," arguments "a, b," and return value "a + b." The server also uses OCR technology to analyze the screen information and identify text and UI elements. For example, it extracts characteristics such as a "number input form" and a "calculation button" from a screenshot.

[0119] Automated generation of specifications:

[0120] The server automatically generates a specification template based on the results of the code and screen analysis, applying these characteristics to the template. Specifically, it uses a template engine to incorporate the analysis results into a predetermined format and generates the following specification:

[0121] Code Specifications

[0122] Function name: add

[0123] Arguments: a, b

[0124] Returns: a + b

[0125] Related screens

[0126] URL: (URL goes here)

[0127] Description: A screen with a form for entering numbers and a "Calculate" button

[0128] Processing flow

[0129] 1. User enters a number into a form

[0130] 2. Press the "Calculate" button

[0131] 3. The add function is called and the result is displayed

[0132] How to save the specification:

[0133] The server saves the automatically generated specifications in a database. When saved, the specifications are given an identifiable ID and managed for easy retrieval later. The fact that the specifications have been saved is also recorded in a log.

[0134] How to search for specifications:

[0135] The user enters a search query into the system interface and sends a request to search for specifications. For example, the user might enter, "I want to know the specifications of the add function." The terminal sends this query to the server, which then searches the database for the relevant specifications. The search results are displayed to the user via the terminal.

[0136] Examples and prompts:

[0137] The user uploads a JavaScript file called "add.js" and a screenshot called "form_screen.png" to the system on their device.

[0138] The terminal transmits these files to the server and displays the transmission status.

[0139] The server analyzes "add.js" to extract function names, arguments, and return values. It also analyzes "form_screen.png" using OCR technology to identify UI elements.

[0140] The server generates a specification based on these analysis results, assigns it an identification ID of "add_function_spec_001," and stores it in the database.

[0141] Later, when the user sends a search request for "add function specification" from the system interface, the server searches the database for "add_function_spec_001" and provides the results to the user.

[0142] Example prompt sentence:

[0143] Upload the code file and screen information. For example, select the file add.js and the screenshot form_screen.png and upload them to the system. A specification document will be automatically generated based on this information.

[0144] This invention allows users to significantly reduce the time and effort associated with manual analysis and documentation, enabling the generation of accurate and consistent specifications. Furthermore, the system's search function allows users to quickly reference existing specifications, improving development efficiency and maintainability.

[0145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0146] Step 1:

[0147] User selects and uploads code files and screen information

[0148] Input: Code file (e.g. add.js), screen information (e.g. form_screen.png)

[0149] Specific operation: The user opens the system interface and clicks the upload button. The user selects the file to upload from their device and clicks "Upload".

[0150] Output: The uploaded file is sent from the user's device to the server.

[0151] Step 2:

[0152] The device receives the file and sends it to the server

[0153] Input: User selected file and upload request

[0154] Specific operation: The device reads the file selected by the user from local storage, creates an HTTP POST request, and sends it to the server.

[0155] Output: The status of the HTTP request sent to the server is displayed on the terminal to inform the user of progress.

[0156] Step 3:

[0157] The server receives the code file and screen information.

[0158] Input: File data included in the HTTP request from the terminal

[0159] Specific operation: The server receives the HTTP request, temporarily stores the code file and screen information in storage, checks the integrity of the file, and if there are no problems, starts the analysis process.

[0160] Output: The file is saved and ready for analysis.

[0161] Step 4:

[0162] The server begins parsing the code file

[0163] Input: Saved code file

[0164] Specific operation: The server starts an analysis program to analyze the contents of the code file, extracting characteristics such as function names, arguments, and return values.

[0165] Output: Characteristic data of the extracted code (e.g., "Function name: add", "Arguments: a, b", "Return value: a + b")

[0166] Step 5:

[0167] The server analyzes the screen information

[0168] Input: Saved screen information

[0169] What it does: The server uses OCR technology to identify text and UI elements from the screenshot, and then categorizes the identified UI elements.

[0170] Output: Extracted screen information characteristics data (e.g. "Number input form", "Calculation button")

[0171] Step 6:

[0172] The server generates specifications based on the analysis results

[0173] Input: Code characteristic data, Screen information characteristic data

[0174] Specific operation: The server uses a template engine to incorporate the analysis results into a predetermined format and automatically generate a specification document.

[0175] Output: Automatically generated specification (e.g., "Function name: add", "Arguments: a, b", "Return value: a + b", "Related screen: Screen with a numeric input form and a calculation button")

[0176] Step 7:

[0177] The server stores the specifications in a database

[0178] Input: Auto-generated specification

[0179] Specific behavior: The server saves the specification in the database using an INSERT query and assigns it an identifiable ID.

[0180] Output: Notification of completion of saving and generation of identification ID (e.g. "Specification ID: add_function_spec_001")

[0181] Step 8:

[0182] User submits a search request for specifications

[0183] Input: Search query (e.g. "I want to know the specifications of the add function")

[0184] Specific actions: The user enters a query into the search box in the system interface and clicks the search button.

[0185] Output: A search request is sent to the server.

[0186] Step 9:

[0187] The server searches for the specification and returns the results

[0188] Input: A search request from the user

[0189] Specific operation: The server searches the database, extracts the relevant specifications, and returns the extracted specifications to the terminal.

[0190] Output: The search result, a specification document, is displayed to the user (e.g., detailed information for "add_function_spec_001").

[0191] (Application example 1)

[0192] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0193] When maintaining and repairing industrial equipment, there is a lack of a way to quickly and accurately generate specifications for related code and screen information. This challenge makes it difficult for on-site technicians to perform their work efficiently, increasing the likelihood of mistakes and rework. In particular, there is a need to address the need to quickly obtain information on-site using smart devices.

[0194] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0195] In this invention, the server includes a means for uploading a code file, a means for uploading screen information, and a means for automatically generating specifications based on the uploaded code file and screen information, thereby enabling specifications to be generated quickly and accurately for maintenance and repair of industrial equipment and displayed on a smart device.

[0196] - "Code File" means a file containing a program description, which contains the set of instructions necessary to run the program.

[0197] "Screen information" is visual information about the user interface, and includes data such as screenshots and URLs.

[0198] A "specification" is a technical document that is automatically generated based on code files and screen information, and contains detailed descriptions of the program's characteristics and screen characteristics.

[0199] "Optical character recognition technology" is a technology that extracts characters from an image and converts them into text data.

[0200] The "storage means" is a function for storing the generated specifications in a database or other storage device.

[0201] "Search facility" is a function that allows you to search saved specifications by keywords or other criteria to quickly access specific information.

[0202] The "uploading means" is a function that allows a user to use a terminal to send code files and screen information to a server.

[0203] "Industrial equipment" refers to machinery and equipment used in various industries, and is controlled using advanced technology.

[0204] "Maintenance and repair" refers to the work of inspecting and repairing industrial equipment to maintain its normal operation.

[0205] "Smart device" refers to a mobile terminal device that can connect to the Internet and use multi-function applications.

[0206] The system for implementing this invention mainly involves three entities: a server, a terminal, and a user. The server analyzes the code file and screen information uploaded by the user and generates a specification document based on that. The server then saves the specification document and displays it on the smart device for use in maintenance and repair.

[0207] Uploading code files and screen information

[0208] The user selects the code file and related screen information (screenshots or URLs) from the terminal and uploads them to the system. The terminal receives the file selected by the user and sends it to the server.

[0209] Analysis of information

[0210] The server receives the code file and screen information sent from the device. The server analyzes the received code file and extracts function names, arguments, return values, etc. It also uses OCR technology to identify text and UI elements in the screen information and extracts characteristics such as "numeric input forms" and "buttons."

[0211] Automatic generation of specifications

[0212] The server automatically generates specifications required for the maintenance and repair of industrial equipment based on the analyzed code and screen information. The generated specifications contain detailed descriptions of the code characteristics and user interface characteristics.

[0213] Save and search specifications

[0214] The generated specifications are stored in a database on the server and assigned an identifiable ID. Users can search for specifications through their smart devices and quickly obtain information for efficient maintenance and repair of industrial equipment.

[0215] Hardware and software used

[0216] The server is a computer system capable of high performance analysis, and is preferably a distributed system if necessary.

[0217] To use OCR technology, we use Tesseract (an open source OCR engine).

[0218] Code analysis uses Python's AST module and specific programming language analysis tools.

[0219] The database uses an RDBMS such as MySQL (registered trademark) or PostgreSQL.

[0220] Examples and prompts

[0221] For example, you could use the following prompt as input to a generative AI model:

[0222] "I'd like to know more about how the move_forward(steps) function works.

[0223] The code file contains the following:

[0224] function move_forward(steps) {

[0225] for(int i = 0; i < steps; i++) {

[0226] move_one_step();

[0227] }

[0228] }

[0229] 2. The screenshot of the screen information includes the following elements:

[0230] "Start" button

[0231] An input field labeled "Steps"

[0232] "Enter the number of steps"

[0233] This prompt allows the server to analyze the code and screen information and automatically generate detailed specifications required for maintenance and repair of industrial equipment.

[0234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0235] Step 1:

[0236] The user uploads the code file and screen information from the terminal. The user selects the code file and screen information (screenshots or URLs) to upload using the interface of the smart device. The terminal sends the files received from the user to the server.

[0237] Input: User selected code file and screen information

[0238] Output: File data sent to the server

[0239] Step 2:

[0240] The server receives the code file and screen information sent from the terminal and stores the received file in a temporary storage area.

[0241] Input: Code file and screen information sent from the terminal

[0242] Output: A file stored in the temporary storage area on the server

[0243] Step 3:

[0244] The server analyzes the code file, specifically using Python's AST module to parse the code and extract properties such as function names, arguments, and return values.

[0245] Input: Code file stored in temporary storage area

[0246] Output: Analysis results such as function name, arguments, and return value

[0247] Step 4:

[0248] The server analyzes the screen information, using OCR technology (Tesseract) to extract text and UI elements from the screenshot and identify screen characteristics.

[0249] Input: Screen information stored in temporary storage area

[0250] Output: Analysis results for UI elements (buttons, input fields, labels, etc.)

[0251] Step 5:

[0252] The server automatically generates specifications based on the analysis results, combining code characteristics and screen characteristics to generate detailed specifications suitable for the maintenance and repair of industrial equipment.

[0253] Input: Analyzed code characteristics and screen characteristics

[0254] Output: Generated specification

[0255] Step 6:

[0256] The server stores the generated specifications in a database, assigns them identifiable IDs, and manages them for easy retrieval later.

[0257] Input: Generated specification

[0258] Output: The specifications stored in the database and their IDs

[0259] Step 7:

[0260] A user searches for specifications using a smart device. The user inputs a query using the search interface and sends a search request to the server. The server searches for the relevant specifications from the database and provides them to the user.

[0261] Input: Search query from user

[0262] Output: Search results for matching specifications

[0263] This series of processing steps makes it possible to quickly and accurately generate specifications required for the maintenance and repair of industrial equipment and display them on a smart device.

[0264] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0265] This invention is a system that automatically generates specifications based on uploaded code files and screen information, and further combines it with an emotion engine that recognizes user emotions. This system enables efficient and accurate specification creation, and the addition of the emotion engine improves usability. Specific embodiments of this system are described below.

[0266] The program used in this system analyzes code files and screen information through multiple steps to generate specifications. Three main parties are involved: the server, the terminal, and the user, and an emotion engine is also integrated.

[0267] How to upload code and screen information

[0268] The user selects the code file and associated screen information (e.g., screenshots or URLs) through the system interface and performs the upload.

[0269] The terminal receives the files selected by the user and transmits the data to the server.

[0270] Means of analyzing information

[0271] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[0272] The server parses the received code file and extracts code characteristics such as function names, arguments, and return values.

[0273] For example, if you receive the following code:

[0274] javascript

[0275] function add(a, b) {

[0276] return a + b;

[0277] }

[0278] The server analyzes information such as "function name: add," "arguments: a, b," and "return value: a + b."

[0279] The server also analyzes the separately received screen information, using OCR technology to extract text and UI elements (e.g., input forms, buttons) from the screenshot.

[0280] Emotion engine integration

[0281] The device uses an emotion engine to analyze the user's facial expressions and voice and recognize the user's emotional state.

[0282] For example, it detects whether a user is confused or satisfied from their facial expression or tone of voice.

[0283] Automatic generation of specifications

[0284] The server automatically generates specifications based on the results of code and screen analysis. Specifically, it integrates the results of code analysis (function names, arguments, return values) and screen analysis (UI elements) and formats them into a specification format.

[0285] For example, it generates the following specification:

[0286] Code Specifications

[0287] Function name: add

[0288] Arguments: a, b

[0289] Returns: a + b

[0290] Related screens

[0291] URL: (URL goes here)

[0292] Description: A screen with a form for entering numbers and a "Calculate" button

[0293] Processing flow

[0294] 1. User enters a number into a form

[0295] 2. Press the "Calculate" button

[0296] 3. The add function is called and the result is displayed

[0297] Storage method for specifications

[0298] The server stores the automatically generated specifications in a database, assigning an ID to each specification to make it easier to search for later.

[0299] How to search for specifications

[0300] A user sends a specification search request from the system interface, for example, a query such as "I want to know the specification of the add function."

[0301] The terminal receives a query from a user and sends it to the server.

[0302] The server searches the stored database for the relevant specifications and provides the results to the user.

[0303] Emotion-based feedback measures

[0304] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine.

[0305] For example, if the user is confused, more detailed help information can be provided.

[0306] The server provides appropriate feedback depending on the user's emotions, such as displaying a positive message if the user is satisfied.

[0307] This system allows users to avoid code duplication and separate implementations of similar functions, improving development efficiency and maintainability. Furthermore, the integration of an emotion engine improves the user experience and makes the system easier to use. Appropriate feedback based on emotions and adjustments to the operation interface allow users to use the system more comfortably.

[0308] The processing flow will be explained below.

[0309] Step 1:

[0310] The user selects the code file and associated screen information (e.g., screenshots or URLs) through the system interface and performs the upload.

[0311] Step 2:

[0312] The terminal receives the code file and screen information selected by the user and transmits the data to the server.

[0313] Step 3:

[0314] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[0315] Step 4:

[0316] The server analyzes the received code file to identify the function name, arguments, and return value of the code. For example, if the server receives the following code:

[0317] javascript

[0318] function add(a, b) {

[0319] return a + b;

[0320] }

[0321] The server extracts "function name: add", "arguments: a, b", and "return value: a + b" from this code.

[0322] Step 5:

[0323] The server analyzes the received screen information. It uses OCR technology to identify text and UI elements from the screenshot and extract their characteristics. For example, it detects "numerical input forms" and "calculation buttons."

[0324] Step 6:

[0325] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize the user's emotional state, for example, detecting whether the user is confused or satisfied from their facial expressions and tone of voice.

[0326] Step 7:

[0327] The server automatically generates a specification based on the analysis results. Specifically, it combines the code analysis results (function names, arguments, return values) with the screen information analysis results (UI elements) and formats them into the following specification format:

[0328] Code Specifications

[0329] Function name: add

[0330] Arguments: a, b

[0331] Returns: a + b

[0332] Related screens

[0333] URL: (URL goes here)

[0334] Description: A screen with a form for entering numbers and a "Calculate" button

[0335] Processing flow

[0336] 1. User enters a number into a form

[0337] 2. Press the "Calculate" button

[0338] 3. The add function is called and the result is displayed

[0339] Step 8:

[0340] The server saves the automatically generated specifications in a database. When saving, it assigns an ID to each specification and manages it so that it can be easily searched for later.

[0341] Step 9:

[0342] A user uses the system interface to submit a specification search request (e.g., a query such as "I want to know the specification of the add function").

[0343] Step 10:

[0344] The terminal receives a query from a user and sends it to the server.

[0345] Step 11:

[0346] The server searches the database for specifications that match the query and finds the relevant specifications.

[0347] Step 12:

[0348] The server provides the user with the relevant specifications as a search result.

[0349] Step 13:

[0350] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine, for example, displaying detailed help information if the user is confused.

[0351] Step 14:

[0352] The server provides appropriate feedback depending on the user's emotions, for example, if the user is satisfied, it displays a positive message.

[0353] In this way, the system prevents code duplication and separate implementations of similar functions, improving development efficiency and maintainability, while also improving the user experience by adding an emotion engine.

[0354] Example 2

[0355] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] Conventional specification creation systems require manual analysis of code files and screen information to create specifications, which is extremely time-consuming, inefficient, and prone to human error. Furthermore, the system lacks usability because it does not reflect the user's emotional state. Therefore, there is a need for a system that can solve these issues, create specifications efficiently and accurately, and adjust the interface according to the user's emotional state.

[0357] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0358] In this invention, the server includes means for a user to upload a code file, means for uploading screen information, means for analyzing the uploaded code file to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating a specification based on the code file and the screen information, means for saving the automatically generated specification, means for searching the saved specification, means for analyzing the user's facial expression and voice to recognize the user's emotional state, and means for automatically adjusting the operation interface based on the recognized emotional state. This enables efficient analysis of the code and screen information and automatic generation of the specification, and further improves usability by adjusting the interface taking the user's emotional state into consideration.

[0359] A "user" is an entity that uses the system to upload code files and screen information, and automatically generate and search specifications.

[0360] A "code file" is a file in which the source code of a program is written, and which has properties such as function names, arguments, and return values.

[0361] "Screen information" is data that indicates the state and structure of the user interface, such as a screenshot of the screen or a URL.

[0362] "Uploading means" refers to an interface that allows a user to send code files and screen information to the system, and a function that performs the processing.

[0363] "Analysis means" refers to a function that analyzes the received code file and screen information and extracts their characteristics.

[0364] A "specification" is a document generated based on the analysis results, which describes in detail the operation and structure of the system.

[0365] "Automatic generation means" refers to a function that automatically creates specifications based on the analysis results.

[0366] "Storage means" refers to a function that stores automatically generated specifications in a database or the like, allowing them to be searched for later.

[0367] "Search means" refers to a function that searches saved specifications based on a query and provides the results to the user.

[0368] The "emotional state" is a psychological state analyzed from the user's facial expression and voice, and includes confusion, satisfaction, anger, and the like.

[0369] "Emotion analysis means" refers to a function that captures the user's facial expressions and voice, analyzes them, and recognizes their emotional state.

[0370] The "operation interface adjusting means" refers to a function that optimizes the interface operated by the user based on the recognized emotional state.

[0371] This invention is a system that allows users to upload code files and screen information, and automatically generates specifications based on that data. The system of this invention involves the server, terminal, and user, and further improves usability by integrating an emotion engine.

[0372] How to upload code and screen information

[0373] The user selects and uploads the code file and screen information (e.g., screenshots or URLs) through the system interface. The device receives the file selected by the user and sends this data to the server. Specifically, the device checks the file format and size, and if there are no problems, transfers the file to the server.

[0374] Means of analyzing information

[0375] The server receives the code file and screen information sent from the device and temporarily stores it. Next, the server analyzes the received code file. The specific software used here is a parser (e.g., ANTLR) that extracts code characteristics such as function names, arguments, and return values. For example, if the following code is received:

[0376] javascript

[0377] function add(a, b) {

[0378] return a + b;

[0379] }

[0380] The server extracts information such as "function name: add", "arguments: a, b", and "return value: a + b".

[0381] Furthermore, the server uses OCR technology (e.g., Tesseract OCR) to extract text and UI elements (e.g., input forms, buttons) from the screen information (screenshots).

[0382] Emotion engine integration

[0383] The device uses a built-in camera and microphone to capture the user's facial expressions and voice in real time. The device then uses emotion analysis software (e.g., Microsoft® Azure® Emotion API) to analyze the user's emotional state. For example, it can detect whether the user is confused or happy from their facial expressions and tone of voice.

[0384] Automatic generation of specifications

[0385] The server automatically generates a specification based on the results of code and screen analysis. Specifically, it combines the results of code analysis (function names, arguments, return values) with the results of screen analysis (UI elements) and formats them into a specification. For example, the following specification is generated:

[0386] markdown

[0387] Code Specifications

[0388] Function name: add

[0389] Arguments: a, b

[0390] Returns: a + b

[0391] Related screens

[0392] URL: (URL goes here)

[0393] Description: A screen with a form for entering numbers and a "Calculate" button

[0394] Processing flow

[0395] 1. User enters a number into a form

[0396] 2. Press the "Calculate" button

[0397] 3. The add function is called and the result is displayed

[0398] Storage and retrieval of specifications

[0399] The server saves the automatically generated specifications in a database. When saved, the specifications are assigned an identification ID, which is managed so that they can be easily searched for later. When the user sends a specification search request from the interface, the terminal receives this and sends it to the server. The server searches the saved database for the corresponding specifications and provides the results to the user.

[0400] Emotion-based feedback measures

[0401] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, it provides more detailed help information. If the user is satisfied, it provides feedback by displaying a positive message.

[0402] Examples and Prompts

[0403] A user logs into the system and uploads the project's code files and screenshots. The device checks them and sends them to the server. The server analyzes the code and extracts function names, arguments, return values, etc. It also uses OCR technology to analyze text and UI elements from screenshots. Based on these results, a specification document is automatically generated and stored in a database.

[0404] Example prompt sentence:

[0405] In this system, I uploaded the following JavaScript code file: function multiply(x, y) { return xy;} Please auto-generate the specification.

[0406] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0407] Step 1:

[0408] Through the system interface, the user clicks the upload button, selects the code file and screen information (screenshots or URLs), and uploads them.

[0409] Input: Code file, screen information

[0410] Behavior: The user selects a file through a file selection dialog and initiates the upload.

[0411] Output: The device receives the selected file.

[0412] Step 2:

[0413] The device checks the code file and screen information received from the user to ensure there are no problems with the format or size, and then sends this data to the server.

[0414] Input: Code file, screen information

[0415] Action: The device checks the file format and whether it exceeds the maximum capacity. If there are no problems, it transfers the file to the server.

[0416] Output: The server receives the file and stores it temporarily.

[0417] Step 3:

[0418] The server analyzes the received code file and extracts code characteristics such as function names, arguments, return values, etc. Specifically, it analyzes the code using a parser (e.g., ANTLR).

[0419] Input: Code file

[0420] How it works: The server parses the code through a parser and extracts the function name, arguments, and return value. For example, if you parse the code function add(a, b) { return a + b;}, it extracts "function name: add", "arguments: a, b", and "return value: a + b".

[0421] Output: Code characteristics information (function names, arguments, return values, etc.)

[0422] Step 4:

[0423] The server simultaneously analyzes the received screen information and uses OCR technology (e.g., Tesseract OCR) to extract text and UI elements from the screenshot.

[0424] Input: Screen information (screenshots, URL, etc.)

[0425] How it works: The server uses OCR technology to analyze the screenshot and extract text and UI elements (input forms, buttons, etc.).

[0426] Output: Screen characteristics (text, UI elements, etc.)

[0427] Step 5:

[0428] The server integrates the code characteristics information and the screen characteristics information to automatically generate a specification document, which includes the function name, arguments, return values, and related screen information.

[0429] Input: Code characteristics information, screen characteristics information

[0430] How it works: The server combines this information and formats it into a specification. For example, it puts function names and argument information in the code section, and screen information in a separate section.

[0431] Output: Auto-generated specification

[0432] Step 6:

[0433] The server stores the automatically generated specifications in a database, assigns them an ID, and manages them for easy retrieval later.

[0434] Input: Auto-generated specification

[0435] Operation: The server assigns an ID to the specification and stores it in the database.

[0436] Output: Specification saved in database

[0437] Step 7:

[0438] A user sends a specification search request through the system interface, for example, by entering a query such as "I want to know the specification of the add function."

[0439] Input: Search query (e.g. "add function specification")

[0440] How it works: A user enters a query into the search box and clicks the search button. The device receives the query and sends it to the server.

[0441] Output: The server receives the search request.

[0442] Step 8:

[0443] The server searches the database for the relevant specifications based on the search request and provides the results to the user.

[0444] Input: Search request

[0445] How it works: The server searches the database for the query, finds the relevant specifications, and returns the search results to the device.

[0446] Output: Search results (relevant specifications)

[0447] Step 9:

[0448] The terminal displays the search results received from the server to the user.

[0449] Input: Search results from the server

[0450] Operation: The device receives the search results and displays them on the user's interface.

[0451] Output: Specification information displayed in the interface

[0452] Step 10:

[0453] The device uses an emotion engine to capture the user's facial expressions and voice and recognize their emotional state, for example, detecting whether the user is confused or happy from their facial expressions or tone of voice.

[0454] Input: User's facial expression, voice

[0455] How it works: The device captures the user's facial expressions and voice through the camera and microphone, and performs emotion analysis.

[0456] Output: User's emotional state (confused, satisfied, etc.)

[0457] Step 11:

[0458] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, it provides detailed help information. If the user is satisfied, it displays a positive message.

[0459] Input: User's emotional state

[0460] Action: The server makes appropriate user interface adjustments and feedback based on the recognized emotional state.

[0461] Output: Adjusting the interface, providing feedback (e.g., detailed help information, positive messages)

[0462] (Application example 2)

[0463] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0464] Currently, programming factory robots and creating operation specifications is done manually using complex tools, which requires a great deal of time and effort. In addition, users can become confused when operating the robots, hindering efficient work. Furthermore, real-time feedback on the inconvenience or confusion experienced by users is not provided, creating a need for improved usability.

[0465] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0466] In this invention, the server includes means for uploading code files, means for uploading screen information, means for analyzing the uploaded code files to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating specifications based on the code files and the screen information, means for saving the automatically generated specifications, means for searching the saved specifications, means for analyzing emotional information to recognize the emotional state of a user, and means for adjusting the operation interface and providing feedback based on the emotional information. This allows users to significantly reduce the time and effort required to program factory robots, and provides real-time feedback according to emotions, improving usability and enabling more efficient and comfortable work.

[0467] A "code file" is a file in which the source code of a program is written.

[0468] "Screen information" refers to data that includes visual information about the user interface, such as screenshots and URLs.

[0469] "Analysis" is the process used to extract characteristics and structure from code files and screen information.

[0470] A "specification" is a document that describes in detail the functions and operations of software or a system.

[0471] "Emotion information" refers to data on emotions detected from the user's facial expressions, voice, etc.

[0472] An "operation interface" refers to the screen or input means that a user uses to operate a system or application.

[0473] "Feedback" refers to the system providing real-time responses to the user's actions and state.

[0474] A "server" is a computer that provides data and services to other terminals over a network.

[0475] "Optical character recognition technology" is a technology that extracts character information from an image.

[0476] MODE FOR CARRYING OUT THE INVENTION

[0477] This invention is a system that automatically generates specifications based on uploaded code files and screen information, and also combines it with an emotion engine that recognizes user emotions. This system enables efficient and accurate specification creation, and the addition of the emotion engine improves usability.

[0478] System configuration and functions

[0479] This system consists of three components: a server, a terminal, and a user. Furthermore, by integrating an emotion engine, it has been added with the function of providing appropriate feedback based on the user's emotional state.

[0480] Uploading code and screen information

[0481] The user selects the code file and related screen information (e.g., screenshots or URLs) through the system interface and performs the upload. The terminal receives the files selected by the user and sends the data to the server. This part can be a general computer or smart device.

[0482] Analysis of information

[0483] The server receives the code files and screen information sent from the device and temporarily stores them. The server then analyzes the received code files and extracts code characteristics such as function names, arguments, and return values. It also analyzes the separately received screen information. Optical character recognition (OCR) is used to extract text and UI elements (e.g., input forms, buttons) from the screenshots. This is done using software libraries such as pytesseract and OpenCV.

[0484] Emotion engine integration

[0485] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize the user's emotional state. Specifically, a library called EmotionRecognizer is used. For example, it can detect whether the user is confused or happy from their facial expressions and tone of voice.

[0486] Automatic generation of specifications

[0487] The server automatically generates specifications based on the results of code and screen information analysis. Specifically, it integrates the results of code analysis (function name, arguments, return value) and screen information analysis (UI elements) and formats them into a specification format. An example of the generated specification is shown below.

[0488] Function name: move_forward

[0489] Argument: distance

[0490] Returns: None

[0491] Function Name: rotate

[0492] Argument: angle

[0493] Returns: None

[0494] Related screen: A screen with a form for entering numbers and an "Execute" button

[0495] Process flow:

[0496] 1. User enters a number into a form

[0497] 2. Press the "Run" button

[0498] 3. The functions move_forward and rotate are called and the results are displayed.

[0499] Save and search specifications

[0500] The server stores the automatically generated specifications in a database. At this time, it assigns an identification ID to the specifications and manages them so that they can be easily searched for later. The user sends a specification search request from the system interface. For example, they send a query such as "I want to know the specifications of the move_forward function." The terminal receives the query from the user and sends it to the server. The server searches the stored database for the relevant specifications and provides the results to the user.

[0501] Emotion-based feedback measures

[0502] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, the server will provide more detailed help information. The server also provides appropriate feedback according to the user's emotions. For example, if the user is satisfied, the server will display a positive message.

[0503] Specific examples

[0504] Below are some examples of specific prompt sentences.

[0505] Please generate a specification for the "move_forward" function, and provide specific instructions if users are confused.

[0506] As described above, this system allows users to automatically generate efficient and accurate specifications simply by uploading code files and screen information. Furthermore, the emotion engine provides appropriate feedback based on the user's emotional state, improving usability.

[0507] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0508] Step 1:

[0509] The user selects the code file and screen information through the system interface and performs the upload. For example, the user selects the source code file of a program that controls the operation of a factory robot and screenshots of the program's operation screen. This is the input. The terminal receives the selected files and sends the data to the server.

[0510] Step 2:

[0511] The server receives the code file and screen information sent from the device and temporarily stores it. It then analyzes the received code file and extracts code characteristics such as function names, arguments, and return values. For example, it analyzes the "distance" argument and return value "None" of the "move_forward" function. This is data processing, and the analyzed information is obtained as output.

[0512] Step 3:

[0513] The server analyzes the screen information. Specifically, it uses optical character recognition (OCR) technology to extract text and UI elements from the screenshot. For example, it extracts the "numeric input form" and the text of the "Execute" button contained in the screenshot. This is the input, and the extracted text information is obtained as the output.

[0514] Step 4:

[0515] The device uses an emotion engine to recognize the user's emotional state. For example, it captures the user's facial expressions with a camera and uses the EmotionRecognizer library to detect emotions such as confusion or satisfaction. Emotion data is the input, and the recognized emotional state is the output.

[0516] Step 5:

[0517] The server automatically generates specifications based on the results of analyzing the code and screen information. Specifically, it integrates the analysis results and formats them into a specification format. For example, it documents the operation flow based on the "move_forward" function specification and screen information. This is the input, and the automatically generated specification is obtained as the output.

[0518] Step 6:

[0519] The server stores the automatically generated specifications in a database. At this time, it assigns an ID to each specification and manages it so that it can be easily searched. The specifications are input, and the state of the specifications stored in the database is obtained as output.

[0520] Step 7:

[0521] The user sends a specification search request from the system interface. For example, they send a query such as "I want to know the specifications of the move_forward function." This is the input. The terminal receives the query from the user and sends it to the server.

[0522] Step 8:

[0523] The server searches for the relevant specifications from the stored database and provides the results to the user. For example, it searches for specifications related to the "move_forward function" and returns them to the user. The query is the input, and the search results are the output.

[0524] Step 9:

[0525] The server automatically adjusts the operation interface and provides feedback based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, detailed help information is displayed. The recognized emotional state is the input, and the adjusted interface is the output.

[0526] Through these steps, users can efficiently create program specifications for factory robots and receive real-time emotional feedback.

[0527] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0528] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0529] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0530] [Second embodiment]

[0531] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0532] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0533] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0534] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0535] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0536] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0537] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0538] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0539] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0540] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0541] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0542] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0543] The present invention relates to a system for automatically generating specifications by uploading code files and screen information to the system. A specific embodiment of this system will be described below.

[0544] The program used in this system analyzes code files and screen information through multiple steps to generate specifications. Three main parties are involved: the server, the terminal, and the user.

[0545] How to upload code and screen information

[0546] The user uses their own terminal through the system interface to select the code file and associated screen information (e.g., screenshots and URLs) and upload them to the system.

[0547] The terminal receives the file selected by the user and transmits it to the server.

[0548] Means of analyzing information

[0549] The server receives the code file and screen information sent from the terminal.

[0550] The server parses the received code file and extracts characteristics of the code, such as function names, arguments, and return values.

[0551] For example, if you receive the following code:

[0552] javascript

[0553] function add(a, b) {

[0554] return a + b;

[0555] }

[0556] The server analyzes information such as "function name: add," "arguments: a, b," and "return value: a + b."

[0557] The server also analyzes the screen information it receives separately. For example, it uses OCR technology to identify text and UI elements from screenshots and extracts characteristics such as "numeric input form" and "calculation button."

[0558] Automatic generation of specifications

[0559] The server automatically generates specifications based on the results of analyzing the code and screen information.

[0560] For example, the analysis results can be used to generate the following specification:

[0561] Code Specifications

[0562] Function name: add

[0563] Arguments: a, b

[0564] Returns: a + b

[0565] Related screens

[0566] URL: (URL goes here)

[0567] Description: A screen with a form for entering numbers and a "Calculate" button

[0568] Processing flow

[0569] 1. User enters a number into a form

[0570] 2. Press the "Calculate" button

[0571] 3. The add function is called and the result is displayed

[0572] Storage method for specifications

[0573] The server stores the automatically generated specifications in a database.

[0574] When saving, the specification is given an identifiable ID and managed so that it can be easily searched for later.

[0575] How to search for specifications

[0576] A user sends a specification search request from the system interface, for example, a query such as "I want to know the specification of the add function."

[0577] The terminal receives a query from a user and sends it to the server.

[0578] The server searches the stored database for the relevant specifications and provides the results to the user.

[0579] This system allows users to avoid code duplication and multiple implementations of similar functionality, improving development efficiency and maintainability. It also automates the specification creation process, saving significant time and effort. This also improves technical accuracy and the productivity of the entire development team.

[0580] The processing flow will be explained below.

[0581] Step 1:

[0582] The user selects the code file and related screen information (screenshots or URLs) through the system interface and performs the upload.

[0583] Step 2:

[0584] The terminal receives the code file and screen information selected by the user and transmits the data to the server.

[0585] Step 3:

[0586] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[0587] Step 4:

[0588] The server analyzes the received code file. Specifically, it identifies the function names, arguments, and return values ​​in the code. For example, it extracts the function "add" from the code, its arguments "a, b", and its return value "a + b".

[0589] Step 5:

[0590] The server analyzes the received screen information and extracts text and UI elements (e.g., input forms, buttons) from the screenshot using OCR technology.

[0591] Step 6:

[0592] The server automatically generates a specification based on the analysis results. Specifically, it integrates the code analysis results (function names, arguments, return values) and the screen information analysis results (UI elements) and formats them into a specification format.

[0593] Step 7:

[0594] The server stores the automatically generated specifications in a database, assigning an ID to each specification to make it easy to search for later.

[0595] Step 8:

[0596] A user submits a specification search request using the system's interface (e.g., a query such as "I want to know the specification of the add function").

[0597] Step 9:

[0598] The terminal transmits a search query from the user to the server.

[0599] Step 10:

[0600] The server searches the database for the relevant specifications, finds a match with the query, and finds the corresponding specifications.

[0601] Step 11:

[0602] The server provides the user with the relevant specifications as a search result.

[0603] Example 1

[0604] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] In existing software development environments, the manual analysis of code files and screen information and the creation of specifications requires a great deal of time and effort. Furthermore, manual analysis and documentation has a high risk of human error, leading to incomplete specifications and reduced development efficiency. To solve these problems, a system is needed that can automatically analyze code files and screen information and generate specifications quickly and accurately.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0607] In this invention, the server includes means for uploading a code file to an electronic computing device, means for uploading screen information to the electronic computing device, means for analyzing the uploaded code file to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating a specification based on the code file and the screen information, means for saving the automatically generated specification in a database, means for searching the saved specification, means for displaying progress and notifying the user, and means for applying the analysis results to a template to generate a specification. This significantly reduces the time and effort associated with manual analysis and document creation, and enables the generation of accurate and consistent specifications.

[0608] A "code file" is electronic data containing the source code of a program or script.

[0609] "Screen information" refers to data that includes information about the user interface, the layout of the screen, and its elements.

[0610] "Computing device" refers to a device that processes and stores data, such as a computer or server.

[0611] "Analysis" is the process of breaking down input data and extracting its components and characteristics.

[0612] "Characteristics" refers to specific attributes or functional characteristics extracted from code or screen information.

[0613] A "specification" is a document that describes in detail the operation, structure, and functions of a system or program.

[0614] A "database" is a management system for systematically storing large amounts of data and making them efficiently searchable and accessible.

[0615] "Search" refers to the process of locating specific information from stored data.

[0616] "Optical character recognition technology" is a technology that analyzes characters in an image and converts them into text data.

[0617] A "template" is a standard format for generating documents in a consistent format.

[0618] "Progress" is information that indicates the current stage of data analysis and processing.

[0619] The term "user" refers to a person who uses the system of the present invention.

[0620] This invention relates to a system that uploads code files and screen information to a system, automatically generates specifications, and stores and searches them. The following steps and techniques are used to realize this system.

[0621] Upload method:

[0622] A user uses his / her device to select a code file (e.g., add.js) and related screen information (e.g., form_screen.png) from the system interface and upload them to the system. The device receives these files and sends them to the server via an HTTP request. The user interface displays the file sending status to the user.

[0623] Information analysis methods:

[0624] The server receives the code file and screen information sent from the device and temporarily stores it. The server analyzes the received code file and extracts characteristics such as function names, arguments, and return values. For example, if the code is JavaScript code, the server identifies the function name "add," arguments "a, b," and return value "a + b." The server also uses OCR technology to analyze the screen information and identify text and UI elements. For example, it extracts characteristics such as a "number input form" and a "calculation button" from a screenshot.

[0625] Automated generation of specifications:

[0626] The server automatically generates a specification template based on the results of the code and screen analysis, applying these characteristics to the template. Specifically, it uses a template engine to incorporate the analysis results into a predetermined format and generates the following specification:

[0627] Code Specifications

[0628] Function name: add

[0629] Arguments: a, b

[0630] Returns: a + b

[0631] Related screens

[0632] URL: (URL goes here)

[0633] Description: A screen with a form for entering numbers and a "Calculate" button

[0634] Processing flow

[0635] 1. User enters a number into a form

[0636] 2. Press the "Calculate" button

[0637] 3. The add function is called and the result is displayed

[0638] How to save the specification:

[0639] The server saves the automatically generated specifications in a database. When saved, the specifications are given an identifiable ID and managed for easy retrieval later. The fact that the specifications have been saved is also recorded in a log.

[0640] How to search for specifications:

[0641] The user enters a search query into the system interface and sends a request to search for specifications. For example, the user might enter, "I want to know the specifications of the add function." The terminal sends this query to the server, which then searches the database for the relevant specifications. The search results are displayed to the user via the terminal.

[0642] Examples and prompts:

[0643] The user uploads a JavaScript file called "add.js" and a screenshot called "form_screen.png" to the system on their device.

[0644] The terminal transmits these files to the server and displays the transmission status.

[0645] The server analyzes "add.js" to extract function names, arguments, and return values. It also analyzes "form_screen.png" using OCR technology to identify UI elements.

[0646] The server generates a specification based on these analysis results, assigns it an identification ID of "add_function_spec_001," and stores it in the database.

[0647] Later, when the user sends a search request for "add function specification" from the system interface, the server searches the database for "add_function_spec_001" and provides the results to the user.

[0648] Example prompt sentence:

[0649] Upload the code file and screen information. For example, select the file add.js and the screenshot form_screen.png and upload them to the system. A specification document will be automatically generated based on this information.

[0650] This invention allows users to significantly reduce the time and effort associated with manual analysis and documentation, enabling the generation of accurate and consistent specifications. Furthermore, the system's search function allows users to quickly reference existing specifications, improving development efficiency and maintainability.

[0651] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0652] Step 1:

[0653] User selects and uploads code files and screen information

[0654] Input: Code file (e.g. add.js), screen information (e.g. form_screen.png)

[0655] Specific operation: The user opens the system interface and clicks the upload button. The user selects the file to upload from their device and clicks "Upload".

[0656] Output: The uploaded file is sent from the user's device to the server.

[0657] Step 2:

[0658] The device receives the file and sends it to the server

[0659] Input: User selected file and upload request

[0660] Specific operation: The device reads the file selected by the user from local storage, creates an HTTP POST request, and sends it to the server.

[0661] Output: The status of the HTTP request sent to the server is displayed on the terminal to inform the user of progress.

[0662] Step 3:

[0663] The server receives the code file and screen information.

[0664] Input: File data included in the HTTP request from the terminal

[0665] Specific operation: The server receives the HTTP request, temporarily stores the code file and screen information in storage, checks the integrity of the file, and if there are no problems, starts the analysis process.

[0666] Output: The file is saved and ready for analysis.

[0667] Step 4:

[0668] The server begins parsing the code file

[0669] Input: Saved code file

[0670] Specific operation: The server starts an analysis program to analyze the contents of the code file, extracting characteristics such as function names, arguments, and return values.

[0671] Output: Characteristic data of the extracted code (e.g., "Function name: add", "Arguments: a, b", "Return value: a + b")

[0672] Step 5:

[0673] The server analyzes the screen information

[0674] Input: Saved screen information

[0675] What it does: The server uses OCR technology to identify text and UI elements from the screenshot, and then categorizes the identified UI elements.

[0676] Output: Extracted screen information characteristics data (e.g. "Number input form", "Calculation button")

[0677] Step 6:

[0678] The server generates specifications based on the analysis results

[0679] Input: Code characteristic data, Screen information characteristic data

[0680] Specific operation: The server uses a template engine to incorporate the analysis results into a predetermined format and automatically generate a specification document.

[0681] Output: Automatically generated specification (e.g., "Function name: add", "Arguments: a, b", "Return value: a + b", "Related screen: Screen with a numeric input form and a calculation button")

[0682] Step 7:

[0683] The server stores the specifications in a database

[0684] Input: Auto-generated specification

[0685] Specific behavior: The server saves the specification in the database using an INSERT query and assigns it an identifiable ID.

[0686] Output: Notification of completion of saving and generation of identification ID (e.g. "Specification ID: add_function_spec_001")

[0687] Step 8:

[0688] User submits a search request for specifications

[0689] Input: Search query (e.g. "I want to know the specifications of the add function")

[0690] Specific actions: The user enters a query into the search box in the system interface and clicks the search button.

[0691] Output: A search request is sent to the server.

[0692] Step 9:

[0693] The server searches for the specification and returns the results

[0694] Input: A search request from the user

[0695] Specific operation: The server searches the database, extracts the relevant specifications, and returns the extracted specifications to the terminal.

[0696] Output: The search result, a specification document, is displayed to the user (e.g., detailed information for "add_function_spec_001").

[0697] (Application example 1)

[0698] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0699] When maintaining and repairing industrial equipment, there is a lack of a way to quickly and accurately generate specifications for related code and screen information. This challenge makes it difficult for on-site technicians to perform their work efficiently, increasing the likelihood of mistakes and rework. In particular, there is a need to address the need to quickly obtain information on-site using smart devices.

[0700] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0701] In this invention, the server includes a means for uploading a code file, a means for uploading screen information, and a means for automatically generating specifications based on the uploaded code file and screen information, thereby enabling specifications to be generated quickly and accurately for maintenance and repair of industrial equipment and displayed on a smart device.

[0702] - "Code File" means a file containing a program description, which contains the set of instructions necessary to run the program.

[0703] "Screen information" is visual information about the user interface, and includes data such as screenshots and URLs.

[0704] A "specification" is a technical document that is automatically generated based on code files and screen information, and contains detailed descriptions of the program's characteristics and screen characteristics.

[0705] "Optical character recognition technology" is a technology that extracts characters from an image and converts them into text data.

[0706] The "storage means" is a function for storing the generated specifications in a database or other storage device.

[0707] "Search facility" is a function that allows you to search saved specifications by keywords or other criteria to quickly access specific information.

[0708] The "uploading means" is a function that allows a user to use a terminal to send code files and screen information to a server.

[0709] "Industrial equipment" refers to machinery and equipment used in various industries, and is controlled using advanced technology.

[0710] "Maintenance and repair" refers to the work of inspecting and repairing industrial equipment to maintain its normal operation.

[0711] "Smart device" refers to a mobile terminal device that can connect to the Internet and use multi-function applications.

[0712] The system for implementing this invention mainly involves three entities: a server, a terminal, and a user. The server analyzes the code file and screen information uploaded by the user and generates a specification document based on that. The server then saves the specification document and displays it on the smart device for use in maintenance and repair.

[0713] Uploading code files and screen information

[0714] The user selects the code file and related screen information (screenshots or URLs) from the terminal and uploads them to the system. The terminal receives the file selected by the user and sends it to the server.

[0715] Analysis of information

[0716] The server receives the code file and screen information sent from the device. The server analyzes the received code file and extracts function names, arguments, return values, etc. It also uses OCR technology to identify text and UI elements in the screen information and extracts characteristics such as "numeric input forms" and "buttons."

[0717] Automatic generation of specifications

[0718] The server automatically generates specifications required for the maintenance and repair of industrial equipment based on the analyzed code and screen information. The generated specifications contain detailed descriptions of the code characteristics and user interface characteristics.

[0719] Save and search specifications

[0720] The generated specifications are stored in a database on the server and assigned an identifiable ID. Users can search for specifications through their smart devices and quickly obtain information for efficient maintenance and repair of industrial equipment.

[0721] Hardware and software used

[0722] The server is a computer system capable of high performance analysis, and is preferably a distributed system if necessary.

[0723] To use OCR technology, we use Tesseract (an open source OCR engine).

[0724] Code analysis uses Python's AST module and specific programming language analysis tools.

[0725] The database uses an RDBMS such as MySQL or PostgreSQL.

[0726] Examples and prompts

[0727] For example, you could use the following prompt as input to a generative AI model:

[0728] "I'd like to know more about how the move_forward(steps) function works.

[0729] The code file contains the following:

[0730] function move_forward(steps) {

[0731] for(int i = 0; i < steps; i++) {

[0732] move_one_step();

[0733] }

[0734] }

[0735] 2. The screenshot of the screen information includes the following elements:

[0736] "Start" button

[0737] An input field labeled "Steps"

[0738] "Enter the number of steps"

[0739] This prompt allows the server to analyze the code and screen information and automatically generate detailed specifications required for maintenance and repair of industrial equipment.

[0740] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0741] Step 1:

[0742] The user uploads the code file and screen information from the terminal. The user selects the code file and screen information (screenshots or URLs) to upload using the interface of the smart device. The terminal sends the files received from the user to the server.

[0743] Input: User selected code file and screen information

[0744] Output: File data sent to the server

[0745] Step 2:

[0746] The server receives the code file and screen information sent from the terminal and stores the received file in a temporary storage area.

[0747] Input: Code file and screen information sent from the terminal

[0748] Output: A file stored in the temporary storage area on the server

[0749] Step 3:

[0750] The server analyzes the code file, specifically using Python's AST module to parse the code and extract properties such as function names, arguments, and return values.

[0751] Input: Code file stored in temporary storage area

[0752] Output: Analysis results such as function name, arguments, and return value

[0753] Step 4:

[0754] The server analyzes the screen information, using OCR technology (Tesseract) to extract text and UI elements from the screenshot and identify screen characteristics.

[0755] Input: Screen information stored in temporary storage area

[0756] Output: Analysis results for UI elements (buttons, input fields, labels, etc.)

[0757] Step 5:

[0758] The server automatically generates specifications based on the analysis results, combining code characteristics and screen characteristics to generate detailed specifications suitable for the maintenance and repair of industrial equipment.

[0759] Input: Analyzed code characteristics and screen characteristics

[0760] Output: Generated specification

[0761] Step 6:

[0762] The server stores the generated specifications in a database, assigns them identifiable IDs, and manages them for easy retrieval later.

[0763] Input: Generated specification

[0764] Output: The specifications stored in the database and their IDs

[0765] Step 7:

[0766] A user searches for specifications using a smart device. The user inputs a query using the search interface and sends a search request to the server. The server searches for the relevant specifications from the database and provides them to the user.

[0767] Input: Search query from user

[0768] Output: Search results for matching specifications

[0769] This series of processing steps makes it possible to quickly and accurately generate specifications required for the maintenance and repair of industrial equipment and display them on a smart device.

[0770] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0771] This invention is a system that automatically generates specifications based on uploaded code files and screen information, and further combines it with an emotion engine that recognizes user emotions. This system enables efficient and accurate specification creation, and the addition of the emotion engine improves usability. Specific embodiments of this system are described below.

[0772] The program used in this system analyzes code files and screen information through multiple steps to generate specifications. Three main parties are involved: the server, the terminal, and the user, and an emotion engine is also integrated.

[0773] How to upload code and screen information

[0774] The user selects the code file and associated screen information (e.g., screenshots or URLs) through the system interface and performs the upload.

[0775] The terminal receives the files selected by the user and transmits the data to the server.

[0776] Means of analyzing information

[0777] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[0778] The server parses the received code file and extracts code characteristics such as function names, arguments, and return values.

[0779] For example, if you receive the following code:

[0780] javascript

[0781] function add(a, b) {

[0782] return a + b;

[0783] }

[0784] The server analyzes information such as "function name: add," "arguments: a, b," and "return value: a + b."

[0785] The server also analyzes the separately received screen information, using OCR technology to extract text and UI elements (e.g., input forms, buttons) from the screenshot.

[0786] Emotion engine integration

[0787] The device uses an emotion engine to analyze the user's facial expressions and voice and recognize the user's emotional state.

[0788] For example, it detects whether a user is confused or satisfied from their facial expression or tone of voice.

[0789] Automatic generation of specifications

[0790] The server automatically generates specifications based on the results of code and screen analysis. Specifically, it integrates the results of code analysis (function names, arguments, return values) and screen analysis (UI elements) and formats them into a specification format.

[0791] For example, it generates the following specification:

[0792] Code Specifications

[0793] Function name: add

[0794] Arguments: a, b

[0795] Returns: a + b

[0796] Related screens

[0797] URL: (URL goes here)

[0798] Description: A screen with a form for entering numbers and a "Calculate" button

[0799] Processing flow

[0800] 1. User enters a number into a form

[0801] 2. Press the "Calculate" button

[0802] 3. The add function is called and the result is displayed

[0803] Storage method for specifications

[0804] The server stores the automatically generated specifications in a database, assigning an ID to each specification to make it easier to search for later.

[0805] How to search for specifications

[0806] A user sends a specification search request from the system interface, for example, a query such as "I want to know the specification of the add function."

[0807] The terminal receives a query from a user and sends it to the server.

[0808] The server searches the stored database for the relevant specifications and provides the results to the user.

[0809] Emotion-based feedback measures

[0810] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine.

[0811] For example, if the user is confused, more detailed help information can be provided.

[0812] The server provides appropriate feedback depending on the user's emotions, such as displaying a positive message if the user is satisfied.

[0813] This system allows users to avoid code duplication and separate implementations of similar functions, improving development efficiency and maintainability. Furthermore, the integration of an emotion engine improves the user experience and makes the system easier to use. Appropriate feedback based on emotions and adjustments to the operation interface allow users to use the system more comfortably.

[0814] The processing flow will be explained below.

[0815] Step 1:

[0816] The user selects the code file and associated screen information (e.g., screenshots or URLs) through the system interface and performs the upload.

[0817] Step 2:

[0818] The terminal receives the code file and screen information selected by the user and transmits the data to the server.

[0819] Step 3:

[0820] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[0821] Step 4:

[0822] The server analyzes the received code file to identify the function name, arguments, and return value of the code. For example, if the server receives the following code:

[0823] javascript

[0824] function add(a, b) {

[0825] return a + b;

[0826] }

[0827] The server extracts "function name: add", "arguments: a, b", and "return value: a + b" from this code.

[0828] Step 5:

[0829] The server analyzes the received screen information. It uses OCR technology to identify text and UI elements from the screenshot and extract their characteristics. For example, it detects "numerical input forms" and "calculation buttons."

[0830] Step 6:

[0831] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize the user's emotional state, for example, detecting whether the user is confused or satisfied from their facial expressions and tone of voice.

[0832] Step 7:

[0833] The server automatically generates a specification based on the analysis results. Specifically, it combines the code analysis results (function names, arguments, return values) with the screen information analysis results (UI elements) and formats them into the following specification format:

[0834] Code Specifications

[0835] Function name: add

[0836] Arguments: a, b

[0837] Returns: a + b

[0838] Related screens

[0839] URL: (URL goes here)

[0840] Description: A screen with a form for entering numbers and a "Calculate" button

[0841] Processing flow

[0842] 1. User enters a number into a form

[0843] 2. Press the "Calculate" button

[0844] 3. The add function is called and the result is displayed

[0845] Step 8:

[0846] The server saves the automatically generated specifications in a database. When saving, it assigns an ID to each specification and manages it so that it can be easily searched for later.

[0847] Step 9:

[0848] A user uses the system interface to submit a specification search request (e.g., a query such as "I want to know the specification of the add function").

[0849] Step 10:

[0850] The terminal receives a query from a user and sends it to the server.

[0851] Step 11:

[0852] The server searches the database for specifications that match the query and finds the relevant specifications.

[0853] Step 12:

[0854] The server provides the user with the relevant specifications as a search result.

[0855] Step 13:

[0856] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine, for example, displaying detailed help information if the user is confused.

[0857] Step 14:

[0858] The server provides appropriate feedback depending on the user's emotions, for example, if the user is satisfied, it displays a positive message.

[0859] In this way, the system prevents code duplication and separate implementations of similar functions, improving development efficiency and maintainability, while also improving the user experience by adding an emotion engine.

[0860] Example 2

[0861] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0862] Conventional specification creation systems require manual analysis of code files and screen information to create specifications, which is extremely time-consuming, inefficient, and prone to human error. Furthermore, the system lacks usability because it does not reflect the user's emotional state. Therefore, there is a need for a system that can solve these issues, create specifications efficiently and accurately, and adjust the interface according to the user's emotional state.

[0863] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0864] In this invention, the server includes means for a user to upload a code file, means for uploading screen information, means for analyzing the uploaded code file to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating a specification based on the code file and the screen information, means for saving the automatically generated specification, means for searching the saved specification, means for analyzing the user's facial expression and voice to recognize the user's emotional state, and means for automatically adjusting the operation interface based on the recognized emotional state. This enables efficient analysis of the code and screen information and automatic generation of the specification, and further improves usability by adjusting the interface taking the user's emotional state into consideration.

[0865] A "user" is an entity that uses the system to upload code files and screen information, and automatically generate and search specifications.

[0866] A "code file" is a file in which the source code of a program is written, and which has properties such as function names, arguments, and return values.

[0867] "Screen information" is data that indicates the state and structure of the user interface, such as a screenshot of the screen or a URL.

[0868] "Uploading means" refers to an interface that allows a user to send code files and screen information to the system, and a function that performs the processing.

[0869] "Analysis means" refers to a function that analyzes the received code file and screen information and extracts their characteristics.

[0870] A "specification" is a document generated based on the analysis results, which describes in detail the operation and structure of the system.

[0871] "Automatic generation means" refers to a function that automatically creates specifications based on the analysis results.

[0872] "Storage means" refers to a function that stores automatically generated specifications in a database or the like, allowing them to be searched for later.

[0873] "Search means" refers to a function that searches saved specifications based on a query and provides the results to the user.

[0874] The "emotional state" is a psychological state analyzed from the user's facial expression and voice, and includes confusion, satisfaction, anger, and the like.

[0875] "Emotion analysis means" refers to a function that captures the user's facial expressions and voice, analyzes them, and recognizes their emotional state.

[0876] The "operation interface adjusting means" refers to a function that optimizes the interface operated by the user based on the recognized emotional state.

[0877] This invention is a system that allows users to upload code files and screen information, and automatically generates specifications based on that data. The system of this invention involves the server, terminal, and user, and further improves usability by integrating an emotion engine.

[0878] How to upload code and screen information

[0879] The user selects and uploads the code file and screen information (e.g., screenshots or URLs) through the system interface. The device receives the file selected by the user and sends this data to the server. Specifically, the device checks the file format and size, and if there are no problems, transfers the file to the server.

[0880] Means of analyzing information

[0881] The server receives the code file and screen information sent from the device and temporarily stores it. Next, the server analyzes the received code file. The specific software used here is a parser (e.g., ANTLR) that extracts code characteristics such as function names, arguments, and return values. For example, if the following code is received:

[0882] javascript

[0883] function add(a, b) {

[0884] return a + b;

[0885] }

[0886] The server extracts information such as "function name: add", "arguments: a, b", and "return value: a + b".

[0887] Furthermore, the server uses OCR technology (e.g., Tesseract OCR) to extract text and UI elements (e.g., input forms, buttons) from the screen information (screenshots).

[0888] Emotion engine integration

[0889] The device uses a built-in camera and microphone to capture the user's facial expressions and voice in real time. The device then uses emotion analysis software (e.g., Microsoft Azure Emotion API) to analyze the user's emotional state. For example, it can detect whether the user is confused or happy from their facial expressions and tone of voice.

[0890] Automatic generation of specifications

[0891] The server automatically generates a specification based on the results of code and screen analysis. Specifically, it combines the results of code analysis (function names, arguments, return values) with the results of screen analysis (UI elements) and formats them into a specification. For example, the following specification is generated:

[0892] markdown

[0893] Code Specifications

[0894] Function name: add

[0895] Arguments: a, b

[0896] Returns: a + b

[0897] Related screens

[0898] URL: (URL goes here)

[0899] Description: A screen with a form for entering numbers and a "Calculate" button

[0900] Processing flow

[0901] 1. User enters a number into a form

[0902] 2. Press the "Calculate" button

[0903] 3. The add function is called and the result is displayed

[0904] Storage and retrieval of specifications

[0905] The server saves the automatically generated specifications in a database. When saved, the specifications are assigned an identification ID, which is managed so that they can be easily searched for later. When the user sends a specification search request from the interface, the terminal receives this and sends it to the server. The server searches the saved database for the corresponding specifications and provides the results to the user.

[0906] Emotion-based feedback measures

[0907] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, it provides more detailed help information. If the user is satisfied, it provides feedback by displaying a positive message.

[0908] Examples and Prompts

[0909] A user logs into the system and uploads the project's code files and screenshots. The device checks them and sends them to the server. The server analyzes the code and extracts function names, arguments, return values, etc. It also uses OCR technology to analyze text and UI elements from screenshots. Based on these results, a specification document is automatically generated and stored in a database.

[0910] Example prompt sentence:

[0911] In this system, I uploaded the following JavaScript code file: function multiply(x, y) { return xy;} Please auto-generate the specification.

[0912] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0913] Step 1:

[0914] Through the system interface, the user clicks the upload button, selects the code file and screen information (screenshots or URLs), and uploads them.

[0915] Input: Code file, screen information

[0916] Behavior: The user selects a file through a file selection dialog and initiates the upload.

[0917] Output: The device receives the selected file.

[0918] Step 2:

[0919] The device checks the code file and screen information received from the user to ensure there are no problems with the format or size, and then sends this data to the server.

[0920] Input: Code file, screen information

[0921] Action: The device checks the file format and whether it exceeds the maximum capacity. If there are no problems, it transfers the file to the server.

[0922] Output: The server receives the file and stores it temporarily.

[0923] Step 3:

[0924] The server analyzes the received code file and extracts code characteristics such as function names, arguments, return values, etc. Specifically, it analyzes the code using a parser (e.g., ANTLR).

[0925] Input: Code file

[0926] How it works: The server parses the code through a parser and extracts the function name, arguments, and return value. For example, if you parse the code function add(a, b) { return a + b;}, it extracts "function name: add", "arguments: a, b", and "return value: a + b".

[0927] Output: Code characteristics information (function names, arguments, return values, etc.)

[0928] Step 4:

[0929] The server simultaneously analyzes the received screen information and uses OCR technology (e.g., Tesseract OCR) to extract text and UI elements from the screenshot.

[0930] Input: Screen information (screenshots, URL, etc.)

[0931] How it works: The server uses OCR technology to analyze the screenshot and extract text and UI elements (input forms, buttons, etc.).

[0932] Output: Screen characteristics (text, UI elements, etc.)

[0933] Step 5:

[0934] The server integrates the code characteristics information and the screen characteristics information to automatically generate a specification document, which includes the function name, arguments, return values, and related screen information.

[0935] Input: Code characteristics information, screen characteristics information

[0936] How it works: The server combines this information and formats it into a specification. For example, it puts function names and argument information in the code section, and screen information in a separate section.

[0937] Output: Auto-generated specification

[0938] Step 6:

[0939] The server stores the automatically generated specifications in a database, assigns them an ID, and manages them for easy retrieval later.

[0940] Input: Auto-generated specification

[0941] Operation: The server assigns an ID to the specification and stores it in the database.

[0942] Output: Specification saved in database

[0943] Step 7:

[0944] A user sends a specification search request through the system interface, for example, by entering a query such as "I want to know the specification of the add function."

[0945] Input: Search query (e.g. "add function specification")

[0946] How it works: A user enters a query into the search box and clicks the search button. The device receives the query and sends it to the server.

[0947] Output: The server receives the search request.

[0948] Step 8:

[0949] The server searches the database for the relevant specifications based on the search request and provides the results to the user.

[0950] Input: Search request

[0951] How it works: The server searches the database for the query, finds the relevant specifications, and returns the search results to the device.

[0952] Output: Search results (relevant specifications)

[0953] Step 9:

[0954] The terminal displays the search results received from the server to the user.

[0955] Input: Search results from the server

[0956] Operation: The device receives the search results and displays them on the user's interface.

[0957] Output: Specification information displayed in the interface

[0958] Step 10:

[0959] The device uses an emotion engine to capture the user's facial expressions and voice and recognize their emotional state, for example, detecting whether the user is confused or happy from their facial expressions or tone of voice.

[0960] Input: User's facial expression, voice

[0961] How it works: The device captures the user's facial expressions and voice through the camera and microphone, and performs emotion analysis.

[0962] Output: User's emotional state (confused, satisfied, etc.)

[0963] Step 11:

[0964] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, it provides detailed help information. If the user is satisfied, it displays a positive message.

[0965] Input: User's emotional state

[0966] Action: The server makes appropriate user interface adjustments and feedback based on the recognized emotional state.

[0967] Output: Adjusting the interface, providing feedback (e.g., detailed help information, positive messages)

[0968] (Application example 2)

[0969] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0970] Currently, programming factory robots and creating operation specifications is done manually using complex tools, which requires a great deal of time and effort. In addition, users can become confused when operating the robots, hindering efficient work. Furthermore, real-time feedback on the inconvenience or confusion experienced by users is not provided, creating a need for improved usability.

[0971] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0972] In this invention, the server includes means for uploading code files, means for uploading screen information, means for analyzing the uploaded code files to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating specifications based on the code files and the screen information, means for saving the automatically generated specifications, means for searching the saved specifications, means for analyzing emotional information to recognize the emotional state of a user, and means for adjusting the operation interface and providing feedback based on the emotional information. This allows users to significantly reduce the time and effort required to program factory robots, and provides real-time feedback according to emotions, improving usability and enabling more efficient and comfortable work.

[0973] A "code file" is a file in which the source code of a program is written.

[0974] "Screen information" refers to data that includes visual information about the user interface, such as screenshots and URLs.

[0975] "Analysis" is the process used to extract characteristics and structure from code files and screen information.

[0976] A "specification" is a document that describes in detail the functions and operations of software or a system.

[0977] "Emotion information" refers to data on emotions detected from the user's facial expressions, voice, etc.

[0978] An "operation interface" refers to the screen or input means that a user uses to operate a system or application.

[0979] "Feedback" refers to the system providing real-time responses to the user's actions and state.

[0980] A "server" is a computer that provides data and services to other terminals over a network.

[0981] "Optical character recognition technology" is a technology that extracts character information from an image.

[0982] MODE FOR CARRYING OUT THE INVENTION

[0983] This invention is a system that automatically generates specifications based on uploaded code files and screen information, and also combines it with an emotion engine that recognizes user emotions. This system enables efficient and accurate specification creation, and the addition of the emotion engine improves usability.

[0984] System configuration and functions

[0985] This system consists of three components: a server, a terminal, and a user. Furthermore, by integrating an emotion engine, it has been added with the function of providing appropriate feedback based on the user's emotional state.

[0986] Uploading code and screen information

[0987] The user selects the code file and related screen information (e.g., screenshots or URLs) through the system interface and performs the upload. The terminal receives the files selected by the user and sends the data to the server. This part can be a general computer or smart device.

[0988] Analysis of information

[0989] The server receives the code files and screen information sent from the device and temporarily stores them. The server then analyzes the received code files and extracts code characteristics such as function names, arguments, and return values. It also analyzes the separately received screen information. Optical character recognition (OCR) is used to extract text and UI elements (e.g., input forms, buttons) from the screenshots. This is done using software libraries such as pytesseract and OpenCV.

[0990] Emotion engine integration

[0991] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize the user's emotional state. Specifically, a library called EmotionRecognizer is used. For example, it can detect whether the user is confused or happy from their facial expressions and tone of voice.

[0992] Automatic generation of specifications

[0993] The server automatically generates specifications based on the results of code and screen information analysis. Specifically, it integrates the results of code analysis (function name, arguments, return value) and screen information analysis (UI elements) and formats them into a specification format. An example of the generated specification is shown below.

[0994] Function name: move_forward

[0995] Argument: distance

[0996] Returns: None

[0997] Function Name: rotate

[0998] Argument: angle

[0999] Returns: None

[1000] Related screen: A screen with a form for entering numbers and an "Execute" button

[1001] Process flow:

[1002] 1. User enters a number into a form

[1003] 2. Press the "Run" button

[1004] 3. The functions move_forward and rotate are called and the results are displayed.

[1005] Save and search specifications

[1006] The server stores the automatically generated specifications in a database. At this time, it assigns an identification ID to the specifications and manages them so that they can be easily searched for later. The user sends a specification search request from the system interface. For example, they send a query such as "I want to know the specifications of the move_forward function." The terminal receives the query from the user and sends it to the server. The server searches the stored database for the relevant specifications and provides the results to the user.

[1007] Emotion-based feedback measures

[1008] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, the server will provide more detailed help information. The server also provides appropriate feedback according to the user's emotions. For example, if the user is satisfied, the server will display a positive message.

[1009] Specific examples

[1010] Below are some examples of specific prompt sentences.

[1011] Please generate a specification for the "move_forward" function, and provide specific instructions if users are confused.

[1012] As described above, this system allows users to automatically generate efficient and accurate specifications simply by uploading code files and screen information. Furthermore, the emotion engine provides appropriate feedback based on the user's emotional state, improving usability.

[1013] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1014] Step 1:

[1015] The user selects the code file and screen information through the system interface and performs the upload. For example, the user selects the source code file of a program that controls the operation of a factory robot and screenshots of the program's operation screen. This is the input. The terminal receives the selected files and sends the data to the server.

[1016] Step 2:

[1017] The server receives the code file and screen information sent from the device and temporarily stores it. It then analyzes the received code file and extracts code characteristics such as function names, arguments, and return values. For example, it analyzes the "distance" argument and return value "None" of the "move_forward" function. This is data processing, and the analyzed information is obtained as output.

[1018] Step 3:

[1019] The server analyzes the screen information. Specifically, it uses optical character recognition (OCR) technology to extract text and UI elements from the screenshot. For example, it extracts the "numeric input form" and the text of the "Execute" button contained in the screenshot. This is the input, and the extracted text information is obtained as the output.

[1020] Step 4:

[1021] The device uses an emotion engine to recognize the user's emotional state. For example, it captures the user's facial expressions with a camera and uses the EmotionRecognizer library to detect emotions such as confusion or satisfaction. Emotion data is the input, and the recognized emotional state is the output.

[1022] Step 5:

[1023] The server automatically generates specifications based on the results of analyzing the code and screen information. Specifically, it integrates the analysis results and formats them into a specification format. For example, it documents the operation flow based on the "move_forward" function specification and screen information. This is the input, and the automatically generated specification is obtained as the output.

[1024] Step 6:

[1025] The server stores the automatically generated specifications in a database. At this time, it assigns an ID to each specification and manages it so that it can be easily searched. The specifications are input, and the state of the specifications stored in the database is obtained as output.

[1026] Step 7:

[1027] The user sends a specification search request from the system interface. For example, they send a query such as "I want to know the specifications of the move_forward function." This is the input. The terminal receives the query from the user and sends it to the server.

[1028] Step 8:

[1029] The server searches for the relevant specifications from the stored database and provides the results to the user. For example, it searches for specifications related to the "move_forward function" and returns them to the user. The query is the input, and the search results are the output.

[1030] Step 9:

[1031] The server automatically adjusts the operation interface and provides feedback based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, detailed help information is displayed. The recognized emotional state is the input, and the adjusted interface is the output.

[1032] Through these steps, users can efficiently create program specifications for factory robots and receive real-time emotional feedback.

[1033] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1034] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1035] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1036] [Third embodiment]

[1037] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1038] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1039] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1040] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1041] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1042] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1043] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1044] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1045] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1046] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1047] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1048] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1049] The present invention relates to a system for automatically generating specifications by uploading code files and screen information to the system. A specific embodiment of this system will be described below.

[1050] The program used in this system analyzes code files and screen information through multiple steps to generate specifications. Three main parties are involved: the server, the terminal, and the user.

[1051] How to upload code and screen information

[1052] The user uses their own terminal through the system interface to select the code file and associated screen information (e.g., screenshots and URLs) and upload them to the system.

[1053] The terminal receives the file selected by the user and transmits it to the server.

[1054] Means of analyzing information

[1055] The server receives the code file and screen information sent from the terminal.

[1056] The server parses the received code file and extracts characteristics of the code, such as function names, arguments, and return values.

[1057] For example, if you receive the following code:

[1058] javascript

[1059] function add(a, b) {

[1060] return a + b;

[1061] }

[1062] The server analyzes information such as "function name: add," "arguments: a, b," and "return value: a + b."

[1063] The server also analyzes the screen information it receives separately. For example, it uses OCR technology to identify text and UI elements from screenshots and extracts characteristics such as "numeric input form" and "calculation button."

[1064] Automatic generation of specifications

[1065] The server automatically generates specifications based on the results of analyzing the code and screen information.

[1066] For example, the analysis results can be used to generate the following specification:

[1067] Code Specifications

[1068] Function name: add

[1069] Arguments: a, b

[1070] Returns: a + b

[1071] Related screens

[1072] URL: (URL goes here)

[1073] Description: A screen with a form for entering numbers and a "Calculate" button

[1074] Processing flow

[1075] 1. User enters a number into a form

[1076] 2. Press the "Calculate" button

[1077] 3. The add function is called and the result is displayed

[1078] Storage method for specifications

[1079] The server stores the automatically generated specifications in a database.

[1080] When saving, the specification is given an identifiable ID and managed so that it can be easily searched for later.

[1081] How to search for specifications

[1082] A user sends a specification search request from the system interface, for example, a query such as "I want to know the specification of the add function."

[1083] The terminal receives a query from a user and sends it to the server.

[1084] The server searches the stored database for the relevant specifications and provides the results to the user.

[1085] This system allows users to avoid code duplication and multiple implementations of similar functionality, improving development efficiency and maintainability. It also automates the specification creation process, saving significant time and effort. This also improves technical accuracy and the productivity of the entire development team.

[1086] The processing flow will be explained below.

[1087] Step 1:

[1088] The user selects the code file and related screen information (screenshots or URLs) through the system interface and performs the upload.

[1089] Step 2:

[1090] The terminal receives the code file and screen information selected by the user and transmits the data to the server.

[1091] Step 3:

[1092] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[1093] Step 4:

[1094] The server analyzes the received code file. Specifically, it identifies the function names, arguments, and return values ​​in the code. For example, it extracts the function "add" from the code, its arguments "a, b", and its return value "a + b".

[1095] Step 5:

[1096] The server analyzes the received screen information and extracts text and UI elements (e.g., input forms, buttons) from the screenshot using OCR technology.

[1097] Step 6:

[1098] The server automatically generates a specification based on the analysis results. Specifically, it integrates the code analysis results (function names, arguments, return values) and the screen information analysis results (UI elements) and formats them into a specification format.

[1099] Step 7:

[1100] The server stores the automatically generated specifications in a database, assigning an ID to each specification to make it easy to search for later.

[1101] Step 8:

[1102] A user submits a specification search request using the system's interface (e.g., a query such as "I want to know the specification of the add function").

[1103] Step 9:

[1104] The terminal transmits a search query from the user to the server.

[1105] Step 10:

[1106] The server searches the database for the relevant specifications, finds a match with the query, and finds the corresponding specifications.

[1107] Step 11:

[1108] The server provides the user with the relevant specifications as a search result.

[1109] Example 1

[1110] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1111] In existing software development environments, the manual analysis of code files and screen information and the creation of specifications requires a great deal of time and effort. Furthermore, manual analysis and documentation has a high risk of human error, leading to incomplete specifications and reduced development efficiency. To solve these problems, a system is needed that can automatically analyze code files and screen information and generate specifications quickly and accurately.

[1112] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1113] In this invention, the server includes means for uploading a code file to an electronic computing device, means for uploading screen information to the electronic computing device, means for analyzing the uploaded code file to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating a specification based on the code file and the screen information, means for saving the automatically generated specification in a database, means for searching the saved specification, means for displaying progress and notifying the user, and means for applying the analysis results to a template to generate a specification. This significantly reduces the time and effort associated with manual analysis and document creation, and enables the generation of accurate and consistent specifications.

[1114] A "code file" is electronic data containing the source code of a program or script.

[1115] "Screen information" refers to data that includes information about the user interface, the layout of the screen, and its elements.

[1116] "Computing device" refers to a device that processes and stores data, such as a computer or server.

[1117] "Analysis" is the process of breaking down input data and extracting its components and characteristics.

[1118] "Characteristics" refers to specific attributes or functional characteristics extracted from code or screen information.

[1119] A "specification" is a document that describes in detail the operation, structure, and functions of a system or program.

[1120] A "database" is a management system for systematically storing large amounts of data and making them efficiently searchable and accessible.

[1121] "Search" refers to the process of locating specific information from stored data.

[1122] "Optical character recognition technology" is a technology that analyzes characters in an image and converts them into text data.

[1123] A "template" is a standard format for generating documents in a consistent format.

[1124] "Progress" is information that indicates the current stage of data analysis and processing.

[1125] The term "user" refers to a person who uses the system of the present invention.

[1126] This invention relates to a system that uploads code files and screen information to a system, automatically generates specifications, and stores and searches them. The following steps and techniques are used to realize this system.

[1127] Upload method:

[1128] A user uses his / her device to select a code file (e.g., add.js) and related screen information (e.g., form_screen.png) from the system interface and upload them to the system. The device receives these files and sends them to the server via an HTTP request. The user interface displays the file sending status to the user.

[1129] Information analysis methods:

[1130] The server receives the code file and screen information sent from the device and temporarily stores it. The server analyzes the received code file and extracts characteristics such as function names, arguments, and return values. For example, if the code is JavaScript code, the server identifies the function name "add," arguments "a, b," and return value "a + b." The server also uses OCR technology to analyze the screen information and identify text and UI elements. For example, it extracts characteristics such as a "number input form" and a "calculation button" from a screenshot.

[1131] Automated generation of specifications:

[1132] The server automatically generates a specification template based on the results of the code and screen analysis, applying these characteristics to the template. Specifically, it uses a template engine to incorporate the analysis results into a predetermined format and generates the following specification:

[1133] Code Specifications

[1134] Function name: add

[1135] Arguments: a, b

[1136] Returns: a + b

[1137] Related screens

[1138] URL: (URL goes here)

[1139] Description: A screen with a form for entering numbers and a "Calculate" button

[1140] Processing flow

[1141] 1. User enters a number into a form

[1142] 2. Press the "Calculate" button

[1143] 3. The add function is called and the result is displayed

[1144] How to save the specification:

[1145] The server saves the automatically generated specifications in a database. When saved, the specifications are given an identifiable ID and managed for easy retrieval later. The fact that the specifications have been saved is also recorded in a log.

[1146] How to search for specifications:

[1147] The user enters a search query into the system interface and sends a request to search for specifications. For example, the user might enter, "I want to know the specifications of the add function." The terminal sends this query to the server, which then searches the database for the relevant specifications. The search results are displayed to the user via the terminal.

[1148] Examples and prompts:

[1149] The user uploads a JavaScript file called "add.js" and a screenshot called "form_screen.png" to the system on their device.

[1150] The terminal transmits these files to the server and displays the transmission status.

[1151] The server analyzes "add.js" to extract function names, arguments, and return values. It also analyzes "form_screen.png" using OCR technology to identify UI elements.

[1152] The server generates a specification based on these analysis results, assigns it an identification ID of "add_function_spec_001," and stores it in the database.

[1153] Later, when the user sends a search request for "add function specification" from the system interface, the server searches the database for "add_function_spec_001" and provides the results to the user.

[1154] Example prompt sentence:

[1155] Upload the code file and screen information. For example, select the file add.js and the screenshot form_screen.png and upload them to the system. A specification document will be automatically generated based on this information.

[1156] This invention allows users to significantly reduce the time and effort associated with manual analysis and documentation, enabling the generation of accurate and consistent specifications. Furthermore, the system's search function allows users to quickly reference existing specifications, improving development efficiency and maintainability.

[1157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1158] Step 1:

[1159] User selects and uploads code files and screen information

[1160] Input: Code file (e.g. add.js), screen information (e.g. form_screen.png)

[1161] Specific operation: The user opens the system interface and clicks the upload button. The user selects the file to upload from their device and clicks "Upload".

[1162] Output: The uploaded file is sent from the user's device to the server.

[1163] Step 2:

[1164] The device receives the file and sends it to the server

[1165] Input: User selected file and upload request

[1166] Specific operation: The device reads the file selected by the user from local storage, creates an HTTP POST request, and sends it to the server.

[1167] Output: The status of the HTTP request sent to the server is displayed on the terminal to inform the user of progress.

[1168] Step 3:

[1169] The server receives the code file and screen information.

[1170] Input: File data included in the HTTP request from the terminal

[1171] Specific operation: The server receives the HTTP request, temporarily stores the code file and screen information in storage, checks the integrity of the file, and if there are no problems, starts the analysis process.

[1172] Output: The file is saved and ready for analysis.

[1173] Step 4:

[1174] The server begins parsing the code file

[1175] Input: Saved code file

[1176] Specific operation: The server starts an analysis program to analyze the contents of the code file, extracting characteristics such as function names, arguments, and return values.

[1177] Output: Characteristic data of the extracted code (e.g., "Function name: add", "Arguments: a, b", "Return value: a + b")

[1178] Step 5:

[1179] The server analyzes the screen information

[1180] Input: Saved screen information

[1181] What it does: The server uses OCR technology to identify text and UI elements from the screenshot, and then categorizes the identified UI elements.

[1182] Output: Extracted screen information characteristics data (e.g. "Number input form", "Calculation button")

[1183] Step 6:

[1184] The server generates specifications based on the analysis results

[1185] Input: Code characteristic data, Screen information characteristic data

[1186] Specific operation: The server uses a template engine to incorporate the analysis results into a predetermined format and automatically generate a specification document.

[1187] Output: Automatically generated specification (e.g., "Function name: add", "Arguments: a, b", "Return value: a + b", "Related screen: Screen with a numeric input form and a calculation button")

[1188] Step 7:

[1189] The server stores the specifications in a database

[1190] Input: Auto-generated specification

[1191] Specific behavior: The server saves the specification in the database using an INSERT query and assigns it an identifiable ID.

[1192] Output: Notification of completion of saving and generation of identification ID (e.g. "Specification ID: add_function_spec_001")

[1193] Step 8:

[1194] User submits a search request for specifications

[1195] Input: Search query (e.g. "I want to know the specifications of the add function")

[1196] Specific actions: The user enters a query into the search box in the system interface and clicks the search button.

[1197] Output: A search request is sent to the server.

[1198] Step 9:

[1199] The server searches for the specification and returns the results

[1200] Input: A search request from the user

[1201] Specific operation: The server searches the database, extracts the relevant specifications, and returns the extracted specifications to the terminal.

[1202] Output: The search result, a specification document, is displayed to the user (e.g., detailed information for "add_function_spec_001").

[1203] (Application example 1)

[1204] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1205] When maintaining and repairing industrial equipment, there is a lack of a way to quickly and accurately generate specifications for related code and screen information. This challenge makes it difficult for on-site technicians to perform their work efficiently, increasing the likelihood of mistakes and rework. In particular, there is a need to address the need to quickly obtain information on-site using smart devices.

[1206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1207] In this invention, the server includes a means for uploading a code file, a means for uploading screen information, and a means for automatically generating specifications based on the uploaded code file and screen information, thereby enabling specifications to be generated quickly and accurately for maintenance and repair of industrial equipment and displayed on a smart device.

[1208] - "Code File" means a file containing a program description, which contains the set of instructions necessary to run the program.

[1209] "Screen information" is visual information about the user interface, and includes data such as screenshots and URLs.

[1210] A "specification" is a technical document that is automatically generated based on code files and screen information, and contains detailed descriptions of the program's characteristics and screen characteristics.

[1211] "Optical character recognition technology" is a technology that extracts characters from an image and converts them into text data.

[1212] The "storage means" is a function for storing the generated specifications in a database or other storage device.

[1213] "Search facility" is a function that allows you to search saved specifications by keywords or other criteria to quickly access specific information.

[1214] The "uploading means" is a function that allows a user to use a terminal to send code files and screen information to a server.

[1215] "Industrial equipment" refers to machinery and equipment used in various industries, and is controlled using advanced technology.

[1216] "Maintenance and repair" refers to the work of inspecting and repairing industrial equipment to maintain its normal operation.

[1217] "Smart device" refers to a mobile terminal device that can connect to the Internet and use multi-function applications.

[1218] The system for implementing this invention mainly involves three entities: a server, a terminal, and a user. The server analyzes the code file and screen information uploaded by the user and generates a specification document based on that. The server then saves the specification document and displays it on the smart device for use in maintenance and repair.

[1219] Uploading code files and screen information

[1220] The user selects the code file and related screen information (screenshots or URLs) from the terminal and uploads them to the system. The terminal receives the file selected by the user and sends it to the server.

[1221] Analysis of information

[1222] The server receives the code file and screen information sent from the device. The server analyzes the received code file and extracts function names, arguments, return values, etc. It also uses OCR technology to identify text and UI elements in the screen information and extracts characteristics such as "numeric input forms" and "buttons."

[1223] Automatic generation of specifications

[1224] The server automatically generates specifications required for the maintenance and repair of industrial equipment based on the analyzed code and screen information. The generated specifications contain detailed descriptions of the code characteristics and user interface characteristics.

[1225] Save and search specifications

[1226] The generated specifications are stored in a database on the server and assigned an identifiable ID. Users can search for specifications through their smart devices and quickly obtain information for efficient maintenance and repair of industrial equipment.

[1227] Hardware and software used

[1228] The server is a computer system capable of high performance analysis, and is preferably a distributed system if necessary.

[1229] To use OCR technology, we use Tesseract (an open source OCR engine).

[1230] Code analysis uses Python's AST module and specific programming language analysis tools.

[1231] The database uses an RDBMS such as MySQL or PostgreSQL.

[1232] Examples and prompts

[1233] For example, you could use the following prompt as input to a generative AI model:

[1234] "I'd like to know more about how the move_forward(steps) function works.

[1235] The code file contains the following:

[1236] function move_forward(steps) {

[1237] for(int i = 0; i < steps; i++) {

[1238] move_one_step();

[1239] }

[1240] }

[1241] 2. The screenshot of the screen information includes the following elements:

[1242] "Start" button

[1243] An input field labeled "Steps"

[1244] "Enter the number of steps"

[1245] This prompt allows the server to analyze the code and screen information and automatically generate detailed specifications required for maintenance and repair of industrial equipment.

[1246] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1247] Step 1:

[1248] The user uploads the code file and screen information from the terminal. The user selects the code file and screen information (screenshots or URLs) to upload using the interface of the smart device. The terminal sends the files received from the user to the server.

[1249] Input: User selected code file and screen information

[1250] Output: File data sent to the server

[1251] Step 2:

[1252] The server receives the code file and screen information sent from the terminal and stores the received file in a temporary storage area.

[1253] Input: Code file and screen information sent from the terminal

[1254] Output: A file stored in the temporary storage area on the server

[1255] Step 3:

[1256] The server analyzes the code file, specifically using Python's AST module to parse the code and extract properties such as function names, arguments, and return values.

[1257] Input: Code file stored in temporary storage area

[1258] Output: Analysis results such as function name, arguments, and return value

[1259] Step 4:

[1260] The server analyzes the screen information, using OCR technology (Tesseract) to extract text and UI elements from the screenshot and identify screen characteristics.

[1261] Input: Screen information stored in temporary storage area

[1262] Output: Analysis results for UI elements (buttons, input fields, labels, etc.)

[1263] Step 5:

[1264] The server automatically generates specifications based on the analysis results, combining code characteristics and screen characteristics to generate detailed specifications suitable for the maintenance and repair of industrial equipment.

[1265] Input: Analyzed code characteristics and screen characteristics

[1266] Output: Generated specification

[1267] Step 6:

[1268] The server stores the generated specifications in a database, assigns them identifiable IDs, and manages them for easy retrieval later.

[1269] Input: Generated specification

[1270] Output: The specifications stored in the database and their IDs

[1271] Step 7:

[1272] A user searches for specifications using a smart device. The user inputs a query using the search interface and sends a search request to the server. The server searches for the relevant specifications from the database and provides them to the user.

[1273] Input: Search query from user

[1274] Output: Search results for matching specifications

[1275] This series of processing steps makes it possible to quickly and accurately generate specifications required for the maintenance and repair of industrial equipment and display them on a smart device.

[1276] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1277] This invention is a system that automatically generates specifications based on uploaded code files and screen information, and further combines it with an emotion engine that recognizes user emotions. This system enables efficient and accurate specification creation, and the addition of the emotion engine improves usability. Specific embodiments of this system are described below.

[1278] The program used in this system analyzes code files and screen information through multiple steps to generate specifications. Three main parties are involved: the server, the terminal, and the user, and an emotion engine is also integrated.

[1279] How to upload code and screen information

[1280] The user selects the code file and associated screen information (e.g., screenshots or URLs) through the system interface and performs the upload.

[1281] The terminal receives the files selected by the user and transmits the data to the server.

[1282] Means of analyzing information

[1283] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[1284] The server parses the received code file and extracts code characteristics such as function names, arguments, and return values.

[1285] For example, if you receive the following code:

[1286] javascript

[1287] function add(a, b) {

[1288] return a + b;

[1289] }

[1290] The server analyzes information such as "function name: add," "arguments: a, b," and "return value: a + b."

[1291] The server also analyzes the separately received screen information, using OCR technology to extract text and UI elements (e.g., input forms, buttons) from the screenshot.

[1292] Emotion engine integration

[1293] The device uses an emotion engine to analyze the user's facial expressions and voice and recognize the user's emotional state.

[1294] For example, it detects whether a user is confused or satisfied from their facial expression or tone of voice.

[1295] Automatic generation of specifications

[1296] The server automatically generates specifications based on the results of code and screen analysis. Specifically, it integrates the results of code analysis (function names, arguments, return values) and screen analysis (UI elements) and formats them into a specification format.

[1297] For example, it generates the following specification:

[1298] Code Specifications

[1299] Function name: add

[1300] Arguments: a, b

[1301] Returns: a + b

[1302] Related screens

[1303] URL: (URL goes here)

[1304] Description: A screen with a form for entering numbers and a "Calculate" button

[1305] Processing flow

[1306] 1. User enters a number into a form

[1307] 2. Press the "Calculate" button

[1308] 3. The add function is called and the result is displayed

[1309] Storage method for specifications

[1310] The server stores the automatically generated specifications in a database, assigning an ID to each specification to make it easier to search for later.

[1311] How to search for specifications

[1312] A user sends a specification search request from the system interface, for example, a query such as "I want to know the specification of the add function."

[1313] The terminal receives a query from a user and sends it to the server.

[1314] The server searches the stored database for the relevant specifications and provides the results to the user.

[1315] Emotion-based feedback measures

[1316] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine.

[1317] For example, if the user is confused, more detailed help information can be provided.

[1318] The server provides appropriate feedback depending on the user's emotions, such as displaying a positive message if the user is satisfied.

[1319] This system allows users to avoid code duplication and separate implementations of similar functions, improving development efficiency and maintainability. Furthermore, the integration of an emotion engine improves the user experience and makes the system easier to use. Appropriate feedback based on emotions and adjustments to the operation interface allow users to use the system more comfortably.

[1320] The processing flow will be explained below.

[1321] Step 1:

[1322] The user selects the code file and associated screen information (e.g., screenshots or URLs) through the system interface and performs the upload.

[1323] Step 2:

[1324] The terminal receives the code file and screen information selected by the user and transmits the data to the server.

[1325] Step 3:

[1326] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[1327] Step 4:

[1328] The server analyzes the received code file to identify the function name, arguments, and return value of the code. For example, if the server receives the following code:

[1329] javascript

[1330] function add(a, b) {

[1331] return a + b;

[1332] }

[1333] The server extracts "function name: add", "arguments: a, b", and "return value: a + b" from this code.

[1334] Step 5:

[1335] The server analyzes the received screen information. It uses OCR technology to identify text and UI elements from the screenshot and extract their characteristics. For example, it detects "numerical input forms" and "calculation buttons."

[1336] Step 6:

[1337] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize the user's emotional state, for example, detecting whether the user is confused or satisfied from their facial expressions and tone of voice.

[1338] Step 7:

[1339] The server automatically generates a specification based on the analysis results. Specifically, it combines the code analysis results (function names, arguments, return values) with the screen information analysis results (UI elements) and formats them into the following specification format:

[1340] Code Specifications

[1341] Function name: add

[1342] Arguments: a, b

[1343] Returns: a + b

[1344] Related screens

[1345] URL: (URL goes here)

[1346] Description: A screen with a form for entering numbers and a "Calculate" button

[1347] Processing flow

[1348] 1. User enters a number into a form

[1349] 2. Press the "Calculate" button

[1350] 3. The add function is called and the result is displayed

[1351] Step 8:

[1352] The server saves the automatically generated specifications in a database. When saving, it assigns an ID to each specification and manages it so that it can be easily searched for later.

[1353] Step 9:

[1354] A user uses the system interface to submit a specification search request (e.g., a query such as "I want to know the specification of the add function").

[1355] Step 10:

[1356] The terminal receives a query from a user and sends it to the server.

[1357] Step 11:

[1358] The server searches the database for specifications that match the query and finds the relevant specifications.

[1359] Step 12:

[1360] The server provides the user with the relevant specifications as a search result.

[1361] Step 13:

[1362] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine, for example, displaying detailed help information if the user is confused.

[1363] Step 14:

[1364] The server provides appropriate feedback depending on the user's emotions, for example, if the user is satisfied, it displays a positive message.

[1365] In this way, the system prevents code duplication and separate implementations of similar functions, improving development efficiency and maintainability, while also improving the user experience by adding an emotion engine.

[1366] Example 2

[1367] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1368] Conventional specification creation systems require manual analysis of code files and screen information to create specifications, which is extremely time-consuming, inefficient, and prone to human error. Furthermore, the system lacks usability because it does not reflect the user's emotional state. Therefore, there is a need for a system that can solve these issues, create specifications efficiently and accurately, and adjust the interface according to the user's emotional state.

[1369] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1370] In this invention, the server includes means for a user to upload a code file, means for uploading screen information, means for analyzing the uploaded code file to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating a specification based on the code file and the screen information, means for saving the automatically generated specification, means for searching the saved specification, means for analyzing the user's facial expression and voice to recognize the user's emotional state, and means for automatically adjusting the operation interface based on the recognized emotional state. This enables efficient analysis of the code and screen information and automatic generation of the specification, and further improves usability by adjusting the interface taking the user's emotional state into consideration.

[1371] A "user" is an entity that uses the system to upload code files and screen information, and automatically generate and search specifications.

[1372] A "code file" is a file in which the source code of a program is written, and which has properties such as function names, arguments, and return values.

[1373] "Screen information" is data that indicates the state and structure of the user interface, such as a screenshot of the screen or a URL.

[1374] "Uploading means" refers to an interface that allows a user to send code files and screen information to the system, and a function that performs the processing.

[1375] "Analysis means" refers to a function that analyzes the received code file and screen information and extracts their characteristics.

[1376] A "specification" is a document generated based on the analysis results, which describes in detail the operation and structure of the system.

[1377] "Automatic generation means" refers to a function that automatically creates specifications based on the analysis results.

[1378] "Storage means" refers to a function that stores automatically generated specifications in a database or the like, allowing them to be searched for later.

[1379] "Search means" refers to a function that searches saved specifications based on a query and provides the results to the user.

[1380] The "emotional state" is a psychological state analyzed from the user's facial expression and voice, and includes confusion, satisfaction, anger, and the like.

[1381] "Emotion analysis means" refers to a function that captures the user's facial expressions and voice, analyzes them, and recognizes their emotional state.

[1382] The "operation interface adjusting means" refers to a function that optimizes the interface operated by the user based on the recognized emotional state.

[1383] This invention is a system that allows users to upload code files and screen information, and automatically generates specifications based on that data. The system of this invention involves the server, terminal, and user, and further improves usability by integrating an emotion engine.

[1384] How to upload code and screen information

[1385] The user selects and uploads the code file and screen information (e.g., screenshots or URLs) through the system interface. The device receives the file selected by the user and sends this data to the server. Specifically, the device checks the file format and size, and if there are no problems, transfers the file to the server.

[1386] Means of analyzing information

[1387] The server receives the code file and screen information sent from the device and temporarily stores it. Next, the server analyzes the received code file. The specific software used here is a parser (e.g., ANTLR) that extracts code characteristics such as function names, arguments, and return values. For example, if the following code is received:

[1388] javascript

[1389] function add(a, b) {

[1390] return a + b;

[1391] }

[1392] The server extracts information such as "function name: add", "arguments: a, b", and "return value: a + b".

[1393] Furthermore, the server uses OCR technology (e.g., Tesseract OCR) to extract text and UI elements (e.g., input forms, buttons) from the screen information (screenshots).

[1394] Emotion engine integration

[1395] The device uses a built-in camera and microphone to capture the user's facial expressions and voice in real time. The device then uses emotion analysis software (e.g., Microsoft Azure Emotion API) to analyze the user's emotional state. For example, it can detect whether the user is confused or happy from their facial expressions and tone of voice.

[1396] Automatic generation of specifications

[1397] The server automatically generates a specification based on the results of code and screen analysis. Specifically, it combines the results of code analysis (function names, arguments, return values) with the results of screen analysis (UI elements) and formats them into a specification. For example, the following specification is generated:

[1398] markdown

[1399] Code Specifications

[1400] Function name: add

[1401] Arguments: a, b

[1402] Returns: a + b

[1403] Related screens

[1404] URL: (URL goes here)

[1405] Description: A screen with a form for entering numbers and a "Calculate" button

[1406] Processing flow

[1407] 1. User enters a number into a form

[1408] 2. Press the "Calculate" button

[1409] 3. The add function is called and the result is displayed

[1410] Storage and retrieval of specifications

[1411] The server saves the automatically generated specifications in a database. When saved, the specifications are assigned an identification ID, which is managed so that they can be easily searched for later. When the user sends a specification search request from the interface, the terminal receives this and sends it to the server. The server searches the saved database for the corresponding specifications and provides the results to the user.

[1412] Emotion-based feedback measures

[1413] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, it provides more detailed help information. If the user is satisfied, it provides feedback by displaying a positive message.

[1414] Examples and Prompts

[1415] A user logs into the system and uploads the project's code files and screenshots. The device checks them and sends them to the server. The server analyzes the code and extracts function names, arguments, return values, etc. It also uses OCR technology to analyze text and UI elements from screenshots. Based on these results, a specification document is automatically generated and stored in a database.

[1416] Example prompt sentence:

[1417] In this system, I uploaded the following JavaScript code file: function multiply(x, y) { return xy;} Please auto-generate the specification.

[1418] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1419] Step 1:

[1420] Through the system interface, the user clicks the upload button, selects the code file and screen information (screenshots or URLs), and uploads them.

[1421] Input: Code file, screen information

[1422] Behavior: The user selects a file through a file selection dialog and initiates the upload.

[1423] Output: The device receives the selected file.

[1424] Step 2:

[1425] The device checks the code file and screen information received from the user to ensure there are no problems with the format or size, and then sends this data to the server.

[1426] Input: Code file, screen information

[1427] Action: The device checks the file format and whether it exceeds the maximum capacity. If there are no problems, it transfers the file to the server.

[1428] Output: The server receives the file and stores it temporarily.

[1429] Step 3:

[1430] The server analyzes the received code file and extracts code characteristics such as function names, arguments, return values, etc. Specifically, it analyzes the code using a parser (e.g., ANTLR).

[1431] Input: Code file

[1432] How it works: The server parses the code through a parser and extracts the function name, arguments, and return value. For example, if you parse the code function add(a, b) { return a + b;}, it extracts "function name: add", "arguments: a, b", and "return value: a + b".

[1433] Output: Code characteristics information (function names, arguments, return values, etc.)

[1434] Step 4:

[1435] The server simultaneously analyzes the received screen information and uses OCR technology (e.g., Tesseract OCR) to extract text and UI elements from the screenshot.

[1436] Input: Screen information (screenshots, URL, etc.)

[1437] How it works: The server uses OCR technology to analyze the screenshot and extract text and UI elements (input forms, buttons, etc.).

[1438] Output: Screen characteristics (text, UI elements, etc.)

[1439] Step 5:

[1440] The server integrates the code characteristics information and the screen characteristics information to automatically generate a specification document, which includes the function name, arguments, return values, and related screen information.

[1441] Input: Code characteristics information, screen characteristics information

[1442] How it works: The server combines this information and formats it into a specification. For example, it puts function names and argument information in the code section, and screen information in a separate section.

[1443] Output: Auto-generated specification

[1444] Step 6:

[1445] The server stores the automatically generated specifications in a database, assigns them an ID, and manages them for easy retrieval later.

[1446] Input: Auto-generated specification

[1447] Operation: The server assigns an ID to the specification and stores it in the database.

[1448] Output: Specification saved in database

[1449] Step 7:

[1450] A user sends a specification search request through the system interface, for example, by entering a query such as "I want to know the specification of the add function."

[1451] Input: Search query (e.g. "add function specification")

[1452] How it works: A user enters a query into the search box and clicks the search button. The device receives the query and sends it to the server.

[1453] Output: The server receives the search request.

[1454] Step 8:

[1455] The server searches the database for the relevant specifications based on the search request and provides the results to the user.

[1456] Input: Search request

[1457] How it works: The server searches the database for the query, finds the relevant specifications, and returns the search results to the device.

[1458] Output: Search results (relevant specifications)

[1459] Step 9:

[1460] The terminal displays the search results received from the server to the user.

[1461] Input: Search results from the server

[1462] Operation: The device receives the search results and displays them on the user's interface.

[1463] Output: Specification information displayed in the interface

[1464] Step 10:

[1465] The device uses an emotion engine to capture the user's facial expressions and voice and recognize their emotional state, for example, detecting whether the user is confused or happy from their facial expressions or tone of voice.

[1466] Input: User's facial expression, voice

[1467] How it works: The device captures the user's facial expressions and voice through the camera and microphone, and performs emotion analysis.

[1468] Output: User's emotional state (confused, satisfied, etc.)

[1469] Step 11:

[1470] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, it provides detailed help information. If the user is satisfied, it displays a positive message.

[1471] Input: User's emotional state

[1472] Action: The server makes appropriate user interface adjustments and feedback based on the recognized emotional state.

[1473] Output: Adjusting the interface, providing feedback (e.g., detailed help information, positive messages)

[1474] (Application example 2)

[1475] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1476] Currently, programming factory robots and creating operation specifications is done manually using complex tools, which requires a great deal of time and effort. In addition, users can become confused when operating the robots, hindering efficient work. Furthermore, real-time feedback on the inconvenience or confusion experienced by users is not provided, creating a need for improved usability.

[1477] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1478] In this invention, the server includes means for uploading code files, means for uploading screen information, means for analyzing the uploaded code files to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating specifications based on the code files and the screen information, means for saving the automatically generated specifications, means for searching the saved specifications, means for analyzing emotional information to recognize the emotional state of a user, and means for adjusting the operation interface and providing feedback based on the emotional information. This allows users to significantly reduce the time and effort required to program factory robots, and provides real-time feedback according to emotions, improving usability and enabling more efficient and comfortable work.

[1479] A "code file" is a file in which the source code of a program is written.

[1480] "Screen information" refers to data that includes visual information about the user interface, such as screenshots and URLs.

[1481] "Analysis" is the process used to extract characteristics and structure from code files and screen information.

[1482] A "specification" is a document that describes in detail the functions and operations of software or a system.

[1483] "Emotion information" refers to data on emotions detected from the user's facial expressions, voice, etc.

[1484] An "operation interface" refers to the screen or input means that a user uses to operate a system or application.

[1485] "Feedback" refers to the system providing real-time responses to the user's actions and state.

[1486] A "server" is a computer that provides data and services to other terminals over a network.

[1487] "Optical character recognition technology" is a technology that extracts character information from an image.

[1488] MODE FOR CARRYING OUT THE INVENTION

[1489] This invention is a system that automatically generates specifications based on uploaded code files and screen information, and also combines it with an emotion engine that recognizes user emotions. This system enables efficient and accurate specification creation, and the addition of the emotion engine improves usability.

[1490] System configuration and functions

[1491] This system consists of three components: a server, a terminal, and a user. Furthermore, by integrating an emotion engine, it has been added with the function of providing appropriate feedback based on the user's emotional state.

[1492] Uploading code and screen information

[1493] The user selects the code file and related screen information (e.g., screenshots or URLs) through the system interface and performs the upload. The terminal receives the files selected by the user and sends the data to the server. This part can be a general computer or smart device.

[1494] Analysis of information

[1495] The server receives the code files and screen information sent from the device and temporarily stores them. The server then analyzes the received code files and extracts code characteristics such as function names, arguments, and return values. It also analyzes the separately received screen information. Optical character recognition (OCR) is used to extract text and UI elements (e.g., input forms, buttons) from the screenshots. This is done using software libraries such as pytesseract and OpenCV.

[1496] Emotion engine integration

[1497] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize the user's emotional state. Specifically, a library called EmotionRecognizer is used. For example, it can detect whether the user is confused or happy from their facial expressions and tone of voice.

[1498] Automatic generation of specifications

[1499] The server automatically generates specifications based on the results of code and screen information analysis. Specifically, it integrates the results of code analysis (function name, arguments, return value) and screen information analysis (UI elements) and formats them into a specification format. An example of the generated specification is shown below.

[1500] Function name: move_forward

[1501] Argument: distance

[1502] Returns: None

[1503] Function Name: rotate

[1504] Argument: angle

[1505] Returns: None

[1506] Related screen: A screen with a form for entering numbers and an "Execute" button

[1507] Process flow:

[1508] 1. User enters a number into a form

[1509] 2. Press the "Run" button

[1510] 3. The functions move_forward and rotate are called and the results are displayed.

[1511] Save and search specifications

[1512] The server stores the automatically generated specifications in a database. At this time, it assigns an identification ID to the specifications and manages them so that they can be easily searched for later. The user sends a specification search request from the system interface. For example, they send a query such as "I want to know the specifications of the move_forward function." The terminal receives the query from the user and sends it to the server. The server searches the stored database for the relevant specifications and provides the results to the user.

[1513] Emotion-based feedback measures

[1514] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, the server will provide more detailed help information. The server also provides appropriate feedback according to the user's emotions. For example, if the user is satisfied, the server will display a positive message.

[1515] Specific examples

[1516] Below are some examples of specific prompt sentences.

[1517] Please generate a specification for the "move_forward" function, and provide specific instructions if users are confused.

[1518] As described above, this system allows users to automatically generate efficient and accurate specifications simply by uploading code files and screen information. Furthermore, the emotion engine provides appropriate feedback based on the user's emotional state, improving usability.

[1519] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1520] Step 1:

[1521] The user selects the code file and screen information through the system interface and performs the upload. For example, the user selects the source code file of a program that controls the operation of a factory robot and screenshots of the program's operation screen. This is the input. The terminal receives the selected files and sends the data to the server.

[1522] Step 2:

[1523] The server receives the code file and screen information sent from the device and temporarily stores it. It then analyzes the received code file and extracts code characteristics such as function names, arguments, and return values. For example, it analyzes the "distance" argument and return value "None" of the "move_forward" function. This is data processing, and the analyzed information is obtained as output.

[1524] Step 3:

[1525] The server analyzes the screen information. Specifically, it uses optical character recognition (OCR) technology to extract text and UI elements from the screenshot. For example, it extracts the "numeric input form" and the text of the "Execute" button contained in the screenshot. This is the input, and the extracted text information is obtained as the output.

[1526] Step 4:

[1527] The device uses an emotion engine to recognize the user's emotional state. For example, it captures the user's facial expressions with a camera and uses the EmotionRecognizer library to detect emotions such as confusion or satisfaction. Emotion data is the input, and the recognized emotional state is the output.

[1528] Step 5:

[1529] The server automatically generates specifications based on the results of analyzing the code and screen information. Specifically, it integrates the analysis results and formats them into a specification format. For example, it documents the operation flow based on the "move_forward" function specification and screen information. This is the input, and the automatically generated specification is obtained as the output.

[1530] Step 6:

[1531] The server stores the automatically generated specifications in a database. At this time, it assigns an ID to each specification and manages it so that it can be easily searched. The specifications are input, and the state of the specifications stored in the database is obtained as output.

[1532] Step 7:

[1533] The user sends a specification search request from the system interface. For example, they send a query such as "I want to know the specifications of the move_forward function." This is the input. The terminal receives the query from the user and sends it to the server.

[1534] Step 8:

[1535] The server searches for the relevant specifications from the stored database and provides the results to the user. For example, it searches for specifications related to the "move_forward function" and returns them to the user. The query is the input, and the search results are the output.

[1536] Step 9:

[1537] The server automatically adjusts the operation interface and provides feedback based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, detailed help information is displayed. The recognized emotional state is the input, and the adjusted interface is the output.

[1538] Through these steps, users can efficiently create program specifications for factory robots and receive real-time emotional feedback.

[1539] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1540] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1541] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1542] [Fourth embodiment]

[1543] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1544] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1545] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1546] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1547] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1548] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1549] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1550] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1551] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1552] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1553] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1554] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1555] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1556] The present invention relates to a system for automatically generating specifications by uploading code files and screen information to the system. A specific embodiment of this system will be described below.

[1557] The program used in this system analyzes code files and screen information through multiple steps to generate specifications. Three main parties are involved: the server, the terminal, and the user.

[1558] How to upload code and screen information

[1559] The user uses their own terminal through the system interface to select the code file and associated screen information (e.g., screenshots and URLs) and upload them to the system.

[1560] The terminal receives the file selected by the user and transmits it to the server.

[1561] Means of analyzing information

[1562] The server receives the code file and screen information sent from the terminal.

[1563] The server parses the received code file and extracts characteristics of the code, such as function names, arguments, and return values.

[1564] For example, if you receive the following code:

[1565] javascript

[1566] function add(a, b) {

[1567] return a + b;

[1568] }

[1569] The server analyzes information such as "function name: add," "arguments: a, b," and "return value: a + b."

[1570] The server also analyzes the screen information it receives separately. For example, it uses OCR technology to identify text and UI elements from screenshots and extracts characteristics such as "numeric input form" and "calculation button."

[1571] Automatic generation of specifications

[1572] The server automatically generates specifications based on the results of analyzing the code and screen information.

[1573] For example, the analysis results can be used to generate the following specification:

[1574] Code Specifications

[1575] Function name: add

[1576] Arguments: a, b

[1577] Returns: a + b

[1578] Related screens

[1579] URL: (URL goes here)

[1580] Description: A screen with a form for entering numbers and a "Calculate" button

[1581] Processing flow

[1582] 1. User enters a number into a form

[1583] 2. Press the "Calculate" button

[1584] 3. The add function is called and the result is displayed

[1585] Storage method for specifications

[1586] The server stores the automatically generated specifications in a database.

[1587] When saving, the specification is given an identifiable ID and managed so that it can be easily searched for later.

[1588] How to search for specifications

[1589] A user sends a specification search request from the system interface, for example, a query such as "I want to know the specification of the add function."

[1590] The terminal receives a query from a user and sends it to the server.

[1591] The server searches the stored database for the relevant specifications and provides the results to the user.

[1592] This system allows users to avoid code duplication and multiple implementations of similar functionality, improving development efficiency and maintainability. It also automates the specification creation process, saving significant time and effort. This also improves technical accuracy and the productivity of the entire development team.

[1593] The processing flow will be explained below.

[1594] Step 1:

[1595] The user selects the code file and related screen information (screenshots or URLs) through the system interface and performs the upload.

[1596] Step 2:

[1597] The terminal receives the code file and screen information selected by the user and transmits the data to the server.

[1598] Step 3:

[1599] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[1600] Step 4:

[1601] The server analyzes the received code file. Specifically, it identifies the function names, arguments, and return values ​​in the code. For example, it extracts the function "add" from the code, its arguments "a, b", and its return value "a + b".

[1602] Step 5:

[1603] The server analyzes the received screen information and extracts text and UI elements (e.g., input forms, buttons) from the screenshot using OCR technology.

[1604] Step 6:

[1605] The server automatically generates a specification based on the analysis results. Specifically, it integrates the code analysis results (function names, arguments, return values) and the screen information analysis results (UI elements) and formats them into a specification format.

[1606] Step 7:

[1607] The server stores the automatically generated specifications in a database, assigning an ID to each specification to make it easy to search for later.

[1608] Step 8:

[1609] A user submits a specification search request using the system's interface (e.g., a query such as "I want to know the specification of the add function").

[1610] Step 9:

[1611] The terminal transmits a search query from the user to the server.

[1612] Step 10:

[1613] The server searches the database for the relevant specifications, finds a match with the query, and finds the corresponding specifications.

[1614] Step 11:

[1615] The server provides the user with the relevant specifications as a search result.

[1616] Example 1

[1617] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1618] In existing software development environments, the manual analysis of code files and screen information and the creation of specifications requires a great deal of time and effort. Furthermore, manual analysis and documentation has a high risk of human error, leading to incomplete specifications and reduced development efficiency. To solve these problems, a system is needed that can automatically analyze code files and screen information and generate specifications quickly and accurately.

[1619] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1620] In this invention, the server includes means for uploading a code file to an electronic computing device, means for uploading screen information to the electronic computing device, means for analyzing the uploaded code file to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating a specification based on the code file and the screen information, means for saving the automatically generated specification in a database, means for searching the saved specification, means for displaying progress and notifying the user, and means for applying the analysis results to a template to generate a specification. This significantly reduces the time and effort associated with manual analysis and document creation, and enables the generation of accurate and consistent specifications.

[1621] A "code file" is electronic data containing the source code of a program or script.

[1622] "Screen information" refers to data that includes information about the user interface, the layout of the screen, and its elements.

[1623] "Computing device" refers to a device that processes and stores data, such as a computer or server.

[1624] "Analysis" is the process of breaking down input data and extracting its components and characteristics.

[1625] "Characteristics" refers to specific attributes or functional characteristics extracted from code or screen information.

[1626] A "specification" is a document that describes in detail the operation, structure, and functions of a system or program.

[1627] A "database" is a management system for systematically storing large amounts of data and making them efficiently searchable and accessible.

[1628] "Search" refers to the process of locating specific information from stored data.

[1629] "Optical character recognition technology" is a technology that analyzes characters in an image and converts them into text data.

[1630] A "template" is a standard format for generating documents in a consistent format.

[1631] "Progress" is information that indicates the current stage of data analysis and processing.

[1632] The term "user" refers to a person who uses the system of the present invention.

[1633] This invention relates to a system that uploads code files and screen information to a system, automatically generates specifications, and stores and searches them. The following steps and techniques are used to realize this system.

[1634] Upload method:

[1635] A user uses his / her device to select a code file (e.g., add.js) and related screen information (e.g., form_screen.png) from the system interface and upload them to the system. The device receives these files and sends them to the server via an HTTP request. The user interface displays the file sending status to the user.

[1636] Information analysis methods:

[1637] The server receives the code file and screen information sent from the device and temporarily stores it. The server analyzes the received code file and extracts characteristics such as function names, arguments, and return values. For example, if the code is JavaScript code, the server identifies the function name "add," arguments "a, b," and return value "a + b." The server also uses OCR technology to analyze the screen information and identify text and UI elements. For example, it extracts characteristics such as a "number input form" and a "calculation button" from a screenshot.

[1638] Automated generation of specifications:

[1639] The server automatically generates a specification template based on the results of the code and screen analysis, applying these characteristics to the template. Specifically, it uses a template engine to incorporate the analysis results into a predetermined format and generates the following specification:

[1640] Code Specifications

[1641] Function name: add

[1642] Arguments: a, b

[1643] Returns: a + b

[1644] Related screens

[1645] URL: (URL goes here)

[1646] Description: A screen with a form for entering numbers and a "Calculate" button

[1647] Processing flow

[1648] 1. User enters a number into a form

[1649] 2. Press the "Calculate" button

[1650] 3. The add function is called and the result is displayed

[1651] How to save the specification:

[1652] The server saves the automatically generated specifications in a database. When saved, the specifications are given an identifiable ID and managed for easy retrieval later. The fact that the specifications have been saved is also recorded in a log.

[1653] How to search for specifications:

[1654] The user enters a search query into the system interface and sends a request to search for specifications. For example, the user might enter, "I want to know the specifications of the add function." The terminal sends this query to the server, which then searches the database for the relevant specifications. The search results are displayed to the user via the terminal.

[1655] Examples and prompts:

[1656] The user uploads a JavaScript file called "add.js" and a screenshot called "form_screen.png" to the system on their device.

[1657] The terminal transmits these files to the server and displays the transmission status.

[1658] The server analyzes "add.js" to extract function names, arguments, and return values. It also analyzes "form_screen.png" using OCR technology to identify UI elements.

[1659] The server generates a specification based on these analysis results, assigns it an identification ID of "add_function_spec_001," and stores it in the database.

[1660] Later, when the user sends a search request for "add function specification" from the system interface, the server searches the database for "add_function_spec_001" and provides the results to the user.

[1661] Example prompt sentence:

[1662] Upload the code file and screen information. For example, select the file add.js and the screenshot form_screen.png and upload them to the system. A specification document will be automatically generated based on this information.

[1663] This invention allows users to significantly reduce the time and effort associated with manual analysis and documentation, enabling the generation of accurate and consistent specifications. Furthermore, the system's search function allows users to quickly reference existing specifications, improving development efficiency and maintainability.

[1664] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1665] Step 1:

[1666] User selects and uploads code files and screen information

[1667] Input: Code file (e.g. add.js), screen information (e.g. form_screen.png)

[1668] Specific operation: The user opens the system interface and clicks the upload button. The user selects the file to upload from their device and clicks "Upload".

[1669] Output: The uploaded file is sent from the user's device to the server.

[1670] Step 2:

[1671] The device receives the file and sends it to the server

[1672] Input: User selected file and upload request

[1673] Specific operation: The device reads the file selected by the user from local storage, creates an HTTP POST request, and sends it to the server.

[1674] Output: The status of the HTTP request sent to the server is displayed on the terminal to inform the user of progress.

[1675] Step 3:

[1676] The server receives the code file and screen information.

[1677] Input: File data included in the HTTP request from the terminal

[1678] Specific operation: The server receives the HTTP request, temporarily stores the code file and screen information in storage, checks the integrity of the file, and if there are no problems, starts the analysis process.

[1679] Output: The file is saved and ready for analysis.

[1680] Step 4:

[1681] The server begins parsing the code file

[1682] Input: Saved code file

[1683] Specific operation: The server starts an analysis program to analyze the contents of the code file, extracting characteristics such as function names, arguments, and return values.

[1684] Output: Characteristic data of the extracted code (e.g., "Function name: add", "Arguments: a, b", "Return value: a + b")

[1685] Step 5:

[1686] The server analyzes the screen information

[1687] Input: Saved screen information

[1688] What it does: The server uses OCR technology to identify text and UI elements from the screenshot, and then categorizes the identified UI elements.

[1689] Output: Extracted screen information characteristics data (e.g. "Number input form", "Calculation button")

[1690] Step 6:

[1691] The server generates specifications based on the analysis results

[1692] Input: Code characteristic data, Screen information characteristic data

[1693] Specific operation: The server uses a template engine to incorporate the analysis results into a predetermined format and automatically generate a specification document.

[1694] Output: Automatically generated specification (e.g., "Function name: add", "Arguments: a, b", "Return value: a + b", "Related screen: Screen with a numeric input form and a calculation button")

[1695] Step 7:

[1696] The server stores the specifications in a database

[1697] Input: Auto-generated specification

[1698] Specific behavior: The server saves the specification in the database using an INSERT query and assigns it an identifiable ID.

[1699] Output: Notification of completion of saving and generation of identification ID (e.g. "Specification ID: add_function_spec_001")

[1700] Step 8:

[1701] User submits a search request for specifications

[1702] Input: Search query (e.g. "I want to know the specifications of the add function")

[1703] Specific actions: The user enters a query into the search box in the system interface and clicks the search button.

[1704] Output: A search request is sent to the server.

[1705] Step 9:

[1706] The server searches for the specification and returns the results

[1707] Input: A search request from the user

[1708] Specific operation: The server searches the database, extracts the relevant specifications, and returns the extracted specifications to the terminal.

[1709] Output: The search result, a specification document, is displayed to the user (e.g., detailed information for "add_function_spec_001").

[1710] (Application example 1)

[1711] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1712] When maintaining and repairing industrial equipment, there is a lack of a way to quickly and accurately generate specifications for related code and screen information. This challenge makes it difficult for on-site technicians to perform their work efficiently, increasing the likelihood of mistakes and rework. In particular, there is a need to address the need to quickly obtain information on-site using smart devices.

[1713] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1714] In this invention, the server includes a means for uploading a code file, a means for uploading screen information, and a means for automatically generating specifications based on the uploaded code file and screen information, thereby enabling specifications to be generated quickly and accurately for maintenance and repair of industrial equipment and displayed on a smart device.

[1715] - "Code File" means a file containing a program description, which contains the set of instructions necessary to run the program.

[1716] "Screen information" is visual information about the user interface, and includes data such as screenshots and URLs.

[1717] A "specification" is a technical document that is automatically generated based on code files and screen information, and contains detailed descriptions of the program's characteristics and screen characteristics.

[1718] "Optical character recognition technology" is a technology that extracts characters from an image and converts them into text data.

[1719] The "storage means" is a function for storing the generated specifications in a database or other storage device.

[1720] "Search facility" is a function that allows you to search saved specifications by keywords or other criteria to quickly access specific information.

[1721] The "uploading means" is a function that allows a user to use a terminal to send code files and screen information to a server.

[1722] "Industrial equipment" refers to machinery and equipment used in various industries, and is controlled using advanced technology.

[1723] "Maintenance and repair" refers to the work of inspecting and repairing industrial equipment to maintain its normal operation.

[1724] "Smart device" refers to a mobile terminal device that can connect to the Internet and use multi-function applications.

[1725] The system for implementing this invention mainly involves three entities: a server, a terminal, and a user. The server analyzes the code file and screen information uploaded by the user and generates a specification document based on that. The server then saves the specification document and displays it on the smart device for use in maintenance and repair.

[1726] Uploading code files and screen information

[1727] The user selects the code file and related screen information (screenshots or URLs) from the terminal and uploads them to the system. The terminal receives the file selected by the user and sends it to the server.

[1728] Analysis of information

[1729] The server receives the code file and screen information sent from the device. The server analyzes the received code file and extracts function names, arguments, return values, etc. It also uses OCR technology to identify text and UI elements in the screen information and extracts characteristics such as "numeric input forms" and "buttons."

[1730] Automatic generation of specifications

[1731] The server automatically generates specifications required for the maintenance and repair of industrial equipment based on the analyzed code and screen information. The generated specifications contain detailed descriptions of the code characteristics and user interface characteristics.

[1732] Save and search specifications

[1733] The generated specifications are stored in a database on the server and assigned an identifiable ID. Users can search for specifications through their smart devices and quickly obtain information for efficient maintenance and repair of industrial equipment.

[1734] Hardware and software used

[1735] The server is a computer system capable of high performance analysis, and is preferably a distributed system if necessary.

[1736] To use OCR technology, we use Tesseract (an open source OCR engine).

[1737] Code analysis uses Python's AST module and specific programming language analysis tools.

[1738] The database uses an RDBMS such as MySQL or PostgreSQL.

[1739] Examples and prompts

[1740] For example, you could use the following prompt as input to a generative AI model:

[1741] "I'd like to know more about how the move_forward(steps) function works.

[1742] The code file contains the following:

[1743] function move_forward(steps) {

[1744] for(int i = 0; i < steps; i++) {

[1745] move_one_step();

[1746] }

[1747] }

[1748] 2. The screenshot of the screen information includes the following elements:

[1749] "Start" button

[1750] An input field labeled "Steps"

[1751] "Enter the number of steps"

[1752] This prompt allows the server to analyze the code and screen information and automatically generate detailed specifications required for maintenance and repair of industrial equipment.

[1753] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1754] Step 1:

[1755] The user uploads the code file and screen information from the terminal. The user selects the code file and screen information (screenshots or URLs) to upload using the interface of the smart device. The terminal sends the files received from the user to the server.

[1756] Input: User selected code file and screen information

[1757] Output: File data sent to the server

[1758] Step 2:

[1759] The server receives the code file and screen information sent from the terminal and stores the received file in a temporary storage area.

[1760] Input: Code file and screen information sent from the terminal

[1761] Output: A file stored in the temporary storage area on the server

[1762] Step 3:

[1763] The server analyzes the code file, specifically using Python's AST module to parse the code and extract properties such as function names, arguments, and return values.

[1764] Input: Code file stored in temporary storage area

[1765] Output: Analysis results such as function name, arguments, and return value

[1766] Step 4:

[1767] The server analyzes the screen information, using OCR technology (Tesseract) to extract text and UI elements from the screenshot and identify screen characteristics.

[1768] Input: Screen information stored in temporary storage area

[1769] Output: Analysis results for UI elements (buttons, input fields, labels, etc.)

[1770] Step 5:

[1771] The server automatically generates specifications based on the analysis results, combining code characteristics and screen characteristics to generate detailed specifications suitable for the maintenance and repair of industrial equipment.

[1772] Input: Analyzed code characteristics and screen characteristics

[1773] Output: Generated specification

[1774] Step 6:

[1775] The server stores the generated specifications in a database, assigns them identifiable IDs, and manages them for easy retrieval later.

[1776] Input: Generated specification

[1777] Output: The specifications stored in the database and their IDs

[1778] Step 7:

[1779] A user searches for specifications using a smart device. The user inputs a query using the search interface and sends a search request to the server. The server searches for the relevant specifications from the database and provides them to the user.

[1780] Input: Search query from user

[1781] Output: Search results for matching specifications

[1782] This series of processing steps makes it possible to quickly and accurately generate specifications required for the maintenance and repair of industrial equipment and display them on a smart device.

[1783] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1784] This invention is a system that automatically generates specifications based on uploaded code files and screen information, and further combines it with an emotion engine that recognizes user emotions. This system enables efficient and accurate specification creation, and the addition of the emotion engine improves usability. Specific embodiments of this system are described below.

[1785] The program used in this system analyzes code files and screen information through multiple steps to generate specifications. Three main parties are involved: the server, the terminal, and the user, and an emotion engine is also integrated.

[1786] How to upload code and screen information

[1787] The user selects the code file and associated screen information (e.g., screenshots or URLs) through the system interface and performs the upload.

[1788] The terminal receives the files selected by the user and transmits the data to the server.

[1789] Means of analyzing information

[1790] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[1791] The server parses the received code file and extracts code characteristics such as function names, arguments, and return values.

[1792] For example, if you receive the following code:

[1793] javascript

[1794] function add(a, b) {

[1795] return a + b;

[1796] }

[1797] The server analyzes information such as "function name: add," "arguments: a, b," and "return value: a + b."

[1798] The server also analyzes the separately received screen information, using OCR technology to extract text and UI elements (e.g., input forms, buttons) from the screenshot.

[1799] Emotion engine integration

[1800] The device uses an emotion engine to analyze the user's facial expressions and voice and recognize the user's emotional state.

[1801] For example, it detects whether a user is confused or satisfied from their facial expression or tone of voice.

[1802] Automatic generation of specifications

[1803] The server automatically generates specifications based on the results of code and screen analysis. Specifically, it integrates the results of code analysis (function names, arguments, return values) and screen analysis (UI elements) and formats them into a specification format.

[1804] For example, it generates the following specification:

[1805] Code Specifications

[1806] Function name: add

[1807] Arguments: a, b

[1808] Returns: a + b

[1809] Related screens

[1810] URL: (URL goes here)

[1811] Description: A screen with a form for entering numbers and a "Calculate" button

[1812] Processing flow

[1813] 1. User enters a number into a form

[1814] 2. Press the "Calculate" button

[1815] 3. The add function is called and the result is displayed

[1816] Storage method for specifications

[1817] The server stores the automatically generated specifications in a database, assigning an ID to each specification to make it easier to search for later.

[1818] How to search for specifications

[1819] A user sends a specification search request from the system interface, for example, a query such as "I want to know the specification of the add function."

[1820] The terminal receives a query from a user and sends it to the server.

[1821] The server searches the stored database for the relevant specifications and provides the results to the user.

[1822] Emotion-based feedback measures

[1823] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine.

[1824] For example, if the user is confused, more detailed help information can be provided.

[1825] The server provides appropriate feedback depending on the user's emotions, such as displaying a positive message if the user is satisfied.

[1826] This system allows users to avoid code duplication and separate implementations of similar functions, improving development efficiency and maintainability. Furthermore, the integration of an emotion engine improves the user experience and makes the system easier to use. Appropriate feedback based on emotions and adjustments to the operation interface allow users to use the system more comfortably.

[1827] The processing flow will be explained below.

[1828] Step 1:

[1829] The user selects the code file and associated screen information (e.g., screenshots or URLs) through the system interface and performs the upload.

[1830] Step 2:

[1831] The terminal receives the code file and screen information selected by the user and transmits the data to the server.

[1832] Step 3:

[1833] The server receives the code file and screen information sent from the terminal and temporarily stores them.

[1834] Step 4:

[1835] The server analyzes the received code file to identify the function name, arguments, and return value of the code. For example, if the server receives the following code:

[1836] javascript

[1837] function add(a, b) {

[1838] return a + b;

[1839] }

[1840] The server extracts "function name: add", "arguments: a, b", and "return value: a + b" from this code.

[1841] Step 5:

[1842] The server analyzes the received screen information. It uses OCR technology to identify text and UI elements from the screenshot and extract their characteristics. For example, it detects "numerical input forms" and "calculation buttons."

[1843] Step 6:

[1844] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize the user's emotional state, for example, detecting whether the user is confused or satisfied from their facial expressions and tone of voice.

[1845] Step 7:

[1846] The server automatically generates a specification based on the analysis results. Specifically, it combines the code analysis results (function names, arguments, return values) with the screen information analysis results (UI elements) and formats them into the following specification format:

[1847] Code Specifications

[1848] Function name: add

[1849] Arguments: a, b

[1850] Returns: a + b

[1851] Related screens

[1852] URL: (URL goes here)

[1853] Description: A screen with a form for entering numbers and a "Calculate" button

[1854] Processing flow

[1855] 1. User enters a number into a form

[1856] 2. Press the "Calculate" button

[1857] 3. The add function is called and the result is displayed

[1858] Step 8:

[1859] The server saves the automatically generated specifications in a database. When saving, it assigns an ID to each specification and manages it so that it can be easily searched for later.

[1860] Step 9:

[1861] A user uses the system interface to submit a specification search request (e.g., a query such as "I want to know the specification of the add function").

[1862] Step 10:

[1863] The terminal receives a query from a user and sends it to the server.

[1864] Step 11:

[1865] The server searches the database for specifications that match the query and finds the relevant specifications.

[1866] Step 12:

[1867] The server provides the user with the relevant specifications as a search result.

[1868] Step 13:

[1869] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine, for example, displaying detailed help information if the user is confused.

[1870] Step 14:

[1871] The server provides appropriate feedback depending on the user's emotions, for example, if the user is satisfied, it displays a positive message.

[1872] In this way, the system prevents code duplication and separate implementations of similar functions, improving development efficiency and maintainability, while also improving the user experience by adding an emotion engine.

[1873] Example 2

[1874] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1875] Conventional specification creation systems require manual analysis of code files and screen information to create specifications, which is extremely time-consuming, inefficient, and prone to human error. Furthermore, the system lacks usability because it does not reflect the user's emotional state. Therefore, there is a need for a system that can solve these issues, create specifications efficiently and accurately, and adjust the interface according to the user's emotional state.

[1876] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1877] In this invention, the server includes means for a user to upload a code file, means for uploading screen information, means for analyzing the uploaded code file to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating a specification based on the code file and the screen information, means for saving the automatically generated specification, means for searching the saved specification, means for analyzing the user's facial expression and voice to recognize the user's emotional state, and means for automatically adjusting the operation interface based on the recognized emotional state. This enables efficient analysis of the code and screen information and automatic generation of the specification, and further improves usability by adjusting the interface taking the user's emotional state into consideration.

[1878] A "user" is an entity that uses the system to upload code files and screen information, and automatically generate and search specifications.

[1879] A "code file" is a file in which the source code of a program is written, and which has properties such as function names, arguments, and return values.

[1880] "Screen information" is data that indicates the state and structure of the user interface, such as a screenshot of the screen or a URL.

[1881] "Uploading means" refers to an interface that allows a user to send code files and screen information to the system, and a function that performs the processing.

[1882] "Analysis means" refers to a function that analyzes the received code file and screen information and extracts their characteristics.

[1883] A "specification" is a document generated based on the analysis results, which describes in detail the operation and structure of the system.

[1884] "Automatic generation means" refers to a function that automatically creates specifications based on the analysis results.

[1885] "Storage means" refers to a function that stores automatically generated specifications in a database or the like, allowing them to be searched for later.

[1886] "Search means" refers to a function that searches saved specifications based on a query and provides the results to the user.

[1887] The "emotional state" is a psychological state analyzed from the user's facial expression and voice, and includes confusion, satisfaction, anger, and the like.

[1888] "Emotion analysis means" refers to a function that captures the user's facial expressions and voice, analyzes them, and recognizes their emotional state.

[1889] The "operation interface adjusting means" refers to a function that optimizes the interface operated by the user based on the recognized emotional state.

[1890] This invention is a system that allows users to upload code files and screen information, and automatically generates specifications based on that data. The system of this invention involves the server, terminal, and user, and further improves usability by integrating an emotion engine.

[1891] How to upload code and screen information

[1892] The user selects and uploads the code file and screen information (e.g., screenshots or URLs) through the system interface. The device receives the file selected by the user and sends this data to the server. Specifically, the device checks the file format and size, and if there are no problems, transfers the file to the server.

[1893] Means of analyzing information

[1894] The server receives the code file and screen information sent from the device and temporarily stores it. Next, the server analyzes the received code file. The specific software used here is a parser (e.g., ANTLR) that extracts code characteristics such as function names, arguments, and return values. For example, if the following code is received:

[1895] javascript

[1896] function add(a, b) {

[1897] return a + b;

[1898] }

[1899] The server extracts information such as "function name: add", "arguments: a, b", and "return value: a + b".

[1900] Furthermore, the server uses OCR technology (e.g., Tesseract OCR) to extract text and UI elements (e.g., input forms, buttons) from the screen information (screenshots).

[1901] Emotion engine integration

[1902] The device uses a built-in camera and microphone to capture the user's facial expressions and voice in real time. The device then uses emotion analysis software (e.g., Microsoft Azure Emotion API) to analyze the user's emotional state. For example, it can detect whether the user is confused or happy from their facial expressions and tone of voice.

[1903] Automatic generation of specifications

[1904] The server automatically generates a specification based on the results of code and screen analysis. Specifically, it combines the results of code analysis (function names, arguments, return values) with the results of screen analysis (UI elements) and formats them into a specification. For example, the following specification is generated:

[1905] markdown

[1906] Code Specifications

[1907] Function name: add

[1908] Arguments: a, b

[1909] Returns: a + b

[1910] Related screens

[1911] URL: (URL goes here)

[1912] Description: A screen with a form for entering numbers and a "Calculate" button

[1913] Processing flow

[1914] 1. User enters a number into a form

[1915] 2. Press the "Calculate" button

[1916] 3. The add function is called and the result is displayed

[1917] Storage and retrieval of specifications

[1918] The server saves the automatically generated specifications in a database. When saved, the specifications are assigned an identification ID, which is managed so that they can be easily searched for later. When the user sends a specification search request from the interface, the terminal receives this and sends it to the server. The server searches the saved database for the corresponding specifications and provides the results to the user.

[1919] Emotion-based feedback measures

[1920] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, it provides more detailed help information. If the user is satisfied, it provides feedback by displaying a positive message.

[1921] Examples and Prompts

[1922] A user logs into the system and uploads the project's code files and screenshots. The device checks them and sends them to the server. The server analyzes the code and extracts function names, arguments, return values, etc. It also uses OCR technology to analyze text and UI elements from screenshots. Based on these results, a specification document is automatically generated and stored in a database.

[1923] Example prompt sentence:

[1924] In this system, I uploaded the following JavaScript code file: function multiply(x, y) { return xy;} Please auto-generate the specification.

[1925] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1926] Step 1:

[1927] Through the system interface, the user clicks the upload button, selects the code file and screen information (screenshots or URLs), and uploads them.

[1928] Input: Code file, screen information

[1929] Behavior: The user selects a file through a file selection dialog and initiates the upload.

[1930] Output: The device receives the selected file.

[1931] Step 2:

[1932] The device checks the code file and screen information received from the user to ensure there are no problems with the format or size, and then sends this data to the server.

[1933] Input: Code file, screen information

[1934] Action: The device checks the file format and whether it exceeds the maximum capacity. If there are no problems, it transfers the file to the server.

[1935] Output: The server receives the file and stores it temporarily.

[1936] Step 3:

[1937] The server analyzes the received code file and extracts code characteristics such as function names, arguments, return values, etc. Specifically, it analyzes the code using a parser (e.g., ANTLR).

[1938] Input: Code file

[1939] How it works: The server parses the code through a parser and extracts the function name, arguments, and return value. For example, if you parse the code function add(a, b) { return a + b;}, it extracts "function name: add", "arguments: a, b", and "return value: a + b".

[1940] Output: Code characteristics information (function names, arguments, return values, etc.)

[1941] Step 4:

[1942] The server simultaneously analyzes the received screen information and uses OCR technology (e.g., Tesseract OCR) to extract text and UI elements from the screenshot.

[1943] Input: Screen information (screenshots, URL, etc.)

[1944] How it works: The server uses OCR technology to analyze the screenshot and extract text and UI elements (input forms, buttons, etc.).

[1945] Output: Screen characteristics (text, UI elements, etc.)

[1946] Step 5:

[1947] The server integrates the code characteristics information and the screen characteristics information to automatically generate a specification document, which includes the function name, arguments, return values, and related screen information.

[1948] Input: Code characteristics information, screen characteristics information

[1949] How it works: The server combines this information and formats it into a specification. For example, it puts function names and argument information in the code section, and screen information in a separate section.

[1950] Output: Auto-generated specification

[1951] Step 6:

[1952] The server stores the automatically generated specifications in a database, assigns them an ID, and manages them for easy retrieval later.

[1953] Input: Auto-generated specification

[1954] Operation: The server assigns an ID to the specification and stores it in the database.

[1955] Output: Specification saved in database

[1956] Step 7:

[1957] A user sends a specification search request through the system interface, for example, by entering a query such as "I want to know the specification of the add function."

[1958] Input: Search query (e.g. "add function specification")

[1959] How it works: A user enters a query into the search box and clicks the search button. The device receives the query and sends it to the server.

[1960] Output: The server receives the search request.

[1961] Step 8:

[1962] The server searches the database for the relevant specifications based on the search request and provides the results to the user.

[1963] Input: Search request

[1964] How it works: The server searches the database for the query, finds the relevant specifications, and returns the search results to the device.

[1965] Output: Search results (relevant specifications)

[1966] Step 9:

[1967] The terminal displays the search results received from the server to the user.

[1968] Input: Search results from the server

[1969] Operation: The device receives the search results and displays them on the user's interface.

[1970] Output: Specification information displayed in the interface

[1971] Step 10:

[1972] The device uses an emotion engine to capture the user's facial expressions and voice and recognize their emotional state, for example, detecting whether the user is confused or happy from their facial expressions or tone of voice.

[1973] Input: User's facial expression, voice

[1974] How it works: The device captures the user's facial expressions and voice through the camera and microphone, and performs emotion analysis.

[1975] Output: User's emotional state (confused, satisfied, etc.)

[1976] Step 11:

[1977] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, it provides detailed help information. If the user is satisfied, it displays a positive message.

[1978] Input: User's emotional state

[1979] Action: The server makes appropriate user interface adjustments and feedback based on the recognized emotional state.

[1980] Output: Adjusting the interface, providing feedback (e.g., detailed help information, positive messages)

[1981] (Application example 2)

[1982] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1983] Currently, programming factory robots and creating operation specifications is done manually using complex tools, which requires a great deal of time and effort. In addition, users can become confused when operating the robots, hindering efficient work. Furthermore, real-time feedback on the inconvenience or confusion experienced by users is not provided, creating a need for improved usability.

[1984] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1985] In this invention, the server includes means for uploading code files, means for uploading screen information, means for analyzing the uploaded code files to extract code characteristics, means for analyzing the screen information to extract screen characteristics, means for automatically generating specifications based on the code files and the screen information, means for saving the automatically generated specifications, means for searching the saved specifications, means for analyzing emotional information to recognize the emotional state of a user, and means for adjusting the operation interface and providing feedback based on the emotional information. This allows users to significantly reduce the time and effort required to program factory robots, and provides real-time feedback according to emotions, improving usability and enabling more efficient and comfortable work.

[1986] A "code file" is a file in which the source code of a program is written.

[1987] "Screen information" refers to data that includes visual information about the user interface, such as screenshots and URLs.

[1988] "Analysis" is the process used to extract characteristics and structure from code files and screen information.

[1989] A "specification" is a document that describes in detail the functions and operations of software or a system.

[1990] "Emotion information" refers to data on emotions detected from the user's facial expressions, voice, etc.

[1991] An "operation interface" refers to the screen or input means that a user uses to operate a system or application.

[1992] "Feedback" refers to the system providing real-time responses to the user's actions and state.

[1993] A "server" is a computer that provides data and services to other terminals over a network.

[1994] "Optical character recognition technology" is a technology that extracts character information from an image.

[1995] MODE FOR CARRYING OUT THE INVENTION

[1996] This invention is a system that automatically generates specifications based on uploaded code files and screen information, and also combines it with an emotion engine that recognizes user emotions. This system enables efficient and accurate specification creation, and the addition of the emotion engine improves usability.

[1997] System configuration and functions

[1998] This system consists of three components: a server, a terminal, and a user. Furthermore, by integrating an emotion engine, it has been added with the function of providing appropriate feedback based on the user's emotional state.

[1999] Uploading code and screen information

[2000] The user selects the code file and related screen information (e.g., screenshots or URLs) through the system interface and performs the upload. The terminal receives the files selected by the user and sends the data to the server. This part can be a general computer or smart device.

[2001] Analysis of information

[2002] The server receives the code files and screen information sent from the device and temporarily stores them. The server then analyzes the received code files and extracts code characteristics such as function names, arguments, and return values. It also analyzes the separately received screen information. Optical character recognition (OCR) is used to extract text and UI elements (e.g., input forms, buttons) from the screenshots. This is done using software libraries such as pytesseract and OpenCV.

[2003] Emotion engine integration

[2004] The device uses an emotion engine to analyze the user's facial expressions and voice to recognize the user's emotional state. Specifically, a library called EmotionRecognizer is used. For example, it can detect whether the user is confused or happy from their facial expressions and tone of voice.

[2005] Automatic generation of specifications

[2006] The server automatically generates specifications based on the results of code and screen information analysis. Specifically, it integrates the results of code analysis (function name, arguments, return value) and screen information analysis (UI elements) and formats them into a specification format. An example of the generated specification is shown below.

[2007] Function name: move_forward

[2008] Argument: distance

[2009] Returns: None

[2010] Function Name: rotate

[2011] Argument: angle

[2012] Returns: None

[2013] Related screen: A screen with a form for entering numbers and an "Execute" button

[2014] Process flow:

[2015] 1. User enters a number into a form

[2016] 2. Press the "Run" button

[2017] 3. The functions move_forward and rotate are called and the results are displayed.

[2018] Save and search specifications

[2019] The server stores the automatically generated specifications in a database. At this time, it assigns an identification ID to the specifications and manages them so that they can be easily searched for later. The user sends a specification search request from the system interface. For example, they send a query such as "I want to know the specifications of the move_forward function." The terminal receives the query from the user and sends it to the server. The server searches the stored database for the relevant specifications and provides the results to the user.

[2020] Emotion-based feedback measures

[2021] The server automatically adjusts the operation interface based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, the server will provide more detailed help information. The server also provides appropriate feedback according to the user's emotions. For example, if the user is satisfied, the server will display a positive message.

[2022] Specific examples

[2023] Below are some examples of specific prompt sentences.

[2024] Please generate a specification for the "move_forward" function, and provide specific instructions if users are confused.

[2025] As described above, this system allows users to automatically generate efficient and accurate specifications simply by uploading code files and screen information. Furthermore, the emotion engine provides appropriate feedback based on the user's emotional state, improving usability.

[2026] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2027] Step 1:

[2028] The user selects the code file and screen information through the system interface and performs the upload. For example, the user selects the source code file of a program that controls the operation of a factory robot and screenshots of the program's operation screen. This is the input. The terminal receives the selected files and sends the data to the server.

[2029] Step 2:

[2030] The server receives the code file and screen information sent from the device and temporarily stores it. It then analyzes the received code file and extracts code characteristics such as function names, arguments, and return values. For example, it analyzes the "distance" argument and return value "None" of the "move_forward" function. This is data processing, and the analyzed information is obtained as output.

[2031] Step 3:

[2032] The server analyzes the screen information. Specifically, it uses optical character recognition (OCR) technology to extract text and UI elements from the screenshot. For example, it extracts the "numeric input form" and the text of the "Execute" button contained in the screenshot. This is the input, and the extracted text information is obtained as the output.

[2033] Step 4:

[2034] The device uses an emotion engine to recognize the user's emotional state. For example, it captures the user's facial expressions with a camera and uses the EmotionRecognizer library to detect emotions such as confusion or satisfaction. Emotion data is the input, and the recognized emotional state is the output.

[2035] Step 5:

[2036] The server automatically generates specifications based on the results of analyzing the code and screen information. Specifically, it integrates the analysis results and formats them into a specification format. For example, it documents the operation flow based on the "move_forward" function specification and screen information. This is the input, and the automatically generated specification is obtained as the output.

[2037] Step 6:

[2038] The server stores the automatically generated specifications in a database. At this time, it assigns an ID to each specification and manages it so that it can be easily searched. The specifications are input, and the state of the specifications stored in the database is obtained as output.

[2039] Step 7:

[2040] The user sends a specification search request from the system interface. For example, they send a query such as "I want to know the specifications of the move_forward function." This is the input. The terminal receives the query from the user and sends it to the server.

[2041] Step 8:

[2042] The server searches for the relevant specifications from the stored database and provides the results to the user. For example, it searches for specifications related to the "move_forward function" and returns them to the user. The query is the input, and the search results are the output.

[2043] Step 9:

[2044] The server automatically adjusts the operation interface and provides feedback based on the user's emotional state recognized by the emotion engine. For example, if the user is confused, detailed help information is displayed. The recognized emotional state is the input, and the adjusted interface is the output.

[2045] Through these steps, users can efficiently create program specifications for factory robots and receive real-time emotional feedback.

[2046] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2047] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2048] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2049] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2050] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2051] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2052] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2053] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2054] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2055] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2056] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2057] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2058] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2059] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2060] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2061] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2062] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2063] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2064] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2065] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2066] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2067] The following is further disclosed regarding the above embodiment.

[2068] (Claim 1)

[2069] A way to upload code files,

[2070] a means for uploading screen information;

[2071] means for analyzing the uploaded code file to extract code characteristics;

[2072] A means for analyzing screen information and extracting screen characteristics;

[2073] A means for automatically generating specifications based on code files and screen information;

[2074] a means for saving the automatically generated specification;

[2075] a means for retrieving the stored specifications;

[2076] A system including:

[2077] (Claim 2)

[2078] The system of claim 1, further comprising means for identifying function names, arguments, and return values ​​of the uploaded code file.

[2079] (Claim 3)

[2080] 10. The system of claim 1, further comprising means for extracting elements of the screen information using optical character recognition techniques.

[2081] "Example 1"

[2082] (Claim 1)

[2083] means for uploading the code file to an electronic computing device;

[2084] means for uploading the screen information to an electronic computing device;

[2085] means for analyzing the uploaded code file to extract code characteristics;

[2086] A means for analyzing screen information and extracting screen characteristics;

[2087] A means for automatically generating specifications based on code files and screen information;

[2088] A means for storing the automatically generated specifications in a database;

[2089] a means for retrieving the stored specifications;

[2090] A system including:

[2091] (Claim 2)

[2092] The system of claim 1, further comprising means for identifying function names, arguments, and return values ​​of the uploaded code file.

[2093] (Claim 3)

[2094] 10. The system of claim 1, further comprising means for extracting elements of the screen information using optical character recognition techniques.

[2095] (Claim 4)

[2096] 2. The system according to claim 1, further comprising means for displaying a progress status and notifying a user when analyzing the code file and the screen information.

[2097] (Claim 5)

[2098] 2. The system according to claim 1, wherein the means for automatically generating the specifications includes means for applying the analysis results to a template to generate the specifications.

[2099] "Application Example 1"

[2100] (Claim 1)

[2101] A way to upload code files,

[2102] a means for uploading screen information;

[2103] means for analyzing the uploaded code file to extract code characteristics;

[2104] A means for analyzing screen information and extracting screen characteristics;

[2105] A means for automatically generating specifications based on code files and screen information;

[2106] a means for saving the automatically generated specification;

[2107] a means for retrieving the stored specifications;

[2108] A means for displaying the stored specifications on a smart device for maintenance and repair of industrial equipment;

[2109] A system including:

[2110] (Claim 2)

[2111] The system of claim 1, further comprising means for identifying function names, arguments, and return values ​​of the uploaded code file.

[2112] (Claim 3)

[2113] 10. The system of claim 1, further comprising means for extracting elements of the screen information using optical character recognition techniques.

[2114] "Example 2: Combining Emotion Engines"

[2115] (Claim 1)

[2116] a means for users to upload code files;

[2117] a means for uploading screen information;

[2118] means for analyzing the uploaded code file to extract code characteristics;

[2119] A means for analyzing screen information and extracting screen characteristics;

[2120] A means for automatically generating specifications based on code files and screen information;

[2121] a means for saving the automatically generated specification;

[2122] a means for retrieving the stored specifications;

[2123] A means for recognizing an emotional state of a user by analyzing the user's facial expressions and voice;

[2124] means for automatically adjusting the operating interface based on the recognized emotional state;

[2125] A system including:

[2126] (Claim 2)

[2127] 2. The system according to claim 1, wherein the means for recognizing the emotional state includes means for capturing the user's facial expressions and voice and performing emotion analysis.

[2128] (Claim 3)

[2129] 2. The system of claim 1, further comprising means for identifying the function name, arguments, and return values ​​of the uploaded code file.

[2130] (Claim 4)

[2131] 2. The system according to claim 1, further comprising means for extracting elements of the screen information using optical character recognition technology.

[2132] (Claim 5)

[2133] 10. The system of claim 1, further comprising means for providing appropriate feedback based on the recognized emotional state.

[2134] "Application example 2 when combining emotion engines"

[2135] (Claim 1)

[2136] A way to upload code files,

[2137] a means for uploading screen information;

[2138] means for analyzing the uploaded code file to extract code characteristics;

[2139] A means for analyzing screen information and extracting screen characteristics;

[2140] A means for automatically generating specifications based on code files and screen information;

[2141] a means for saving the automatically generated specification;

[2142] a means for retrieving the stored specifications;

[2143] a means for analyzing emotional information to recognize the emotional state of a user;

[2144] means for adjusting an operation interface and providing feedback based on the emotion information;

[2145] A system including:

[2146] (Claim 2)

[2147] The system of claim 1, further comprising means for identifying function names, arguments, and return values ​​of the uploaded code file.

[2148] (Claim 3)

[2149] 10. The system of claim 1, further comprising means for extracting elements of the screen information using optical character recognition techniques. [Explanation of symbols]

[2150] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A way to upload code files, a means for uploading screen information; means for analyzing the uploaded code file to extract code characteristics; A means for analyzing screen information and extracting screen characteristics; A means for automatically generating specifications based on code files and screen information; a means for saving the automatically generated specification; a means for retrieving the stored specifications; A system including:

2. The system of claim 1 further comprising means for identifying function names, arguments, and return values ​​of the uploaded code file.

3. 2. The system of claim 1, further comprising means for extracting elements of said screen information using optical character recognition techniques.

Images