system

Abstract

Provide a system. 【Solution means】 Means for receiving a code, Means for analyzing the received code and generating visualization data, Means for transmitting the generated visualization data to a display device, Means for operating the visualization data via a user interface, Means for regenerating the original code based on the operated visualization data, Means for providing the regenerated code to the user, Means for the user to visually change the working process of the mechanical device, A system including.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For users with little programming experience, it is difficult to directly edit and understand code. Also, the maintenance and management of tools described in different programming languages are difficult, and the handover during changes becomes a factor increasing man-hours and stress. As a result, the efficient operation of tools within an organization is hindered. The purpose of this invention is to solve these problems by visualizing programming code and facilitating operations.

Means for Solving the Problems

[0005] This invention provides a system that receives code, analyzes the received code, and generates visualization data. Furthermore, this system includes means for transmitting the generated visualization data to a display device and manipulating the visualization data via a user interface. By automatically regenerating the original code based on the results of the user's visual manipulation and providing it to the user, it becomes possible to intuitively edit and understand the code even without programming expertise. This reduces the handover time during personnel changes and supports the efficient operation of the tool throughout the organization.

[0006] "Code" refers to the instructions and data that make up a computer program; it is created by humans and designed to be executed by computers.

[0007] "Means of receiving" refers to a function or device for receiving data, signals, or information transmitted from an external source.

[0008] "Means of analysis" refers to a function or device for processing received information and understanding its structure and content.

[0009] "Visualized data" refers to data that converts digital information into a format that is easy to understand visually, providing users with visual feedback.

[0010] A "generative means" is a function or device for creating new data or structures based on specific rules or algorithms.

[0011] A "display device" is hardware or software that visually represents digital data or information, allowing users to view and understand it.

[0012] A "user interface" is an interface for exchanging information between a computer system or application and a human being, and a means of operation.

[0013] "Means of operation" refers to a function or device for controlling or managing a system or device.

[0014] "Means of regeneration" refers to a function or device for converting existing data or information back into its original format or a new format.

[0015] "Means of provision" refers to functions or devices that provide or make available necessary information or functions to users. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12]It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the language used in the following description will be explained.

[0019] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a RAM (Random Access Memory) with a reference numeral is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention is implemented as a system that provides an environment for visually manipulating programming code. This system consists of server software operating under specific hardware conditions, a user terminal interface, and a generative AI engine.

[0038] The server receives program code submitted by the user. This receiving process takes place, for example, through a web form or API. The received code is first stored in an integrated database, and then proceeds to the analysis process.

[0039] The analysis uses a generative AI engine. The server passes the received code to the generative AI engine, which performs semantic analysis of the code. Here, each element of the code, such as functions, variables, and control flow, is recognized and converted into data for visualization. This visualized data becomes intermediate data to be provided to the user.

[0040] Next, the server sends this visualization data to the user's terminal. The user's terminal has a dedicated interface that, upon receiving the visualization data, presents the user with a visual representation of the program's flow and structure. This interface is designed to be easy for the user to understand and has an intuitive user interface (UI).

[0041] Users can interact with visual programs provided on their devices. For example, they can use visual blocks to modify the control flow of code or add new conditional branches and loops. This allows users to understand and edit the program's logic without needing to understand the intricacies of programming.

[0042] Once the user has finished editing, their device sends the modified visualization data to the server. The server then automatically regenerates the code in the original programming language based on this visualization data. The regenerated code accurately reflects the visual edits made to the original code, and the user can review and use it.

[0043] This series of processes allows users to easily operate and edit tools, overcoming the barriers of different programming languages, and enables a smooth transition of tools during internal transfers. This invention is expected to significantly improve the efficiency of program management within organizations and reduce technical barriers.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The user enters program code and presses the submit button, sending the code from the terminal. The terminal then sends the entered code to the server in the correct format.

[0047] Step 2:

[0048] The server receives program code sent by the user. The received data is temporarily stored in a database on the server.

[0049] Step 3:

[0050] The server sends the received code to a generation AI engine. This engine analyzes the structural elements of the code and converts them into visualized data.

[0051] Step 4:

[0052] The generative AI engine analyzes the program code and generates visually representable views and models. In this process, various elements of the program structure, such as functions, variables, and conditional expressions, are extracted.

[0053] Step 5:

[0054] The server sends the visualization data obtained from the generated AI to the user's device. The visualization data is packaged in a format suitable for the intuitive UI components that the user interacts with.

[0055] Step 6:

[0056] The system displays the visualized data received by the terminal on the user interface. This allows the user to intuitively manipulate the program's logic in a visualized form.

[0057] Step 7:

[0058] The user edits the code based on the displayed visualization objects. The user adds, deletes, rearranges, and changes the attributes of blocks to modify the program's logic.

[0059] Step 8:

[0060] The user completes editing and sends the changes to the server. The terminal repackages the edited visualization data and sends it to the server again.

[0061] Step 9:

[0062] The server regenerates the code in the original programming language based on the edited visualization data received from the user.

[0063] Step 10:

[0064] The server provides the user with the regenerated code. The user can review this code and download or run it as needed.

[0065] (Example 1)

[0066] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0067] There is a need to provide an environment where users with little programming experience can easily understand, visually manipulate, and edit code written in various programming languages. Furthermore, the challenge lies in achieving real-time visualization of editing and efficient code regeneration.

[0068] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0069] In this invention, the server includes means for receiving information, means for analyzing the received information and converting the data into visual information, and means for transmitting the converted visual information to a device for display. This allows the user to easily understand the program's logic and operate it intuitively.

[0070] "Information" is a general term that refers to the various data and codes processed within a system.

[0071] "Means of receiving" refers to the methods and technologies used by the server to retrieve information provided by the user.

[0072] "Means of analysis and conversion of data into visual information" refers to the process of analyzing received code or information and converting it into a format that visualizes its structure and content.

[0073] "Means for transmitting to a device for display" refers to technologies and methods for transferring converted visual information to a device accessible to the user.

[0074] "Means of manipulating visual information using a user interface" refers to an interface for editing and modifying visualized data in a way that users can directly interact with.

[0075] "Means of regenerating original information based on visual information" refers to the process of reconstructing the original program code or information based on changes made by the user using visualized information.

[0076] "Means of providing regenerated information" refers to methods for providing reconstructed code and information in a form that users can use.

[0077] "Analysis capabilities that support multiple programming languages" refers to a function that has the ability to understand and analyze code written in different programming languages.

[0078] This invention is a system that provides an environment in which users can easily manipulate and visually understand program code. The hardware used consists of a server and a terminal for the user, which communicate via a network. As the main software component, the server manages the process of receiving information, analyzing the data using a generative AI model, and visually transforming it. The analyzed data is sent to the user's terminal in a visually easy-to-understand format. A dedicated user interface is provided on the terminal side, allowing the user to manipulate the visual representation and intuitively understand and edit the program's logic.

[0079] For example, a user might want to improve the inventory management system of an existing e-commerce application. This system allows the user to visually add new logic to automatically send notifications when inventory levels drop. Loops and conditional branches can be easily added through a visual interface, and the resulting new code can be viewed in real time.

[0080] This system, which utilizes a generative AI model, allows users to input prompts such as: "Add a conditional branch to the existing code and visualize the logic that notifies when inventory falls below a certain number." This prompt helps the AI ​​visually realize the program according to the user's wishes.

[0081] This method allows users to easily improve systems and add new features, overcoming the barriers between different programming languages, even without specialized programming knowledge. This invention is expected to reduce technical barriers and significantly improve the efficiency of program management within organizations.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The server receives program code from the user. The input is code provided by the user through web forms or APIs. The server receives this code and temporarily stores it in an integrated database. This prepares the data for robust management for subsequent analysis.

[0085] Step 2:

[0086] The server passes the received code to the generating AI model to begin analysis. The input is stored program code, and the output is visually transformed data. The generating AI model interprets the code, identifies the function's behavior and control flow, and generates data in a format that visualizes the code's structure. In this process, it identifies each element within the code and transforms it into a form that is easy for the user to understand visually.

[0087] Step 3:

[0088] The server sends the generated visualization data to the user's terminal. The input is the visualized data, and the output is its display on the user's terminal. The server transfers this data to the user's terminal, making it ready for user interaction.

[0089] Step 4:

[0090] The user's device interacts with the user through displayed visualization data. The user interface receives visualization data as input and presents it to the user visually as output. Based on this, the user edits the program's logic using mouse or touch gestures. They can add new conditional branches and loops, or adjust existing operations.

[0091] Step 5:

[0092] After completing visual changes, the user sends the updated information to the server. The input is the manipulated visualization data, and the output is the updated data. The user's terminal accurately transmits the changes to the server, which then prepares to reflect them in the original code.

[0093] Step 6:

[0094] The server regenerates the original program code based on the change information sent to the user. The input is the manipulated visualization data, and the final output is the regenerated program code. The server reconstructs the code based on this edit to ensure it is properly reflected in the original programming language. The user can then download and actually use / verify the updated program.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] In factories and manufacturing environments, there is a challenge in the lack of sufficient means to easily and intuitively operate and edit the operating programs of machinery and equipment. In particular, it is difficult for users without specialized programming knowledge to efficiently optimize the movements of robots.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes means for receiving a code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to a user, and means for the user to visually modify the work process of a machine. This makes it possible for a user to effectively modify and optimize the operation of a machine without specialized knowledge.

[0100] "Means for receiving code" refers to a device or method that has the function of receiving program code data transmitted from an external source and holding it for processing.

[0101] "Means for generating visualization data" refers to a device or method that has the function of analyzing received program code and converting it into data for visually representing its structure and operation.

[0102] "Means for transmitting to a display device" refers to a device or method that has the function of transmitting generated visualization data to an output device in a format that can be understood by the user.

[0103] "Means of operation via a user interface" refers to a device or method that has the function of providing a screen or input means for users to intuitively manipulate and edit visualized data.

[0104] "Means for regenerating the original code" refers to a device or method that has the function of converting user-operated and edited visualization data into program code that can be executed by humans or machines.

[0105] "Means for providing regenerated code" refers to a device or method that has the function of presenting regenerated program code in a format accessible to the user.

[0106] "Means for visually modifying the work process of a machine or device" refers to a device or method that has the function of allowing a user to visually observe the operating procedure or flow of a machine or device and adjust or modify it as needed.

[0107] To implement this invention, a system is required that combines a server, a user terminal, and a generative AI engine. The server first receives program code sent by the user. The received code is stored in a database and then analyzed using the generative AI engine. In this analysis, functions, variables, control flow, etc., within the code are recognized and converted into visualized data. The visualized data is structured in an intermediate format and sent to the user terminal.

[0108] The user terminal hardware includes smartphones and tablets, equipped with a dedicated user interface. The software used here includes HTML / CSS and JavaScript (registered trademark) for UI design. Through this interface, users can view the program flow and structure as visualized data and edit the program using blocks. This allows users without programming knowledge to intuitively understand the program's logic and make changes.

[0109] The edited visualization data is sent back to the server and regenerated into code in the original programming language. The generated code is then provided to the user in a state where it can be accessed for further manipulation and verification. This entire process allows users to visually modify the operation of machinery and optimize work processes. For example, when optimizing the operation of a robot on an automotive parts manufacturing line, the order and method of each process can be intuitively changed.

[0110] By utilizing a generative AI model, advanced analysis is performed using prompt messages. By inputting prompt messages such as, "Visually represent the assembly process of this factory and find the optimization points," users can visually obtain the results of analyzing complex code.

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The server receives program code from users. It accepts code submitted by users via APIs or web forms as input and securely stores it in data storage. The data processing performed here involves converting the code into an appropriate format, preparing it for analysis in subsequent stages of the process. The output is a reference to the stored code.

[0114] Step 2:

[0115] The server passes the received code to the generative AI engine for analysis. At this stage, the code reference saved in step 1 is used as input. The generative AI model is used to identify elements such as functions, variables, and control flow, and performs data calculations to generate visualization data. As output, data is created that visually represents each part of the code.

[0116] Step 3:

[0117] The server sends the generated visualization data to the user terminal. The visualization data generated in step 2 is used as input. The data is serialized into an appropriate format (e.g., JSON) and processed to make it viewable on the terminal's display device. The output is in a data format easily processed by the user terminal.

[0118] Step 4:

[0119] The user manipulates the visualization data through the terminal's user interface. The terminal uses the visualization data received in step 3 as input. The user performs specific actions such as dragging and dropping blocks to change the program flow or add new elements. The output is the modified visualization data.

[0120] Step 5:

[0121] The user terminal sends the modified visualization data to the server. The server receives the visualization data edited by the user in step 4 as input. The server analyzes the visualization data and performs data calculations to regenerate program code that reflects the changes. The output is the regenerated program code.

[0122] Step 6:

[0123] The regenerated code is provided to the user. The code generated in step 5 is used as input. The server distributes this code for user access and manages the data to ensure it is ultimately verifiable and usable by the user. As output, the complete program code is provided, available through the user interface.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention provides a program editing environment that takes into account the user's emotional state by incorporating an emotion engine into a system that visually manipulates program code. This system consists of server software, a user terminal interface, a generation AI engine, and an emotion engine.

[0126] First, the user's terminal receives the program code and sends it to the server. The server receives this code and performs analysis using a generation AI engine. As a result of the analysis, visualization data representing the logical structure of the program is generated, and this data is sent from the server to the user's terminal.

[0127] The user's terminal displays the generated visualization data on the user interface. This allows the user to visually confirm and manipulate the program flow. These operations include adding, deleting, and rearranging blocks.

[0128] The emotion engine recognizes the user's emotional state in real time by analyzing their facial expressions and voice. Based on this emotional state, the system dynamically adjusts the displayed content and user interface responses. For example, if the user is feeling stressed, the system may simplify interactions or display encouraging messages.

[0129] After the user visually edits the program again, the changes are sent from the terminal to the server. The server creates regenerated code based on the edits and provides it to the user. The user can then review and execute the regenerated code.

[0130] For example, suppose a user is struggling for a long time because they misunderstand a loop in a program. The emotion engine detects the user's frustration, and the system provides concrete examples of the loop structure and hints for solving it. In this way, the system provides a more intuitive and supportive programming environment, reducing the burden on the user.

[0131] The following describes the processing flow.

[0132] Step 1:

[0133] The user enters program code into the interface using their own device and presses the submit button. This sends the code from the device to the server.

[0134] Step 2:

[0135] The server receives program code sent by the user. First, the server sends the code to a generative AI engine to understand its structure.

[0136] Step 3:

[0137] The generation AI engine analyzes the received code. It identifies functions, variables, and control structures within the code and generates data for visual representation.

[0138] Step 4:

[0139] The server sends visualization data obtained from the AI ​​generation engine to the user's terminal. This visualization data is intended to show the user the structure of the program.

[0140] Step 5:

[0141] The terminal displays the received visualization data on the screen. Based on this visualization data, the user can visually understand and operate the program's flow and structure.

[0142] Step 6:

[0143] The emotion engine runs on the user's device, analyzing the user's facial expressions and voice through the camera and microphone while the user interacts with the program. The user's emotional state is monitored in real time.

[0144] Step 7:

[0145] Users edit the program using a visual interface. For example, they can add new conditional branches or adjust existing loops.

[0146] Step 8:

[0147] When the emotion engine detects user stress or frustration, the device provides corresponding interface support functions. Specifically, it displays user-friendly hints and advice.

[0148] Step 9:

[0149] The user completes the editing process, and the device sends the changes to the server. The server receives this information and constructs regenerated code based on the edited visualization data.

[0150] Step 10:

[0151] The server returns the regenerated code to the user. The user reviews the new code and makes further edits or runs it as needed.

[0152] (Example 2)

[0153] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0154] In recent years, editing program code has become increasingly complex, making it a stressful process, especially for beginners and users unfamiliar with programming. Furthermore, the lack of graphical user interfaces that reflect the user's emotional state leads to inefficient support.

[0155] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0156] In this invention, the server includes means for receiving code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to the user, and means for analyzing emotions and adjusting the interface according to the user's emotional state. This makes it possible to provide a flexible program editing environment that is sensitive to the user's emotions and to reduce the user's stress and burden.

[0157] "Code" is text that describes instructions and commands to be executed by a computer.

[0158] "Analysis" is the process of breaking down received code and understanding its structure and function.

[0159] "Visualized data" refers to information that visually represents the logical structure of a program, converted into a format that users can easily understand and manipulate.

[0160] A "display device" is hardware or software that provides visualized data to the user in a visible format.

[0161] A "user interface" is a mechanism that provides users with a means to interact with a system and manipulate visualized data.

[0162] "Emotions" refer to the psychological state extracted from the user's facial expressions and voice, and are detected by the system.

[0163] "Interface adjustment" refers to dynamically changing the display and functionality of the user interface based on the user's emotional state.

[0164] A "programming language" is a standardized language used to give instructions to a computer, and there are many different types.

[0165] "Code regeneration" is the process of regenerating the original code based on the manipulated visualization data.

[0166] This invention is a system that provides support to users while they visually edit a program, taking into account their emotional state. The system mainly consists of server software, a user terminal, a generative AI engine, and an emotion engine.

[0167] The user's terminal has an interface for inputting program code. Once the user inputs code, the terminal sends it to the server. The server uses a generative AI engine to analyze the received code. This analysis generates the program's logical structure as visualized data. The server then sends this generated visualization data to the user's terminal.

[0168] The terminal displays visualized data on the user interface. This allows the user to visually confirm the program flow and manipulate it freely. These operations include adding, deleting, and rearranging program blocks.

[0169] The emotion engine can analyze the user's facial expressions and voice in real time to recognize their emotional state. Based on this emotional state, the device dynamically adjusts the content and responses of the user interface. For example, if the user is feeling frustrated, the system can simplify the interface and display encouraging messages.

[0170] After editing is complete, the user's terminal sends the changes to the server. The server regenerates the original code according to the changes and provides the result to the user. The user can then review and execute this regenerated code.

[0171] For example, if a user prompts with "I don't know how to use a for loop," the system automatically presents a concrete example of a loop structure to help the user understand. In this way, the system provides an intuitive and emotionally resonant programming environment.

[0172] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0173] Step 1:

[0174] The user's terminal accepts code through an interface for entering program code. The user begins by entering code into this interface. The entered code is processed as data in a format that is sent directly to the server. Specifically, if the user enters the code "I don't know how to use a for loop," the terminal will transfer that information to the server.

[0175] Step 2:

[0176] The server analyzes the received code using a generative AI engine. Here, the server analyzes the input code line by line, determining its structure and purpose. Data calculations include code tokenization and syntax tree generation. The output is data that visualizes the logical structure of the code. This visualization diagrams the structure like a program flowchart.

[0177] Step 3:

[0178] The server sends the generated visualization data to the user's terminal. The terminal displays the received visualization data on its user interface. Specifically, the visualized program flowchart or blocks are displayed on the screen, ready for the user to review.

[0179] Step 4:

[0180] Users manipulate the programming flow on the interface based on visualized data. Here, they can perform specific actions such as adding, deleting, and rearranging blocks. Based on the user's input, the terminal processes the data and generates the manipulated visualized data. This allows users to intuitively build and improve the program's logic.

[0181] Step 5:

[0182] The device uses an emotion engine to analyze the user's voice and facial expressions to determine their emotional state. If the user is feeling anxious, the device adjusts the data to display encouraging messages or simplified instructions on the screen. Specifically, the on-screen instructions and guidance change to clearly show the user what to do next.

[0183] Step 6:

[0184] After the user's actions are finalized, the terminal sends this information to the server. The server then regenerates the original code based on the manipulated visualization data. The input is the modified data resulting from the user's actions, and after data calculations, the new code is output.

[0185] Step 7:

[0186] The server sends the regenerated code to the user's terminal. The user then verifies this regenerated code. Specifically, the newly generated program code is displayed on the screen, allowing the user to review its contents.

[0187] Step 8:

[0188] The user executes the code they have reviewed. This allows them to experience the actual program's operation and verify the results of the visualized program.

[0189] (Application Example 2)

[0190] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0191] Modern programming environments often fail to consider the user's operational capabilities or emotional state, leading to frustration and stress during code editing. Furthermore, understanding and manipulating complex programs requires advanced knowledge, necessitating additional support. This invention aims to address these challenges by understanding the user's emotional state in real time and providing appropriate interface adjustments and support accordingly.

[0192] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0193] In this invention, the server includes means for receiving code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to the user, means for analyzing the user's emotional state, and means for dynamically adjusting the display content based on the user's emotional state. This allows the user to receive optimal support according to their emotional state and reduces the burden during program editing.

[0194] "Means of receiving code" refers to the function by which the system receives program code entered by the user.

[0195] "Means for analyzing received code and generating visualization data" refers to a function that analyzes the received code and creates data to visually represent its logical structure.

[0196] "Means for transmitting generated visualization data to a display device" refers to a function that transmits the generated visualization data to the user's terminal, making it displayable on a display device such as a screen.

[0197] "Means of manipulating visualized data through a user interface" refers to a function that provides an interface that allows users to edit a program through visualized data.

[0198] "Means for regenerating the original code based on manipulated visualization data" refers to a function that rewrites the original program code based on the visual operations performed by the user.

[0199] "Means of providing regenerated code to users" refers to a function that presents and allows users to use the newly generated code.

[0200] "Means for analyzing the user's emotional state" refers to a function that analyzes the user's facial expressions, voice, etc., to understand their emotional state in real time.

[0201] "Means for dynamically adjusting displayed content based on the user's emotional state" refers to a function that appropriately changes the user interface and displayed messages according to the analyzed emotional state, thereby improving the user's experience.

[0202] This invention provides a system that dynamically adjusts the programming environment according to the user's emotional state. The system mainly consists of a server, a user terminal, a generative AI engine, and an emotion engine.

[0203] The server receives the code entered by the user, analyzes it with the help of a generation AI engine, and generates visualization data. This visualization data is designed to allow the user to intuitively understand the structure and flow of the program. This data is sent from the server to a display device and displayed on the user's terminal.

[0204] The user terminal can be, for example, a smartphone or tablet. Through the terminal's user interface, the user can manipulate the visualized data of the program code. Specifically, they can edit it by adding, deleting, and rearranging blocks.

[0205] The emotion engine analyzes the user's facial expressions and voice in real time to understand their emotional state. Based on this emotional data, the system dynamically adjusts the content and interactions of the user interface. For example, if the user is feeling stressed, the system can simplify operations or display encouraging messages.

[0206] As a concrete example, consider a situation where a user is visually editing the robot's operation program to control a robot in their home. If the system detects that the user is confused, the emotion engine communicates this to the server, which then provides step-by-step guidance and specific operation suggestions to the user's terminal.

[0207] This system allows users to receive support tailored to their emotional state, resulting in a more comfortable and efficient programming experience.

[0208] An example of a prompt to input into the generating AI model is: "The user is trying to set a cleaning schedule for a robot, but is showing signs of confusion during the process. Based on sentiment analysis, suggest how to provide support."

[0209] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0210] Step 1:

[0211] The server receives program code entered from the user's terminal. The input is the program code entered by the user on the terminal. Upon receiving this code, the server prepares to proceed to the next analysis process.

[0212] Step 2:

[0213] The server uses a generation AI engine to analyze the received program code. The input is the program code received in step 1. As a result of the analysis, it generates data to visualize the logical structure of the code. This visualized data is the output.

[0214] Step 3:

[0215] The server sends the generated visualization data to the user's terminal. The terminal receives this visualization data as input and prepares to display it on the user interface. As a result, the output visualization data becomes available for viewing on the terminal.

[0216] Step 4:

[0217] The user manipulates the visualized data through the terminal's user interface. The input is the visualized data displayed on the terminal, and the user performs operations such as adding, deleting, and rearranging blocks based on this data. The results of the user's operations are passed on to the next step.

[0218] Step 5:

[0219] The terminal reshapes the visualization data based on user actions and sends that data to the server. The input is the visualization data after user actions, and the output is the instruction for regeneration.

[0220] Step 6:

[0221] The server regenerates the original program code based on the user's re-formed visualization data. The input is the data sent in step 5, and the output is the regenerated program code.

[0222] Step 7:

[0223] The server provides the user with regenerated program code. The user reviews this code on their terminal and executes it if necessary. The input is the regenerated code, and the output is information for the user to review.

[0224] Step 8:

[0225] The emotion engine analyzes the user's facial expressions and voice to understand their emotional state. Input consists of audio and image data obtained through the camera and microphone. Based on these results, it outputs the user's emotional state.

[0226] Step 9:

[0227] The server adjusts the user interface display based on the user's emotional state, which is analyzed by the emotion engine. The input is the user's emotional state, and the output is the adjusted interface display and interaction content.

[0228] As a concrete example of its operation, if a user is experiencing stress while editing a program, the emotion engine can detect this state, and the server can instruct the interface to simplify the operation. In this case, a generative AI model is used to provide guidance that includes prompts that are easy for the user to understand. A sample prompt could be: "The user is trying to set a cleaning schedule for the robot, but is showing signs of confusion in the process. Based on emotion analysis, please suggest how to provide support."

[0229] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0230] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0231] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0232] [Second Embodiment]

[0233] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0234] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0235] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0236] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0237] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0238] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0239] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0240] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0241] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0242] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0243] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0244] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0245] This invention is implemented as a system that provides an environment for visually manipulating programming code. This system consists of server software operating under specific hardware conditions, a user terminal interface, and a generative AI engine.

[0246] The server receives program code submitted by the user. This receiving process takes place, for example, through a web form or API. The received code is first stored in an integrated database, and then proceeds to the analysis process.

[0247] The analysis uses a generative AI engine. The server passes the received code to the generative AI engine, which performs semantic analysis of the code. Here, each element of the code, such as functions, variables, and control flow, is recognized and converted into data for visualization. This visualized data becomes intermediate data to be provided to the user.

[0248] Next, the server sends this visualization data to the user's terminal. The user's terminal has a dedicated interface that, upon receiving the visualization data, presents the user with a visual representation of the program's flow and structure. This interface is designed to be easy for the user to understand and has an intuitive user interface (UI).

[0249] Users can interact with visual programs provided on their devices. For example, they can use visual blocks to modify the control flow of code or add new conditional branches and loops. This allows users to understand and edit the program's logic without needing to understand the intricacies of programming.

[0250] Once the user has finished editing, their device sends the modified visualization data to the server. The server then automatically regenerates the code in the original programming language based on this visualization data. The regenerated code accurately reflects the visual edits made to the original code, and the user can review and use it.

[0251] This series of processes allows users to easily operate and edit tools, overcoming the barriers of different programming languages, and enables a smooth transition of tools during internal transfers. This invention is expected to significantly improve the efficiency of program management within organizations and reduce technical barriers.

[0252] The following describes the processing flow.

[0253] Step 1:

[0254] The user enters program code and presses the submit button, sending the code from the terminal. The terminal then sends the entered code to the server in the correct format.

[0255] Step 2:

[0256] The server receives program code sent by the user. The received data is temporarily stored in a database on the server.

[0257] Step 3:

[0258] The server sends the received code to a generation AI engine. This engine analyzes the structural elements of the code and converts them into visualized data.

[0259] Step 4:

[0260] The generative AI engine analyzes the program code and generates visually representable views and models. In this process, various elements of the program structure, such as functions, variables, and conditional expressions, are extracted.

[0261] Step 5:

[0262] The server sends the visualization data obtained from the generated AI to the user's device. The visualization data is packaged in a format suitable for the intuitive UI components that the user interacts with.

[0263] Step 6:

[0264] The system displays the visualized data received by the terminal on the user interface. This allows the user to intuitively manipulate the program's logic in a visualized form.

[0265] Step 7:

[0266] The user edits the code based on the displayed visualization objects. The user adds, deletes, rearranges, and changes the attributes of blocks to modify the program's logic.

[0267] Step 8:

[0268] The user completes editing and sends the changes to the server. The terminal repackages the edited visualization data and sends it to the server again.

[0269] Step 9:

[0270] The server regenerates the code in the original programming language based on the edited visualization data received from the user.

[0271] Step 10:

[0272] The server provides the user with the regenerated code. The user can review this code and download or run it as needed.

[0273] (Example 1)

[0274] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0275] There is a need to provide an environment where users with little programming experience can easily understand, visually manipulate, and edit code written in various programming languages. Furthermore, the challenge lies in achieving real-time visualization of editing and efficient code regeneration.

[0276] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0277] In this invention, the server includes means for receiving information, means for analyzing the received information and converting the data into visual information, and means for transmitting the converted visual information to a device for display. As a result, the user can easily understand the logic of the program and operate intuitively.

[0278] "Information" is a general term referring to various data and codes processed within the system.

[0279] "Means for receiving" refers to the method and technology for the server to capture the information provided by the user.

[0280] "Means for analyzing and converting data into visual information" refers to the process of analyzing the received code or information and converting it into a form that visualizes its structure and content.

[0281] "Means for transmitting to a device for display" refers to the technology and method for transferring the converted visual information to a device accessible by the user.

[0282] "Means for operating visual information using a user interface" refers to the interface for editing and changing the visualized data in a form directly operable by the user.

[0283] "Means for regenerating the original information based on visual information" refers to the process of reconstructing the original program code or information based on the changes made by the user to the visualized information.

[0284] "Means for providing the regenerated information" refers to the method for providing the reconstructed code or information in a form usable by the user.

[0285] "Analysis function capable of supporting multiple programming languages" refers to the function having the ability to understand and analyze codes in different programming languages.

[0286] This invention is a system for providing an environment where users can easily operate program code and visually understand it. The hardware used includes servers and terminals for users, and these communicate via a network. As the main software components, the server manages the process of receiving information, analyzing data using a generative AI model, and visually converting it. The analyzed data is sent to the user's terminal in a visually understandable form. On the terminal side, a dedicated user interface is provided, and users can operate the visual representation and intuitively understand and edit the program logic.

[0287] As a specific example, a user may want to improve the inventory management system of an existing e-commerce application. Using this system, the user can visually add new logic that automatically sends notifications when the inventory decreases. Loops and conditional branches can be easily added through the visual interface, and the newly generated code can be confirmed in real time.

[0288] In this system that utilizes a generative AI model, examples like the following can be input as prompt texts. "Add the logic to notify when the inventory falls below a specific number as a conditional branch to the existing code and visualize it." This prompt text enables the AI to support the visual realization of the program according to the user's wishes.

[0289] By such a method, even without specialized programming knowledge, users can easily improve the system or add new functions across the barriers of different programming languages. This invention is expected to reduce technical barriers and significantly improve the efficiency of program management within an organization.

[0290] The flow of a specific process in Example 1 will be described using FIG. 11.

[0291] Step 1:

[0292] The server receives program code from the user. The input is code provided by the user through web forms or APIs. The server receives this code and temporarily stores it in an integrated database. This prepares the data for robust management for subsequent analysis.

[0293] Step 2:

[0294] The server passes the received code to the generating AI model to begin analysis. The input is stored program code, and the output is visually transformed data. The generating AI model interprets the code, identifies the function's behavior and control flow, and generates data in a format that visualizes the code's structure. In this process, it identifies each element within the code and transforms it into a form that is easy for the user to understand visually.

[0295] Step 3:

[0296] The server sends the generated visualization data to the user's terminal. The input is the visualized data, and the output is its display on the user's terminal. The server transfers this data to the user's terminal, making it ready for user interaction.

[0297] Step 4:

[0298] The user's device interacts with the user through displayed visualization data. The user interface receives visualization data as input and presents it to the user visually as output. Based on this, the user edits the program's logic using mouse or touch gestures. They can add new conditional branches and loops, or adjust existing operations.

[0299] Step 5:

[0300] After completing visual changes, the user sends the updated information to the server. The input is the manipulated visualization data, and the output is the updated data. The user's terminal accurately transmits the changes to the server, which then prepares to reflect them in the original code.

[0301] Step 6:

[0302] The server regenerates the original program code based on the change information sent to the user. The input is the manipulated visualization data, and the final output is the regenerated program code. The server reconstructs the code based on this edit to ensure it is properly reflected in the original programming language. The user can then download and actually use / verify the updated program.

[0303] (Application Example 1)

[0304] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0305] In factories and manufacturing environments, there is a challenge in the lack of sufficient means to easily and intuitively operate and edit the operating programs of machinery and equipment. In particular, it is difficult for users without specialized programming knowledge to efficiently optimize the movements of robots.

[0306] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0307] In this invention, the server includes means for receiving code, means for analyzing the received code to generate visualization data, means for transmitting the generated visualization data to a display device, means for operating the visualization data via a user interface, means for regenerating the original code based on the operated visualization data, means for providing the regenerated code to the user, and means for the user to visually change the working process of the mechanical device. As a result, even without specialized knowledge, the user can effectively change and optimize the operation of the mechanical device.

[0308] The "means for receiving code" is a device or method having the function of receiving program code data transmitted from the outside and holding it for processing.

[0309] The "means for generating visualization data" is a device or method having the function of analyzing the received program code and converting it into data for visually expressing its structure and operation.

[0310] The "means for transmitting to a display device" is a device or method having the function of transmitting the generated visualization data to an output device in a form understandable by the user.

[0311] The "means for operating via a user interface" is a device or method having the function of providing a screen and input means for the user to intuitively operate and edit the visualization data.

[0312] The "means for regenerating the original code" is a device or method having the function of converting the visualization data operated and edited by the user into a program code executable by humans or machines.

[0313] The "means for providing the regenerated code" is a device or method having the function of presenting the regenerated program code in a form accessible to the user.

[0314] "Means for visually modifying the work process of a machine or device" refers to a device or method that has the function of allowing a user to visually observe the operating procedure or workflow of a machine or device and to adjust or modify it as necessary.

[0315] To implement this invention, a system is required that combines a server, a user terminal, and a generative AI engine. The server first receives program code sent by the user. The received code is stored in a database and then analyzed using the generative AI engine. In this analysis, functions, variables, control flow, etc., within the code are recognized and converted into visualized data. The visualized data is structured in an intermediate format and sent to the user terminal.

[0316] The user terminal hardware includes smartphones and tablets, equipped with a dedicated user interface. The software used here includes HTML / CSS and JavaScript for UI design. Through this interface, users can view the program flow and structure presented as visualized data and edit the program using blocks. This allows users without programming knowledge to intuitively understand the program's logic and make changes.

[0317] The edited visualization data is sent back to the server and regenerated into code in the original programming language. The generated code is then provided to the user in a state where it can be accessed for further manipulation and verification. This entire process allows users to visually modify the operation of machinery and optimize work processes. For example, when optimizing the operation of a robot on an automotive parts manufacturing line, the order and method of each process can be intuitively changed.

[0318] By utilizing a generative AI model, advanced analysis is performed using prompt messages. By inputting prompt messages such as, "Visually represent the assembly process of this factory and find the optimization points," users can visually obtain the results of analyzing complex code.

[0319] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0320] Step 1:

[0321] The server receives program code from users. It accepts code submitted by users via APIs or web forms as input and securely stores it in data storage. The data processing performed here involves converting the code into an appropriate format, preparing it for analysis in subsequent stages of the process. The output is a reference to the stored code.

[0322] Step 2:

[0323] The server passes the received code to the generative AI engine for analysis. At this stage, the code reference saved in step 1 is used as input. The generative AI model is used to identify elements such as functions, variables, and control flow, and performs data calculations to generate visualization data. As output, data is created that visually represents each part of the code.

[0324] Step 3:

[0325] The server sends the generated visualization data to the user terminal. The visualization data generated in step 2 is used as input. The data is serialized into an appropriate format (e.g., JSON) and processed to make it viewable on the terminal's display device. The output is in a data format easily processed by the user terminal.

[0326] Step 4:

[0327] The user manipulates the visualization data through the terminal's user interface. The terminal uses the visualization data received in step 3 as input. The user performs specific actions such as changing the program flow or adding new elements by dragging and dropping blocks. The output is the modified visualization data.

[0328] Step 5:

[0329] The user terminal sends the modified visualization data to the server. The server receives the visualization data edited by the user in step 4 as input. The server analyzes the visualization data and performs data calculations to regenerate program code that reflects the changes. The output is the regenerated program code.

[0330] Step 6:

[0331] The regenerated code is provided to the user. The code generated in step 5 is used as input. The server distributes this code so that the user can access it and manages the data to make it available for the user to finally review and use. As output, the complete program code is provided, which can be used through the user interface.

[0332] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0333] This invention provides a program editing environment that takes into account the user's emotional state by incorporating an emotion engine into a system that visually manipulates program code. This system consists of server software, a user terminal interface, a generation AI engine, and an emotion engine.

[0334] First, the user's terminal receives the program code and sends it to the server. The server receives this code and performs analysis using a generation AI engine. As a result of the analysis, visualization data representing the logical structure of the program is generated, and this data is sent from the server to the user's terminal.

[0335] The user's terminal displays the generated visualization data on the user interface. This allows the user to visually confirm and manipulate the program flow. These operations include adding, deleting, and rearranging blocks.

[0336] The emotion engine recognizes the user's emotional state in real time by analyzing their facial expressions and voice. Based on this emotional state, the system dynamically adjusts the displayed content and user interface responses. For example, if the user is feeling stressed, the system may simplify interactions or display encouraging messages.

[0337] After the user visually edits the program again, the changes are sent from the terminal to the server. The server creates regenerated code based on the edits and provides it to the user. The user can then review and execute the regenerated code.

[0338] For example, suppose a user is struggling for a long time because they misunderstand a loop in a program. The emotion engine detects the user's frustration, and the system provides concrete examples of the loop structure and hints for solving it. In this way, the system provides a more intuitive and supportive programming environment, reducing the burden on the user.

[0339] The following describes the processing flow.

[0340] Step 1:

[0341] The user enters program code into the interface using their own device and presses the submit button. This sends the code from the device to the server.

[0342] Step 2:

[0343] The server receives program code sent by the user. First, the server sends the code to a generative AI engine to understand its structure.

[0344] Step 3:

[0345] The generation AI engine analyzes the received code. It identifies functions, variables, and control structures within the code and generates data for visual representation.

[0346] Step 4:

[0347] The server sends visualization data obtained from the AI ​​generation engine to the user's terminal. This visualization data is intended to show the user the structure of the program.

[0348] Step 5:

[0349] The terminal displays the received visualization data on the screen. Based on this visualization data, the user can visually understand and operate the program's flow and structure.

[0350] Step 6:

[0351] The emotion engine runs on the user's device, analyzing the user's facial expressions and voice through the camera and microphone while the user interacts with the program. The user's emotional state is monitored in real time.

[0352] Step 7:

[0353] Users edit the program using a visual interface. For example, they can add new conditional branches or adjust existing loops.

[0354] Step 8:

[0355] When the emotion engine detects user stress or frustration, the device provides corresponding interface support functions. Specifically, it displays user-friendly hints and advice.

[0356] Step 9:

[0357] The user completes the editing process, and the device sends the changes to the server. The server receives this information and builds regenerated code based on the edited visualization data.

[0358] Step 10:

[0359] The server returns the regenerated code to the user. The user reviews the new code and makes further edits or runs it as needed.

[0360] (Example 2)

[0361] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0362] In recent years, editing program code has become increasingly complex, making it a stressful process, especially for beginners and users unfamiliar with programming. Furthermore, the lack of graphical user interfaces that reflect the user's emotional state leads to inefficient support.

[0363] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0364] In this invention, the server includes means for receiving code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to the user, and means for analyzing emotions and adjusting the interface according to the user's emotional state. This makes it possible to provide a flexible program editing environment that is sensitive to the user's emotions and to reduce the user's stress and burden.

[0365] "Code" is text that describes instructions and commands to be executed by a computer.

[0366] "Analysis" is the process of breaking down received code and understanding its structure and function.

[0367] "Visualized data" refers to information that visually represents the logical structure of a program, converted into a format that users can easily understand and manipulate.

[0368] A "display device" is hardware or software that provides visualized data to the user in a visible format.

[0369] A "user interface" is a mechanism that provides users with a means to interact with a system and manipulate visualized data.

[0370] "Emotions" refer to the psychological state extracted from the user's facial expressions and voice, and are detected by the system.

[0371] "Interface adjustment" refers to dynamically changing the display and functionality of the user interface based on the user's emotional state.

[0372] A "programming language" is a standardized language used to give instructions to a computer, and there are many different types.

[0373] "Code regeneration" is the process of regenerating the original code based on the manipulated visualization data.

[0374] This invention is a system that provides support to users while they visually edit a program, taking into account their emotional state. The system mainly consists of server software, a user terminal, a generative AI engine, and an emotion engine.

[0375] The user's terminal has an interface for inputting program code. Once the user inputs code, the terminal sends it to the server. The server uses a generative AI engine to analyze the received code. This analysis generates the program's logical structure as visualized data. The server then sends this generated visualization data to the user's terminal.

[0376] The terminal displays visualized data on the user interface. This allows the user to visually confirm the program flow and manipulate it freely. These operations include adding, deleting, and rearranging program blocks.

[0377] The emotion engine can analyze the user's facial expressions and voice in real time to recognize their emotional state. Based on this emotional state, the device dynamically adjusts the content and responses of the user interface. For example, if the user is feeling frustrated, the system can simplify the interface and display encouraging messages.

[0378] After editing is complete, the user's terminal sends the changes to the server. The server regenerates the original code according to the changes and provides the result to the user. The user can then review and execute this regenerated code.

[0379] For example, if a user prompts with "I don't know how to use a for loop," the system automatically presents a concrete example of a loop structure to help the user understand. In this way, the system provides an intuitive and emotionally resonant programming environment.

[0380] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0381] Step 1:

[0382] The user's terminal accepts code through an interface for entering program code. The user begins by entering code into this interface. The entered code is processed as data in a format that is sent directly to the server. Specifically, if the user enters the code "I don't know how to use a for loop," the terminal will transfer that information to the server.

[0383] Step 2:

[0384] The server analyzes the received code using a generative AI engine. Here, the server analyzes the input code line by line, determining its structure and purpose. Data calculations include code tokenization and syntax tree generation. The output is data that visualizes the logical structure of the code. This visualization diagrams the structure like a program flowchart.

[0385] Step 3:

[0386] The server sends the generated visualization data to the user's terminal. The terminal displays the received visualization data on its user interface. Specifically, the visualized program flowchart or blocks are displayed on the screen, ready for the user to review.

[0387] Step 4:

[0388] Users manipulate the programming flow on the interface based on visualized data. Here, they can perform specific actions such as adding, deleting, and rearranging blocks. Based on the user's input, the terminal processes the data and generates the manipulated visualized data. This allows users to intuitively build and improve the program's logic.

[0389] Step 5:

[0390] The device uses an emotion engine to analyze the user's voice and facial expressions to determine their emotional state. If the user is feeling anxious, the device adjusts the data to display encouraging messages or simplified instructions on the screen. Specifically, the on-screen instructions and guidance change to clearly show the user what to do next.

[0391] Step 6:

[0392] After the user's actions are finalized, the terminal sends this information to the server. The server then regenerates the original code based on the manipulated visualization data. The input is the data modified by the user's actions, and after data calculations, the new code is output.

[0393] Step 7:

[0394] The server sends the regenerated code to the user's terminal. The user then verifies this regenerated code. Specifically, the newly generated program code is displayed on the screen, allowing the user to review its contents.

[0395] Step 8:

[0396] The user executes the code they have reviewed. This allows them to experience the actual program's operation and verify the results of the visualized program.

[0397] (Application Example 2)

[0398] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0399] Modern programming environments often fail to consider the user's operational capabilities or emotional state, leading to frustration and stress during code editing. Furthermore, understanding and manipulating complex programs requires advanced knowledge, necessitating additional support. This invention aims to address these challenges by understanding the user's emotional state in real time and providing appropriate interface adjustments and support accordingly.

[0400] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0401] In this invention, the server includes means for receiving code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to the user, means for analyzing the user's emotional state, and means for dynamically adjusting the display content based on the user's emotional state. This allows the user to receive optimal support according to their emotional state and reduces the burden during program editing.

[0402] "Means of receiving code" refers to the function by which the system receives program code entered by the user.

[0403] "Means for analyzing received code and generating visualization data" refers to a function that analyzes the received code and creates data to visually represent its logical structure.

[0404] "Means for transmitting generated visualization data to a display device" refers to a function that transmits the generated visualization data to the user's terminal, making it displayable on a display device such as a screen.

[0405] "Means of manipulating visualized data through a user interface" refers to a function that provides an interface that allows users to edit a program through visualized data.

[0406] "Means for regenerating the original code based on manipulated visualization data" refers to a function that rewrites the original program code based on the visual operations performed by the user.

[0407] "Means of providing regenerated code to users" refers to a function that presents and allows users to use the newly generated code.

[0408] "Means for analyzing the user's emotional state" refers to a function that analyzes the user's facial expressions, voice, etc., to understand their emotional state in real time.

[0409] "Means for dynamically adjusting displayed content based on the user's emotional state" refers to a function that appropriately changes the user interface and displayed messages according to the analyzed emotional state, thereby improving the user's experience.

[0410] This invention provides a system that dynamically adjusts the programming environment according to the user's emotional state. The system mainly consists of a server, a user terminal, a generative AI engine, and an emotion engine.

[0411] The server receives the code entered by the user, analyzes it with the help of a generation AI engine, and generates visualization data. This visualization data is designed to allow the user to intuitively understand the structure and flow of the program. This data is sent from the server to a display device and displayed on the user's terminal.

[0412] The user terminal can be, for example, a smartphone or tablet. Through the terminal's user interface, the user can manipulate the visualized data of the program code. Specifically, they can edit it by adding, deleting, and rearranging blocks.

[0413] The emotion engine analyzes the user's facial expressions and voice in real time to understand their emotional state. Based on this emotional data, the system dynamically adjusts the content and interactions of the user interface. For example, if the user is feeling stressed, the system can simplify operations or display encouraging messages.

[0414] As a concrete example, consider a situation where a user is visually editing the robot's operation program to control a robot in their home. If the system detects that the user is confused, the emotion engine communicates this to the server, which then provides step-by-step guidance and specific operation suggestions to the user's terminal.

[0415] This system allows users to receive support tailored to their emotional state, resulting in a more comfortable and efficient programming experience.

[0416] An example of a prompt to input into the generating AI model is: "The user is trying to set a cleaning schedule for a robot, but is showing signs of confusion during the process. Based on sentiment analysis, suggest how to provide support."

[0417] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0418] Step 1:

[0419] The server receives program code entered from the user's terminal. The input is the program code entered by the user on the terminal. Upon receiving this code, the server prepares to proceed to the next analysis process.

[0420] Step 2:

[0421] The server uses a generation AI engine to analyze the received program code. The input is the program code received in step 1. As a result of the analysis, it generates data to visualize the logical structure of the code. This visualized data is the output.

[0422] Step 3:

[0423] The server sends the generated visualization data to the user's terminal. The terminal receives this visualization data as input and prepares to display it on the user interface. As a result, the output visualization data becomes available for viewing on the terminal.

[0424] Step 4:

[0425] The user manipulates the visualized data through the terminal's user interface. The input is the visualized data displayed on the terminal, and the user performs operations such as adding, deleting, and rearranging blocks based on this data. The results of the user's operations are passed on to the next step.

[0426] Step 5:

[0427] The terminal reshapes the visualization data based on user actions and sends that data to the server. The input is the visualization data after user actions, and the output is the instruction for regeneration.

[0428] Step 6:

[0429] The server regenerates the original program code based on the user's re-formed visualization data. The input is the data sent in step 5, and the output is the regenerated program code.

[0430] Step 7:

[0431] The server provides the user with regenerated program code. The user reviews this code on their terminal and executes it if necessary. The input is the regenerated code, and the output is information for the user to review.

[0432] Step 8:

[0433] The emotion engine analyzes the user's facial expressions and voice to understand their emotional state. Input consists of audio and image data obtained through the camera and microphone. Based on these results, it outputs the user's emotional state.

[0434] Step 9:

[0435] The server adjusts the user interface display based on the user's emotional state, which is analyzed by the emotion engine. The input is the user's emotional state, and the output is the adjusted interface display and interaction content.

[0436] As a concrete example of its operation, if a user is experiencing stress while editing a program, the emotion engine can detect this state, and the server can instruct the interface to simplify the operation. In this case, a generative AI model is used to provide guidance that includes prompts that are easy for the user to understand. A sample prompt could be: "The user is trying to set a cleaning schedule for the robot, but is showing signs of confusion in the process. Based on emotion analysis, please suggest how to provide support."

[0437] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0438] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0439] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0440] [Third Embodiment]

[0441] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0442] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0443] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0444] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0445] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0446] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0447] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0448] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0449] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0450] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0451] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0452] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0453] This invention is implemented as a system that provides an environment for visually manipulating programming code. This system consists of server software operating under specific hardware conditions, a user terminal interface, and a generative AI engine.

[0454] The server receives program code submitted by the user. This receiving process takes place, for example, through a web form or API. The received code is first stored in an integrated database, and then proceeds to the analysis process.

[0455] The analysis uses a generative AI engine. The server passes the received code to the generative AI engine, which performs semantic analysis of the code. Here, each element of the code, such as functions, variables, and control flow, is recognized and converted into data for visualization. This visualized data becomes intermediate data to be provided to the user.

[0456] Next, the server sends this visualization data to the user's terminal. The user's terminal has a dedicated interface that, upon receiving the visualization data, presents the user with a visual representation of the program's flow and structure. This interface is designed to be easy for the user to understand and has an intuitive user interface (UI).

[0457] Users can interact with visual programs provided on their devices. For example, they can use visual blocks to modify the control flow of code or add new conditional branches and loops. This allows users to understand and edit the program's logic without needing to understand the intricacies of programming.

[0458] Once the user has finished editing, their device sends the modified visualization data to the server. The server then automatically regenerates the code in the original programming language based on this visualization data. The regenerated code accurately reflects the visual edits made to the original code, and the user can review and use it.

[0459] This series of processes allows users to easily operate and edit tools, overcoming the barriers of different programming languages, and enables a smooth transition of tools during internal transfers. This invention is expected to significantly improve the efficiency of program management within organizations and reduce technical barriers.

[0460] The following describes the processing flow.

[0461] Step 1:

[0462] The user enters program code and presses the submit button, sending the code from the terminal. The terminal then sends the entered code to the server in the correct format.

[0463] Step 2:

[0464] The server receives program code sent by the user. The received data is temporarily stored in a database on the server.

[0465] Step 3:

[0466] The server sends the received code to a generation AI engine. This engine analyzes the structural elements of the code and converts them into visualized data.

[0467] Step 4:

[0468] The generative AI engine analyzes the program code and generates visually representable views and models. In this process, various elements of the program structure, such as functions, variables, and conditional expressions, are extracted.

[0469] Step 5:

[0470] The server sends the visualization data obtained from the generated AI to the user's device. The visualization data is packaged in a format suitable for the intuitive UI components that the user interacts with.

[0471] Step 6:

[0472] The system displays the visualized data received by the terminal on the user interface. This allows the user to intuitively manipulate the program's logic in a visualized form.

[0473] Step 7:

[0474] The user edits the code based on the displayed visualization objects. The user adds, deletes, rearranges, and changes the attributes of blocks to modify the program's logic.

[0475] Step 8:

[0476] The user completes editing and sends the changes to the server. The terminal repackages the edited visualization data and sends it to the server again.

[0477] Step 9:

[0478] The server regenerates the code in the original programming language based on the edited visualization data received from the user.

[0479] Step 10:

[0480] The server provides the user with the regenerated code. The user can review this code and download or run it as needed.

[0481] (Example 1)

[0482] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0483] There is a need to provide an environment where users with little programming experience can easily understand, visually manipulate, and edit code written in various programming languages. Furthermore, the challenge lies in achieving real-time visualization of editing and efficient code regeneration.

[0484] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0485] In this invention, the server includes means for receiving information, means for analyzing the received information and converting the data into visual information, and means for transmitting the converted visual information to a device for display. This allows the user to easily understand the program's logic and operate it intuitively.

[0486] "Information" is a general term that refers to the various data and codes processed within a system.

[0487] "Means of receiving" refers to the methods and technologies used by the server to retrieve information provided by the user.

[0488] "Means of analysis and conversion of data into visual information" refers to the process of analyzing received code or information and converting its structure and content into a format that can be visualized.

[0489] "Means for transmitting to a device for display" refers to technologies and methods for transferring converted visual information to a device accessible to the user.

[0490] "Means of manipulating visual information using a user interface" refers to an interface for editing and modifying visualized data in a way that users can directly interact with.

[0491] "Means of regenerating original information based on visual information" refers to the process of reconstructing the original program code or information based on changes made by the user using visualized information.

[0492] "Means of providing regenerated information" refers to methods for providing reconstructed code and information in a form that users can use.

[0493] "Analysis capabilities that support multiple programming languages" refers to a function that has the ability to understand and analyze code written in different programming languages.

[0494] This invention is a system that provides an environment in which users can easily manipulate and visually understand program code. The hardware used consists of a server and a terminal for the user, which communicate via a network. As the main software component, the server manages the process of receiving information, analyzing the data using a generative AI model, and visually transforming it. The analyzed data is sent to the user's terminal in a visually easy-to-understand format. A dedicated user interface is provided on the terminal side, allowing the user to manipulate the visual representation and intuitively understand and edit the program's logic.

[0495] For example, a user might want to improve the inventory management system of an existing e-commerce application. This system allows the user to visually add new logic to automatically send notifications when inventory levels drop. Loops and conditional branches can be easily added through a visual interface, and the resulting new code can be viewed in real time.

[0496] This system, which utilizes a generative AI model, allows users to input prompts such as: "Add a conditional branch to the existing code and visualize the logic that notifies when inventory falls below a certain number." This prompt helps the AI ​​visually realize the program according to the user's wishes.

[0497] This method allows users to easily improve systems and add new features, overcoming the barriers between different programming languages, even without specialized programming knowledge. This invention is expected to reduce technical barriers and significantly improve the efficiency of program management within organizations.

[0498] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0499] Step 1:

[0500] The server receives program code from the user. The input is code provided by the user through web forms or APIs. The server receives this code and temporarily stores it in an integrated database. This prepares the data for robust management for subsequent analysis.

[0501] Step 2:

[0502] The server passes the received code to the generating AI model to begin analysis. The input is stored program code, and the output is visually transformed data. The generating AI model interprets the code, identifies the function's behavior and control flow, and generates data in a format that visualizes the code's structure. In this process, it identifies each element within the code and transforms it into a form that is easy for the user to understand visually.

[0503] Step 3:

[0504] The server sends the generated visualization data to the user's terminal. The input is the visualized data, and the output is its display on the user's terminal. The server transfers this data to the user's terminal, making it ready for user interaction.

[0505] Step 4:

[0506] The user's device interacts with the user through displayed visualization data. The user interface receives visualization data as input and presents it to the user visually as output. Based on this, the user edits the program's logic using mouse or touch gestures. They can add new conditional branches and loops, or adjust existing operations.

[0507] Step 5:

[0508] After completing visual changes, the user sends the updated information to the server. The input is the manipulated visualization data, and the output is the updated data. The user's terminal accurately transmits the changes to the server, which then prepares to reflect them in the original code.

[0509] Step 6:

[0510] The server regenerates the original program code based on the change information sent to the user. The input is the manipulated visualization data, and the final output is the regenerated program code. The server reconstructs the code based on this edit to ensure it is properly reflected in the original programming language. The user can then download and actually use / verify the updated program.

[0511] (Application Example 1)

[0512] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0513] In factories and manufacturing environments, there is a challenge in the lack of sufficient means to easily and intuitively operate and edit the operating programs of machinery and equipment. In particular, it is difficult for users without specialized programming knowledge to efficiently optimize the movements of robots.

[0514] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0515] In this invention, the server includes means for receiving a code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to a user, and means for the user to visually modify the work process of a machine. This makes it possible for a user to effectively modify and optimize the operation of a machine without specialized knowledge.

[0516] "Means for receiving code" refers to a device or method that has the function of receiving program code data transmitted from an external source and holding it for processing.

[0517] "Means for generating visualization data" refers to a device or method that has the function of analyzing received program code and converting it into data for visually representing its structure and operation.

[0518] "Means for transmitting to a display device" refers to a device or method that has the function of transmitting generated visualization data to an output device in a format that can be understood by the user.

[0519] "Means of operation via a user interface" refers to a device or method that has the function of providing a screen or input means for users to intuitively manipulate and edit visualized data.

[0520] "Means for regenerating the original code" refers to a device or method that has the function of converting user-operated and edited visualization data into program code that can be executed by humans or machines.

[0521] "Means for providing regenerated code" refers to a device or method that has the function of presenting regenerated program code in a format accessible to the user.

[0522] "Means for visually modifying the work process of a machine or device" refers to a device or method that has the function of allowing a user to visually observe the operating procedure or workflow of a machine or device and to adjust or modify it as necessary.

[0523] To implement this invention, a system is required that combines a server, a user terminal, and a generative AI engine. The server first receives program code sent by the user. The received code is stored in a database and then analyzed using the generative AI engine. In this analysis, functions, variables, control flow, etc., within the code are recognized and converted into visualized data. The visualized data is structured in an intermediate format and sent to the user terminal.

[0524] The user terminal hardware includes smartphones and tablets, equipped with a dedicated user interface. The software used here includes HTML / CSS and JavaScript for UI design. Through this interface, users can view the program flow and structure presented as visualized data and edit the program using blocks. This allows users without programming knowledge to intuitively understand the program's logic and make changes.

[0525] The edited visualization data is sent back to the server and regenerated into code in the original programming language. The generated code is then provided to the user in a state where it can be accessed for further manipulation and verification. This entire process allows users to visually modify the operation of machinery and optimize work processes. For example, when optimizing the operation of a robot on an automotive parts manufacturing line, the order and method of each process can be intuitively changed.

[0526] By utilizing a generative AI model, advanced analysis is performed using prompt messages. By inputting prompt messages such as, "Visually represent the assembly process of this factory and find the optimization points," users can visually obtain the results of analyzing complex code.

[0527] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0528] Step 1:

[0529] The server receives program code from users. It accepts code submitted by users via APIs or web forms as input and securely stores it in data storage. The data processing performed here involves converting the code into an appropriate format, preparing it for analysis in subsequent stages of the process. The output is a reference to the stored code.

[0530] Step 2:

[0531] The server passes the received code to the generative AI engine for analysis. At this stage, the code reference saved in step 1 is used as input. The generative AI model is used to identify elements such as functions, variables, and control flow, and performs data calculations to generate visualization data. As output, data is created that visually represents each part of the code.

[0532] Step 3:

[0533] The server sends the generated visualization data to the user terminal. The visualization data generated in step 2 is used as input. The data is serialized into an appropriate format (e.g., JSON) and processed to make it viewable on the terminal's display device. The output is in a data format easily processed by the user terminal.

[0534] Step 4:

[0535] The user manipulates the visualization data through the terminal's user interface. The terminal uses the visualization data received in step 3 as input. The user performs specific actions such as dragging and dropping blocks to change the program flow or add new elements. The output is the modified visualization data.

[0536] Step 5:

[0537] The user terminal sends the modified visualization data to the server. The server receives the visualization data edited by the user in step 4 as input. The server analyzes the visualization data and performs data calculations to regenerate program code that reflects the changes. The output is the regenerated program code.

[0538] Step 6:

[0539] The regenerated code is provided to the user. The code generated in step 5 is used as input. The server distributes this code for user access and manages the data to ensure it is ultimately verifiable and usable by the user. As output, the complete program code is provided, available through the user interface.

[0540] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0541] This invention provides a program editing environment that takes into account the user's emotional state by incorporating an emotion engine into a system that visually manipulates program code. This system consists of server software, a user terminal interface, a generation AI engine, and an emotion engine.

[0542] First, the user's terminal receives the program code and sends it to the server. The server receives this code and performs analysis using a generation AI engine. As a result of the analysis, visualization data representing the logical structure of the program is generated, and this data is sent from the server to the user's terminal.

[0543] The user's terminal displays the generated visualization data on the user interface. This allows the user to visually confirm and manipulate the program flow. These operations include adding, deleting, and rearranging blocks.

[0544] The emotion engine recognizes the user's emotional state in real time by analyzing their facial expressions and voice. Based on this emotional state, the system dynamically adjusts the displayed content and user interface responses. For example, if the user is feeling stressed, the system may simplify interactions or display encouraging messages.

[0545] After the user visually edits the program again, the changes are sent from the terminal to the server. The server creates regenerated code based on the edits and provides it to the user. The user can then review and execute the regenerated code.

[0546] For example, suppose a user is struggling for a long time because they misunderstand a loop in a program. The emotion engine detects the user's frustration, and the system provides concrete examples of the loop structure and hints for solving it. In this way, the system provides a more intuitive and supportive programming environment, reducing the burden on the user.

[0547] The following describes the processing flow.

[0548] Step 1:

[0549] The user enters program code into the interface using their own device and presses the submit button. This sends the code from the device to the server.

[0550] Step 2:

[0551] The server receives program code sent by the user. First, the server sends the code to a generative AI engine to understand its structure.

[0552] Step 3:

[0553] The generation AI engine analyzes the received code. It identifies functions, variables, and control structures within the code and generates data for visual representation.

[0554] Step 4:

[0555] The server sends visualization data obtained from the AI ​​generation engine to the user's terminal. This visualization data is intended to show the user the structure of the program.

[0556] Step 5:

[0557] The terminal displays the received visualization data on the screen. Based on this visualization data, the user can visually understand and operate the program's flow and structure.

[0558] Step 6:

[0559] The emotion engine runs on the user's device, analyzing the user's facial expressions and voice through the camera and microphone while the user interacts with the program. The user's emotional state is monitored in real time.

[0560] Step 7:

[0561] Users edit the program using a visual interface. For example, they can add new conditional branches or adjust existing loops.

[0562] Step 8:

[0563] When the emotion engine detects user stress or frustration, the device provides corresponding interface support functions. Specifically, it displays user-friendly hints and advice.

[0564] Step 9:

[0565] The user completes the editing process, and the device sends the changes to the server. The server receives this information and builds regenerated code based on the edited visualization data.

[0566] Step 10:

[0567] The server returns the regenerated code to the user. The user reviews the new code and makes further edits or runs it as needed.

[0568] (Example 2)

[0569] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0570] In recent years, editing program code has become increasingly complex, making it a stressful process, especially for beginners and users unfamiliar with programming. Furthermore, the lack of graphical user interfaces that reflect the user's emotional state leads to inefficient support.

[0571] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0572] In this invention, the server includes means for receiving code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to the user, and means for analyzing emotions and adjusting the interface according to the user's emotional state. This makes it possible to provide a flexible program editing environment that is sensitive to the user's emotions and to reduce the user's stress and burden.

[0573] "Code" is text that describes instructions and commands to be executed by a computer.

[0574] "Analysis" is the process of breaking down received code and understanding its structure and function.

[0575] "Visualized data" refers to information that visually represents the logical structure of a program, converted into a format that users can easily understand and manipulate.

[0576] A "display device" is hardware or software that provides visualized data to the user in a visible format.

[0577] A "user interface" is a mechanism that provides users with a means to interact with a system and manipulate visualized data.

[0578] "Emotions" refer to the psychological state extracted from the user's facial expressions and voice, and are detected by the system.

[0579] "Interface adjustment" refers to dynamically changing the display and functionality of the user interface based on the user's emotional state.

[0580] A "programming language" is a standardized language used to give instructions to a computer, and there are many different types.

[0581] "Code regeneration" is the process of regenerating the original code based on the manipulated visualization data.

[0582] This invention is a system that provides support to users while they visually edit a program, taking into account their emotional state. The system mainly consists of server software, a user terminal, a generative AI engine, and an emotion engine.

[0583] The user's terminal has an interface for inputting program code. Once the user inputs code, the terminal sends it to the server. The server uses a generative AI engine to analyze the received code. This analysis generates the program's logical structure as visualized data. The server then sends this generated visualization data to the user's terminal.

[0584] The terminal displays visualized data on the user interface. This allows the user to visually confirm the program flow and manipulate it freely. These operations include adding, deleting, and rearranging program blocks.

[0585] The emotion engine can analyze the user's facial expressions and voice in real time to recognize their emotional state. Based on this emotional state, the device dynamically adjusts the content and responses of the user interface. For example, if the user is feeling frustrated, the system can simplify the interface and display encouraging messages.

[0586] After editing is complete, the user's terminal sends the changes to the server. The server regenerates the original code according to the changes and provides the result to the user. The user can then review and execute this regenerated code.

[0587] For example, if a user prompts with "I don't know how to use a for loop," the system automatically presents a concrete example of a loop structure to help the user understand. In this way, the system provides an intuitive and emotionally resonant programming environment.

[0588] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0589] Step 1:

[0590] The user's terminal accepts code through an interface for entering program code. The user begins by entering code into this interface. The entered code is processed as data in a format that is sent directly to the server. Specifically, if the user enters the code "I don't know how to use a for loop," the terminal will transfer that information to the server.

[0591] Step 2:

[0592] The server analyzes the received code using a generative AI engine. Here, the server analyzes the input code line by line, determining its structure and purpose. Data calculations include code tokenization and syntax tree generation. The output is data that visualizes the logical structure of the code. This visualization diagrams the structure like a program flowchart.

[0593] Step 3:

[0594] The server sends the generated visualization data to the user's terminal. The terminal displays the received visualization data on its user interface. Specifically, the visualized program flowchart or blocks are displayed on the screen, ready for the user to review.

[0595] Step 4:

[0596] Users manipulate the programming flow on the interface based on visualized data. Here, they can perform specific actions such as adding, deleting, and rearranging blocks. Based on the user's input, the terminal processes the data and generates the manipulated visualized data. This allows users to intuitively build and improve the program's logic.

[0597] Step 5:

[0598] The device uses an emotion engine to analyze the user's voice and facial expressions to determine their emotional state. If the user is feeling anxious, the device adjusts the data to display encouraging messages or simplified instructions on the screen. Specifically, the on-screen instructions and guidance change to clearly show the user what to do next.

[0599] Step 6:

[0600] After the user's actions are finalized, the terminal sends this information to the server. The server then regenerates the original code based on the manipulated visualization data. The input is the data modified by the user's actions, and after data calculations, the new code is output.

[0601] Step 7:

[0602] The server sends the regenerated code to the user's terminal. The user then verifies this regenerated code. Specifically, the newly generated program code is displayed on the screen, allowing the user to review its contents.

[0603] Step 8:

[0604] The user executes the code they have reviewed. This allows them to experience the actual program's operation and verify the results of the visualized program.

[0605] (Application Example 2)

[0606] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0607] Modern programming environments often fail to consider the user's operational capabilities or emotional state, leading to frustration and stress during code editing. Furthermore, understanding and manipulating complex programs requires advanced knowledge, necessitating additional support. This invention aims to address these challenges by understanding the user's emotional state in real time and providing appropriate interface adjustments and support accordingly.

[0608] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0609] In this invention, the server includes means for receiving code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to the user, means for analyzing the user's emotional state, and means for dynamically adjusting the display content based on the user's emotional state. This allows the user to receive optimal support according to their emotional state and reduces the burden during program editing.

[0610] "Means of receiving code" refers to the function by which the system receives program code entered by the user.

[0611] "Means for analyzing received code and generating visualization data" refers to a function that analyzes the received code and creates data to visually represent its logical structure.

[0612] "Means for transmitting generated visualization data to a display device" refers to a function that transmits the generated visualization data to the user's terminal, making it displayable on a display device such as a screen.

[0613] "Means of manipulating visualized data through a user interface" refers to a function that provides an interface that allows users to edit a program through visualized data.

[0614] "Means for regenerating the original code based on manipulated visualization data" refers to a function that rewrites the original program code based on the visual operations performed by the user.

[0615] "Means of providing regenerated code to users" refers to a function that presents and allows users to use the newly generated code.

[0616] "Means for analyzing the user's emotional state" refers to a function that analyzes the user's facial expressions, voice, etc., to understand their emotional state in real time.

[0617] "Means for dynamically adjusting displayed content based on the user's emotional state" refers to a function that appropriately changes the user interface and displayed messages according to the analyzed emotional state, thereby improving the user's experience.

[0618] This invention provides a system that dynamically adjusts the programming environment according to the user's emotional state. The system mainly consists of a server, a user terminal, a generative AI engine, and an emotion engine.

[0619] The server receives the code entered by the user, analyzes it with the help of a generation AI engine, and generates visualization data. This visualization data is designed to allow the user to intuitively understand the structure and flow of the program. This data is sent from the server to a display device and displayed on the user's terminal.

[0620] The user terminal can be, for example, a smartphone or tablet. Through the terminal's user interface, the user can manipulate the visualized data of the program code. Specifically, they can edit it by adding, deleting, and rearranging blocks.

[0621] The emotion engine analyzes the user's facial expressions and voice in real time to understand their emotional state. Based on this emotional data, the system dynamically adjusts the content and interactions of the user interface. For example, if the user is feeling stressed, the system can simplify operations or display encouraging messages.

[0622] As a concrete example, consider a situation where a user is visually editing the robot's operation program to control a robot in their home. If the system detects that the user is confused, the emotion engine communicates this to the server, which then provides step-by-step guidance and specific operation suggestions to the user's terminal.

[0623] This system allows users to receive support tailored to their emotional state, resulting in a more comfortable and efficient programming experience.

[0624] An example of a prompt to input into the generating AI model is: "The user is trying to set a cleaning schedule for a robot, but is showing signs of confusion during the process. Based on sentiment analysis, suggest how to provide support."

[0625] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0626] Step 1:

[0627] The server receives program code entered from the user's terminal. The input is the program code entered by the user on the terminal. Upon receiving this code, the server prepares to proceed to the next analysis process.

[0628] Step 2:

[0629] The server uses a generation AI engine to analyze the received program code. The input is the program code received in step 1. As a result of the analysis, it generates data to visualize the logical structure of the code. This visualized data is the output.

[0630] Step 3:

[0631] The server sends the generated visualization data to the user's terminal. The terminal receives this visualization data as input and prepares to display it on the user interface. As a result, the output visualization data becomes available for viewing on the terminal.

[0632] Step 4:

[0633] The user manipulates the visualized data through the terminal's user interface. The input is the visualized data displayed on the terminal, and the user performs operations such as adding, deleting, and rearranging blocks based on this data. The results of the user's operations are passed on to the next step.

[0634] Step 5:

[0635] The terminal reshapes the visualization data based on user actions and sends that data to the server. The input is the visualization data after user actions, and the output is the instruction for regeneration.

[0636] Step 6:

[0637] The server regenerates the original program code based on the user's re-formed visualization data. The input is the data sent in step 5, and the output is the regenerated program code.

[0638] Step 7:

[0639] The server provides the user with regenerated program code. The user reviews this code on their terminal and executes it if necessary. The input is the regenerated code, and the output is information for the user to review.

[0640] Step 8:

[0641] The emotion engine analyzes the user's facial expressions and voice to understand their emotional state. Input consists of audio and image data obtained through the camera and microphone. Based on these results, it outputs the user's emotional state.

[0642] Step 9:

[0643] The server adjusts the user interface display based on the user's emotional state, which is analyzed by the emotion engine. The input is the user's emotional state, and the output is the adjusted interface display and interaction content.

[0644] As a concrete example of its operation, if a user is experiencing stress while editing a program, the emotion engine can detect this state, and the server can instruct the interface to simplify the operation. In this case, a generative AI model is used to provide guidance that includes prompts that are easy for the user to understand. A sample prompt could be: "The user is trying to set a cleaning schedule for the robot, but is showing signs of confusion in the process. Based on emotion analysis, please suggest how to provide support."

[0645] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0646] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0647] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0648] [Fourth Embodiment]

[0649] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0650] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0651] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0652] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0653] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0654] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0655] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0656] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0657] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0658] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0659] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0660] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0661] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0662] This invention is implemented as a system that provides an environment for visually manipulating programming code. This system consists of server software operating under specific hardware conditions, a user terminal interface, and a generative AI engine.

[0663] The server receives program code submitted by the user. This receiving process takes place, for example, through a web form or API. The received code is first stored in an integrated database, and then proceeds to the analysis process.

[0664] The analysis uses a generative AI engine. The server passes the received code to the generative AI engine, which performs semantic analysis of the code. Here, each element of the code, such as functions, variables, and control flow, is recognized and converted into data for visualization. This visualized data becomes intermediate data to be provided to the user.

[0665] Next, the server sends this visualization data to the user's terminal. The user's terminal has a dedicated interface that, upon receiving the visualization data, presents the user with a visual representation of the program's flow and structure. This interface is designed to be easy for the user to understand and has an intuitive user interface (UI).

[0666] Users can interact with visual programs provided on their devices. For example, they can use visual blocks to modify the control flow of code or add new conditional branches and loops. This allows users to understand and edit the program's logic without needing to understand the intricacies of programming.

[0667] Once the user has finished editing, their device sends the modified visualization data to the server. The server then automatically regenerates the code in the original programming language based on this visualization data. The regenerated code accurately reflects the visual edits made to the original code, and the user can review and use it.

[0668] This series of processes allows users to easily operate and edit tools, overcoming the barriers of different programming languages, and enables a smooth transition of tools during internal transfers. This invention is expected to significantly improve the efficiency of program management within organizations and reduce technical barriers.

[0669] The following describes the processing flow.

[0670] Step 1:

[0671] The user enters program code and presses the submit button, sending the code from the terminal. The terminal then sends the entered code to the server in the correct format.

[0672] Step 2:

[0673] The server receives program code sent by the user. The received data is temporarily stored in a database on the server.

[0674] Step 3:

[0675] The server sends the received code to a generation AI engine. This engine analyzes the structural elements of the code and converts them into visualized data.

[0676] Step 4:

[0677] The generative AI engine analyzes the program code and generates visually representable views and models. In this process, various elements of the program structure, such as functions, variables, and conditional expressions, are extracted.

[0678] Step 5:

[0679] The server sends the visualization data obtained from the generated AI to the user's device. The visualization data is packaged in a format suitable for the intuitive UI components that the user interacts with.

[0680] Step 6:

[0681] The system displays the visualized data received by the terminal on the user interface. This allows the user to intuitively manipulate the program's logic in a visualized form.

[0682] Step 7:

[0683] The user edits the code based on the displayed visualization objects. The user adds, deletes, rearranges, and changes the attributes of blocks to modify the program's logic.

[0684] Step 8:

[0685] The user completes editing and sends the changes to the server. The terminal repackages the edited visualization data and sends it to the server again.

[0686] Step 9:

[0687] The server regenerates the code in the original programming language based on the edited visualization data received from the user.

[0688] Step 10:

[0689] The server provides the user with the regenerated code. The user can review this code and download or run it as needed.

[0690] (Example 1)

[0691] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0692] There is a need to provide an environment where users with little programming experience can easily understand, visually manipulate, and edit code written in various programming languages. Furthermore, the challenge lies in achieving real-time visualization of editing and efficient code regeneration.

[0693] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0694] In this invention, the server includes means for receiving information, means for analyzing the received information and converting the data into visual information, and means for transmitting the converted visual information to a device for display. This allows the user to easily understand the program's logic and operate it intuitively.

[0695] "Information" is a general term that refers to the various data and codes processed within a system.

[0696] "Means of receiving" refers to the methods and technologies used by the server to retrieve information provided by the user.

[0697] "Means of analysis and conversion of data into visual information" refers to the process of analyzing received code or information and converting its structure and content into a format that can be visualized.

[0698] "Means for transmitting to a device for display" refers to technologies and methods for transferring converted visual information to a device accessible to the user.

[0699] "Means of manipulating visual information using a user interface" refers to an interface for editing and modifying visualized data in a way that users can directly interact with.

[0700] "Means of regenerating original information based on visual information" refers to the process of reconstructing the original program code or information based on changes made by the user using visualized information.

[0701] "Means of providing regenerated information" refers to methods for providing reconstructed code and information in a form that users can use.

[0702] "Analysis capabilities that support multiple programming languages" refers to a function that has the ability to understand and analyze code written in different programming languages.

[0703] This invention is a system that provides an environment in which users can easily manipulate and visually understand program code. The hardware used consists of a server and a terminal for the user, which communicate via a network. As the main software component, the server manages the process of receiving information, analyzing the data using a generative AI model, and visually transforming it. The analyzed data is sent to the user's terminal in a visually easy-to-understand format. A dedicated user interface is provided on the terminal side, allowing the user to manipulate the visual representation and intuitively understand and edit the program's logic.

[0704] For example, a user might want to improve the inventory management system of an existing e-commerce application. This system allows the user to visually add new logic to automatically send notifications when inventory levels drop. Loops and conditional branches can be easily added through a visual interface, and the resulting new code can be viewed in real time.

[0705] This system, which utilizes a generative AI model, allows users to input prompts such as: "Add a conditional branch to the existing code and visualize the logic that notifies when inventory falls below a certain number." This prompt helps the AI ​​visually realize the program according to the user's wishes.

[0706] This method allows users to easily improve systems and add new features, overcoming the barriers between different programming languages, even without specialized programming knowledge. This invention is expected to reduce technical barriers and significantly improve the efficiency of program management within organizations.

[0707] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0708] Step 1:

[0709] The server receives program code from the user. The input is code provided by the user through web forms or APIs. The server receives this code and temporarily stores it in an integrated database. This prepares the data for robust management for subsequent analysis.

[0710] Step 2:

[0711] The server passes the received code to the generating AI model to begin analysis. The input is stored program code, and the output is visually transformed data. The generating AI model interprets the code, identifies the function's behavior and control flow, and generates data in a format that visualizes the code's structure. In this process, it identifies each element within the code and transforms it into a form that is easy for the user to understand visually.

[0712] Step 3:

[0713] The server sends the generated visualization data to the user's terminal. The input is the visualized data, and the output is its display on the user's terminal. The server transfers this data to the user's terminal, making it ready for user interaction.

[0714] Step 4:

[0715] The user's device interacts with the user through displayed visualization data. The user interface receives visualization data as input and presents it to the user visually as output. Based on this, the user edits the program's logic using mouse or touch gestures. They can add new conditional branches and loops, or adjust existing operations.

[0716] Step 5:

[0717] After completing visual changes, the user sends the updated information to the server. The input is the manipulated visualization data, and the output is the updated data. The user's terminal accurately transmits the changes to the server, which then prepares to reflect them in the original code.

[0718] Step 6:

[0719] The server regenerates the original program code based on the change information sent to the user. The input is the manipulated visualization data, and the final output is the regenerated program code. The server reconstructs the code based on this edit to ensure it is properly reflected in the original programming language. The user can then download and actually use / verify the updated program.

[0720] (Application Example 1)

[0721] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0722] In factories and manufacturing environments, there is a challenge in the lack of sufficient means to easily and intuitively operate and edit the operating programs of machinery and equipment. In particular, it is difficult for users without specialized programming knowledge to efficiently optimize the movements of robots.

[0723] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0724] In this invention, the server includes means for receiving a code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to a user, and means for the user to visually modify the work process of a machine. This makes it possible for a user to effectively modify and optimize the operation of a machine without specialized knowledge.

[0725] "Means for receiving code" refers to a device or method that has the function of receiving program code data transmitted from an external source and holding it for processing.

[0726] "Means for generating visualization data" refers to a device or method that has the function of analyzing received program code and converting it into data for visually representing its structure and operation.

[0727] "Means for transmitting to a display device" refers to a device or method that has the function of transmitting generated visualization data to an output device in a format that can be understood by the user.

[0728] "Means of operation via a user interface" refers to a device or method that has the function of providing a screen or input means for users to intuitively manipulate and edit visualized data.

[0729] "Means for regenerating the original code" refers to a device or method that has the function of converting user-operated and edited visualization data into program code that can be executed by humans or machines.

[0730] "Means for providing regenerated code" refers to a device or method that has the function of presenting regenerated program code in a format accessible to the user.

[0731] "Means for visually modifying the work process of a machine or device" refers to a device or method that has the function of allowing a user to visually observe the operating procedure or workflow of a machine or device and to adjust or modify it as necessary.

[0732] To implement this invention, a system is required that combines a server, a user terminal, and a generative AI engine. The server first receives program code sent by the user. The received code is stored in a database and then analyzed using the generative AI engine. In this analysis, functions, variables, control flow, etc., within the code are recognized and converted into visualized data. The visualized data is structured in an intermediate format and sent to the user terminal.

[0733] The user terminal hardware includes smartphones and tablets, equipped with a dedicated user interface. The software used here includes HTML / CSS and JavaScript for UI design. Through this interface, users can view the program flow and structure presented as visualized data and edit the program using blocks. This allows users without programming knowledge to intuitively understand the program's logic and make changes.

[0734] The edited visualization data is sent back to the server and regenerated into code in the original programming language. The generated code is then provided to the user in a state where it can be accessed for further manipulation and verification. This entire process allows users to visually modify the operation of machinery and optimize work processes. For example, when optimizing the operation of a robot on an automotive parts manufacturing line, the order and method of each process can be intuitively changed.

[0735] By utilizing a generative AI model, advanced analysis is performed using prompt messages. By inputting prompt messages such as, "Visually represent the assembly process of this factory and find the optimization points," users can visually obtain the results of analyzing complex code.

[0736] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0737] Step 1:

[0738] The server receives program code from users. It accepts code submitted by users via APIs or web forms as input and securely stores it in data storage. The data processing performed here involves converting the code into an appropriate format, preparing it for analysis in subsequent stages of the process. The output is a reference to the stored code.

[0739] Step 2:

[0740] The server passes the received code to the generative AI engine for analysis. At this stage, the code reference saved in step 1 is used as input. The generative AI model is used to identify elements such as functions, variables, and control flow, and performs data calculations to generate visualization data. As output, data is created that visually represents each part of the code.

[0741] Step 3:

[0742] The server sends the generated visualization data to the user terminal. The visualization data generated in step 2 is used as input. The data is serialized into an appropriate format (e.g., JSON) and processed to make it viewable on the terminal's display device. The output is in a data format easily processed by the user terminal.

[0743] Step 4:

[0744] The user manipulates the visualization data through the terminal's user interface. The terminal uses the visualization data received in step 3 as input. The user performs specific actions such as dragging and dropping blocks to change the program flow or add new elements. The output is the modified visualization data.

[0745] Step 5:

[0746] The user terminal sends the modified visualization data to the server. The server receives the visualization data edited by the user in step 4 as input. The server analyzes the visualization data and performs data calculations to regenerate program code that reflects the changes. The output is the regenerated program code.

[0747] Step 6:

[0748] The regenerated code is provided to the user. The code generated in step 5 is used as input. The server distributes this code for user access and manages the data to ensure it is ultimately verifiable and usable by the user. As output, the complete program code is provided, available through the user interface.

[0749] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0750] This invention provides a program editing environment that takes into account the user's emotional state by incorporating an emotion engine into a system that visually manipulates program code. This system consists of server software, a user terminal interface, a generation AI engine, and an emotion engine.

[0751] First, the user's terminal receives the program code and sends it to the server. The server receives this code and performs analysis using a generation AI engine. As a result of the analysis, visualization data representing the logical structure of the program is generated, and this data is sent from the server to the user's terminal.

[0752] The user's terminal displays the generated visualization data on the user interface. This allows the user to visually confirm and manipulate the program flow. These operations include adding, deleting, and rearranging blocks.

[0753] The emotion engine recognizes the user's emotional state in real time by analyzing their facial expressions and voice. Based on this emotional state, the system dynamically adjusts the displayed content and user interface responses. For example, if the user is feeling stressed, the system may simplify interactions or display encouraging messages.

[0754] After the user visually edits the program again, the changes are sent from the terminal to the server. The server creates regenerated code based on the edits and provides it to the user. The user can then review and execute the regenerated code.

[0755] For example, suppose a user is struggling for a long time because they misunderstand a loop in a program. The emotion engine detects the user's frustration, and the system provides concrete examples of the loop structure and hints for solving it. In this way, the system provides a more intuitive and supportive programming environment, reducing the burden on the user.

[0756] The following describes the processing flow.

[0757] Step 1:

[0758] The user enters program code into the interface using their own device and presses the submit button. This sends the code from the device to the server.

[0759] Step 2:

[0760] The server receives program code sent by the user. First, the server sends the code to a generative AI engine to understand its structure.

[0761] Step 3:

[0762] The generation AI engine analyzes the received code. It identifies functions, variables, and control structures within the code and generates data for visual representation.

[0763] Step 4:

[0764] The server sends visualization data obtained from the AI ​​generation engine to the user's terminal. This visualization data is intended to show the user the structure of the program.

[0765] Step 5:

[0766] The terminal displays the received visualization data on the screen. Based on this visualization data, the user can visually understand and operate the program's flow and structure.

[0767] Step 6:

[0768] The emotion engine runs on the user's device, analyzing the user's facial expressions and voice through the camera and microphone while the user interacts with the program. The user's emotional state is monitored in real time.

[0769] Step 7:

[0770] Users edit the program using a visual interface. For example, they can add new conditional branches or adjust existing loops.

[0771] Step 8:

[0772] When the emotion engine detects user stress or frustration, the device provides corresponding interface support functions. Specifically, it displays user-friendly hints and advice.

[0773] Step 9:

[0774] The user completes the editing process, and the device sends the changes to the server. The server receives this information and builds regenerated code based on the edited visualization data.

[0775] Step 10:

[0776] The server returns the regenerated code to the user. The user reviews the new code and makes further edits or runs it as needed.

[0777] (Example 2)

[0778] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0779] In recent years, editing program code has become increasingly complex, making it a stressful process, especially for beginners and users unfamiliar with programming. Furthermore, the lack of graphical user interfaces that reflect the user's emotional state leads to inefficient support.

[0780] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0781] In this invention, the server includes means for receiving code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to the user, and means for analyzing emotions and adjusting the interface according to the user's emotional state. This makes it possible to provide a flexible program editing environment that is sensitive to the user's emotions and to reduce the user's stress and burden.

[0782] "Code" is text that describes instructions and commands to be executed by a computer.

[0783] "Analysis" is the process of breaking down received code and understanding its structure and function.

[0784] "Visualized data" refers to information that visually represents the logical structure of a program, converted into a format that users can easily understand and manipulate.

[0785] A "display device" is hardware or software that provides visualized data to the user in a visible format.

[0786] A "user interface" is a mechanism that provides users with a means to interact with a system and manipulate visualized data.

[0787] "Emotions" refer to the psychological state extracted from the user's facial expressions and voice, and are detected by the system.

[0788] "Interface adjustment" refers to dynamically changing the display and functionality of the user interface based on the user's emotional state.

[0789] A "programming language" is a standardized language used to give instructions to a computer, and there are many different types.

[0790] "Code regeneration" is the process of regenerating the original code based on the manipulated visualization data.

[0791] This invention is a system that provides support to users while they visually edit a program, taking into account their emotional state. The system mainly consists of server software, a user terminal, a generative AI engine, and an emotion engine.

[0792] The user's terminal has an interface for inputting program code. Once the user inputs code, the terminal sends it to the server. The server uses a generative AI engine to analyze the received code. This analysis generates the program's logical structure as visualized data. The server then sends this generated visualization data to the user's terminal.

[0793] The terminal displays visualized data on the user interface. This allows the user to visually confirm the program flow and manipulate it freely. These operations include adding, deleting, and rearranging program blocks.

[0794] The emotion engine can analyze the user's facial expressions and voice in real time to recognize their emotional state. Based on this emotional state, the device dynamically adjusts the content and responses of the user interface. For example, if the user is feeling frustrated, the system can simplify the interface and display encouraging messages.

[0795] After editing is complete, the user's terminal sends the changes to the server. The server regenerates the original code according to the changes and provides the result to the user. The user can then review and execute this regenerated code.

[0796] For example, if a user prompts with "I don't know how to use a for loop," the system automatically presents a concrete example of a loop structure to help the user understand. In this way, the system provides an intuitive and emotionally resonant programming environment.

[0797] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0798] Step 1:

[0799] The user's terminal accepts code through an interface for entering program code. The user begins by entering code into this interface. The entered code is processed as data in a format that is sent directly to the server. Specifically, if the user enters the code "I don't know how to use a for loop," the terminal will transfer that information to the server.

[0800] Step 2:

[0801] The server analyzes the received code using a generative AI engine. Here, the server analyzes the input code line by line, determining its structure and purpose. Data calculations include code tokenization and syntax tree generation. The output is data that visualizes the logical structure of the code. This visualization diagrams the structure like a program flowchart.

[0802] Step 3:

[0803] The server sends the generated visualization data to the user's terminal. The terminal displays the received visualization data on its user interface. Specifically, the visualized program flowchart or blocks are displayed on the screen, ready for the user to review.

[0804] Step 4:

[0805] Users manipulate the programming flow on the interface based on visualized data. Here, they can perform specific actions such as adding, deleting, and rearranging blocks. Based on the user's input, the terminal processes the data and generates the manipulated visualized data. This allows users to intuitively build and improve the program's logic.

[0806] Step 5:

[0807] The device uses an emotion engine to analyze the user's voice and facial expressions to determine their emotional state. If the user is feeling anxious, the device adjusts the data to display encouraging messages or simplified instructions on the screen. Specifically, the on-screen instructions and guidance change to clearly show the user what to do next.

[0808] Step 6:

[0809] After the user's actions are finalized, the terminal sends this information to the server. The server then regenerates the original code based on the manipulated visualization data. The input is the data modified by the user's actions, and after data calculations, the new code is output.

[0810] Step 7:

[0811] The server sends the regenerated code to the user's terminal. The user then verifies this regenerated code. Specifically, the newly generated program code is displayed on the screen, allowing the user to review its contents.

[0812] Step 8:

[0813] The user executes the code they have reviewed. This allows them to experience the actual program's operation and verify the results of the visualized program.

[0814] (Application Example 2)

[0815] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0816] Modern programming environments often fail to consider the user's operational capabilities or emotional state, leading to frustration and stress during code editing. Furthermore, understanding and manipulating complex programs requires advanced knowledge, necessitating additional support. This invention aims to address these challenges by understanding the user's emotional state in real time and providing appropriate interface adjustments and support accordingly.

[0817] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0818] In this invention, the server includes means for receiving code, means for analyzing the received code and generating visualization data, means for transmitting the generated visualization data to a display device, means for manipulating the visualization data via a user interface, means for regenerating the original code based on the manipulated visualization data, means for providing the regenerated code to the user, means for analyzing the user's emotional state, and means for dynamically adjusting the display content based on the user's emotional state. This allows the user to receive optimal support according to their emotional state and reduces the burden during program editing.

[0819] "Means of receiving code" refers to the function by which the system receives program code entered by the user.

[0820] "Means for analyzing received code and generating visualization data" refers to a function that analyzes the received code and creates data to visually represent its logical structure.

[0821] "Means for transmitting generated visualization data to a display device" refers to a function that transmits the generated visualization data to the user's terminal, making it displayable on a display device such as a screen.

[0822] "Means of manipulating visualized data through a user interface" refers to a function that provides an interface that allows users to edit a program through visualized data.

[0823] "Means for regenerating the original code based on manipulated visualization data" refers to a function that rewrites the original program code based on the visual operations performed by the user.

[0824] "Means of providing regenerated code to users" refers to a function that presents and allows users to use the newly generated code.

[0825] "Means for analyzing the user's emotional state" refers to a function that analyzes the user's facial expressions, voice, etc., to understand their emotional state in real time.

[0826] "Means for dynamically adjusting displayed content based on the user's emotional state" refers to a function that appropriately changes the user interface and displayed messages according to the analyzed emotional state, thereby improving the user's experience.

[0827] This invention provides a system that dynamically adjusts the programming environment according to the user's emotional state. The system mainly consists of a server, a user terminal, a generative AI engine, and an emotion engine.

[0828] The server receives the code entered by the user, analyzes it with the help of a generation AI engine, and generates visualization data. This visualization data is designed to allow the user to intuitively understand the structure and flow of the program. This data is sent from the server to a display device and displayed on the user's terminal.

[0829] The user terminal can be, for example, a smartphone or tablet. Through the terminal's user interface, the user can manipulate the visualized data of the program code. Specifically, they can edit it by adding, deleting, and rearranging blocks.

[0830] The emotion engine analyzes the user's facial expressions and voice in real time to understand their emotional state. Based on this emotional data, the system dynamically adjusts the content and interactions of the user interface. For example, if the user is feeling stressed, the system can simplify operations or display encouraging messages.

[0831] As a concrete example, consider a situation where a user is visually editing the robot's operation program to control a robot in their home. If the system detects that the user is confused, the emotion engine communicates this to the server, which then provides step-by-step guidance and specific operation suggestions to the user's terminal.

[0832] This system allows users to receive support tailored to their emotional state, resulting in a more comfortable and efficient programming experience.

[0833] An example of a prompt to input into the generating AI model is: "The user is trying to set a cleaning schedule for a robot, but is showing signs of confusion during the process. Based on sentiment analysis, suggest how to provide support."

[0834] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0835] Step 1:

[0836] The server receives program code entered from the user's terminal. The input is the program code entered by the user on the terminal. Upon receiving this code, the server prepares to proceed to the next analysis process.

[0837] Step 2:

[0838] The server uses a generation AI engine to analyze the received program code. The input is the program code received in step 1. As a result of the analysis, it generates data to visualize the logical structure of the code. This visualized data is the output.

[0839] Step 3:

[0840] The server sends the generated visualization data to the user's terminal. The terminal receives this visualization data as input and prepares to display it on the user interface. As a result, the output visualization data becomes available for viewing on the terminal.

[0841] Step 4:

[0842] The user manipulates the visualized data through the terminal's user interface. The input is the visualized data displayed on the terminal, and the user performs operations such as adding, deleting, and rearranging blocks based on this data. The results of the user's operations are passed on to the next step.

[0843] Step 5:

[0844] The terminal reshapes the visualization data based on user actions and sends that data to the server. The input is the visualization data after user actions, and the output is the instruction for regeneration.

[0845] Step 6:

[0846] The server regenerates the original program code based on the user's re-formed visualization data. The input is the data sent in step 5, and the output is the regenerated program code.

[0847] Step 7:

[0848] The server provides the user with regenerated program code. The user reviews this code on their terminal and executes it if necessary. The input is the regenerated code, and the output is information for the user to review.

[0849] Step 8:

[0850] The emotion engine analyzes the user's facial expressions and voice to understand their emotional state. Input consists of audio and image data obtained through the camera and microphone. Based on these results, it outputs the user's emotional state.

[0851] Step 9:

[0852] The server adjusts the user interface display based on the user's emotional state, which is analyzed by the emotion engine. The input is the user's emotional state, and the output is the adjusted interface display and interaction content.

[0853] As a concrete example of its operation, if a user is experiencing stress while editing a program, the emotion engine can detect this state, and the server can instruct the interface to simplify the operation. In this case, a generative AI model is used to provide guidance that includes prompts that are easy for the user to understand. A sample prompt could be: "The user is trying to set a cleaning schedule for the robot, but is showing signs of confusion in the process. Based on emotion analysis, please suggest how to provide support."

[0854] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0855] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0856] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0857] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0858] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0859] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0860] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0861] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0862] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0863] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0864] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0865] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0866] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0867] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0868] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0869] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0870] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0871] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0872] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0873] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0874] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0875] The following is further disclosed regarding the embodiments described above.

[0876] (Claim 1)

[0877] A means of receiving a code,

[0878] A means of analyzing the received code and generating visualization data,

[0879] A means for transmitting the generated visualization data to a display device,

[0880] A means of manipulating visualization data through a user interface,

[0881] A means for regenerating the original code based on manipulated visualization data,

[0882] A means of providing the regenerated code to the user,

[0883] A system that includes this.

[0884] (Claim 2)

[0885] The system according to claim 1, comprising means for visualizing changes in real time when a user edits program code.

[0886] (Claim 3)

[0887] The system according to claim 1, comprising an analysis means that enables support for multiple programming languages.

[0888] "Example 1"

[0889] (Claim 1)

[0890] Means of receiving information,

[0891] A means of analyzing received information and converting the data into visual information,

[0892] Means for transmitting converted visual information to a device for display,

[0893] A means of manipulating visual information using a user interface,

[0894] A means of regenerating the original information based on manipulated visual information,

[0895] Means for providing regenerated information,

[0896] A system that includes this.

[0897] (Claim 2)

[0898] The system according to claim 1, comprising means for instantly visualizing changes when a user edits information.

[0899] (Claim 3)

[0900] The system according to claim 1, which is equipped with an analysis function that can support a large number of programming languages.

[0901] "Application Example 1"

[0902] (Claim 1)

[0903] A means of receiving a code,

[0904] A means of analyzing the received code and generating visualization data,

[0905] A means for transmitting the generated visualization data to a display device,

[0906] A means of manipulating visualization data through a user interface,

[0907] A means for regenerating the original code based on manipulated visualization data,

[0908] A means of providing the regenerated code to the user,

[0909] A means for users to visually change the work process of a machine,

[0910] A system that includes this.

[0911] (Claim 2)

[0912] The system according to claim 1, comprising means for visualizing changes in real time when a user edits program code.

[0913] (Claim 3)

[0914] The system according to claim 1, comprising an analysis means that enables support for multiple programming languages.

[0915] "Example 2 of combining an emotion engine"

[0916] (Claim 1)

[0917] A means of receiving a code,

[0918] A means of analyzing the received code and generating visualization data,

[0919] A means for transmitting the generated visualization data to a display device,

[0920] A means of manipulating visualization data through a user interface,

[0921] A means for regenerating the original code based on manipulated visualization data,

[0922] A means of providing the regenerated code to the user,

[0923] A means of analyzing emotions and adjusting the interface according to the user's emotional state,

[0924] A system that includes this.

[0925] (Claim 2)

[0926] The system according to claim 1, comprising means for visualizing changes in real time when a user edits program code and adjusting the user interface based on the user's emotional state.

[0927] (Claim 3)

[0928] The system according to claim 1, comprising an analysis means that enables support for multiple programming languages, and a means that supports program editing while taking into account emotional states.

[0929] "Application example 2 when combining with an emotional engine"

[0930] (Claim 1)

[0931] A means of receiving a code,

[0932] A means of analyzing the received code and generating visualization data,

[0933] A means for transmitting the generated visualization data to a display device,

[0934] A means of manipulating visualization data through a user interface,

[0935] A means for regenerating the original code based on manipulated visualization data,

[0936] A means of providing the regenerated code to the user,

[0937] A means of analyzing the user's emotional state,

[0938] A means of dynamically adjusting the displayed content based on the user's emotional state,

[0939] A system that includes this.

[0940] (Claim 2)

[0941] The system according to claim 1, comprising means for visualizing changes in real time when a user edits program code and providing feedback according to the user's emotional state.

[0942] (Claim 3)

[0943] The system according to claim 1, comprising an analysis means that enables support for multiple programming languages ​​and a means that adapts the user interface while taking into account the user's emotional state. [Explanation of Symbols]

[0944] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving a code, A means of analyzing the received code and generating visualization data, A means for transmitting the generated visualization data to a display device, A means of manipulating visualization data through a user interface, A means for regenerating the original code based on manipulated visualization data, A means of providing the regenerated code to the user, A means for users to visually change the work process of a machine, A system that includes this.

2. The system according to claim 1, comprising means for visualizing changes in real time when a user edits program code.

3. The system according to claim 1, comprising an analysis means that enables support for multiple programming languages.

Images