System

Abstract

A system is provided.SOLUTION: This system includes a means for analyzing a natural language inquiry and retrieving a code piece matched with a purpose, a means for visually expressing the operation of the retrieved code piece, a means for generating a detailed document including a test case and a use example related to the code piece, and a means for transmitting a retrieval result, the visual expression and the detailed document to the terminal of a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Previously, when programmers and engineers were searching for code to achieve a specific function or behavior, they had to manually search through fragments of code using the same words and examine them in detail. This was time-consuming and inefficient, and often resulted in them not finding the most appropriate code fragment. Another problem was that there were limited ways to visually understand the code's behavior, which made it time-consuming to understand the code. Furthermore, there was no system for integrating test cases, use cases, and bug reports related to a code fragment and providing them as a single document, which made it time-consuming to check the quality and scope of the code. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system including: means for analyzing a natural language query and searching for code fragments that match a purpose; means for visually representing the behavior of the searched code fragments; means for generating detailed documentation including test cases and use cases related to the code fragments; and means for transmitting the search results, the visual representation, and the detailed documentation to a user's terminal. In particular, the means for visually representing the code behavior analyzes the code structure to generate a flowchart or animation, and generates documentation that integrates not only test cases and use cases related to the code fragments, but also related bug reports and improvement suggestions. This configuration enables users to quickly and efficiently search for code fragments that match a purpose, visually understand their behavior, and comprehensively grasp related information.

[0006] A "natural language query" refers to a natural language-based question or request entered by a user.

[0007] "Parsing" is the process of breaking down a natural language query and extracting its intent and keywords.

[0008] A "code fragment" refers to a part of a program that achieves a specific function or operation.

[0009] "Visual representation" refers to displaying the behavior and structure of a code fragment in a visual format such as a flowchart or animation.

[0010] A "test case" is a set of specific input data and expected results used to verify the behavior or functionality of a piece of code.

[0011] "Use Case" refers to a concrete example code or scenario that shows how a piece of code should be used.

[0012] "Detailed documentation" refers to a comprehensive document that integrates explanations, test cases, use cases, and other relevant information related to a piece of code.

[0013] "User device" refers to an electronic device, such as a computer or smartphone, on which a user enters a search query and displays search results, visual representations, and detailed documents.

[0014] "Server" refers to a central computer system that processes natural language queries, searches for code fragments, analyzes them, generates visual representations, and creates detailed documentation.

[0015] A "flowchart" is a diagram that shows the order in which a piece of code operates, visually indicating each processing step and conditional branch.

[0016] "Animation" refers to a visual representation of the behavior or process of a piece of code through movement.

[0017] A "bug report" is a report of a defect in a piece of code, including details of the problem and steps to reproduce it.

[0018] An "improvement proposal" is a proposal aimed at improving the performance of a code fragment or adding functionality, and refers to a document that describes specific changes and expected effects. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention provides a system for searching code in natural language, displaying code fragments in a visually understandable format, and providing related detailed information. Specific embodiments of the system are described below.

[0041] Overall system overview

[0042] This system consists of users, terminals, and a server.

[0043] User operations

[0044] A user uses a device (such as a computer or smartphone) to search for code related to a specific function or operation using natural language, for example, typing "What is the Python code to sort a list?"

[0045] Device Role

[0046] The terminal receives input from the user, transmits it to a server via the Internet, and displays search results, visual representations, and detailed documents returned by the server, allowing the user to quickly and efficiently obtain the desired information.

[0047] Server Roles

[0048] The server has the following main functions:

[0049] 1. Natural Language Query Parsing:

[0050] The server receives the natural language query sent by the user and analyzes its intent. Specifically, it uses natural language processing technology to extract keywords and contexts, which are then used as search criteria.

[0051] 2. Search for code fragments:

[0052] Based on the analysis results, it searches for code fragments that match the purpose from an internal database or an external code repository, thereby finding the appropriate code to achieve the functionality or operation the user desires.

[0053] 3. Generate visual representation:

[0054] The behavior of the found code fragments is converted into a form that can be understood visually. Specifically, the code process is expressed as a flowchart or animation, allowing users to intuitively understand the structure and behavior of the code.

[0055] 4. Generate detailed documentation:

[0056] It collects test cases, use cases, bug reports and improvement suggestions related to a piece of code and generates a consolidated and detailed documentation, which gives users a comprehensive understanding of the code's quality and coverage.

[0057] 5. Sending the results:

[0058] The visual representation and detailed documentation are sent to the terminal, which displays it and provides it to the user.

[0059] Specific examples

[0060] Below is a specific example where a user searches for "Python code to sort a list."

[0061] 1. User input:

[0062] The user types "Please tell me the Python code to sort a list" into the terminal and clicks the search button.

[0063] 2. Sending by terminal:

[0064] The terminal sends this input to the server.

[0065] 3. Query parsing and code lookup by the server:

[0066] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches its internal database for relevant code fragments.

[0067] 4. Generating a visual representation:

[0068] The server analyzes the steps in the code fragment it finds and generates a flowchart that shows the process of taking an input list, sorting the elements, and returning the result.

[0069] 5. Generate detailed documentation:

[0070] Generate documentation including relevant test cases (e.g., sorting a list of numbers or a list of strings), use cases (specific sorting methods using particular lists), bug reports (issues that have been encountered and how they have been resolved), and improvement suggestions (e.g., a more efficient sorting algorithm).

[0071] 6. Sending and viewing results:

[0072] The server sends this information to the device, which displays the results to the user, who can review code snippets, flowcharts, and detailed documentation and incorporate them into their own projects as needed.

[0073] The above is a specific embodiment of the present invention, and this system enables users to efficiently find code fragments that suit their purpose, understand them visually, and obtain related information comprehensively.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] A user enters a search query into a terminal, such as "Can you give me the Python code to sort a list?" in natural language, and clicks the search button.

[0077] Step 2:

[0078] The device sends the query entered by the user to the server as is, and then communicates via the Internet to pass the data to the server.

[0079] Step 3:

[0080] The server analyzes the received natural language query, and the query analysis engine uses natural language processing techniques to extract key keywords (in this case, "list," "sort," and "Python").

[0081] Step 4:

[0082] The server searches for relevant code fragments in its internal database or external code repository based on the extracted keywords. The search engine finds code fragments that match the keywords and displays them in a list.

[0083] Step 5:

[0084] The server selects the most appropriate code fragment from the searched code fragments and visually represents its behavior. The visual representation generation component analyzes the code structure and generates visual data as flowcharts or animations.

[0085] Step 6:

[0086] The server retrieves test cases and use cases related to a piece of code from a database, as well as related bug reports and improvement suggestions, and synthesizes them to generate comprehensive documentation. The documentation generation component organizes this information and presents it in a user-friendly format.

[0087] Step 7:

[0088] The server sends the generated visual representation and detailed documentation to the device, which receives the data and converts it into a format that can be displayed to the user.

[0089] Step 8:

[0090] The terminal displays the received visual representation and detailed documentation to the user, who can then review the code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[0091] The above is the processing flow from the user's natural language query to code search, analysis, visual representation generation, document generation, and result display.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Conventional code search systems make it difficult for users to quickly find the code fragment they are looking for and intuitively understand its behavior. They also lack the functionality to provide detailed information such as related test cases and use cases. As a result, users have to spend a lot of time finding the appropriate code fragment, and it is difficult to fully understand the quality and scope of the code.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes: means for analyzing a user's query using natural language processing technology; means for retrieving code fragments from an internal database or an external repository; means for converting the retrieved code procedures into a flowchart or visual diagram; means for generating detailed documentation using a generative AI model; means for analyzing the natural language query and searching for code fragments that match a purpose; means for visually representing the behavior of the retrieved code fragments; means for generating detailed documentation including test cases and use cases related to the code fragments; and means for transmitting the search results, the visual representation, and the detailed documentation to the user's terminal. This allows the user to efficiently find code fragments that match their purpose, visually understand them, and comprehensively obtain related information.

[0097] A "natural language query" is a question or request made by a user to a computer system in natural human language.

[0098] A "query" is a statement or command that describes instructions for retrieving information from a database or information system.

[0099] "Analysis" is the process of separating and analyzing data and information to understand its intent, content, and structure.

[0100] A "code fragment" is a part of a program or script, a fragment of source code that realizes a specific function or operation.

[0101] "Visual representation" refers to the display of information or data in visual formats such as graphs, flow charts, animations, etc.

[0102] A "flowchart" is a diagram that visually shows the flow of an algorithm or process by representing each step with shapes and arrows.

[0103] "Animation" is a technique for expressing movement by displaying a series of still images in succession over time.

[0104] A "test case" is a set of input data and expected results designed to verify a particular function or behavior of software or a system.

[0105] A "use example" is a concrete example that shows how to actually use a specific function or operation.

[0106] A "document" is a document or material created to record information and communicate it to others.

[0107] A "bug report" is a document used to report defects or problems within software or a system, and describes the incident, the steps to reproduce it, the scope of the impact, etc.

[0108] An "improvement proposal" is a document that describes specific proposals and methods for improving the performance or usability of an existing system or process.

[0109] A "generative AI model" is an artificial intelligence model that is generated using machine learning or deep learning techniques and is designed to solve a specific task or problem.

[0110] A "search result" is a set of information returned based on a search query.

[0111] The "Internet" is a global information and communications network of interconnected computer networks.

[0112] "API" is an abbreviation for Application Programming Interface, and is a set of definitions and rules for exchanging functions and data between software programs.

[0113] An "internal database" is a database managed within the system, used to store and manage user and system data.

[0114] An "external repository" is a database or information resource that exists outside the system, where code and data that can be accessed from outside are managed.

[0115] MODE FOR CARRYING OUT THE INVENTION

[0116] The present invention provides a system for searching code in natural language, displaying code fragments in a visually understandable format, and providing related detailed information. Specific embodiments of the system are described below.

[0117] Overall system overview

[0118] This system consists of a user, a terminal, and a server. The user uses the terminal to search for codes related to specific functions or operations in natural language. The terminal receives input from the user and sends it to the server. The server analyzes the received input and returns the search results. Specifically, the server operates based on the following steps.

[0119] User operations

[0120] A user uses a device (such as a computer or smartphone) to search for code related to a specific function or operation using natural language, for example, typing "What is the Python code to sort a list?"

[0121] Device Role

[0122] The device receives input from the user and sends it to a server over the Internet, for example, as text data in JSON format. It also has the ability to display search results, visual representations, and detailed documents returned by the server, allowing the user to quickly and efficiently obtain the desired information.

[0123] Server Roles

[0124] The server has the following main functions:

[0125] 1. Natural Language Query Parsing:

[0126] The server receives natural language queries sent by users and analyzes their intent. It uses natural language processing techniques to extract keywords and contexts and use them as search criteria. This process uses generative AI models such as BERT and GPT-4.

[0127] 2. Search for code fragments:

[0128] Based on the analysis results, it searches for matching code fragments from an internal database or external code repositories (e.g., GitHub or GitLab), thereby finding the appropriate code to achieve the functionality or operation the user is looking for.

[0129] 3. Generate visual representation:

[0130] The behavior of the found code fragments is then translated into a visually understandable form, specifically using tools like PlantUML or Draw.io to represent the code's processes as flowcharts or animations, allowing users to intuitively understand the code's structure and behavior.

[0131] 4. Generate detailed documentation:

[0132] It collects test cases, use cases, bug reports, and improvement suggestions related to a piece of code and generates a consolidated and detailed document. It uses a generative AI model to generate the document by inputting a prompt such as:

[0133] "Generate documentation containing test cases, use cases, bug reports, and improvement suggestions related to the code snippets found in the search results. Format should be Markdown."

[0134] 5. Sending the results:

[0135] The visual representation and detailed documentation are sent to the terminal, which displays it and provides it to the user.

[0136] Specific examples

[0137] Below is a specific example where a user searches for "Python code to sort a list."

[0138] 1. User input:

[0139] The user types "Please tell me the Python code to sort a list" into the terminal and clicks the search button.

[0140] 2. Sending by terminal:

[0141] The terminal sends this input to the server.

[0142] 3. Query parsing and code lookup by the server:

[0143] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches its internal database for relevant code fragments.

[0144] 4. Generating a visual representation:

[0145] The server analyzes the steps in the code fragments it finds and generates a flowchart using PlantUML that shows the process of receiving an input list, sorting the elements, and returning the results.

[0146] 5. Generate detailed documentation:

[0147] The server generates documentation in LaTeX and Markdown format containing test cases, use cases, bug reports, and improvement suggestions. For example, the following prompt is input to the generative AI model:

[0148] "Generate documentation containing test cases, use cases, bug reports, and improvement suggestions related to the code snippets found in the search results. Format should be Markdown."

[0149] 6. Sending and viewing results:

[0150] The server sends this information to the device, which displays the results to the user: code snippets, flowcharts, and detailed documentation that the user can review and, if desired, incorporate into their own projects.

[0151] The above is an embodiment of the present invention, and this system enables users to efficiently find code fragments that suit their purpose, understand them visually, and obtain related information comprehensively.

[0152] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0153] Step 1:

[0154] Accepting user input

[0155] The user types into the terminal, "Can you give me the Python code to sort a list?"

[0156] Input: Natural language text input by the user

[0157] Output: User-entered text data

[0158] Specific operation: A user enters the required query into a text input field in a web browser or mobile app and clicks the search button. The device retrieves this input using JavaScript or similar tools and prepares it for further processing.

[0159] Step 2:

[0160] Terminal input transmission

[0161] The terminal receives input from the user and sends it to the server.

[0162] Input: User-entered text data

[0163] Output: Sending text data to the server

[0164] What it does: The device uses JavaScript or other front-end technologies to convert user input into JSON format and send it to the server via an AJAX request, while displaying a sending indicator to the user.

[0165] Step 3:

[0166] Query analysis by the server

[0167] The server parses the received user input.

[0168] Input: Received text data (query)

[0169] Output: Keyword and context extraction results

[0170] How it works: The server parses the incoming data and analyzes it using natural language processing techniques. It uses generative AI models such as BERT and GPT-4 to analyze the query and extract key keywords and intent. This process extracts keywords such as "list," "sort," and "Python."

[0171] Step 4:

[0172] Server lookup of code fragments

[0173] The server uses the analysis results to search for the code fragment.

[0174] Input: Keyword and context extraction results

[0175] Output: relevant code fragment

[0176] What happens: The server executes SQL queries to search its internal database, and optionally retrieves relevant code fragments from external code repositories (e.g., GitHub or GitLab) using API requests. As a result, the relevant code fragments are stored in the server's temporary memory.

[0177] Step 5:

[0178] Generating a visual representation

[0179] The server provides a visual representation of the retrieved code fragment.

[0180] Input: The searched code fragment

[0181] Output: Visual flowcharts and animations

[0182] How it works: The server analyzes the structure of a code fragment, calls the APIs of PlantUML and Draw.io, and generates flowcharts and animations. The generated visual data is saved as SVG or PNG format files.

[0183] Step 6:

[0184] Generate detailed documentation

[0185] The server generates detailed documentation related to the code fragment.

[0186] Input: The searched code fragment

[0187] Output: Detailed documentation (Markdown or LaTeX format)

[0188] How it works: The following prompt is input to the generative AI model to generate a document: "Please generate a document containing test cases, use cases, bug reports, and improvement suggestions related to the code fragment found in the search results. Please use Markdown format." The generated document is saved on the server in Markdown or LaTeX format.

[0189] Step 7:

[0190] Sending and viewing results

[0191] The server sends the visual representation and detailed documentation to the terminal, which displays the results to the user.

[0192] Input: Visual flowcharts, animations, detailed documentation

[0193] Output: The results displayed on the user's terminal

[0194] What it does: The server encodes the generated data in JSON format and sends it to the device as an HTTP response. The device receives it, renders the data using HTML and JavaScript, and displays it to the user. The user can then view the displayed code fragments, flowcharts, and detailed documentation and incorporate them into their own projects as needed.

[0195] The above are the specific processing steps of this system, which allow users to efficiently obtain the information they require.

[0196] (Application example 1)

[0197] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0198] While conventional code search systems analyze natural language and provide relevant code fragments, they lack sufficient visual understanding support. They also struggle to provide comprehensive, relevant detailed information. Furthermore, they have limitations in searching for code fragments specialized for specific applications and generating visual representations, making it difficult for developers to quickly and accurately obtain the information they need. For this reason, further improvements are needed in specialized fields such as electronic payments.

[0199] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0200] In this invention, the server includes means for analyzing a natural language query and searching for code fragments that match a purpose, means for visually representing the behavior of the searched code fragments, means for generating detailed documentation including test cases and use examples related to the code fragments, means for transmitting the search results, the visual representation, and the detailed documentation to a user's terminal, means for using a generative AI model to search for code fragments related to the query content and providing the code fragments together with their visual representations, and means for analyzing the user's intent using prompt sentences and generating related code fragments and documentation. This enables a user to quickly obtain specific code fragments related to electronic payment based on a natural language query, visually understand them, and comprehensively obtain detailed documentation.

[0201] A "natural language query" is a question or command entered by a user in human language.

[0202] A "code fragment" is a portion of program code that implements a specific function or operation.

[0203] "Visually representing" means displaying textual information in graphical form, such as a flow chart or animation.

[0204] A "test case" is a set of specific inputs and their expected outputs that are executed to verify the behavior of a program.

[0205] "Detailed documentation" refers to comprehensive information related to a piece of code, including explanations, use cases, bug reports, improvement suggestions, etc.

[0206] A "generative AI model" is an artificial intelligence program that uses machine learning techniques to analyze natural language and generate or find appropriate code fragments.

[0207] A "prompt sentence" is an input sentence for conveying the user's intention, and is a sentence used as a natural language query.

[0208] A "terminal" is a device that can be operated by a user, and specifically includes a personal computer, a smartphone, etc.

[0209] A "server" is a computer system that processes and stores data and provides information in response to requests from terminals.

[0210] "User" refers to a person who uses this system.

[0211] This invention provides a system that allows a user to input a query in natural language using a terminal (such as a smartphone or computer), search for code fragments that match the user's purpose, and present the results in a visually easy-to-understand format. This system is mainly composed of the following elements:

[0212] 1. Parsing natural language queries

[0213] The user uses the device to provide input in natural language, such as a prompt such as "What is the Python code to process a credit card payment?" The device then sends this input to the server, which uses a generative AI model to analyze the query and extract relevant keywords and intent. This generative AI model uses the Hugging Face transformers library.

[0214] 2. Searching for code fragments

[0215] Based on the extracted keywords and intent, the server searches for relevant code fragments from an internal database or external repository that contains code fragments written in various programming languages, including, for example, Python code for processing credit card payments.

[0216] 3. Generating Visual Representations

[0217] For each code fragment found, the server converts its behavior into a visually understandable representation, using Matplotlib and NetworkX to represent the code steps as flowcharts and animations. This visual representation helps users intuitively understand the code's flow and structure.

[0218] 4. Generate detailed documentation

[0219] The server generates detailed documentation such as test cases, use cases, bug reports, and improvement suggestions related to the retrieved code fragments, allowing users to gain a comprehensive understanding of the code's quality and coverage.

[0220] 5. Sending and displaying results

[0221] The server sends the search results, visual representations, and detailed documentation to the terminal, which then displays them to the user, who can quickly and efficiently obtain the information they need based on the displayed code fragments, flowcharts, and detailed documentation.

[0222] As a concrete example, consider the case where a user uses the following prompt text:

[0223] "Can you give me some Python code to process credit card payments?"

[0224] In response, the system searches for relevant code fragments, generates visual representations and detailed documentation, and provides them to users, allowing them to quickly extract the code they need and apply it to their own projects.

[0225] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0226] Step 1:

[0227] Accepting natural language inquiries

[0228] The user uses a terminal to input a query in natural language. At this stage, the user enters a prompt sentence such as "Please give me the Python code to process credit card payments." The input data is sent over the Internet to the server.

[0229] Input: Natural language query (prompt)

[0230] Output: The query sent to the server

[0231] Step 2:

[0232] Natural Language Analysis

[0233] The server analyzes the received query using a generative AI model (Hugging Face's transformers library). This model extracts keywords and user intent from the input natural language text. Specifically, keywords such as "credit card," "payment," and "Python" are extracted.

[0234] Input: Received natural language query

[0235] Output: Extracted keywords and user intent

[0236] Step 3:

[0237] Searching for code fragments

[0238] The server then searches for relevant code fragments from an internal database or external repository based on the extracted keywords and intent. This database contains code fragments written in various programming languages, and the search result is "Python code to process credit card payments."

[0239] Input: Extracted keywords and user intent

[0240] Output: relevant code fragment

[0241] Step 4:

[0242] Generating a visual representation

[0243] The server then converts the code fragments it finds into a visually understandable representation of their behavior, using Matplotlib and NetworkX to generate flowcharts and animations of the code steps, providing a visual representation of the code's flow and structure.

[0244] Input: The searched code fragment

[0245] Output: Generated flowchart or animation

[0246] Step 5:

[0247] Generate detailed documentation

[0248] The server generates detailed documentation containing test cases, use cases, bug reports, and improvement suggestions associated with the code fragment, providing data for a comprehensive understanding of the code's quality and coverage.

[0249] Input: The searched code fragment

[0250] Output: Detailed documentation (test cases, use cases, bug reports, improvement suggestions)

[0251] Step 6:

[0252] Sending and displaying results

[0253] The server sends the search results, visual representations, and detailed documentation to the terminal, which displays them to the user, allowing the user to quickly and efficiently obtain the information they need based on the displayed code fragments, flowcharts, and detailed documentation.

[0254] Input: Search results, visual representation, detailed documentation

[0255] Output: what is displayed on the user's device (code fragments, visual representations, detailed documentation)

[0256] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0257] The present invention incorporates an emotion engine that recognizes the user's emotions into a system that performs code searches in natural language, displays the results in a visually easy-to-understand format, and provides related information. Specific embodiments of the present invention are described below.

[0258] Overall system overview

[0259] This system consists of a user, a terminal, a server, and an emotion engine.

[0260] User operations

[0261] Users use the terminal to search for code related to a particular function or operation using natural language, for example by entering a query such as "What is the Python code to sort a list?" and clicking the search button.

[0262] Device Role

[0263] The device receives input from the user and sends it to the server, displays search results, visual representations, and detailed documents sent from the server, and collects emotional data from the user's facial expressions and voice and sends it to the server.

[0264] Server Roles

[0265] The server has the following main functions:

[0266] 1. Natural Language Query Parsing:

[0267] The server analyzes the natural language queries sent by users and extracts keywords and context using natural language processing techniques.

[0268] 2. Search for code fragments:

[0269] Based on the analysis results, the search engine searches for relevant code fragments in internal databases and external code repositories, and finds code fragments that match the keywords.

[0270] 3. Generate visual representation:

[0271] The behavior of the found code fragments is converted into a visually understandable form (flowchart or animation) by the visual representation generation component, which analyzes the code structure.

[0272] 4. Generate detailed documentation:

[0273] It collects test cases, use cases, bug reports and improvement suggestions related to a piece of code and generates a single comprehensive document.

[0274] 5. User sentiment analysis:

[0275] The emotion engine analyzes the user's emotional data sent from the device and adjusts search results and visual presentation accordingly, for example by adding more detailed explanations if the user is confused.

[0276] 6. Sending the results:

[0277] Finally, adjustment information that takes into account the user's emotion data is sent to the terminal together with the visual representation and detailed documentation.

[0278] Specific examples

[0279] Below is a concrete example of how the sentiment engine works in the process of a user searching for "Python code to sort a list."

[0280] 1. User input:

[0281] The user enters "Please tell me the Python code to sort the list" into the terminal and clicks the search button. The terminal simultaneously collects the user's facial expression and voice data.

[0282] 2. Sending by terminal:

[0283] The terminal transmits the query and the user's emotion data to the server.

[0284] 3. Query parsing and code lookup by the server:

[0285] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches the database for the corresponding code fragment.

[0286] 4. Generating a visual representation:

[0287] The server analyzes the steps in the code fragment it finds and generates a flowchart that shows the process of taking an input list, sorting the elements, and returning the result.

[0288] 5. Generate detailed documentation:

[0289] Generate documentation including relevant test cases (e.g., sorting a list of numbers or a list of strings), use cases (specific sorting methods using particular lists), bug reports (issues that have been encountered and how they have been resolved), and improvement suggestions (e.g., a more efficient sorting algorithm).

[0290] 6. User sentiment analysis:

[0291] The server analyzes the user's emotional data through an emotion engine and determines that the user is confused, and accordingly adjusts the generated documentation to include additional explanations and examples.

[0292] 7. Submitting and Viewing Results:

[0293] The server sends visual representations and detailed documentation to the user's device, reflecting the emotion-based adjustment information, and the device displays it, allowing the user to review code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[0294] The above is a specific embodiment of the present invention incorporating an emotion engine. This system allows users to efficiently find code fragments that suit their purpose, visually understand them, and comprehensively obtain related information, while also receiving appropriate support based on their emotions.

[0295] The processing flow will be explained below.

[0296] Step 1:

[0297] Using a terminal, a user enters a natural language query such as "Can you give me the Python code to sort a list?" and clicks the search button.

[0298] Step 2:

[0299] The device collects the user's facial expressions and voice data along with natural language queries from the user, and these data are simultaneously sent to the server as emotion data for analyzing the user's emotions.

[0300] Step 3:

[0301] The server analyzes the received natural language query, and the query analysis engine uses natural language processing techniques to extract key keywords such as "list," "sort," and "Python."

[0302] Step 4:

[0303] The server searches for relevant code fragments from an internal database or external code repository based on the analyzed keywords, and the search engine extracts matching code fragments in a list.

[0304] Step 5:

[0305] The server selects the most suitable code fragment from those found and then uses a visual representation generation component to visually represent the behavior of the code fragment, generating a flowchart or animation of the code fragment.

[0306] Step 6:

[0307] The server collects test cases and use cases related to a piece of code from a database, as well as bug reports and improvement suggestions, and generates comprehensive documentation out of this information. This process is handled by the documentation generation component.

[0308] Step 7:

[0309] The server runs an emotion engine to analyze the user's emotion data sent from the device. This engine analyzes how the user feels about the displayed information, and if a negative emotion such as confusion is detected, the engine includes a more detailed explanation or additional examples.

[0310] Step 8:

[0311] The server transmits the generated visual representation, detailed documentation, and information adjusted based on the emotion to the terminal.

[0312] Step 9:

[0313] The terminal displays the received information to the user, showing visual flowcharts, detailed code descriptions, documentation with associated test cases, use cases, bug reports, and improvement suggestions, allowing the user to understand the code fragment and efficiently incorporate it into their own projects.

[0314] This series of steps allows users to obtain code fragments and related information in a more relevant and understandable way, based on the additional information of emotions.

[0315] Example 2

[0316] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0317] Conventional program code search systems often produce insufficient search results even when users input queries in natural language. Furthermore, providing information through code fragments that are difficult to understand visually or incomplete documentation contributes to a lack of user convenience. Furthermore, providing search results without considering the user's emotional state can be confusing, especially for novice users. To address these issues, systems are required that can analyze queries in natural language, display information that is easy to understand visually, and provide information that takes the user's emotions into account.

[0318] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0319] In this invention, the server includes means for analyzing a natural language query and searching for program code that matches a purpose, means for visually representing the behavior of the searched program code, means for generating detailed documentation including test cases and use cases related to the program code, means for collecting and analyzing emotional data, and means for adjusting the visual representation and the content of the detailed documentation based on the emotional data. This allows the user to obtain search results that are more accurate and visually easy to understand, and also makes it possible to provide appropriate information according to the user's emotional state.

[0320] A "natural language query" refers to a query entered by a user using a human language (e.g., Japanese, English, etc.).

[0321] "Program Code" means source code that contains instructions and procedures for performing a particular task or function.

[0322] A "search tool" refers to a mechanism that analyzes natural language queries and uses them to find appropriate program code from databases or external repositories.

[0323] "Visual representation means" refers to a mechanism for displaying the behavior and structure of program code using visualization techniques such as flowcharts and animations.

[0324] "Detailed documentation" refers to comprehensive documentation containing test cases, use cases, bug reports, and improvement suggestions related to the program code.

[0325] "Emotional data" refers to data that indicates the user's emotional state, collected from the user's facial expressions, voice, etc.

[0326] "Emotion analysis means" refers to a mechanism for analyzing collected emotion data and determining the user's emotional state.

[0327] "Adjustment means" refers to a mechanism that appropriately changes the visual presentation and detailed document content based on the results of sentiment analysis to provide information in a form that is easier for users to understand.

[0328] "User terminal" refers to a device such as a computer, smartphone, or tablet that is actually operated by a user.

[0329] The present invention provides a system that allows a user to search for program code in natural language, displays the code in a visually easy-to-understand format, and provides information based on the user's emotions. Specific embodiments will be described below.

[0330] System configuration

[0331] This system consists of a user, a terminal, a server, and an emotion engine.

[0332] User operations

[0333] Users use the terminal to search for program code related to a specific function or operation in natural language, for example, by entering a query such as "What is the Python code to sort a list?" and clicking the search button.

[0334] Device Role

[0335] The device receives input queries from the user and sends them to the server. It also has the function of collecting emotional data from the user's facial expressions and voice and transmitting it to the server. For example, it acquires the user's emotional data using a camera or microphone. It also displays search results, visual representations, and detailed documents received from the server to the user.

[0336] Server Roles

[0337] The server has the following main functions:

[0338] 1. Natural Language Query Parsing:

[0339] The server analyzes the natural language query submitted by the user, for example by using natural language processing tools such as SpaCy or NLTK to extract keywords such as "list," "sort," and "Python."

[0340] 2. Search for the program code:

[0341] Based on the analysis results, the relevant program code is searched for in an internal database (e.g., PostgreSQL) or an external code repository (e.g., GitHub).

[0342] 3. Generate visual representation:

[0343] Convert the searched program code into a visually understandable format (flowchart or animation). For example, generate a flow chart using D3.js.

[0344] 4. Generate detailed documentation:

[0345] It collects test cases, use cases, bug reports, and improvement suggestions related to program code and generates detailed documentation in Markdown format.

[0346] 5. Emotion analysis:

[0347] The server's emotion engine analyzes the user's emotion data sent from the device, for example, using an emotion recognition API to determine whether the user is confused.

[0348] 6. Sending the results:

[0349] The server generates a visual representation and detailed documentation, and transmits the information adjusted based on the sentiment analysis results to the terminal.

[0350] Specific examples

[0351] Here is a specific example of what happens when a user searches for "Can you give me the Python code to sort a list?"

[0352] 1. User input:

[0353] The user enters "Please tell me the Python code to sort the list" into the terminal and clicks the search button. The terminal simultaneously collects the user's facial expression and voice data.

[0354] 2. Sending by terminal:

[0355] The terminal transmits the query and the user's emotion data to the server.

[0356] 3. Query parsing and code lookup by the server:

[0357] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches the database for the corresponding code fragment.

[0358] 4. Generating a visual representation:

[0359] The server analyzes the steps in the code fragments it finds and generates a flowchart using D3.js.

[0360] 5. Generate detailed documentation:

[0361] The server creates a detailed document in Markdown containing relevant test cases, use cases, bug reports, and improvement suggestions.

[0362] 6. User sentiment analysis:

[0363] The server's emotion engine detects user confusion and adjusts the generated documentation to include additional explanations and examples.

[0364] 7. Submitting and Viewing Results:

[0365] The server sends the final visual representation and detailed documentation to the device, which displays it, allowing users to review code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[0366] Prompt Sentence Examples

[0367] Here are some example prompts for searches using this system:

[0368] "Enter 'Please tell me the Python code to sort a list' in the terminal and search."

[0369] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0370] Step 1:

[0371] The user enters a natural language query into the text input field of the terminal and clicks the search button. For example, they might type "Can you give me the Python code to sort a list?" This is the input data. The output they get from the terminal is the query.

[0372] Step 2:

[0373] The device sends queries entered by the user to the server. In addition, the device's built-in camera and microphone capture the user's facial expressions and voice data, which are also sent to the server. The input is the user's query and emotion data, and the output is an HTTP request to the server. Specifically, the device converts the query and emotion data into JSON format and sends it.

[0374] Step 3:

[0375] The server analyzes the received natural language query using a natural language processing tool (e.g., SpaCy or NLTK) to extract keywords and context. The input is the query sent from the terminal, and the output is the extracted keywords (e.g., "list," "sort," "Python"). Specifically, the server calls a text analysis engine to analyze the query.

[0376] Step 4:

[0377] The server searches for relevant program code in a database (e.g., PostgreSQL) or an external repository (e.g., GitHub) based on the extracted keywords. The input is the parsed keywords, and the output is the search results, which are program code fragments. Specifically, it executes SQL queries or API requests to retrieve data.

[0378] Step 5:

[0379] The server analyzes the searched program code and generates visual representations such as flowcharts and animations. This is done using D3.js or other visualization tools. The input is the program code fragments as search results, and the output is the visual representation. Specifically, each step of the program code is input into the visualization tool to generate a flowchart.

[0380] Step 6:

[0381] The server collects test cases, use cases, bug reports, and improvement suggestions related to the searched program code and generates detailed documentation. The documentation format is Markdown. The input is a fragment of program code, and the output is a detailed documentation. Specifically, it collects and integrates related information and generates a document in Markdown format.

[0382] Step 7:

[0383] The emotion engine on the server analyzes the user's emotion data collected from the device and determines the user's emotional state (e.g., confusion, satisfaction, etc.). The input is the emotion data sent from the device, and the output is the user's emotional state as a result of the analysis. Specifically, it calls the emotion recognition API and analyzes the data.

[0384] Step 8:

[0385] The server adjusts the content of the visual representation and detailed text based on the user's emotional state. For example, if the user is confused, it adds more detailed explanations and examples. The input is the user's emotional state, the visual representation, and the detailed text, and the output is the adjusted visual representation and detailed text. Specific operations include updating the text and visualization data.

[0386] Step 9:

[0387] The server sends the adjusted visual representation and detailed document to the terminal. The input is the adjusted visual representation and detailed document, and the output is an HTTP response to the terminal. As a specific operation, the server generates an HTTP response for sending data to the terminal.

[0388] Step 10:

[0389] The terminal displays the visual representation and detailed document received from the server to the user. The input is data from the server, and the output is the display to the user. Specifically, the terminal uses a browser or a dedicated application to render and display the visual representation and document.

[0390] (Application example 2)

[0391] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0392] In modern factories and production lines, robot programming and troubleshooting are extremely important, but they require specialized knowledge and experience. Furthermore, conventional code search systems and support tools provide only one-way information without considering the user's feelings. This often leaves users confused and reduces work efficiency. Another issue is the lack of visual support, which makes it time-consuming to understand and operate the code. A system that can solve these problems and provide more intuitive and efficient support is needed.

[0393] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing a natural language query and searching for code fragments that match the purpose, means for visually expressing the behavior of the searched code fragments, and means for analyzing user emotion data and adjusting the display method of the search results and visual expression based on the results. This makes it possible to provide optimal support that is easy for the user to intuitively understand and that is tailored to their emotions.

[0394] A "natural language query" is a question or instruction written in a language normally used by humans, with the goal of converting it into a form that the system can understand and process.

[0395] A "code fragment" is a part of a program that realizes a specific function or operation, and refers to a short piece of code that is used in actual programming work.

[0396] A "visual representation" is a visually easy-to-understand representation of the behavior of the retrieved code fragment, which can take the form of a flowchart or animation.

[0397] "Detailed documentation" refers to materials that contain supplementary and comprehensive information, such as test cases, use cases, bug reports, and improvement suggestions related to a piece of code.

[0398] "Emotional data" refers to emotional information extracted from a user's facial expressions and voice, and is used by the system to analyze and understand the user's emotional state.

[0399] "Adjusting the display method" refers to changing the format and content of search results and visual representations based on the user's emotional data to make them easier for the user to understand.

[0400] This system analyzes natural language queries, searches for code fragments that match the query, and displays the code's behavior in a visually understandable format. It also analyzes user sentiment and adjusts the display of search results and visual representations based on the results.

[0401] System configuration

[0402] This system consists of a user, a terminal, a server, and an emotion engine.

[0403] 1. Users

[0404] Users use devices such as smartphones or head-mounted displays to search for code fragments in natural language, for example by entering queries such as "What code can I use to change the speed of a robot?"

[0405] 2. Terminal

[0406] The device receives input from the user and sends it to the server. It also collects emotional data from the user's facial expressions and voice and sends this data to the server.

[0407] 3. Server

[0408] The server has the following functions:

[0409] Natural language query analysis: The server analyzes the natural language query sent by the user and extracts keywords and context using natural language processing techniques.

[0410] Searching for code fragments: Based on the analysis results, the database is searched for the relevant code fragments.

[0411] Visual representation generation: Convert the behavior of the retrieved code fragments into a visually understandable form (flowchart or animation).

[0412] Generate detailed documentation: Collect test cases, use cases, bug reports and improvement suggestions related to a piece of code and generate one comprehensive document.

[0413] User sentiment analysis: Analyzes user sentiment data sent from the device through the emotion engine and adjusts the way search results and visual representations are displayed based on the results.

[0414] 4. Emotion Engine

[0415] The emotion engine analyzes the user's facial expressions and voice data to determine the user's emotional state. This analysis is performed using, for example, the "Transformers" pipeline.

[0416] Processing flow

[0417] As a concrete example, a user enters the query "Please tell me the code to change the robot's movement speed," and in the process explains how the emotion engine works.

[0418] 1. Query input: The user inputs a query into the device, and the device simultaneously collects the user's facial expressions and voice.

[0419] 2. Data transmission: The device sends the query and emotion data to the server.

[0420] 3. Query analysis and code search: The server analyzes the query, extracts the keywords "operation speed," "change," "code," and "robot," and searches the database for the corresponding code fragment.

[0421] 4. Visual representation generation: The server analyzes the procedures in the code fragments it finds and generates a flowchart that shows the sequence of processes that change the speed of the execution.

[0422] 5. Detailed documentation generation: Generate documentation including code examples, test cases, improvement suggestions, etc.

[0423] 6. Sentiment Analysis: The server analyzes the user's emotional data through an emotion engine and determines that the user is confused. Based on this, it adjusts the generated documentation and visual representations to include additional explanations and examples.

[0424] 7. Result transmission and display: The server transmits the tailored visual representation and detailed documentation to the user's device, which displays them, allowing the user to review and incorporate code snippets, flowcharts, and related information into their own projects.

[0425] Prompt Sentence Examples

[0426] "Please provide code snippets for the following natural language queries, and add detailed documentation and visual representations if the user's sentiment is confused: Query: 'What code can I use to change the speed of the robot?'"

[0427] The above is a concrete example of how to implement the invention. This system allows users to efficiently find code fragments that suit their purpose, visually understand them, and comprehensively obtain related information, while also receiving appropriate support based on their emotions.

[0428] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0429] Step 1:

[0430] The user inputs a query in natural language. For example, "Please tell me the code to change the robot's movement speed." At this time, the device simultaneously collects the user's facial expression and voice data. The inputs in this step are the natural language query, facial expression data, and voice data, and the output is a data package of these.

[0431] Step 2:

[0432] The device sends the query and collected emotion data to the server. The input of this step is the data package collected in step 1, and the output is the data transmission to the server.

[0433] Step 3:

[0434] The server analyzes the received natural language query and extracts keywords and context. Here, natural language processing technology is used to extract important keywords such as "robot," "movement speed," and "change." The input is the user's query, and the output is a list of keywords as the analysis result.

[0435] Step 4:

[0436] The server searches the database for relevant code fragments based on the keywords extracted in step 3. For example, it finds "code to change the robot's movement speed." The input of this step is a list of keywords, and the output is the searched code fragments.

[0437] Step 5:

[0438] The server analyzes the behavior of the retrieved code fragments and generates visually understandable flowcharts and animations. Specifically, it processes the data to visually display the flow of each statement and loop in the code. The input for this step is the code fragment, and the output is a flowchart or animation.

[0439] Step 6:

[0440] The server generates detailed documentation for a code fragment, including test cases, use cases, bug reports, and improvement suggestions. The documentation generation process involves collecting relevant information and organizing it in a systematic way. The input of this step is the code fragment, and the output is detailed documentation.

[0441] Step 7:

[0442] The server analyzes the emotion data received in step 2 and determines the user's emotional state. For example, it uses an emotion engine to detect a "confused" state. The input of this step is emotion data, and the output is the emotion analysis result.

[0443] Step 8:

[0444] The server adjusts the search results, visual representation, and detailed document content based on the sentiment analysis results obtained in step 7. For example, if the user is confused, the server may include more detailed explanations or additional examples in the document. The inputs to this step are the sentiment analysis results, visual representation, and detailed document, and the output is the final adjusted visual representation and detailed document.

[0445] Step 9:

[0446] The server finally sends the adjusted visual representation and detailed document to the terminal. The input of this step is the adjusted visual representation and detailed document, and the output is data transmission to the terminal.

[0447] Step 10:

[0448] The terminal displays the visual representation and detailed documentation sent from the server to the user, allowing the user to identify the necessary code fragments and integrate them into their project. The input of this step is the data sent from the server, and the output is the display to the user.

[0449] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0450] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0451] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0452] [Second embodiment]

[0453] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0454] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0455] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0456] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0457] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0458] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0459] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0460] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0461] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0462] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0463] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0464] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0465] The present invention provides a system for searching code in natural language, displaying code fragments in a visually understandable format, and providing related detailed information. Specific embodiments of the system are described below.

[0466] Overall system overview

[0467] This system consists of users, terminals, and a server.

[0468] User operations

[0469] A user uses a device (such as a computer or smartphone) to search for code related to a specific function or operation using natural language, for example, typing "What is the Python code to sort a list?"

[0470] Device Role

[0471] The terminal receives input from the user, transmits it to a server via the Internet, and displays search results, visual representations, and detailed documents returned by the server, allowing the user to quickly and efficiently obtain the desired information.

[0472] Server Roles

[0473] The server has the following main functions:

[0474] 1. Natural Language Query Parsing:

[0475] The server receives the natural language query sent by the user and analyzes its intent. Specifically, it uses natural language processing technology to extract keywords and contexts, which are then used as search criteria.

[0476] 2. Search for code fragments:

[0477] Based on the analysis results, it searches for code fragments that match the purpose from an internal database or an external code repository, thereby finding the appropriate code to achieve the functionality or operation the user desires.

[0478] 3. Generate visual representation:

[0479] The behavior of the found code fragments is converted into a form that can be understood visually. Specifically, the code process is expressed as a flowchart or animation, allowing users to intuitively understand the structure and behavior of the code.

[0480] 4. Generate detailed documentation:

[0481] It collects test cases, use cases, bug reports and improvement suggestions related to a piece of code and generates a consolidated and detailed documentation, which gives users a comprehensive understanding of the code's quality and coverage.

[0482] 5. Sending the results:

[0483] The visual representation and detailed documentation are sent to the terminal, which displays it and provides it to the user.

[0484] Specific examples

[0485] Below is a specific example where a user searches for "Python code to sort a list."

[0486] 1. User input:

[0487] The user types "Please tell me the Python code to sort a list" into the terminal and clicks the search button.

[0488] 2. Sending by terminal:

[0489] The terminal sends this input to the server.

[0490] 3. Query parsing and code lookup by the server:

[0491] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches its internal database for relevant code fragments.

[0492] 4. Generating a visual representation:

[0493] The server analyzes the steps in the code fragment it finds and generates a flowchart that shows the process of taking an input list, sorting the elements, and returning the result.

[0494] 5. Generate detailed documentation:

[0495] Generate documentation including relevant test cases (e.g., sorting a list of numbers or a list of strings), use cases (specific sorting methods using particular lists), bug reports (issues that have been encountered and how they have been resolved), and improvement suggestions (e.g., a more efficient sorting algorithm).

[0496] 6. Sending and viewing results:

[0497] The server sends this information to the device, which displays the results to the user, who can review code snippets, flowcharts, and detailed documentation and incorporate them into their own projects as needed.

[0498] The above is a specific embodiment of the present invention, and this system enables users to efficiently find code fragments that suit their purpose, understand them visually, and obtain related information comprehensively.

[0499] The processing flow will be explained below.

[0500] Step 1:

[0501] A user enters a search query into a terminal, such as "Can you give me the Python code to sort a list?" in natural language, and clicks the search button.

[0502] Step 2:

[0503] The device sends the query entered by the user to the server as is, and then communicates via the Internet to pass the data to the server.

[0504] Step 3:

[0505] The server analyzes the received natural language query, and the query analysis engine uses natural language processing techniques to extract key keywords (in this case, "list," "sort," and "Python").

[0506] Step 4:

[0507] The server searches for relevant code fragments in its internal database or external code repository based on the extracted keywords. The search engine finds code fragments that match the keywords and displays them in a list.

[0508] Step 5:

[0509] The server selects the most appropriate code fragment from the searched code fragments and visually represents its behavior. The visual representation generation component analyzes the code structure and generates visual data as flowcharts or animations.

[0510] Step 6:

[0511] The server retrieves test cases and use cases related to a piece of code from a database, as well as related bug reports and improvement suggestions, and synthesizes them to generate comprehensive documentation. The documentation generation component organizes this information and presents it in a user-friendly format.

[0512] Step 7:

[0513] The server sends the generated visual representation and detailed documentation to the device, which receives the data and converts it into a format that can be displayed to the user.

[0514] Step 8:

[0515] The terminal displays the received visual representation and detailed documentation to the user, who can then review the code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[0516] The above is the processing flow from the user's natural language query to code search, analysis, visual representation generation, document generation, and result display.

[0517] Example 1

[0518] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0519] Conventional code search systems make it difficult for users to quickly find the code fragment they are looking for and intuitively understand its behavior. They also lack the functionality to provide detailed information such as related test cases and use cases. As a result, users have to spend a lot of time finding the appropriate code fragment, and it is difficult to fully understand the quality and scope of the code.

[0520] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0521] In this invention, the server includes: means for analyzing a user's query using natural language processing technology; means for retrieving code fragments from an internal database or an external repository; means for converting the retrieved code procedures into a flowchart or visual diagram; means for generating detailed documentation using a generative AI model; means for analyzing the natural language query and searching for code fragments that match a purpose; means for visually representing the behavior of the retrieved code fragments; means for generating detailed documentation including test cases and use cases related to the code fragments; and means for transmitting the search results, the visual representation, and the detailed documentation to the user's terminal. This allows the user to efficiently find code fragments that match their purpose, visually understand them, and comprehensively obtain related information.

[0522] A "natural language query" is a question or request made by a user to a computer system in natural human language.

[0523] A "query" is a statement or command that describes instructions for retrieving information from a database or information system.

[0524] "Analysis" is the process of separating and analyzing data and information to understand its intent, content, and structure.

[0525] A "code fragment" is a part of a program or script, a fragment of source code that realizes a specific function or operation.

[0526] "Visual representation" refers to the display of information or data in visual formats such as graphs, flow charts, animations, etc.

[0527] A "flowchart" is a diagram that visually shows the flow of an algorithm or process by representing each step with shapes and arrows.

[0528] "Animation" is a technique for expressing movement by displaying a series of still images in succession over time.

[0529] A "test case" is a set of input data and expected results designed to verify a particular function or behavior of software or a system.

[0530] A "use example" is a concrete example that shows how to actually use a specific function or operation.

[0531] A "document" is a document or material created to record information and communicate it to others.

[0532] A "bug report" is a document used to report defects or problems within software or a system, and describes the incident, the steps to reproduce it, the scope of the impact, etc.

[0533] An "improvement proposal" is a document that describes specific proposals and methods for improving the performance or usability of an existing system or process.

[0534] A "generative AI model" is an artificial intelligence model that is generated using machine learning or deep learning techniques and is designed to solve a specific task or problem.

[0535] A "search result" is a set of information returned based on a search query.

[0536] The "Internet" is a global information and communications network of interconnected computer networks.

[0537] "API" is an abbreviation for Application Programming Interface, and is a set of definitions and rules for exchanging functions and data between software programs.

[0538] An "internal database" is a database managed within the system, used to store and manage user and system data.

[0539] An "external repository" is a database or information resource that exists outside the system, where code and data that can be accessed from outside are managed.

[0540] MODE FOR CARRYING OUT THE INVENTION

[0541] The present invention provides a system for searching code in natural language, displaying code fragments in a visually understandable format, and providing related detailed information. Specific embodiments of the system are described below.

[0542] Overall system overview

[0543] This system consists of a user, a terminal, and a server. The user uses the terminal to search for codes related to specific functions or operations in natural language. The terminal receives input from the user and sends it to the server. The server analyzes the received input and returns the search results. Specifically, the server operates based on the following steps.

[0544] User operations

[0545] A user uses a device (such as a computer or smartphone) to search for code related to a specific function or operation using natural language, for example, typing "What is the Python code to sort a list?"

[0546] Device Role

[0547] The device receives input from the user and sends it to a server over the Internet, for example, as text data in JSON format. It also has the ability to display search results, visual representations, and detailed documents returned by the server, allowing the user to quickly and efficiently obtain the desired information.

[0548] Server Roles

[0549] The server has the following main functions:

[0550] 1. Natural Language Query Parsing:

[0551] The server receives natural language queries sent by users and analyzes their intent. It uses natural language processing techniques to extract keywords and contexts and use them as search criteria. This process uses generative AI models such as BERT and GPT-4.

[0552] 2. Search for code fragments:

[0553] Based on the analysis results, it searches for matching code fragments from an internal database or external code repositories (e.g., GitHub or GitLab), thereby finding the appropriate code to achieve the functionality or operation the user is looking for.

[0554] 3. Generate visual representation:

[0555] The behavior of the found code fragments is then translated into a visually understandable form, specifically using tools like PlantUML or Draw.io to represent the code's processes as flowcharts or animations, allowing users to intuitively understand the code's structure and behavior.

[0556] 4. Generate detailed documentation:

[0557] It collects test cases, use cases, bug reports, and improvement suggestions related to a piece of code and generates a consolidated and detailed document. It uses a generative AI model to generate the document by inputting a prompt such as:

[0558] "Generate documentation containing test cases, use cases, bug reports, and improvement suggestions related to the code snippets found in the search results. Format should be Markdown."

[0559] 5. Sending the results:

[0560] The visual representation and detailed documentation are sent to the terminal, which displays it and provides it to the user.

[0561] Specific examples

[0562] Below is a specific example where a user searches for "Python code to sort a list."

[0563] 1. User input:

[0564] The user types "Please tell me the Python code to sort a list" into the terminal and clicks the search button.

[0565] 2. Sending by terminal:

[0566] The terminal sends this input to the server.

[0567] 3. Query parsing and code lookup by the server:

[0568] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches its internal database for relevant code fragments.

[0569] 4. Generating a visual representation:

[0570] The server analyzes the steps in the code fragments it finds and generates a flowchart using PlantUML that shows the process of receiving an input list, sorting the elements, and returning the results.

[0571] 5. Generate detailed documentation:

[0572] The server generates documentation in LaTeX and Markdown format containing test cases, use cases, bug reports, and improvement suggestions. For example, the following prompt is input to the generative AI model:

[0573] "Generate documentation containing test cases, use cases, bug reports, and improvement suggestions related to the code snippets found in the search results. Format should be Markdown."

[0574] 6. Sending and viewing results:

[0575] The server sends this information to the device, which displays the results to the user: code snippets, flowcharts, and detailed documentation that the user can review and, if desired, incorporate into their own projects.

[0576] The above is an embodiment of the present invention, and this system enables users to efficiently find code fragments that suit their purpose, understand them visually, and obtain related information comprehensively.

[0577] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0578] Step 1:

[0579] Accepting user input

[0580] The user types into the terminal, "Can you give me the Python code to sort a list?"

[0581] Input: Natural language text input by the user

[0582] Output: User-entered text data

[0583] Specific operation: A user enters the required query into a text input field in a web browser or mobile app and clicks the search button. The device retrieves this input using JavaScript or similar tools and prepares it for further processing.

[0584] Step 2:

[0585] Terminal input transmission

[0586] The terminal receives input from the user and sends it to the server.

[0587] Input: User-entered text data

[0588] Output: Sending text data to the server

[0589] What it does: The device uses JavaScript or other front-end technologies to convert user input into JSON format and send it to the server via an AJAX request, while displaying a sending indicator to the user.

[0590] Step 3:

[0591] Query analysis by the server

[0592] The server parses the received user input.

[0593] Input: Received text data (query)

[0594] Output: Keyword and context extraction results

[0595] How it works: The server parses the incoming data and analyzes it using natural language processing techniques. It uses generative AI models such as BERT and GPT-4 to analyze the query and extract key keywords and intent. This process extracts keywords such as "list," "sort," and "Python."

[0596] Step 4:

[0597] Server lookup of code fragments

[0598] The server uses the analysis results to search for the code fragment.

[0599] Input: Keyword and context extraction results

[0600] Output: relevant code fragment

[0601] What happens: The server executes SQL queries to search its internal database, and optionally retrieves relevant code fragments from external code repositories (e.g., GitHub or GitLab) using API requests. As a result, the relevant code fragments are stored in the server's temporary memory.

[0602] Step 5:

[0603] Generating a visual representation

[0604] The server provides a visual representation of the retrieved code fragment.

[0605] Input: The searched code fragment

[0606] Output: Visual flowcharts and animations

[0607] How it works: The server analyzes the structure of a code fragment, calls the APIs of PlantUML and Draw.io, and generates flowcharts and animations. The generated visual data is saved as SVG or PNG format files.

[0608] Step 6:

[0609] Generate detailed documentation

[0610] The server generates detailed documentation related to the code fragment.

[0611] Input: The searched code fragment

[0612] Output: Detailed documentation (Markdown or LaTeX format)

[0613] How it works: The following prompt is input to the generative AI model to generate a document: "Please generate a document containing test cases, use cases, bug reports, and improvement suggestions related to the code fragment found in the search results. Please use Markdown format." The generated document is saved on the server in Markdown or LaTeX format.

[0614] Step 7:

[0615] Sending and viewing results

[0616] The server sends the visual representation and detailed documentation to the terminal, which displays the results to the user.

[0617] Input: Visual flowcharts, animations, detailed documentation

[0618] Output: The results displayed on the user's terminal

[0619] What it does: The server encodes the generated data in JSON format and sends it to the device as an HTTP response. The device receives it, renders the data using HTML and JavaScript, and displays it to the user. The user can then view the displayed code fragments, flowcharts, and detailed documentation and incorporate them into their own projects as needed.

[0620] The above are the specific processing steps of this system, which allow users to efficiently obtain the information they require.

[0621] (Application example 1)

[0622] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0623] While conventional code search systems analyze natural language and provide relevant code fragments, they lack sufficient visual understanding support. They also struggle to provide comprehensive, relevant detailed information. Furthermore, they have limitations in searching for code fragments specialized for specific applications and generating visual representations, making it difficult for developers to quickly and accurately obtain the information they need. For this reason, further improvements are needed in specialized fields such as electronic payments.

[0624] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0625] In this invention, the server includes means for analyzing a natural language query and searching for code fragments that match a purpose, means for visually representing the behavior of the searched code fragments, means for generating detailed documentation including test cases and use examples related to the code fragments, means for transmitting the search results, the visual representation, and the detailed documentation to a user's terminal, means for using a generative AI model to search for code fragments related to the query content and providing the code fragments together with their visual representations, and means for analyzing the user's intent using prompt sentences and generating related code fragments and documentation. This enables a user to quickly obtain specific code fragments related to electronic payment based on a natural language query, visually understand them, and comprehensively obtain detailed documentation.

[0626] A "natural language query" is a question or command entered by a user in human language.

[0627] A "code fragment" is a portion of program code that implements a specific function or operation.

[0628] "Visually representing" means displaying textual information in graphical form, such as a flow chart or animation.

[0629] A "test case" is a set of specific inputs and their expected outputs that are executed to verify the behavior of a program.

[0630] "Detailed documentation" refers to comprehensive information related to a piece of code, including explanations, use cases, bug reports, improvement suggestions, etc.

[0631] A "generative AI model" is an artificial intelligence program that uses machine learning techniques to analyze natural language and generate or find appropriate code fragments.

[0632] A "prompt sentence" is an input sentence for conveying the user's intention, and is a sentence used as a natural language query.

[0633] A "terminal" is a device that can be operated by a user, and specifically includes a personal computer, a smartphone, etc.

[0634] A "server" is a computer system that processes and stores data and provides information in response to requests from terminals.

[0635] "User" refers to a person who uses this system.

[0636] This invention provides a system that allows a user to input a query in natural language using a terminal (such as a smartphone or computer), search for code fragments that match the user's purpose, and present the results in a visually easy-to-understand format. This system is mainly composed of the following elements:

[0637] 1. Parsing natural language queries

[0638] The user uses the device to provide input in natural language, such as a prompt such as "What is the Python code to process a credit card payment?" The device then sends this input to the server, which uses a generative AI model to analyze the query and extract relevant keywords and intent. This generative AI model uses the Hugging Face transformers library.

[0639] 2. Searching for code fragments

[0640] Based on the extracted keywords and intent, the server searches for relevant code fragments from an internal database or external repository that contains code fragments written in various programming languages, including, for example, Python code for processing credit card payments.

[0641] 3. Generating Visual Representations

[0642] For each code fragment found, the server converts its behavior into a visually understandable representation, using Matplotlib and NetworkX to represent the code steps as flowcharts and animations. This visual representation helps users intuitively understand the code's flow and structure.

[0643] 4. Generate detailed documentation

[0644] The server generates detailed documentation such as test cases, use cases, bug reports, and improvement suggestions related to the retrieved code fragments, allowing users to gain a comprehensive understanding of the code's quality and coverage.

[0645] 5. Sending and displaying results

[0646] The server sends the search results, visual representations, and detailed documentation to the terminal, which then displays them to the user, who can quickly and efficiently obtain the information they need based on the displayed code fragments, flowcharts, and detailed documentation.

[0647] As a concrete example, consider the case where a user uses the following prompt text:

[0648] "Can you give me some Python code to process credit card payments?"

[0649] In response, the system searches for relevant code fragments, generates visual representations and detailed documentation, and provides them to users, allowing them to quickly extract the code they need and apply it to their own projects.

[0650] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0651] Step 1:

[0652] Accepting natural language inquiries

[0653] The user uses a terminal to input a query in natural language. At this stage, the user enters a prompt sentence such as "Please give me the Python code to process credit card payments." The input data is sent over the Internet to the server.

[0654] Input: Natural language query (prompt)

[0655] Output: The query sent to the server

[0656] Step 2:

[0657] Natural Language Analysis

[0658] The server analyzes the received query using a generative AI model (Hugging Face's transformers library). This model extracts keywords and user intent from the input natural language text. Specifically, keywords such as "credit card," "payment," and "Python" are extracted.

[0659] Input: Received natural language query

[0660] Output: Extracted keywords and user intent

[0661] Step 3:

[0662] Searching for code fragments

[0663] The server then searches for relevant code fragments from an internal database or external repository based on the extracted keywords and intent. This database contains code fragments written in various programming languages, and the search result is "Python code to process credit card payments."

[0664] Input: Extracted keywords and user intent

[0665] Output: relevant code fragment

[0666] Step 4:

[0667] Generating a visual representation

[0668] The server then converts the code fragments it finds into a visually understandable representation of their behavior, using Matplotlib and NetworkX to generate flowcharts and animations of the code steps, providing a visual representation of the code's flow and structure.

[0669] Input: The searched code fragment

[0670] Output: Generated flowchart or animation

[0671] Step 5:

[0672] Generate detailed documentation

[0673] The server generates detailed documentation containing test cases, use cases, bug reports, and improvement suggestions associated with the code fragment, providing data for a comprehensive understanding of the code's quality and coverage.

[0674] Input: The searched code fragment

[0675] Output: Detailed documentation (test cases, use cases, bug reports, improvement suggestions)

[0676] Step 6:

[0677] Sending and displaying results

[0678] The server sends the search results, visual representations, and detailed documentation to the terminal, which displays them to the user, allowing the user to quickly and efficiently obtain the information they need based on the displayed code fragments, flowcharts, and detailed documentation.

[0679] Input: Search results, visual representation, detailed documentation

[0680] Output: what is displayed on the user's device (code fragments, visual representations, detailed documentation)

[0681] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0682] The present invention incorporates an emotion engine that recognizes the user's emotions into a system that performs code searches in natural language, displays the results in a visually easy-to-understand format, and provides related information. Specific embodiments of the present invention are described below.

[0683] Overall system overview

[0684] This system consists of a user, a terminal, a server, and an emotion engine.

[0685] User operations

[0686] Users use the terminal to search for code related to a particular function or operation using natural language, for example by entering a query such as "What is the Python code to sort a list?" and clicking the search button.

[0687] Device Role

[0688] The device receives input from the user and sends it to the server, displays search results, visual representations, and detailed documents sent from the server, and collects emotional data from the user's facial expressions and voice and sends it to the server.

[0689] Server Roles

[0690] The server has the following main functions:

[0691] 1. Natural Language Query Parsing:

[0692] The server analyzes the natural language queries sent by users and extracts keywords and context using natural language processing techniques.

[0693] 2. Search for code fragments:

[0694] Based on the analysis results, the search engine searches for relevant code fragments in internal databases and external code repositories, and finds code fragments that match the keywords.

[0695] 3. Generate visual representation:

[0696] The behavior of the found code fragments is converted into a visually understandable form (flowchart or animation) by the visual representation generation component, which analyzes the code structure.

[0697] 4. Generate detailed documentation:

[0698] It collects test cases, use cases, bug reports and improvement suggestions related to a piece of code and generates a single comprehensive document.

[0699] 5. User sentiment analysis:

[0700] The emotion engine analyzes the user's emotional data sent from the device and adjusts search results and visual presentation accordingly, for example by adding more detailed explanations if the user is confused.

[0701] 6. Sending the results:

[0702] Finally, adjustment information that takes into account the user's emotion data is sent to the terminal together with the visual representation and detailed documentation.

[0703] Specific examples

[0704] Below is a concrete example of how the sentiment engine works in the process of a user searching for "Python code to sort a list."

[0705] 1. User input:

[0706] The user enters "Please tell me the Python code to sort the list" into the terminal and clicks the search button. The terminal simultaneously collects the user's facial expression and voice data.

[0707] 2. Sending by terminal:

[0708] The terminal transmits the query and the user's emotion data to the server.

[0709] 3. Query parsing and code lookup by the server:

[0710] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches the database for the corresponding code fragment.

[0711] 4. Generating a visual representation:

[0712] The server analyzes the steps in the code fragment it finds and generates a flowchart that shows the process of taking an input list, sorting the elements, and returning the result.

[0713] 5. Generate detailed documentation:

[0714] Generate documentation including relevant test cases (e.g., sorting a list of numbers or a list of strings), use cases (specific sorting methods using particular lists), bug reports (issues that have been encountered and how they have been resolved), and improvement suggestions (e.g., a more efficient sorting algorithm).

[0715] 6. User sentiment analysis:

[0716] The server analyzes the user's emotional data through an emotion engine and determines that the user is confused, and accordingly adjusts the generated documentation to include additional explanations and examples.

[0717] 7. Submitting and Viewing Results:

[0718] The server sends visual representations and detailed documentation to the user's device, reflecting the emotion-based adjustment information, and the device displays it, allowing the user to review code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[0719] The above is a specific embodiment of the present invention incorporating an emotion engine. This system allows users to efficiently find code fragments that suit their purpose, visually understand them, and comprehensively obtain related information, while also receiving appropriate support based on their emotions.

[0720] The processing flow will be explained below.

[0721] Step 1:

[0722] Using a terminal, a user enters a natural language query such as "Can you give me the Python code to sort a list?" and clicks the search button.

[0723] Step 2:

[0724] The device collects the user's facial expressions and voice data along with natural language queries from the user, and these data are simultaneously sent to the server as emotion data for analyzing the user's emotions.

[0725] Step 3:

[0726] The server analyzes the received natural language query, and the query analysis engine uses natural language processing techniques to extract key keywords such as "list," "sort," and "Python."

[0727] Step 4:

[0728] The server searches for relevant code fragments from an internal database or external code repository based on the analyzed keywords, and the search engine extracts matching code fragments in a list.

[0729] Step 5:

[0730] The server selects the most suitable code fragment from those found and then uses a visual representation generation component to visually represent the behavior of the code fragment, generating a flowchart or animation of the code fragment.

[0731] Step 6:

[0732] The server collects test cases and use cases related to a piece of code from a database, as well as bug reports and improvement suggestions, and generates comprehensive documentation out of this information. This process is handled by the documentation generation component.

[0733] Step 7:

[0734] The server runs an emotion engine to analyze the user's emotion data sent from the device. This engine analyzes how the user feels about the displayed information, and if a negative emotion such as confusion is detected, the engine includes a more detailed explanation or additional examples.

[0735] Step 8:

[0736] The server transmits the generated visual representation, detailed documentation, and information adjusted based on the emotion to the terminal.

[0737] Step 9:

[0738] The terminal displays the received information to the user, showing visual flowcharts, detailed code descriptions, documentation with associated test cases, use cases, bug reports, and improvement suggestions, allowing the user to understand the code fragment and efficiently incorporate it into their own projects.

[0739] This series of steps allows users to obtain code fragments and related information in a more relevant and understandable way, based on the additional information of emotions.

[0740] Example 2

[0741] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0742] Conventional program code search systems often produce insufficient search results even when users input queries in natural language. Furthermore, providing information through code fragments that are difficult to understand visually or incomplete documentation contributes to a lack of user convenience. Furthermore, providing search results without considering the user's emotional state can be confusing, especially for novice users. To address these issues, systems are required that can analyze queries in natural language, display information that is easy to understand visually, and provide information that takes the user's emotions into account.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0744] In this invention, the server includes means for analyzing a natural language query and searching for program code that matches a purpose, means for visually representing the behavior of the searched program code, means for generating detailed documentation including test cases and use cases related to the program code, means for collecting and analyzing emotional data, and means for adjusting the visual representation and the content of the detailed documentation based on the emotional data. This allows the user to obtain search results that are more accurate and visually easy to understand, and also makes it possible to provide appropriate information according to the user's emotional state.

[0745] A "natural language query" refers to a query entered by a user using a human language (e.g., Japanese, English, etc.).

[0746] "Program Code" means source code that contains instructions and procedures for performing a particular task or function.

[0747] A "search tool" refers to a mechanism that analyzes natural language queries and uses them to find appropriate program code from databases or external repositories.

[0748] "Visual representation means" refers to a mechanism for displaying the behavior and structure of program code using visualization techniques such as flowcharts and animations.

[0749] "Detailed documentation" refers to comprehensive documentation containing test cases, use cases, bug reports, and improvement suggestions related to the program code.

[0750] "Emotional data" refers to data that indicates the user's emotional state, collected from the user's facial expressions, voice, etc.

[0751] "Emotion analysis means" refers to a mechanism for analyzing collected emotion data and determining the user's emotional state.

[0752] "Adjustment means" refers to a mechanism that appropriately changes the visual presentation and detailed document content based on the results of sentiment analysis to provide information in a form that is easier for users to understand.

[0753] "User terminal" refers to a device such as a computer, smartphone, or tablet that is actually operated by a user.

[0754] The present invention provides a system that allows a user to search for program code in natural language, displays the code in a visually easy-to-understand format, and provides information based on the user's emotions. Specific embodiments will be described below.

[0755] System configuration

[0756] This system consists of a user, a terminal, a server, and an emotion engine.

[0757] User operations

[0758] Users use the terminal to search for program code related to a specific function or operation in natural language, for example, by entering a query such as "What is the Python code to sort a list?" and clicking the search button.

[0759] Device Role

[0760] The device receives input queries from the user and sends them to the server. It also has the function of collecting emotional data from the user's facial expressions and voice and transmitting it to the server. For example, it acquires the user's emotional data using a camera or microphone. It also displays search results, visual representations, and detailed documents received from the server to the user.

[0761] Server Roles

[0762] The server has the following main functions:

[0763] 1. Natural Language Query Parsing:

[0764] The server analyzes the natural language query submitted by the user, for example by using natural language processing tools such as SpaCy or NLTK to extract keywords such as "list," "sort," and "Python."

[0765] 2. Search for the program code:

[0766] Based on the analysis results, the relevant program code is searched for in an internal database (e.g., PostgreSQL) or an external code repository (e.g., GitHub).

[0767] 3. Generate visual representation:

[0768] Convert the searched program code into a visually understandable format (flowchart or animation). For example, generate a flow chart using D3.js.

[0769] 4. Generate detailed documentation:

[0770] It collects test cases, use cases, bug reports, and improvement suggestions related to program code and generates detailed documentation in Markdown format.

[0771] 5. Emotion analysis:

[0772] The server's emotion engine analyzes the user's emotion data sent from the device, for example, using an emotion recognition API to determine whether the user is confused.

[0773] 6. Sending the results:

[0774] The server generates a visual representation and detailed documentation, and transmits the information adjusted based on the sentiment analysis results to the terminal.

[0775] Specific examples

[0776] Here is a specific example of what happens when a user searches for "Can you give me the Python code to sort a list?"

[0777] 1. User input:

[0778] The user enters "Please tell me the Python code to sort the list" into the terminal and clicks the search button. The terminal simultaneously collects the user's facial expression and voice data.

[0779] 2. Sending by terminal:

[0780] The terminal transmits the query and the user's emotion data to the server.

[0781] 3. Query parsing and code lookup by the server:

[0782] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches the database for the corresponding code fragment.

[0783] 4. Generating a visual representation:

[0784] The server analyzes the steps in the code fragments it finds and generates a flowchart using D3.js.

[0785] 5. Generate detailed documentation:

[0786] The server creates a detailed document in Markdown containing relevant test cases, use cases, bug reports, and improvement suggestions.

[0787] 6. User sentiment analysis:

[0788] The server's emotion engine detects user confusion and adjusts the generated documentation to include additional explanations and examples.

[0789] 7. Submitting and Viewing Results:

[0790] The server sends the final visual representation and detailed documentation to the device, which displays it, allowing users to review code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[0791] Prompt Sentence Examples

[0792] Here are some example prompts for searches using this system:

[0793] "Enter 'Please tell me the Python code to sort a list' in the terminal and search."

[0794] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0795] Step 1:

[0796] The user enters a natural language query into the text input field of the terminal and clicks the search button. For example, they might type "Can you give me the Python code to sort a list?" This is the input data. The output they get from the terminal is the query.

[0797] Step 2:

[0798] The device sends queries entered by the user to the server. In addition, the device's built-in camera and microphone capture the user's facial expressions and voice data, which are also sent to the server. The input is the user's query and emotion data, and the output is an HTTP request to the server. Specifically, the device converts the query and emotion data into JSON format and sends it.

[0799] Step 3:

[0800] The server analyzes the received natural language query using a natural language processing tool (e.g., SpaCy or NLTK) to extract keywords and context. The input is the query sent from the terminal, and the output is the extracted keywords (e.g., "list," "sort," "Python"). Specifically, the server calls a text analysis engine to analyze the query.

[0801] Step 4:

[0802] The server searches for relevant program code in a database (e.g., PostgreSQL) or an external repository (e.g., GitHub) based on the extracted keywords. The input is the parsed keywords, and the output is the search results, which are program code fragments. Specifically, it executes SQL queries or API requests to retrieve data.

[0803] Step 5:

[0804] The server analyzes the searched program code and generates visual representations such as flowcharts and animations. This is done using D3.js or other visualization tools. The input is the program code fragments as search results, and the output is the visual representation. Specifically, each step of the program code is input into the visualization tool to generate a flowchart.

[0805] Step 6:

[0806] The server collects test cases, use cases, bug reports, and improvement suggestions related to the searched program code and generates detailed documentation. The documentation format is Markdown. The input is a fragment of program code, and the output is a detailed documentation. Specifically, it collects and integrates related information and generates a document in Markdown format.

[0807] Step 7:

[0808] The emotion engine on the server analyzes the user's emotion data collected from the device and determines the user's emotional state (e.g., confusion, satisfaction, etc.). The input is the emotion data sent from the device, and the output is the user's emotional state as a result of the analysis. Specifically, it calls the emotion recognition API and analyzes the data.

[0809] Step 8:

[0810] The server adjusts the content of the visual representation and detailed text based on the user's emotional state. For example, if the user is confused, it adds more detailed explanations and examples. The input is the user's emotional state, the visual representation, and the detailed text, and the output is the adjusted visual representation and detailed text. Specific operations include updating the text and visualization data.

[0811] Step 9:

[0812] The server sends the adjusted visual representation and detailed document to the terminal. The input is the adjusted visual representation and detailed document, and the output is an HTTP response to the terminal. As a specific operation, the server generates an HTTP response for sending data to the terminal.

[0813] Step 10:

[0814] The terminal displays the visual representation and detailed document received from the server to the user. The input is data from the server, and the output is the display to the user. Specifically, the terminal uses a browser or a dedicated application to render and display the visual representation and document.

[0815] (Application example 2)

[0816] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0817] In modern factories and production lines, robot programming and troubleshooting are extremely important, but they require specialized knowledge and experience. Furthermore, conventional code search systems and support tools provide only one-way information without considering the user's feelings. This often leaves users confused and reduces work efficiency. Another issue is the lack of visual support, which makes it time-consuming to understand and operate the code. A system that can solve these problems and provide more intuitive and efficient support is needed.

[0818] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing a natural language query and searching for code fragments that match the purpose, means for visually expressing the behavior of the searched code fragments, and means for analyzing user emotion data and adjusting the display method of the search results and visual expression based on the results. This makes it possible to provide optimal support that is easy for the user to intuitively understand and that is tailored to their emotions.

[0819] A "natural language query" is a question or instruction written in a language normally used by humans, with the goal of converting it into a form that the system can understand and process.

[0820] A "code fragment" is a part of a program that realizes a specific function or operation, and refers to a short piece of code that is used in actual programming work.

[0821] A "visual representation" is a visually easy-to-understand representation of the behavior of the retrieved code fragment, which can take the form of a flowchart or animation.

[0822] "Detailed documentation" refers to materials that contain supplementary and comprehensive information, such as test cases, use cases, bug reports, and improvement suggestions related to a piece of code.

[0823] "Emotional data" refers to emotional information extracted from a user's facial expressions and voice, and is used by the system to analyze and understand the user's emotional state.

[0824] "Adjusting the display method" refers to changing the format and content of search results and visual representations based on the user's emotional data to make them easier for the user to understand.

[0825] This system analyzes natural language queries, searches for code fragments that match the query, and displays the code's behavior in a visually understandable format. It also analyzes user sentiment and adjusts the display of search results and visual representations based on the results.

[0826] System configuration

[0827] This system consists of a user, a terminal, a server, and an emotion engine.

[0828] 1. Users

[0829] Users use devices such as smartphones or head-mounted displays to search for code fragments in natural language, for example by entering queries such as "What code can I use to change the speed of a robot?"

[0830] 2. Terminal

[0831] The device receives input from the user and sends it to the server. It also collects emotional data from the user's facial expressions and voice and sends this data to the server.

[0832] 3. Server

[0833] The server has the following functions:

[0834] Natural language query analysis: The server analyzes the natural language query sent by the user and extracts keywords and context using natural language processing techniques.

[0835] Searching for code fragments: Based on the analysis results, the database is searched for the relevant code fragments.

[0836] Visual representation generation: Convert the behavior of the retrieved code fragments into a visually understandable form (flowchart or animation).

[0837] Generate detailed documentation: Collect test cases, use cases, bug reports and improvement suggestions related to a piece of code and generate one comprehensive document.

[0838] User sentiment analysis: Analyzes user sentiment data sent from the device through the emotion engine and adjusts the way search results and visual representations are displayed based on the results.

[0839] 4. Emotion Engine

[0840] The emotion engine analyzes the user's facial expressions and voice data to determine the user's emotional state. This analysis is performed using, for example, the "Transformers" pipeline.

[0841] Processing flow

[0842] As a concrete example, a user enters the query "Please tell me the code to change the robot's movement speed," and in the process explains how the emotion engine works.

[0843] 1. Query input: The user inputs a query into the device, and the device simultaneously collects the user's facial expressions and voice.

[0844] 2. Data transmission: The device sends the query and emotion data to the server.

[0845] 3. Query analysis and code search: The server analyzes the query, extracts the keywords "operation speed," "change," "code," and "robot," and searches the database for the corresponding code fragment.

[0846] 4. Visual representation generation: The server analyzes the procedures in the code fragments it finds and generates a flowchart that shows the sequence of processes that change the speed of the execution.

[0847] 5. Detailed documentation generation: Generate documentation including code examples, test cases, improvement suggestions, etc.

[0848] 6. Sentiment Analysis: The server analyzes the user's emotional data through an emotion engine and determines that the user is confused. Based on this, it adjusts the generated documentation and visual representations to include additional explanations and examples.

[0849] 7. Result transmission and display: The server transmits the tailored visual representation and detailed documentation to the user's device, which displays them, allowing the user to review and incorporate code snippets, flowcharts, and related information into their own projects.

[0850] Prompt Sentence Examples

[0851] "Please provide code snippets for the following natural language queries, and add detailed documentation and visual representations if the user's sentiment is confused: Query: 'What code can I use to change the speed of the robot?'"

[0852] The above is a concrete example of how to implement the invention. This system allows users to efficiently find code fragments that suit their purpose, visually understand them, and comprehensively obtain related information, while also receiving appropriate support based on their emotions.

[0853] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0854] Step 1:

[0855] The user inputs a query in natural language. For example, "Please tell me the code to change the robot's movement speed." At this time, the device simultaneously collects the user's facial expression and voice data. The inputs in this step are the natural language query, facial expression data, and voice data, and the output is a data package of these.

[0856] Step 2:

[0857] The device sends the query and collected emotion data to the server. The input of this step is the data package collected in step 1, and the output is the data transmission to the server.

[0858] Step 3:

[0859] The server analyzes the received natural language query and extracts keywords and context. Here, natural language processing technology is used to extract important keywords such as "robot," "movement speed," and "change." The input is the user's query, and the output is a list of keywords as the analysis result.

[0860] Step 4:

[0861] The server searches the database for relevant code fragments based on the keywords extracted in step 3. For example, it finds "code to change the robot's movement speed." The input of this step is a list of keywords, and the output is the searched code fragments.

[0862] Step 5:

[0863] The server analyzes the behavior of the retrieved code fragments and generates visually understandable flowcharts and animations. Specifically, it processes the data to visually display the flow of each statement and loop in the code. The input for this step is the code fragment, and the output is a flowchart or animation.

[0864] Step 6:

[0865] The server generates detailed documentation for a code fragment, including test cases, use cases, bug reports, and improvement suggestions. The documentation generation process involves collecting relevant information and organizing it in a systematic way. The input of this step is the code fragment, and the output is detailed documentation.

[0866] Step 7:

[0867] The server analyzes the emotion data received in step 2 and determines the user's emotional state. For example, it uses an emotion engine to detect a "confused" state. The input of this step is emotion data, and the output is the emotion analysis result.

[0868] Step 8:

[0869] The server adjusts the search results, visual representation, and detailed document content based on the sentiment analysis results obtained in step 7. For example, if the user is confused, the server may include more detailed explanations or additional examples in the document. The inputs to this step are the sentiment analysis results, visual representation, and detailed document, and the output is the final adjusted visual representation and detailed document.

[0870] Step 9:

[0871] The server finally sends the adjusted visual representation and detailed document to the terminal. The input of this step is the adjusted visual representation and detailed document, and the output is data transmission to the terminal.

[0872] Step 10:

[0873] The terminal displays the visual representation and detailed documentation sent from the server to the user, allowing the user to identify the necessary code fragments and integrate them into their project. The input of this step is the data sent from the server, and the output is the display to the user.

[0874] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0875] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0876] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0877] [Third embodiment]

[0878] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0879] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0880] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0881] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0882] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0883] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0884] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0885] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0886] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0887] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0888] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0889] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0890] The present invention provides a system for searching code in natural language, displaying code fragments in a visually understandable format, and providing related detailed information. Specific embodiments of the system are described below.

[0891] Overall system overview

[0892] This system consists of users, terminals, and a server.

[0893] User operations

[0894] A user uses a device (such as a computer or smartphone) to search for code related to a specific function or operation using natural language, for example, typing "What is the Python code to sort a list?"

[0895] Device Role

[0896] The terminal receives input from the user, transmits it to a server via the Internet, and displays search results, visual representations, and detailed documents returned by the server, allowing the user to quickly and efficiently obtain the desired information.

[0897] Server Roles

[0898] The server has the following main functions:

[0899] 1. Natural Language Query Parsing:

[0900] The server receives the natural language query sent by the user and analyzes its intent. Specifically, it uses natural language processing technology to extract keywords and contexts, which are then used as search criteria.

[0901] 2. Search for code fragments:

[0902] Based on the analysis results, it searches for code fragments that match the purpose from an internal database or an external code repository, thereby finding the appropriate code to achieve the functionality or operation the user desires.

[0903] 3. Generate visual representation:

[0904] The behavior of the found code fragments is converted into a form that can be understood visually. Specifically, the code process is expressed as a flowchart or animation, allowing users to intuitively understand the structure and behavior of the code.

[0905] 4. Generate detailed documentation:

[0906] It collects test cases, use cases, bug reports and improvement suggestions related to a piece of code and generates a consolidated and detailed documentation, which gives users a comprehensive understanding of the code's quality and coverage.

[0907] 5. Sending the results:

[0908] The visual representation and detailed documentation are sent to the terminal, which displays it and provides it to the user.

[0909] Specific examples

[0910] Below is a specific example where a user searches for "Python code to sort a list."

[0911] 1. User input:

[0912] The user types "Please tell me the Python code to sort a list" into the terminal and clicks the search button.

[0913] 2. Sending by terminal:

[0914] The terminal sends this input to the server.

[0915] 3. Query parsing and code lookup by the server:

[0916] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches its internal database for relevant code fragments.

[0917] 4. Generating a visual representation:

[0918] The server analyzes the steps in the code fragment it finds and generates a flowchart that shows the process of taking an input list, sorting the elements, and returning the result.

[0919] 5. Generate detailed documentation:

[0920] Generate documentation including relevant test cases (e.g., sorting a list of numbers or a list of strings), use cases (specific sorting methods using particular lists), bug reports (issues that have been encountered and how they have been resolved), and improvement suggestions (e.g., a more efficient sorting algorithm).

[0921] 6. Sending and viewing results:

[0922] The server sends this information to the device, which displays the results to the user, who can review code snippets, flowcharts, and detailed documentation and incorporate them into their own projects as needed.

[0923] The above is a specific embodiment of the present invention, and this system enables users to efficiently find code fragments that suit their purpose, understand them visually, and obtain related information comprehensively.

[0924] The processing flow will be explained below.

[0925] Step 1:

[0926] A user enters a search query into a terminal, such as "Can you give me the Python code to sort a list?" in natural language, and clicks the search button.

[0927] Step 2:

[0928] The device sends the query entered by the user to the server as is, and then communicates via the Internet to pass the data to the server.

[0929] Step 3:

[0930] The server analyzes the received natural language query, and the query analysis engine uses natural language processing techniques to extract key keywords (in this case, "list," "sort," and "Python").

[0931] Step 4:

[0932] The server searches for relevant code fragments in its internal database or external code repository based on the extracted keywords. The search engine finds code fragments that match the keywords and displays them in a list.

[0933] Step 5:

[0934] The server selects the most appropriate code fragment from the searched code fragments and visually represents its behavior. The visual representation generation component analyzes the code structure and generates visual data as flowcharts or animations.

[0935] Step 6:

[0936] The server retrieves test cases and use cases related to a piece of code from a database, as well as related bug reports and improvement suggestions, and synthesizes them to generate comprehensive documentation. The documentation generation component organizes this information and presents it in a user-friendly format.

[0937] Step 7:

[0938] The server sends the generated visual representation and detailed documentation to the device, which receives the data and converts it into a format that can be displayed to the user.

[0939] Step 8:

[0940] The terminal displays the received visual representation and detailed documentation to the user, who can then review the code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[0941] The above is the processing flow from the user's natural language query to code search, analysis, visual representation generation, document generation, and result display.

[0942] Example 1

[0943] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0944] Conventional code search systems make it difficult for users to quickly find the code fragment they are looking for and intuitively understand its behavior. They also lack the functionality to provide detailed information such as related test cases and use cases. As a result, users have to spend a lot of time finding the appropriate code fragment, and it is difficult to fully understand the quality and scope of the code.

[0945] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0946] In this invention, the server includes: means for analyzing a user's query using natural language processing technology; means for retrieving code fragments from an internal database or an external repository; means for converting the retrieved code procedures into a flowchart or visual diagram; means for generating detailed documentation using a generative AI model; means for analyzing the natural language query and searching for code fragments that match a purpose; means for visually representing the behavior of the retrieved code fragments; means for generating detailed documentation including test cases and use cases related to the code fragments; and means for transmitting the search results, the visual representation, and the detailed documentation to the user's terminal. This allows the user to efficiently find code fragments that match their purpose, visually understand them, and comprehensively obtain related information.

[0947] A "natural language query" is a question or request made by a user to a computer system in natural human language.

[0948] A "query" is a statement or command that describes instructions for retrieving information from a database or information system.

[0949] "Analysis" is the process of separating and analyzing data and information to understand its intent, content, and structure.

[0950] A "code fragment" is a part of a program or script, a fragment of source code that realizes a specific function or operation.

[0951] "Visual representation" refers to the display of information or data in visual formats such as graphs, flow charts, animations, etc.

[0952] A "flowchart" is a diagram that visually shows the flow of an algorithm or process by representing each step with shapes and arrows.

[0953] "Animation" is a technique for expressing movement by displaying a series of still images in succession over time.

[0954] A "test case" is a set of input data and expected results designed to verify a particular function or behavior of software or a system.

[0955] A "use example" is a concrete example that shows how to actually use a specific function or operation.

[0956] A "document" is a document or material created to record information and communicate it to others.

[0957] A "bug report" is a document used to report defects or problems within software or a system, and describes the incident, the steps to reproduce it, the scope of the impact, etc.

[0958] An "improvement proposal" is a document that describes specific proposals and methods for improving the performance or usability of an existing system or process.

[0959] A "generative AI model" is an artificial intelligence model that is generated using machine learning or deep learning techniques and is designed to solve a specific task or problem.

[0960] A "search result" is a set of information returned based on a search query.

[0961] The "Internet" is a global information and communications network of interconnected computer networks.

[0962] "API" is an abbreviation for Application Programming Interface, and is a set of definitions and rules for exchanging functions and data between software programs.

[0963] An "internal database" is a database managed within the system, used to store and manage user and system data.

[0964] An "external repository" is a database or information resource that exists outside the system, where code and data that can be accessed from outside are managed.

[0965] MODE FOR CARRYING OUT THE INVENTION

[0966] The present invention provides a system for searching code in natural language, displaying code fragments in a visually understandable format, and providing related detailed information. Specific embodiments of the system are described below.

[0967] Overall system overview

[0968] This system consists of a user, a terminal, and a server. The user uses the terminal to search for codes related to specific functions or operations in natural language. The terminal receives input from the user and sends it to the server. The server analyzes the received input and returns the search results. Specifically, the server operates based on the following steps.

[0969] User operations

[0970] A user uses a device (such as a computer or smartphone) to search for code related to a specific function or operation using natural language, for example, typing "What is the Python code to sort a list?"

[0971] Device Role

[0972] The device receives input from the user and sends it to a server over the Internet, for example, as text data in JSON format. It also has the ability to display search results, visual representations, and detailed documents returned by the server, allowing the user to quickly and efficiently obtain the desired information.

[0973] Server Roles

[0974] The server has the following main functions:

[0975] 1. Natural Language Query Parsing:

[0976] The server receives natural language queries sent by users and analyzes their intent. It uses natural language processing techniques to extract keywords and contexts and use them as search criteria. This process uses generative AI models such as BERT and GPT-4.

[0977] 2. Search for code fragments:

[0978] Based on the analysis results, it searches for matching code fragments from an internal database or external code repositories (e.g., GitHub or GitLab), thereby finding the appropriate code to achieve the functionality or operation the user is looking for.

[0979] 3. Generate visual representation:

[0980] The behavior of the found code fragments is then translated into a visually understandable form, specifically using tools like PlantUML or Draw.io to represent the code's processes as flowcharts or animations, allowing users to intuitively understand the code's structure and behavior.

[0981] 4. Generate detailed documentation:

[0982] It collects test cases, use cases, bug reports, and improvement suggestions related to a piece of code and generates a consolidated and detailed document. It uses a generative AI model to generate the document by inputting a prompt such as:

[0983] "Generate documentation containing test cases, use cases, bug reports, and improvement suggestions related to the code snippets found in the search results. Format should be Markdown."

[0984] 5. Sending the results:

[0985] The visual representation and detailed documentation are sent to the terminal, which displays it and provides it to the user.

[0986] Specific examples

[0987] Below is a specific example where a user searches for "Python code to sort a list."

[0988] 1. User input:

[0989] The user types "Please tell me the Python code to sort a list" into the terminal and clicks the search button.

[0990] 2. Sending by terminal:

[0991] The terminal sends this input to the server.

[0992] 3. Query parsing and code lookup by the server:

[0993] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches its internal database for relevant code fragments.

[0994] 4. Generating a visual representation:

[0995] The server analyzes the steps in the code fragments it finds and generates a flowchart using PlantUML that shows the process of receiving an input list, sorting the elements, and returning the results.

[0996] 5. Generate detailed documentation:

[0997] The server generates documentation in LaTeX and Markdown format containing test cases, use cases, bug reports, and improvement suggestions. For example, the following prompt is input to the generative AI model:

[0998] "Generate documentation containing test cases, use cases, bug reports, and improvement suggestions related to the code snippets found in the search results. Format should be Markdown."

[0999] 6. Sending and viewing results:

[1000] The server sends this information to the device, which displays the results to the user: code snippets, flowcharts, and detailed documentation that the user can review and, if desired, incorporate into their own projects.

[1001] The above is an embodiment of the present invention, and this system enables users to efficiently find code fragments that suit their purpose, understand them visually, and obtain related information comprehensively.

[1002] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1003] Step 1:

[1004] Accepting user input

[1005] The user types into the terminal, "Can you give me the Python code to sort a list?"

[1006] Input: Natural language text input by the user

[1007] Output: User-entered text data

[1008] Specific operation: A user enters the required query into a text input field in a web browser or mobile app and clicks the search button. The device retrieves this input using JavaScript or similar tools and prepares it for further processing.

[1009] Step 2:

[1010] Terminal input transmission

[1011] The terminal receives input from the user and sends it to the server.

[1012] Input: User-entered text data

[1013] Output: Sending text data to the server

[1014] What it does: The device uses JavaScript or other front-end technologies to convert user input into JSON format and send it to the server via an AJAX request, while displaying a sending indicator to the user.

[1015] Step 3:

[1016] Query analysis by the server

[1017] The server parses the received user input.

[1018] Input: Received text data (query)

[1019] Output: Keyword and context extraction results

[1020] How it works: The server parses the incoming data and analyzes it using natural language processing techniques. It uses generative AI models such as BERT and GPT-4 to analyze the query and extract key keywords and intent. This process extracts keywords such as "list," "sort," and "Python."

[1021] Step 4:

[1022] Server lookup of code fragments

[1023] The server uses the analysis results to search for the code fragment.

[1024] Input: Keyword and context extraction results

[1025] Output: relevant code fragment

[1026] What happens: The server executes SQL queries to search its internal database, and optionally retrieves relevant code fragments from external code repositories (e.g., GitHub or GitLab) using API requests. As a result, the relevant code fragments are stored in the server's temporary memory.

[1027] Step 5:

[1028] Generating a visual representation

[1029] The server provides a visual representation of the retrieved code fragment.

[1030] Input: The searched code fragment

[1031] Output: Visual flowcharts and animations

[1032] How it works: The server analyzes the structure of a code fragment, calls the APIs of PlantUML and Draw.io, and generates flowcharts and animations. The generated visual data is saved as SVG or PNG format files.

[1033] Step 6:

[1034] Generate detailed documentation

[1035] The server generates detailed documentation related to the code fragment.

[1036] Input: The searched code fragment

[1037] Output: Detailed documentation (Markdown or LaTeX format)

[1038] How it works: The following prompt is input to the generative AI model to generate a document: "Please generate a document containing test cases, use cases, bug reports, and improvement suggestions related to the code fragment found in the search results. Please use Markdown format." The generated document is saved on the server in Markdown or LaTeX format.

[1039] Step 7:

[1040] Sending and viewing results

[1041] The server sends the visual representation and detailed documentation to the terminal, which displays the results to the user.

[1042] Input: Visual flowcharts, animations, detailed documentation

[1043] Output: The results displayed on the user's terminal

[1044] What it does: The server encodes the generated data in JSON format and sends it to the device as an HTTP response. The device receives it, renders the data using HTML and JavaScript, and displays it to the user. The user can then view the displayed code fragments, flowcharts, and detailed documentation and incorporate them into their own projects as needed.

[1045] The above are the specific processing steps of this system, which allow users to efficiently obtain the information they require.

[1046] (Application example 1)

[1047] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1048] While conventional code search systems analyze natural language and provide relevant code fragments, they lack sufficient visual understanding support. They also struggle to provide comprehensive, relevant detailed information. Furthermore, they have limitations in searching for code fragments specialized for specific applications and generating visual representations, making it difficult for developers to quickly and accurately obtain the information they need. For this reason, further improvements are needed in specialized fields such as electronic payments.

[1049] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1050] In this invention, the server includes means for analyzing a natural language query and searching for code fragments that match a purpose, means for visually representing the behavior of the searched code fragments, means for generating detailed documentation including test cases and use examples related to the code fragments, means for transmitting the search results, the visual representation, and the detailed documentation to a user's terminal, means for using a generative AI model to search for code fragments related to the query content and providing the code fragments together with their visual representations, and means for analyzing the user's intent using prompt sentences and generating related code fragments and documentation. This enables a user to quickly obtain specific code fragments related to electronic payment based on a natural language query, visually understand them, and comprehensively obtain detailed documentation.

[1051] A "natural language query" is a question or command entered by a user in human language.

[1052] A "code fragment" is a portion of program code that implements a specific function or operation.

[1053] "Visually representing" means displaying textual information in graphical form, such as a flow chart or animation.

[1054] A "test case" is a set of specific inputs and their expected outputs that are executed to verify the behavior of a program.

[1055] "Detailed documentation" refers to comprehensive information related to a piece of code, including explanations, use cases, bug reports, improvement suggestions, etc.

[1056] A "generative AI model" is an artificial intelligence program that uses machine learning techniques to analyze natural language and generate or find appropriate code fragments.

[1057] A "prompt sentence" is an input sentence for conveying the user's intention, and is a sentence used as a natural language query.

[1058] A "terminal" is a device that can be operated by a user, and specifically includes a personal computer, a smartphone, etc.

[1059] A "server" is a computer system that processes and stores data and provides information in response to requests from terminals.

[1060] "User" refers to a person who uses this system.

[1061] This invention provides a system that allows a user to input a query in natural language using a terminal (such as a smartphone or computer), search for code fragments that match the user's purpose, and present the results in a visually easy-to-understand format. This system is mainly composed of the following elements:

[1062] 1. Parsing natural language queries

[1063] The user uses the device to provide input in natural language, such as a prompt such as "What is the Python code to process a credit card payment?" The device then sends this input to the server, which uses a generative AI model to analyze the query and extract relevant keywords and intent. This generative AI model uses the Hugging Face transformers library.

[1064] 2. Searching for code fragments

[1065] Based on the extracted keywords and intent, the server searches for relevant code fragments from an internal database or external repository that contains code fragments written in various programming languages, including, for example, Python code for processing credit card payments.

[1066] 3. Generating Visual Representations

[1067] For each code fragment found, the server converts its behavior into a visually understandable representation, using Matplotlib and NetworkX to represent the code steps as flowcharts and animations. This visual representation helps users intuitively understand the code's flow and structure.

[1068] 4. Generate detailed documentation

[1069] The server generates detailed documentation such as test cases, use cases, bug reports, and improvement suggestions related to the retrieved code fragments, allowing users to gain a comprehensive understanding of the code's quality and coverage.

[1070] 5. Sending and displaying results

[1071] The server sends the search results, visual representations, and detailed documentation to the terminal, which then displays them to the user, who can quickly and efficiently obtain the information they need based on the displayed code fragments, flowcharts, and detailed documentation.

[1072] As a concrete example, consider the case where a user uses the following prompt text:

[1073] "Can you give me some Python code to process credit card payments?"

[1074] In response, the system searches for relevant code fragments, generates visual representations and detailed documentation, and provides them to users, allowing them to quickly extract the code they need and apply it to their own projects.

[1075] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1076] Step 1:

[1077] Accepting natural language inquiries

[1078] The user uses a terminal to input a query in natural language. At this stage, the user enters a prompt sentence such as "Please give me the Python code to process credit card payments." The input data is sent over the Internet to the server.

[1079] Input: Natural language query (prompt)

[1080] Output: The query sent to the server

[1081] Step 2:

[1082] Natural Language Analysis

[1083] The server analyzes the received query using a generative AI model (Hugging Face's transformers library). This model extracts keywords and user intent from the input natural language text. Specifically, keywords such as "credit card," "payment," and "Python" are extracted.

[1084] Input: Received natural language query

[1085] Output: Extracted keywords and user intent

[1086] Step 3:

[1087] Searching for code fragments

[1088] The server then searches for relevant code fragments from an internal database or external repository based on the extracted keywords and intent. This database contains code fragments written in various programming languages, and the search result is "Python code to process credit card payments."

[1089] Input: Extracted keywords and user intent

[1090] Output: relevant code fragment

[1091] Step 4:

[1092] Generating a visual representation

[1093] The server then converts the code fragments it finds into a visually understandable representation of their behavior, using Matplotlib and NetworkX to generate flowcharts and animations of the code steps, providing a visual representation of the code's flow and structure.

[1094] Input: The searched code fragment

[1095] Output: Generated flowchart or animation

[1096] Step 5:

[1097] Generate detailed documentation

[1098] The server generates detailed documentation containing test cases, use cases, bug reports, and improvement suggestions associated with the code fragment, providing data for a comprehensive understanding of the code's quality and coverage.

[1099] Input: The searched code fragment

[1100] Output: Detailed documentation (test cases, use cases, bug reports, improvement suggestions)

[1101] Step 6:

[1102] Sending and displaying results

[1103] The server sends the search results, visual representations, and detailed documentation to the terminal, which displays them to the user, allowing the user to quickly and efficiently obtain the information they need based on the displayed code fragments, flowcharts, and detailed documentation.

[1104] Input: Search results, visual representation, detailed documentation

[1105] Output: what is displayed on the user's device (code fragments, visual representations, detailed documentation)

[1106] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1107] The present invention incorporates an emotion engine that recognizes the user's emotions into a system that performs code searches in natural language, displays the results in a visually easy-to-understand format, and provides related information. Specific embodiments of the present invention are described below.

[1108] Overall system overview

[1109] This system consists of a user, a terminal, a server, and an emotion engine.

[1110] User operations

[1111] Users use the terminal to search for code related to a particular function or operation using natural language, for example by entering a query such as "What is the Python code to sort a list?" and clicking the search button.

[1112] Device Role

[1113] The device receives input from the user and sends it to the server, displays search results, visual representations, and detailed documents sent from the server, and collects emotional data from the user's facial expressions and voice and sends it to the server.

[1114] Server Roles

[1115] The server has the following main functions:

[1116] 1. Natural Language Query Parsing:

[1117] The server analyzes the natural language queries sent by users and extracts keywords and context using natural language processing techniques.

[1118] 2. Search for code fragments:

[1119] Based on the analysis results, the search engine searches for relevant code fragments in internal databases and external code repositories, and finds code fragments that match the keywords.

[1120] 3. Generate visual representation:

[1121] The behavior of the found code fragments is converted into a visually understandable form (flowchart or animation) by the visual representation generation component, which analyzes the code structure.

[1122] 4. Generate detailed documentation:

[1123] It collects test cases, use cases, bug reports and improvement suggestions related to a piece of code and generates a single comprehensive document.

[1124] 5. User sentiment analysis:

[1125] The emotion engine analyzes the user's emotional data sent from the device and adjusts search results and visual presentation accordingly, for example by adding more detailed explanations if the user is confused.

[1126] 6. Sending the results:

[1127] Finally, adjustment information that takes into account the user's emotion data is sent to the terminal together with the visual representation and detailed documentation.

[1128] Specific examples

[1129] Below is a concrete example of how the sentiment engine works in the process of a user searching for "Python code to sort a list."

[1130] 1. User input:

[1131] The user enters "Please tell me the Python code to sort the list" into the terminal and clicks the search button. The terminal simultaneously collects the user's facial expression and voice data.

[1132] 2. Sending by terminal:

[1133] The terminal transmits the query and the user's emotion data to the server.

[1134] 3. Query parsing and code lookup by the server:

[1135] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches the database for the corresponding code fragment.

[1136] 4. Generating a visual representation:

[1137] The server analyzes the steps in the code fragment it finds and generates a flowchart that shows the process of taking an input list, sorting the elements, and returning the result.

[1138] 5. Generate detailed documentation:

[1139] Generate documentation including relevant test cases (e.g., sorting a list of numbers or a list of strings), use cases (specific sorting methods using particular lists), bug reports (issues that have been encountered and how they have been resolved), and improvement suggestions (e.g., a more efficient sorting algorithm).

[1140] 6. User sentiment analysis:

[1141] The server analyzes the user's emotional data through an emotion engine and determines that the user is confused, and accordingly adjusts the generated documentation to include additional explanations and examples.

[1142] 7. Submitting and Viewing Results:

[1143] The server sends visual representations and detailed documentation to the user's device, reflecting the emotion-based adjustment information, and the device displays it, allowing the user to review code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[1144] The above is a specific embodiment of the present invention incorporating an emotion engine. This system allows users to efficiently find code fragments that suit their purpose, visually understand them, and comprehensively obtain related information, while also receiving appropriate support based on their emotions.

[1145] The processing flow will be explained below.

[1146] Step 1:

[1147] Using a terminal, a user enters a natural language query such as "Can you give me the Python code to sort a list?" and clicks the search button.

[1148] Step 2:

[1149] The device collects the user's facial expressions and voice data along with natural language queries from the user, and these data are simultaneously sent to the server as emotion data for analyzing the user's emotions.

[1150] Step 3:

[1151] The server analyzes the received natural language query, and the query analysis engine uses natural language processing techniques to extract key keywords such as "list," "sort," and "Python."

[1152] Step 4:

[1153] The server searches for relevant code fragments from an internal database or external code repository based on the analyzed keywords, and the search engine extracts matching code fragments in a list.

[1154] Step 5:

[1155] The server selects the most suitable code fragment from those found and then uses a visual representation generation component to visually represent the behavior of the code fragment, generating a flowchart or animation of the code fragment.

[1156] Step 6:

[1157] The server collects test cases and use cases related to a piece of code from a database, as well as bug reports and improvement suggestions, and generates comprehensive documentation out of this information. This process is handled by the documentation generation component.

[1158] Step 7:

[1159] The server runs an emotion engine to analyze the user's emotion data sent from the device. This engine analyzes how the user feels about the displayed information, and if a negative emotion such as confusion is detected, the engine includes a more detailed explanation or additional examples.

[1160] Step 8:

[1161] The server transmits the generated visual representation, detailed documentation, and information adjusted based on the emotion to the terminal.

[1162] Step 9:

[1163] The terminal displays the received information to the user, showing visual flowcharts, detailed code descriptions, documentation with associated test cases, use cases, bug reports, and improvement suggestions, allowing the user to understand the code fragment and efficiently incorporate it into their own projects.

[1164] This series of steps allows users to obtain code fragments and related information in a more relevant and understandable way, based on the additional information of emotions.

[1165] Example 2

[1166] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1167] Conventional program code search systems often produce insufficient search results even when users input queries in natural language. Furthermore, providing information through code fragments that are difficult to understand visually or incomplete documentation contributes to a lack of user convenience. Furthermore, providing search results without considering the user's emotional state can be confusing, especially for novice users. To address these issues, systems are required that can analyze queries in natural language, display information that is easy to understand visually, and provide information that takes the user's emotions into account.

[1168] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1169] In this invention, the server includes means for analyzing a natural language query and searching for program code that matches a purpose, means for visually representing the behavior of the searched program code, means for generating detailed documentation including test cases and use cases related to the program code, means for collecting and analyzing emotional data, and means for adjusting the visual representation and the content of the detailed documentation based on the emotional data. This allows the user to obtain search results that are more accurate and visually easy to understand, and also makes it possible to provide appropriate information according to the user's emotional state.

[1170] A "natural language query" refers to a query entered by a user using a human language (e.g., Japanese, English, etc.).

[1171] "Program Code" means source code that contains instructions and procedures for performing a particular task or function.

[1172] A "search tool" refers to a mechanism that analyzes natural language queries and uses them to find appropriate program code from databases or external repositories.

[1173] "Visual representation means" refers to a mechanism for displaying the behavior and structure of program code using visualization techniques such as flowcharts and animations.

[1174] "Detailed documentation" refers to comprehensive documentation containing test cases, use cases, bug reports, and improvement suggestions related to the program code.

[1175] "Emotional data" refers to data that indicates the user's emotional state, collected from the user's facial expressions, voice, etc.

[1176] "Emotion analysis means" refers to a mechanism for analyzing collected emotion data and determining the user's emotional state.

[1177] "Adjustment means" refers to a mechanism that appropriately changes the visual presentation and detailed document content based on the results of sentiment analysis to provide information in a form that is easier for users to understand.

[1178] "User terminal" refers to a device such as a computer, smartphone, or tablet that is actually operated by a user.

[1179] The present invention provides a system that allows a user to search for program code in natural language, displays the code in a visually easy-to-understand format, and provides information based on the user's emotions. Specific embodiments will be described below.

[1180] System configuration

[1181] This system consists of a user, a terminal, a server, and an emotion engine.

[1182] User operations

[1183] Users use the terminal to search for program code related to a specific function or operation in natural language, for example, by entering a query such as "What is the Python code to sort a list?" and clicking the search button.

[1184] Device Role

[1185] The device receives input queries from the user and sends them to the server. It also has the function of collecting emotional data from the user's facial expressions and voice and transmitting it to the server. For example, it acquires the user's emotional data using a camera or microphone. It also displays search results, visual representations, and detailed documents received from the server to the user.

[1186] Server Roles

[1187] The server has the following main functions:

[1188] 1. Natural Language Query Parsing:

[1189] The server analyzes the natural language query submitted by the user, for example by using natural language processing tools such as SpaCy or NLTK to extract keywords such as "list," "sort," and "Python."

[1190] 2. Search for the program code:

[1191] Based on the analysis results, the relevant program code is searched for in an internal database (e.g., PostgreSQL) or an external code repository (e.g., GitHub).

[1192] 3. Generate visual representation:

[1193] Convert the searched program code into a visually understandable format (flowchart or animation). For example, generate a flow chart using D3.js.

[1194] 4. Generate detailed documentation:

[1195] It collects test cases, use cases, bug reports, and improvement suggestions related to program code and generates detailed documentation in Markdown format.

[1196] 5. Emotion analysis:

[1197] The server's emotion engine analyzes the user's emotion data sent from the device, for example, using an emotion recognition API to determine whether the user is confused.

[1198] 6. Sending the results:

[1199] The server generates a visual representation and detailed documentation, and transmits the information adjusted based on the sentiment analysis results to the terminal.

[1200] Specific examples

[1201] Here is a specific example of what happens when a user searches for "Can you give me the Python code to sort a list?"

[1202] 1. User input:

[1203] The user enters "Please tell me the Python code to sort the list" into the terminal and clicks the search button. The terminal simultaneously collects the user's facial expression and voice data.

[1204] 2. Sending by terminal:

[1205] The terminal transmits the query and the user's emotion data to the server.

[1206] 3. Query parsing and code lookup by the server:

[1207] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches the database for the corresponding code fragment.

[1208] 4. Generating a visual representation:

[1209] The server analyzes the steps in the code fragments it finds and generates a flowchart using D3.js.

[1210] 5. Generate detailed documentation:

[1211] The server creates a detailed document in Markdown containing relevant test cases, use cases, bug reports, and improvement suggestions.

[1212] 6. User sentiment analysis:

[1213] The server's emotion engine detects user confusion and adjusts the generated documentation to include additional explanations and examples.

[1214] 7. Submitting and Viewing Results:

[1215] The server sends the final visual representation and detailed documentation to the device, which displays it, allowing users to review code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[1216] Prompt Sentence Examples

[1217] Here are some example prompts for searches using this system:

[1218] "Enter 'Please tell me the Python code to sort a list' in the terminal and search."

[1219] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1220] Step 1:

[1221] The user enters a natural language query into the text input field of the terminal and clicks the search button. For example, they might type "Can you give me the Python code to sort a list?" This is the input data. The output they get from the terminal is the query.

[1222] Step 2:

[1223] The device sends queries entered by the user to the server. In addition, the device's built-in camera and microphone capture the user's facial expressions and voice data, which are also sent to the server. The input is the user's query and emotion data, and the output is an HTTP request to the server. Specifically, the device converts the query and emotion data into JSON format and sends it.

[1224] Step 3:

[1225] The server analyzes the received natural language query using a natural language processing tool (e.g., SpaCy or NLTK) to extract keywords and context. The input is the query sent from the terminal, and the output is the extracted keywords (e.g., "list," "sort," "Python"). Specifically, the server calls a text analysis engine to analyze the query.

[1226] Step 4:

[1227] The server searches for relevant program code in a database (e.g., PostgreSQL) or an external repository (e.g., GitHub) based on the extracted keywords. The input is the parsed keywords, and the output is the search results, which are program code fragments. Specifically, it executes SQL queries or API requests to retrieve data.

[1228] Step 5:

[1229] The server analyzes the searched program code and generates visual representations such as flowcharts and animations. This is done using D3.js or other visualization tools. The input is the program code fragments as search results, and the output is the visual representation. Specifically, each step of the program code is input into the visualization tool to generate a flowchart.

[1230] Step 6:

[1231] The server collects test cases, use cases, bug reports, and improvement suggestions related to the searched program code and generates detailed documentation. The documentation format is Markdown. The input is a fragment of program code, and the output is a detailed documentation. Specifically, it collects and integrates related information and generates a document in Markdown format.

[1232] Step 7:

[1233] The emotion engine on the server analyzes the user's emotion data collected from the device and determines the user's emotional state (e.g., confusion, satisfaction, etc.). The input is the emotion data sent from the device, and the output is the user's emotional state as a result of the analysis. Specifically, it calls the emotion recognition API and analyzes the data.

[1234] Step 8:

[1235] The server adjusts the content of the visual representation and detailed text based on the user's emotional state. For example, if the user is confused, it adds more detailed explanations and examples. The input is the user's emotional state, the visual representation, and the detailed text, and the output is the adjusted visual representation and detailed text. Specific operations include updating the text and visualization data.

[1236] Step 9:

[1237] The server sends the adjusted visual representation and detailed document to the terminal. The input is the adjusted visual representation and detailed document, and the output is an HTTP response to the terminal. As a specific operation, the server generates an HTTP response for sending data to the terminal.

[1238] Step 10:

[1239] The terminal displays the visual representation and detailed document received from the server to the user. The input is data from the server, and the output is the display to the user. Specifically, the terminal uses a browser or a dedicated application to render and display the visual representation and document.

[1240] (Application example 2)

[1241] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1242] In modern factories and production lines, robot programming and troubleshooting are extremely important, but they require specialized knowledge and experience. Furthermore, conventional code search systems and support tools provide only one-way information without considering the user's feelings. This often leaves users confused and reduces work efficiency. Another issue is the lack of visual support, which makes it time-consuming to understand and operate the code. A system that can solve these problems and provide more intuitive and efficient support is needed.

[1243] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing a natural language query and searching for code fragments that match the purpose, means for visually expressing the behavior of the searched code fragments, and means for analyzing user emotion data and adjusting the display method of the search results and visual expression based on the results. This makes it possible to provide optimal support that is easy for the user to intuitively understand and that is tailored to their emotions.

[1244] A "natural language query" is a question or instruction written in a language normally used by humans, with the goal of converting it into a form that the system can understand and process.

[1245] A "code fragment" is a part of a program that realizes a specific function or operation, and refers to a short piece of code that is used in actual programming work.

[1246] A "visual representation" is a visually easy-to-understand representation of the behavior of the retrieved code fragment, which can take the form of a flowchart or animation.

[1247] "Detailed documentation" refers to materials that contain supplementary and comprehensive information, such as test cases, use cases, bug reports, and improvement suggestions related to a piece of code.

[1248] "Emotional data" refers to emotional information extracted from a user's facial expressions and voice, and is used by the system to analyze and understand the user's emotional state.

[1249] "Adjusting the display method" refers to changing the format and content of search results and visual representations based on the user's emotional data to make them easier for the user to understand.

[1250] This system analyzes natural language queries, searches for code fragments that match the query, and displays the code's behavior in a visually understandable format. It also analyzes user sentiment and adjusts the display of search results and visual representations based on the results.

[1251] System configuration

[1252] This system consists of a user, a terminal, a server, and an emotion engine.

[1253] 1. Users

[1254] Users use devices such as smartphones or head-mounted displays to search for code fragments in natural language, for example by entering queries such as "What code can I use to change the speed of a robot?"

[1255] 2. Terminal

[1256] The device receives input from the user and sends it to the server. It also collects emotional data from the user's facial expressions and voice and sends this data to the server.

[1257] 3. Server

[1258] The server has the following functions:

[1259] Natural language query analysis: The server analyzes the natural language query sent by the user and extracts keywords and context using natural language processing techniques.

[1260] Searching for code fragments: Based on the analysis results, the database is searched for the relevant code fragments.

[1261] Visual representation generation: Convert the behavior of the retrieved code fragments into a visually understandable form (flowchart or animation).

[1262] Generate detailed documentation: Collect test cases, use cases, bug reports and improvement suggestions related to a piece of code and generate one comprehensive document.

[1263] User sentiment analysis: Analyzes user sentiment data sent from the device through the emotion engine and adjusts the way search results and visual representations are displayed based on the results.

[1264] 4. Emotion Engine

[1265] The emotion engine analyzes the user's facial expressions and voice data to determine the user's emotional state. This analysis is performed using, for example, the "Transformers" pipeline.

[1266] Processing flow

[1267] As a concrete example, a user enters the query "Please tell me the code to change the robot's movement speed," and in the process explains how the emotion engine works.

[1268] 1. Query input: The user inputs a query into the device, and the device simultaneously collects the user's facial expressions and voice.

[1269] 2. Data transmission: The device sends the query and emotion data to the server.

[1270] 3. Query analysis and code search: The server analyzes the query, extracts the keywords "operation speed," "change," "code," and "robot," and searches the database for the corresponding code fragment.

[1271] 4. Visual representation generation: The server analyzes the procedures in the code fragments it finds and generates a flowchart that shows the sequence of processes that change the speed of the execution.

[1272] 5. Detailed documentation generation: Generate documentation including code examples, test cases, improvement suggestions, etc.

[1273] 6. Sentiment Analysis: The server analyzes the user's emotional data through an emotion engine and determines that the user is confused. Based on this, it adjusts the generated documentation and visual representations to include additional explanations and examples.

[1274] 7. Result transmission and display: The server transmits the tailored visual representation and detailed documentation to the user's device, which displays them, allowing the user to review and incorporate code snippets, flowcharts, and related information into their own projects.

[1275] Prompt Sentence Examples

[1276] "Please provide code snippets for the following natural language queries, and add detailed documentation and visual representations if the user's sentiment is confused: Query: 'What code can I use to change the speed of the robot?'"

[1277] The above is a concrete example of how to implement the invention. This system allows users to efficiently find code fragments that suit their purpose, visually understand them, and comprehensively obtain related information, while also receiving appropriate support based on their emotions.

[1278] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1279] Step 1:

[1280] The user inputs a query in natural language. For example, "Please tell me the code to change the robot's movement speed." At this time, the device simultaneously collects the user's facial expression and voice data. The inputs in this step are the natural language query, facial expression data, and voice data, and the output is a data package of these.

[1281] Step 2:

[1282] The device sends the query and collected emotion data to the server. The input of this step is the data package collected in step 1, and the output is the data transmission to the server.

[1283] Step 3:

[1284] The server analyzes the received natural language query and extracts keywords and context. Here, natural language processing technology is used to extract important keywords such as "robot," "movement speed," and "change." The input is the user's query, and the output is a list of keywords as the analysis result.

[1285] Step 4:

[1286] The server searches the database for relevant code fragments based on the keywords extracted in step 3. For example, it finds "code to change the robot's movement speed." The input of this step is a list of keywords, and the output is the searched code fragments.

[1287] Step 5:

[1288] The server analyzes the behavior of the retrieved code fragments and generates visually understandable flowcharts and animations. Specifically, it processes the data to visually display the flow of each statement and loop in the code. The input for this step is the code fragment, and the output is a flowchart or animation.

[1289] Step 6:

[1290] The server generates detailed documentation for a code fragment, including test cases, use cases, bug reports, and improvement suggestions. The documentation generation process involves collecting relevant information and organizing it in a systematic way. The input of this step is the code fragment, and the output is detailed documentation.

[1291] Step 7:

[1292] The server analyzes the emotion data received in step 2 and determines the user's emotional state. For example, it uses an emotion engine to detect a "confused" state. The input of this step is emotion data, and the output is the emotion analysis result.

[1293] Step 8:

[1294] The server adjusts the search results, visual representation, and detailed document content based on the sentiment analysis results obtained in step 7. For example, if the user is confused, the server may include more detailed explanations or additional examples in the document. The inputs to this step are the sentiment analysis results, visual representation, and detailed document, and the output is the final adjusted visual representation and detailed document.

[1295] Step 9:

[1296] The server finally sends the adjusted visual representation and detailed document to the terminal. The input of this step is the adjusted visual representation and detailed document, and the output is data transmission to the terminal.

[1297] Step 10:

[1298] The terminal displays the visual representation and detailed documentation sent from the server to the user, allowing the user to identify the necessary code fragments and integrate them into their project. The input of this step is the data sent from the server, and the output is the display to the user.

[1299] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1300] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1301] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1302] [Fourth embodiment]

[1303] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1304] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1305] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1306] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1307] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1308] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1309] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1310] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1311] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1312] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1313] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1314] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1315] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1316] The present invention provides a system for searching code in natural language, displaying code fragments in a visually understandable format, and providing related detailed information. Specific embodiments of the system are described below.

[1317] Overall system overview

[1318] This system consists of users, terminals, and a server.

[1319] User operations

[1320] A user uses a device (such as a computer or smartphone) to search for code related to a specific function or operation using natural language, for example, typing "What is the Python code to sort a list?"

[1321] Device Role

[1322] The terminal receives input from the user, transmits it to a server via the Internet, and displays search results, visual representations, and detailed documents returned by the server, allowing the user to quickly and efficiently obtain the desired information.

[1323] Server Roles

[1324] The server has the following main functions:

[1325] 1. Natural Language Query Parsing:

[1326] The server receives the natural language query sent by the user and analyzes its intent. Specifically, it uses natural language processing technology to extract keywords and contexts, which are then used as search criteria.

[1327] 2. Search for code fragments:

[1328] Based on the analysis results, it searches for code fragments that match the purpose from an internal database or an external code repository, thereby finding the appropriate code to achieve the functionality or operation the user desires.

[1329] 3. Generate visual representation:

[1330] The behavior of the found code fragments is converted into a form that can be understood visually. Specifically, the code process is expressed as a flowchart or animation, allowing users to intuitively understand the structure and behavior of the code.

[1331] 4. Generate detailed documentation:

[1332] It collects test cases, use cases, bug reports and improvement suggestions related to a piece of code and generates a consolidated and detailed documentation, which gives users a comprehensive understanding of the code's quality and coverage.

[1333] 5. Sending the results:

[1334] The visual representation and detailed documentation are sent to the terminal, which displays it and provides it to the user.

[1335] Specific examples

[1336] Below is a specific example where a user searches for "Python code to sort a list."

[1337] 1. User input:

[1338] The user types "Please tell me the Python code to sort a list" into the terminal and clicks the search button.

[1339] 2. Sending by terminal:

[1340] The terminal sends this input to the server.

[1341] 3. Query parsing and code lookup by the server:

[1342] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches its internal database for relevant code fragments.

[1343] 4. Generating a visual representation:

[1344] The server analyzes the steps in the code fragment it finds and generates a flowchart that shows the process of taking an input list, sorting the elements, and returning the result.

[1345] 5. Generate detailed documentation:

[1346] Generate documentation including relevant test cases (e.g., sorting a list of numbers or a list of strings), use cases (specific sorting methods using particular lists), bug reports (issues that have been encountered and how they have been resolved), and improvement suggestions (e.g., a more efficient sorting algorithm).

[1347] 6. Sending and viewing results:

[1348] The server sends this information to the device, which displays the results to the user, who can review code snippets, flowcharts, and detailed documentation and incorporate them into their own projects as needed.

[1349] The above is a specific embodiment of the present invention, and this system enables users to efficiently find code fragments that suit their purpose, understand them visually, and obtain related information comprehensively.

[1350] The processing flow will be explained below.

[1351] Step 1:

[1352] A user enters a search query into a terminal, such as "Can you give me the Python code to sort a list?" in natural language, and clicks the search button.

[1353] Step 2:

[1354] The device sends the query entered by the user to the server as is, and then communicates via the Internet to pass the data to the server.

[1355] Step 3:

[1356] The server analyzes the received natural language query, and the query analysis engine uses natural language processing techniques to extract key keywords (in this case, "list," "sort," and "Python").

[1357] Step 4:

[1358] The server searches for relevant code fragments in its internal database or external code repository based on the extracted keywords. The search engine finds code fragments that match the keywords and displays them in a list.

[1359] Step 5:

[1360] The server selects the most appropriate code fragment from the searched code fragments and visually represents its behavior. The visual representation generation component analyzes the code structure and generates visual data as flowcharts or animations.

[1361] Step 6:

[1362] The server retrieves test cases and use cases related to a piece of code from a database, as well as related bug reports and improvement suggestions, and synthesizes them to generate comprehensive documentation. The documentation generation component organizes this information and presents it in a user-friendly format.

[1363] Step 7:

[1364] The server sends the generated visual representation and detailed documentation to the device, which receives the data and converts it into a format that can be displayed to the user.

[1365] Step 8:

[1366] The terminal displays the received visual representation and detailed documentation to the user, who can then review the code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[1367] The above is the processing flow from the user's natural language query to code search, analysis, visual representation generation, document generation, and result display.

[1368] Example 1

[1369] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1370] Conventional code search systems make it difficult for users to quickly find the code fragment they are looking for and intuitively understand its behavior. They also lack the functionality to provide detailed information such as related test cases and use cases. As a result, users have to spend a lot of time finding the appropriate code fragment, and it is difficult to fully understand the quality and scope of the code.

[1371] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1372] In this invention, the server includes: means for analyzing a user's query using natural language processing technology; means for retrieving code fragments from an internal database or an external repository; means for converting the retrieved code procedures into a flowchart or visual diagram; means for generating detailed documentation using a generative AI model; means for analyzing the natural language query and searching for code fragments that match a purpose; means for visually representing the behavior of the retrieved code fragments; means for generating detailed documentation including test cases and use cases related to the code fragments; and means for transmitting the search results, the visual representation, and the detailed documentation to the user's terminal. This allows the user to efficiently find code fragments that match their purpose, visually understand them, and comprehensively obtain related information.

[1373] A "natural language query" is a question or request made by a user to a computer system in natural human language.

[1374] A "query" is a statement or command that describes instructions for retrieving information from a database or information system.

[1375] "Analysis" is the process of separating and analyzing data and information to understand its intent, content, and structure.

[1376] A "code fragment" is a part of a program or script, a fragment of source code that realizes a specific function or operation.

[1377] "Visual representation" refers to the display of information or data in visual formats such as graphs, flow charts, animations, etc.

[1378] A "flowchart" is a diagram that visually shows the flow of an algorithm or process by representing each step with shapes and arrows.

[1379] "Animation" is a technique for expressing movement by displaying a series of still images in succession over time.

[1380] A "test case" is a set of input data and expected results designed to verify a particular function or behavior of software or a system.

[1381] A "use example" is a concrete example that shows how to actually use a specific function or operation.

[1382] A "document" is a document or material created to record information and communicate it to others.

[1383] A "bug report" is a document used to report defects or problems within software or a system, and describes the incident, the steps to reproduce it, the scope of the impact, etc.

[1384] An "improvement proposal" is a document that describes specific proposals and methods for improving the performance or usability of an existing system or process.

[1385] A "generative AI model" is an artificial intelligence model that is generated using machine learning or deep learning techniques and is designed to solve a specific task or problem.

[1386] A "search result" is a set of information returned based on a search query.

[1387] The "Internet" is a global information and communications network of interconnected computer networks.

[1388] "API" is an abbreviation for Application Programming Interface, and is a set of definitions and rules for exchanging functions and data between software programs.

[1389] An "internal database" is a database managed within the system, used to store and manage user and system data.

[1390] An "external repository" is a database or information resource that exists outside the system, where code and data that can be accessed from outside are managed.

[1391] MODE FOR CARRYING OUT THE INVENTION

[1392] The present invention provides a system for searching code in natural language, displaying code fragments in a visually understandable format, and providing related detailed information. Specific embodiments of the system are described below.

[1393] Overall system overview

[1394] This system consists of a user, a terminal, and a server. The user uses the terminal to search for codes related to specific functions or operations in natural language. The terminal receives input from the user and sends it to the server. The server analyzes the received input and returns the search results. Specifically, the server operates based on the following steps.

[1395] User operations

[1396] A user uses a device (such as a computer or smartphone) to search for code related to a specific function or operation using natural language, for example, typing "What is the Python code to sort a list?"

[1397] Device Role

[1398] The device receives input from the user and sends it to a server over the Internet, for example, as text data in JSON format. It also has the ability to display search results, visual representations, and detailed documents returned by the server, allowing the user to quickly and efficiently obtain the desired information.

[1399] Server Roles

[1400] The server has the following main functions:

[1401] 1. Natural Language Query Parsing:

[1402] The server receives natural language queries sent by users and analyzes their intent. It uses natural language processing techniques to extract keywords and contexts and use them as search criteria. This process uses generative AI models such as BERT and GPT-4.

[1403] 2. Search for code fragments:

[1404] Based on the analysis results, it searches for matching code fragments from an internal database or external code repositories (e.g., GitHub or GitLab), thereby finding the appropriate code to achieve the functionality or operation the user is looking for.

[1405] 3. Generate visual representation:

[1406] The behavior of the found code fragments is then translated into a visually understandable form, specifically using tools like PlantUML or Draw.io to represent the code's processes as flowcharts or animations, allowing users to intuitively understand the code's structure and behavior.

[1407] 4. Generate detailed documentation:

[1408] It collects test cases, use cases, bug reports, and improvement suggestions related to a piece of code and generates a consolidated and detailed document. It uses a generative AI model to generate the document by inputting a prompt such as:

[1409] "Generate documentation containing test cases, use cases, bug reports, and improvement suggestions related to the code snippets found in the search results. Format should be Markdown."

[1410] 5. Sending the results:

[1411] The visual representation and detailed documentation are sent to the terminal, which displays it and provides it to the user.

[1412] Specific examples

[1413] Below is a specific example where a user searches for "Python code to sort a list."

[1414] 1. User input:

[1415] The user types "Please tell me the Python code to sort a list" into the terminal and clicks the search button.

[1416] 2. Sending by terminal:

[1417] The terminal sends this input to the server.

[1418] 3. Query parsing and code lookup by the server:

[1419] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches its internal database for relevant code fragments.

[1420] 4. Generating a visual representation:

[1421] The server analyzes the steps in the code fragments it finds and generates a flowchart using PlantUML that shows the process of receiving an input list, sorting the elements, and returning the results.

[1422] 5. Generate detailed documentation:

[1423] The server generates documentation in LaTeX and Markdown format containing test cases, use cases, bug reports, and improvement suggestions. For example, the following prompt is input to the generative AI model:

[1424] "Generate documentation containing test cases, use cases, bug reports, and improvement suggestions related to the code snippets found in the search results. Format should be Markdown."

[1425] 6. Sending and viewing results:

[1426] The server sends this information to the device, which displays the results to the user: code snippets, flowcharts, and detailed documentation that the user can review and, if desired, incorporate into their own projects.

[1427] The above is an embodiment of the present invention, and this system enables users to efficiently find code fragments that suit their purpose, understand them visually, and obtain related information comprehensively.

[1428] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1429] Step 1:

[1430] Accepting user input

[1431] The user types into the terminal, "Can you give me the Python code to sort a list?"

[1432] Input: Natural language text input by the user

[1433] Output: User-entered text data

[1434] Specific operation: A user enters the required query into a text input field in a web browser or mobile app and clicks the search button. The device retrieves this input using JavaScript or similar tools and prepares it for further processing.

[1435] Step 2:

[1436] Terminal input transmission

[1437] The terminal receives input from the user and sends it to the server.

[1438] Input: User-entered text data

[1439] Output: Sending text data to the server

[1440] What it does: The device uses JavaScript or other front-end technologies to convert user input into JSON format and send it to the server via an AJAX request, while displaying a sending indicator to the user.

[1441] Step 3:

[1442] Query analysis by the server

[1443] The server parses the received user input.

[1444] Input: Received text data (query)

[1445] Output: Keyword and context extraction results

[1446] How it works: The server parses the incoming data and analyzes it using natural language processing techniques. It uses generative AI models such as BERT and GPT-4 to analyze the query and extract key keywords and intent. This process extracts keywords such as "list," "sort," and "Python."

[1447] Step 4:

[1448] Server lookup of code fragments

[1449] The server uses the analysis results to search for the code fragment.

[1450] Input: Keyword and context extraction results

[1451] Output: relevant code fragment

[1452] What happens: The server executes SQL queries to search its internal database, and optionally retrieves relevant code fragments from external code repositories (e.g., GitHub or GitLab) using API requests. As a result, the relevant code fragments are stored in the server's temporary memory.

[1453] Step 5:

[1454] Generating a visual representation

[1455] The server provides a visual representation of the retrieved code fragment.

[1456] Input: The searched code fragment

[1457] Output: Visual flowcharts and animations

[1458] How it works: The server analyzes the structure of a code fragment, calls the APIs of PlantUML and Draw.io, and generates flowcharts and animations. The generated visual data is saved as SVG or PNG format files.

[1459] Step 6:

[1460] Generate detailed documentation

[1461] The server generates detailed documentation related to the code fragment.

[1462] Input: The searched code fragment

[1463] Output: Detailed documentation (Markdown or LaTeX format)

[1464] How it works: The following prompt is input to the generative AI model to generate a document: "Please generate a document containing test cases, use cases, bug reports, and improvement suggestions related to the code fragment found in the search results. Please use Markdown format." The generated document is saved on the server in Markdown or LaTeX format.

[1465] Step 7:

[1466] Sending and viewing results

[1467] The server sends the visual representation and detailed documentation to the terminal, which displays the results to the user.

[1468] Input: Visual flowcharts, animations, detailed documentation

[1469] Output: The results displayed on the user's terminal

[1470] What it does: The server encodes the generated data in JSON format and sends it to the device as an HTTP response. The device receives it, renders the data using HTML and JavaScript, and displays it to the user. The user can then view the displayed code fragments, flowcharts, and detailed documentation and incorporate them into their own projects as needed.

[1471] The above are the specific processing steps of this system, which allow users to efficiently obtain the information they require.

[1472] (Application example 1)

[1473] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1474] While conventional code search systems analyze natural language and provide relevant code fragments, they lack sufficient visual understanding support. They also struggle to provide comprehensive, relevant detailed information. Furthermore, they have limitations in searching for code fragments specialized for specific applications and generating visual representations, making it difficult for developers to quickly and accurately obtain the information they need. For this reason, further improvements are needed in specialized fields such as electronic payments.

[1475] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1476] In this invention, the server includes means for analyzing a natural language query and searching for code fragments that match a purpose, means for visually representing the behavior of the searched code fragments, means for generating detailed documentation including test cases and use examples related to the code fragments, means for transmitting the search results, the visual representation, and the detailed documentation to a user's terminal, means for using a generative AI model to search for code fragments related to the query content and providing the code fragments together with their visual representations, and means for analyzing the user's intent using prompt sentences and generating related code fragments and documentation. This enables a user to quickly obtain specific code fragments related to electronic payment based on a natural language query, visually understand them, and comprehensively obtain detailed documentation.

[1477] A "natural language query" is a question or command entered by a user in human language.

[1478] A "code fragment" is a portion of program code that implements a specific function or operation.

[1479] "Visually representing" means displaying textual information in graphical form, such as a flow chart or animation.

[1480] A "test case" is a set of specific inputs and their expected outputs that are executed to verify the behavior of a program.

[1481] "Detailed documentation" refers to comprehensive information related to a piece of code, including explanations, use cases, bug reports, improvement suggestions, etc.

[1482] A "generative AI model" is an artificial intelligence program that uses machine learning techniques to analyze natural language and generate or find appropriate code fragments.

[1483] A "prompt sentence" is an input sentence for conveying the user's intention, and is a sentence used as a natural language query.

[1484] A "terminal" is a device that can be operated by a user, and specifically includes a personal computer, a smartphone, etc.

[1485] A "server" is a computer system that processes and stores data and provides information in response to requests from terminals.

[1486] "User" refers to a person who uses this system.

[1487] This invention provides a system that allows a user to input a query in natural language using a terminal (such as a smartphone or computer), search for code fragments that match the user's purpose, and present the results in a visually easy-to-understand format. This system is mainly composed of the following elements:

[1488] 1. Parsing natural language queries

[1489] The user uses the device to provide input in natural language, such as a prompt such as "What is the Python code to process a credit card payment?" The device then sends this input to the server, which uses a generative AI model to analyze the query and extract relevant keywords and intent. This generative AI model uses the Hugging Face transformers library.

[1490] 2. Searching for code fragments

[1491] Based on the extracted keywords and intent, the server searches for relevant code fragments from an internal database or external repository that contains code fragments written in various programming languages, including, for example, Python code for processing credit card payments.

[1492] 3. Generating Visual Representations

[1493] For each code fragment found, the server converts its behavior into a visually understandable representation, using Matplotlib and NetworkX to represent the code steps as flowcharts and animations. This visual representation helps users intuitively understand the code's flow and structure.

[1494] 4. Generate detailed documentation

[1495] The server generates detailed documentation such as test cases, use cases, bug reports, and improvement suggestions related to the retrieved code fragments, allowing users to gain a comprehensive understanding of the code's quality and coverage.

[1496] 5. Sending and displaying results

[1497] The server sends the search results, visual representations, and detailed documentation to the terminal, which then displays them to the user, who can quickly and efficiently obtain the information they need based on the displayed code fragments, flowcharts, and detailed documentation.

[1498] As a concrete example, consider the case where a user uses the following prompt text:

[1499] "Can you give me some Python code to process credit card payments?"

[1500] In response, the system searches for relevant code fragments, generates visual representations and detailed documentation, and provides them to users, allowing them to quickly extract the code they need and apply it to their own projects.

[1501] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1502] Step 1:

[1503] Accepting natural language inquiries

[1504] The user uses a terminal to input a query in natural language. At this stage, the user enters a prompt sentence such as "Please give me the Python code to process credit card payments." The input data is sent over the Internet to the server.

[1505] Input: Natural language query (prompt)

[1506] Output: The query sent to the server

[1507] Step 2:

[1508] Natural Language Analysis

[1509] The server analyzes the received query using a generative AI model (Hugging Face's transformers library). This model extracts keywords and user intent from the input natural language text. Specifically, keywords such as "credit card," "payment," and "Python" are extracted.

[1510] Input: Received natural language query

[1511] Output: Extracted keywords and user intent

[1512] Step 3:

[1513] Searching for code fragments

[1514] The server then searches for relevant code fragments from an internal database or external repository based on the extracted keywords and intent. This database contains code fragments written in various programming languages, and the search result is "Python code to process credit card payments."

[1515] Input: Extracted keywords and user intent

[1516] Output: relevant code fragment

[1517] Step 4:

[1518] Generating a visual representation

[1519] The server then converts the code fragments it finds into a visually understandable representation of their behavior, using Matplotlib and NetworkX to generate flowcharts and animations of the code steps, providing a visual representation of the code's flow and structure.

[1520] Input: The searched code fragment

[1521] Output: Generated flowchart or animation

[1522] Step 5:

[1523] Generate detailed documentation

[1524] The server generates detailed documentation containing test cases, use cases, bug reports, and improvement suggestions associated with the code fragment, providing data for a comprehensive understanding of the code's quality and coverage.

[1525] Input: The searched code fragment

[1526] Output: Detailed documentation (test cases, use cases, bug reports, improvement suggestions)

[1527] Step 6:

[1528] Sending and displaying results

[1529] The server sends the search results, visual representations, and detailed documentation to the terminal, which displays them to the user, allowing the user to quickly and efficiently obtain the information they need based on the displayed code fragments, flowcharts, and detailed documentation.

[1530] Input: Search results, visual representation, detailed documentation

[1531] Output: what is displayed on the user's device (code fragments, visual representations, detailed documentation)

[1532] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1533] The present invention incorporates an emotion engine that recognizes the user's emotions into a system that performs code searches in natural language, displays the results in a visually easy-to-understand format, and provides related information. Specific embodiments of the present invention are described below.

[1534] Overall system overview

[1535] This system consists of a user, a terminal, a server, and an emotion engine.

[1536] User operations

[1537] Users use the terminal to search for code related to a particular function or operation using natural language, for example by entering a query such as "What is the Python code to sort a list?" and clicking the search button.

[1538] Device Role

[1539] The device receives input from the user and sends it to the server, displays search results, visual representations, and detailed documents sent from the server, and collects emotional data from the user's facial expressions and voice and sends it to the server.

[1540] Server Roles

[1541] The server has the following main functions:

[1542] 1. Natural Language Query Parsing:

[1543] The server analyzes the natural language queries sent by users and extracts keywords and context using natural language processing techniques.

[1544] 2. Search for code fragments:

[1545] Based on the analysis results, the search engine searches for relevant code fragments in internal databases and external code repositories, and finds code fragments that match the keywords.

[1546] 3. Generate visual representation:

[1547] The behavior of the found code fragments is converted into a visually understandable form (flowchart or animation) by the visual representation generation component, which analyzes the code structure.

[1548] 4. Generate detailed documentation:

[1549] It collects test cases, use cases, bug reports and improvement suggestions related to a piece of code and generates a single comprehensive document.

[1550] 5. User sentiment analysis:

[1551] The emotion engine analyzes the user's emotional data sent from the device and adjusts search results and visual presentation accordingly, for example by adding more detailed explanations if the user is confused.

[1552] 6. Sending the results:

[1553] Finally, adjustment information that takes into account the user's emotion data is sent to the terminal together with the visual representation and detailed documentation.

[1554] Specific examples

[1555] Below is a concrete example of how the sentiment engine works in the process of a user searching for "Python code to sort a list."

[1556] 1. User input:

[1557] The user enters "Please tell me the Python code to sort the list" into the terminal and clicks the search button. The terminal simultaneously collects the user's facial expression and voice data.

[1558] 2. Sending by terminal:

[1559] The terminal transmits the query and the user's emotion data to the server.

[1560] 3. Query parsing and code lookup by the server:

[1561] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches the database for the corresponding code fragment.

[1562] 4. Generating a visual representation:

[1563] The server analyzes the steps in the code fragment it finds and generates a flowchart that shows the process of taking an input list, sorting the elements, and returning the result.

[1564] 5. Generate detailed documentation:

[1565] Generate documentation including relevant test cases (e.g., sorting a list of numbers or a list of strings), use cases (specific sorting methods using particular lists), bug reports (issues that have been encountered and how they have been resolved), and improvement suggestions (e.g., a more efficient sorting algorithm).

[1566] 6. User sentiment analysis:

[1567] The server analyzes the user's emotional data through an emotion engine and determines that the user is confused, and accordingly adjusts the generated documentation to include additional explanations and examples.

[1568] 7. Submitting and Viewing Results:

[1569] The server sends visual representations and detailed documentation to the user's device, reflecting the emotion-based adjustment information, and the device displays it, allowing the user to review code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[1570] The above is a specific embodiment of the present invention incorporating an emotion engine. This system allows users to efficiently find code fragments that suit their purpose, visually understand them, and comprehensively obtain related information, while also receiving appropriate support based on their emotions.

[1571] The processing flow will be explained below.

[1572] Step 1:

[1573] Using a terminal, a user enters a natural language query such as "Can you give me the Python code to sort a list?" and clicks the search button.

[1574] Step 2:

[1575] The device collects the user's facial expressions and voice data along with natural language queries from the user, and these data are simultaneously sent to the server as emotion data for analyzing the user's emotions.

[1576] Step 3:

[1577] The server analyzes the received natural language query, and the query analysis engine uses natural language processing techniques to extract key keywords such as "list," "sort," and "Python."

[1578] Step 4:

[1579] The server searches for relevant code fragments from an internal database or external code repository based on the analyzed keywords, and the search engine extracts matching code fragments in a list.

[1580] Step 5:

[1581] The server selects the most suitable code fragment from those found and then uses a visual representation generation component to visually represent the behavior of the code fragment, generating a flowchart or animation of the code fragment.

[1582] Step 6:

[1583] The server collects test cases and use cases related to a piece of code from a database, as well as bug reports and improvement suggestions, and generates comprehensive documentation out of this information. This process is handled by the documentation generation component.

[1584] Step 7:

[1585] The server runs an emotion engine to analyze the user's emotion data sent from the device. This engine analyzes how the user feels about the displayed information, and if a negative emotion such as confusion is detected, the engine includes a more detailed explanation or additional examples.

[1586] Step 8:

[1587] The server transmits the generated visual representation, detailed documentation, and information adjusted based on the emotion to the terminal.

[1588] Step 9:

[1589] The terminal displays the received information to the user, showing visual flowcharts, detailed code descriptions, documentation with associated test cases, use cases, bug reports, and improvement suggestions, allowing the user to understand the code fragment and efficiently incorporate it into their own projects.

[1590] This series of steps allows users to obtain code fragments and related information in a more relevant and understandable way, based on the additional information of emotions.

[1591] Example 2

[1592] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1593] Conventional program code search systems often produce insufficient search results even when users input queries in natural language. Furthermore, providing information through code fragments that are difficult to understand visually or incomplete documentation contributes to a lack of user convenience. Furthermore, providing search results without considering the user's emotional state can be confusing, especially for novice users. To address these issues, systems are required that can analyze queries in natural language, display information that is easy to understand visually, and provide information that takes the user's emotions into account.

[1594] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1595] In this invention, the server includes means for analyzing a natural language query and searching for program code that matches a purpose, means for visually representing the behavior of the searched program code, means for generating detailed documentation including test cases and use cases related to the program code, means for collecting and analyzing emotional data, and means for adjusting the visual representation and the content of the detailed documentation based on the emotional data. This allows the user to obtain search results that are more accurate and visually easy to understand, and also makes it possible to provide appropriate information according to the user's emotional state.

[1596] A "natural language query" refers to a query entered by a user using a human language (e.g., Japanese, English, etc.).

[1597] "Program Code" means source code that contains instructions and procedures for performing a particular task or function.

[1598] A "search tool" refers to a mechanism that analyzes natural language queries and uses them to find appropriate program code from databases or external repositories.

[1599] "Visual representation means" refers to a mechanism for displaying the behavior and structure of program code using visualization techniques such as flowcharts and animations.

[1600] "Detailed documentation" refers to comprehensive documentation containing test cases, use cases, bug reports, and improvement suggestions related to the program code.

[1601] "Emotional data" refers to data that indicates the user's emotional state, collected from the user's facial expressions, voice, etc.

[1602] "Emotion analysis means" refers to a mechanism for analyzing collected emotion data and determining the user's emotional state.

[1603] "Adjustment means" refers to a mechanism that appropriately changes the visual presentation and detailed document content based on the results of sentiment analysis to provide information in a form that is easier for users to understand.

[1604] "User terminal" refers to a device such as a computer, smartphone, or tablet that is actually operated by a user.

[1605] The present invention provides a system that allows a user to search for program code in natural language, displays the code in a visually easy-to-understand format, and provides information based on the user's emotions. Specific embodiments will be described below.

[1606] System configuration

[1607] This system consists of a user, a terminal, a server, and an emotion engine.

[1608] User operations

[1609] Users use the terminal to search for program code related to a specific function or operation in natural language, for example, by entering a query such as "What is the Python code to sort a list?" and clicking the search button.

[1610] Device Role

[1611] The device receives input queries from the user and sends them to the server. It also has the function of collecting emotional data from the user's facial expressions and voice and transmitting it to the server. For example, it acquires the user's emotional data using a camera or microphone. It also displays search results, visual representations, and detailed documents received from the server to the user.

[1612] Server Roles

[1613] The server has the following main functions:

[1614] 1. Natural Language Query Parsing:

[1615] The server analyzes the natural language query submitted by the user, for example by using natural language processing tools such as SpaCy or NLTK to extract keywords such as "list," "sort," and "Python."

[1616] 2. Search for the program code:

[1617] Based on the analysis results, the relevant program code is searched for in an internal database (e.g., PostgreSQL) or an external code repository (e.g., GitHub).

[1618] 3. Generate visual representation:

[1619] Convert the searched program code into a visually understandable format (flowchart or animation). For example, generate a flow chart using D3.js.

[1620] 4. Generate detailed documentation:

[1621] It collects test cases, use cases, bug reports, and improvement suggestions related to program code and generates detailed documentation in Markdown format.

[1622] 5. Emotion analysis:

[1623] The server's emotion engine analyzes the user's emotion data sent from the device, for example, using an emotion recognition API to determine whether the user is confused.

[1624] 6. Sending the results:

[1625] The server generates a visual representation and detailed documentation, and transmits the information adjusted based on the sentiment analysis results to the terminal.

[1626] Specific examples

[1627] Here is a specific example of what happens when a user searches for "Can you give me the Python code to sort a list?"

[1628] 1. User input:

[1629] The user enters "Please tell me the Python code to sort the list" into the terminal and clicks the search button. The terminal simultaneously collects the user's facial expression and voice data.

[1630] 2. Sending by terminal:

[1631] The terminal transmits the query and the user's emotion data to the server.

[1632] 3. Query parsing and code lookup by the server:

[1633] The server analyzes the query, extracts the keywords "list," "sort," and "Python," and searches the database for the corresponding code fragment.

[1634] 4. Generating a visual representation:

[1635] The server analyzes the steps in the code fragments it finds and generates a flowchart using D3.js.

[1636] 5. Generate detailed documentation:

[1637] The server creates a detailed document in Markdown containing relevant test cases, use cases, bug reports, and improvement suggestions.

[1638] 6. User sentiment analysis:

[1639] The server's emotion engine detects user confusion and adjusts the generated documentation to include additional explanations and examples.

[1640] 7. Submitting and Viewing Results:

[1641] The server sends the final visual representation and detailed documentation to the device, which displays it, allowing users to review code snippets, flowcharts, and related information and incorporate them into their own projects as needed.

[1642] Prompt Sentence Examples

[1643] Here are some example prompts for searches using this system:

[1644] "Enter 'Please tell me the Python code to sort a list' in the terminal and search."

[1645] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1646] Step 1:

[1647] The user enters a natural language query into the text input field of the terminal and clicks the search button. For example, they might type "Can you give me the Python code to sort a list?" This is the input data. The output they get from the terminal is the query.

[1648] Step 2:

[1649] The device sends queries entered by the user to the server. In addition, the device's built-in camera and microphone capture the user's facial expressions and voice data, which are also sent to the server. The input is the user's query and emotion data, and the output is an HTTP request to the server. Specifically, the device converts the query and emotion data into JSON format and sends it.

[1650] Step 3:

[1651] The server analyzes the received natural language query using a natural language processing tool (e.g., SpaCy or NLTK) to extract keywords and context. The input is the query sent from the terminal, and the output is the extracted keywords (e.g., "list," "sort," "Python"). Specifically, the server calls a text analysis engine to analyze the query.

[1652] Step 4:

[1653] The server searches for relevant program code in a database (e.g., PostgreSQL) or an external repository (e.g., GitHub) based on the extracted keywords. The input is the parsed keywords, and the output is the search results, which are program code fragments. Specifically, it executes SQL queries or API requests to retrieve data.

[1654] Step 5:

[1655] The server analyzes the searched program code and generates visual representations such as flowcharts and animations. This is done using D3.js or other visualization tools. The input is the program code fragments as search results, and the output is the visual representation. Specifically, each step of the program code is input into the visualization tool to generate a flowchart.

[1656] Step 6:

[1657] The server collects test cases, use cases, bug reports, and improvement suggestions related to the searched program code and generates detailed documentation. The documentation format is Markdown. The input is a fragment of program code, and the output is a detailed documentation. Specifically, it collects and integrates related information and generates a document in Markdown format.

[1658] Step 7:

[1659] The emotion engine on the server analyzes the user's emotion data collected from the device and determines the user's emotional state (e.g., confusion, satisfaction, etc.). The input is the emotion data sent from the device, and the output is the user's emotional state as a result of the analysis. Specifically, it calls the emotion recognition API and analyzes the data.

[1660] Step 8:

[1661] The server adjusts the content of the visual representation and detailed text based on the user's emotional state. For example, if the user is confused, it adds more detailed explanations and examples. The input is the user's emotional state, the visual representation, and the detailed text, and the output is the adjusted visual representation and detailed text. Specific operations include updating the text and visualization data.

[1662] Step 9:

[1663] The server sends the adjusted visual representation and detailed document to the terminal. The input is the adjusted visual representation and detailed document, and the output is an HTTP response to the terminal. As a specific operation, the server generates an HTTP response for sending data to the terminal.

[1664] Step 10:

[1665] The terminal displays the visual representation and detailed document received from the server to the user. The input is data from the server, and the output is the display to the user. Specifically, the terminal uses a browser or a dedicated application to render and display the visual representation and document.

[1666] (Application example 2)

[1667] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1668] In modern factories and production lines, robot programming and troubleshooting are extremely important, but they require specialized knowledge and experience. Furthermore, conventional code search systems and support tools provide only one-way information without considering the user's feelings. This often leaves users confused and reduces work efficiency. Another issue is the lack of visual support, which makes it time-consuming to understand and operate the code. A system that can solve these problems and provide more intuitive and efficient support is needed.

[1669] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing a natural language query and searching for code fragments that match the purpose, means for visually expressing the behavior of the searched code fragments, and means for analyzing user emotion data and adjusting the display method of the search results and visual expression based on the results. This makes it possible to provide optimal support that is easy for the user to intuitively understand and that is tailored to their emotions.

[1670] A "natural language query" is a question or instruction written in a language normally used by humans, with the goal of converting it into a form that the system can understand and process.

[1671] A "code fragment" is a part of a program that realizes a specific function or operation, and refers to a short piece of code that is used in actual programming work.

[1672] A "visual representation" is a visually easy-to-understand representation of the behavior of the retrieved code fragment, which can take the form of a flowchart or animation.

[1673] "Detailed documentation" refers to materials that contain supplementary and comprehensive information, such as test cases, use cases, bug reports, and improvement suggestions related to a piece of code.

[1674] "Emotional data" refers to emotional information extracted from a user's facial expressions and voice, and is used by the system to analyze and understand the user's emotional state.

[1675] "Adjusting the display method" refers to changing the format and content of search results and visual representations based on the user's emotional data to make them easier for the user to understand.

[1676] This system analyzes natural language queries, searches for code fragments that match the query, and displays the code's behavior in a visually understandable format. It also analyzes user sentiment and adjusts the display of search results and visual representations based on the results.

[1677] System configuration

[1678] This system consists of a user, a terminal, a server, and an emotion engine.

[1679] 1. Users

[1680] Users use devices such as smartphones or head-mounted displays to search for code fragments in natural language, for example by entering queries such as "What code can I use to change the speed of a robot?"

[1681] 2. Terminal

[1682] The device receives input from the user and sends it to the server. It also collects emotional data from the user's facial expressions and voice and sends this data to the server.

[1683] 3. Server

[1684] The server has the following functions:

[1685] Natural language query analysis: The server analyzes the natural language query sent by the user and extracts keywords and context using natural language processing techniques.

[1686] Searching for code fragments: Based on the analysis results, the database is searched for the relevant code fragments.

[1687] Visual representation generation: Convert the behavior of the retrieved code fragments into a visually understandable form (flowchart or animation).

[1688] Generate detailed documentation: Collect test cases, use cases, bug reports and improvement suggestions related to a piece of code and generate one comprehensive document.

[1689] User sentiment analysis: Analyzes user sentiment data sent from the device through the emotion engine and adjusts the way search results and visual representations are displayed based on the results.

[1690] 4. Emotion Engine

[1691] The emotion engine analyzes the user's facial expressions and voice data to determine the user's emotional state. This analysis is performed using, for example, the "Transformers" pipeline.

[1692] Processing flow

[1693] As a concrete example, a user enters the query "Please tell me the code to change the robot's movement speed," and in the process explains how the emotion engine works.

[1694] 1. Query input: The user inputs a query into the device, and the device simultaneously collects the user's facial expressions and voice.

[1695] 2. Data transmission: The device sends the query and emotion data to the server.

[1696] 3. Query analysis and code search: The server analyzes the query, extracts the keywords "operation speed," "change," "code," and "robot," and searches the database for the corresponding code fragment.

[1697] 4. Visual representation generation: The server analyzes the procedures in the code fragments it finds and generates a flowchart that shows the sequence of processes that change the speed of the execution.

[1698] 5. Detailed documentation generation: Generate documentation including code examples, test cases, improvement suggestions, etc.

[1699] 6. Sentiment Analysis: The server analyzes the user's emotional data through an emotion engine and determines that the user is confused. Based on this, it adjusts the generated documentation and visual representations to include additional explanations and examples.

[1700] 7. Result transmission and display: The server transmits the tailored visual representation and detailed documentation to the user's device, which displays them, allowing the user to review and incorporate code snippets, flowcharts, and related information into their own projects.

[1701] Prompt Sentence Examples

[1702] "Please provide code snippets for the following natural language queries, and add detailed documentation and visual representations if the user's sentiment is confused: Query: 'What code can I use to change the speed of the robot?'"

[1703] The above is a concrete example of how to implement the invention. This system allows users to efficiently find code fragments that suit their purpose, visually understand them, and comprehensively obtain related information, while also receiving appropriate support based on their emotions.

[1704] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1705] Step 1:

[1706] The user inputs a query in natural language. For example, "Please tell me the code to change the robot's movement speed." At this time, the device simultaneously collects the user's facial expression and voice data. The inputs in this step are the natural language query, facial expression data, and voice data, and the output is a data package of these.

[1707] Step 2:

[1708] The device sends the query and collected emotion data to the server. The input of this step is the data package collected in step 1, and the output is the data transmission to the server.

[1709] Step 3:

[1710] The server analyzes the received natural language query and extracts keywords and context. Here, natural language processing technology is used to extract important keywords such as "robot," "movement speed," and "change." The input is the user's query, and the output is a list of keywords as the analysis result.

[1711] Step 4:

[1712] The server searches the database for relevant code fragments based on the keywords extracted in step 3. For example, it finds "code to change the robot's movement speed." The input of this step is a list of keywords, and the output is the searched code fragments.

[1713] Step 5:

[1714] The server analyzes the behavior of the retrieved code fragments and generates visually understandable flowcharts and animations. Specifically, it processes the data to visually display the flow of each statement and loop in the code. The input for this step is the code fragment, and the output is a flowchart or animation.

[1715] Step 6:

[1716] The server generates detailed documentation for a code fragment, including test cases, use cases, bug reports, and improvement suggestions. The documentation generation process involves collecting relevant information and organizing it in a systematic way. The input of this step is the code fragment, and the output is detailed documentation.

[1717] Step 7:

[1718] The server analyzes the emotion data received in step 2 and determines the user's emotional state. For example, it uses an emotion engine to detect a "confused" state. The input of this step is emotion data, and the output is the emotion analysis result.

[1719] Step 8:

[1720] The server adjusts the search results, visual representation, and detailed document content based on the sentiment analysis results obtained in step 7. For example, if the user is confused, the server may include more detailed explanations or additional examples in the document. The inputs to this step are the sentiment analysis results, visual representation, and detailed document, and the output is the final adjusted visual representation and detailed document.

[1721] Step 9:

[1722] The server finally sends the adjusted visual representation and detailed document to the terminal. The input of this step is the adjusted visual representation and detailed document, and the output is data transmission to the terminal.

[1723] Step 10:

[1724] The terminal displays the visual representation and detailed documentation sent from the server to the user, allowing the user to identify the necessary code fragments and integrate them into their project. The input of this step is the data sent from the server, and the output is the display to the user.

[1725] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1726] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1727] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1728] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1729] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1730] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1731] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1732] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1733] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1734] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1735] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1736] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1737] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1738] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1739] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1740] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1741] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1742] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1743] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1744] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1745] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1746] The following is further disclosed regarding the above embodiment.

[1747] (Claim 1)

[1748] A means for analyzing natural language queries and retrieving matching code fragments;

[1749] a means for visually representing the behavior of the retrieved code fragment;

[1750] A means to generate detailed documentation including test cases and use cases related to a piece of code;

[1751] The system includes means for transmitting search results, visual representations, and detailed documents to a user's terminal.

[1752] (Claim 2)

[1753] 2. The system according to claim 1, wherein the means for visually representing the behavior of a code fragment is a means for analyzing the structure of the code and generating a flowchart or animation.

[1754] (Claim 3)

[1755] 2. The system of claim 1, wherein the means for generating detailed documentation including test cases and use cases related to a piece of code also integrates related bug reports and improvement suggestions to generate the documentation.

[1756] "Example 1"

[1757] (Claim 1)

[1758] A means for analyzing natural language queries and retrieving matching code fragments;

[1759] a means for visually representing the behavior of the retrieved code fragment;

[1760] A means to generate detailed documentation including test cases and use cases related to a piece of code;

[1761] means for transmitting the search results, the visual representation, and the detailed document to the user's terminal;

[1762] means for analyzing a user's query using natural language processing techniques;

[1763] a means of retrieving code fragments from an internal database or an external repository;

[1764] A means of converting the obtained code steps into a flowchart or visual diagram;

[1765] a means for generating detailed documentation using a generative AI model;

[1766] The system includes means for transmitting this information to a terminal and displaying it to the user.

[1767] (Claim 2)

[1768] 2. The system according to claim 1, wherein the means for visually representing the behavior of a code fragment is a means for analyzing the structure of the code and generating a flowchart or animation.

[1769] (Claim 3)

[1770] 2. The system of claim 1, wherein the means for generating detailed documentation including test cases and use cases related to a piece of code also integrates related bug reports and improvement suggestions to generate the documentation.

[1771] "Application Example 1"

[1772] (Claim 1)

[1773] A means for analyzing natural language queries and retrieving matching code fragments;

[1774] a means for visually representing the behavior of the retrieved code fragment;

[1775] A means to generate detailed documentation including test cases and use cases related to a piece of code;

[1776] means for transmitting the search results, the visual representation, and the detailed document to the user's terminal;

[1777] a means for using a generative AI model to find code fragments relevant to the query and providing the code fragments along with a visual representation;

[1778] A means of analyzing user intent using prompt statements and generating relevant code fragments and documentation

[1779] A system including:

[1780] (Claim 2)

[1781] 2. The system according to claim 1, wherein the means for visually representing the behavior of a code fragment is a means for analyzing the structure of the code and generating a flowchart or animation.

[1782] (Claim 3)

[1783] 2. The system of claim 1, wherein the means for generating detailed documentation including test cases and use cases related to a piece of code also integrates related bug reports and improvement suggestions to generate the documentation.

[1784] "Example 2: Combining Emotion Engines"

[1785] (Claim 1)

[1786] a means for analyzing natural language queries and retrieving program code that matches the query;

[1787] a means for visually representing the behavior of the retrieved program code;

[1788] A means for generating detailed documentation including test cases and use cases associated with the program code;

[1789] means for transmitting the search results, the visual representation, and the detailed document to a user terminal;

[1790] A means for collecting user emotion data from the terminal and analyzing the emotion;

[1791] The system includes means for adjusting the visual presentation and detailed document content based on the results of sentiment analysis.

[1792] (Claim 2)

[1793] 2. The system according to claim 1, wherein the means for visually representing the operation of the program code is a means for analyzing the structure of the code and generating a flowchart or animation.

[1794] (Claim 3)

[1795] 2. The system of claim 1, wherein the means for generating detailed documentation including test cases and use cases related to the program code also integrates related bug reports and improvement suggestions to generate the documentation.

[1796] "Application example 2 when combining emotion engines"

[1797] (Claim 1)

[1798] A means for analyzing natural language queries and retrieving matching code fragments;

[1799] a means for visually representing the behavior of the retrieved code fragment;

[1800] A means to generate detailed documentation including test cases and use cases related to a piece of code;

[1801] means for transmitting the search results, the visual representation, and the detailed document to the user's terminal;

[1802] A system that includes means for analyzing a user's emotional data and adjusting the way search results and visual representations are displayed based on the results.

[1803] (Claim 2)

[1804] The means for visually representing the behavior of a piece of code is a means for analyzing the structure of the code and generating a flowchart or animation,

[1805] 10. The system of claim 1, further comprising means for adjusting the generated visual representation in real time based on a user's emotion.

[1806] (Claim 3)

[1807] A means to generate detailed documentation including test cases and use cases related to a piece of code, integrating related bug reports and improvement suggestions,

[1808] The system according to claim 1, further comprising means for adjusting the content and presentation method of the document based on the analysis results of the emotion data. [Explanation of symbols]

[1809] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing natural language queries and retrieving matching code fragments; a means for visually representing the behavior of the retrieved code fragment; A means to generate detailed documentation including test cases and use cases related to a piece of code; The system includes means for transmitting search results, visual representations, and detailed documents to a user's terminal.

2. 2. The system according to claim 1, wherein the means for visually representing the operation of a code fragment is means for analyzing the structure of the code and generating a flowchart or animation.

3. 2. The system of claim 1, wherein the means for generating detailed documentation including test cases and use cases related to a piece of code also integrates related bug reports and improvement suggestions to generate the documentation.

Images