Information processing system

Abstract

The invention provides an information processing system. The information processing system comprises a device for automatically scanning all software codes and identifying the effect and relevance of each code file; the device is used for extracting the overall design concept of the system based on the analyzed code metadata; the device is used for generating a teaching reduced version code convenient to understand on the basis of keeping a project core function; the device is used for generating corresponding answers for questions proposed by a user based on the overall design concept of a project.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech. Summary of the Invention

[0003] To address the aforementioned problems, this invention provides an information processing system, comprising: a device for automatically scanning all software code and identifying the function and relationships of each code file; a device for extracting the overall system design concept based on the parsed code metadata; a device for generating easily understandable, abridged teaching code while maintaining the core functionality of the project; and a device for generating corresponding answers to user questions based on the overall project design concept. Furthermore, the system can provide interactive tutorials for new members based on the analyzed and extracted information and continuously collect user feedback to improve system functionality. Through these methods, reliance on specific technical personnel can be effectively reduced, enabling automatic transfer and iterative updates of project knowledge, thereby efficiently supporting project development and maintenance.

[0004] "Automatic scanning" refers to the process by which the system can autonomously detect and read all software code files in a project without human intervention.

[0005] "The role of code files" refers to the specific functional positioning and implementation purpose of each code file in the entire system.

[0006] "Code file association" refers to the interconnection and collaboration between different code files through relationships such as calls and dependencies.

[0007] "Code metadata" refers to auxiliary data extracted from source code that describes the characteristics of the code, such as its structure, function, and dependencies.

[0008] "System overall design concept" refers to the global design ideas such as system module division, functional division, and architectural patterns obtained through code and structural analysis.

[0009] "Reduced code for teaching" refers to new code that simplifies and refines the original code while retaining the core functionality of the project, and is then used for learning and training.

[0010] "User-submitted questions" refers to various questions that users have entered regarding the project's functions, structure, and implementation details, and that they hope to receive answers to.

[0011] "Answer" refers to the explanation, clarification, or suggestion automatically generated by the system based on its internal knowledge base for the questions raised by users.

[0012] "Interactive tutorials" refer to teaching programs that provide step-by-step guidance to users on learning project knowledge and operational procedures based on dynamic feedback from user operations.

[0013] "User feedback" refers to the evaluations, suggestions, and operational results submitted by users during the actual learning or use of the system. Attached Figure Description

[0014] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0015] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0016] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0017] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0018] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0019] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0020] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0021] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0022] Figure 9 This represents an emotion map that maps multiple emotions.

[0023] Figure 10 This represents an emotion map that maps multiple emotions.

[0024] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of Embodiment 1.

[0025] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0026] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system in Embodiment 2.

[0027] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0028] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0029] First, let me explain the terminology used in the following instructions.

[0030] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0031] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0032] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0033] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0034] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" is used to connect and express more than three items, the same interpretation as "A and / or B" applies.

[0035] First Implementation Method Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0036] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0037] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0038] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0039] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0040] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0041] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0042] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0043] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0044] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0045] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0046] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0047] Example 1 The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0048] In existing software development projects, the overall architecture and design concepts are difficult for new members to quickly understand, and there is a high dependence on specific technical personnel or service providers, resulting in slow project progress and inefficient training of new members. Furthermore, the lack of structured and intelligent methods for solving project problems and transferring knowledge hinders automated teaching and knowledge delivery, impacting team collaboration and improving project quality.

[0049] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0050] In this invention, the server includes means for automatically parsing a dataset containing program information and determining its functional and relational information; means for generating extracted design principle information based on the functional and relational information; means for automatically generating a concise dataset for learning support that retains core functions; means for parsing user queries in natural language and generating answers based on the design principle and functional information using a knowledge generation device; and means for outputting actionable interactive teaching information using the concise dataset for learning support. This helps new members quickly grasp the overall project architecture and core ideas, improves project progress and training efficiency, achieves automated and intelligent knowledge transfer and interactive teaching, and promotes project team collaboration and project quality improvement.

[0051] "Information processing device" refers to a general-purpose or special-purpose hardware system capable of analyzing, processing, storing, and outputting input data, including but not limited to servers, computers, and their network equipment.

[0052] A "data set" refers to an ordered or unordered group of data consisting of multiple data elements, including program information, code files, metadata, and other related information.

[0053] "Functional information" refers to information used to describe the specific operations and uses performed by each unit or module in a dataset.

[0054] "Relationship information" refers to information that reflects the relationships and dependencies between different parts, modules, or other systems in a data set.

[0055] "Design principle information" refers to information obtained by extracting and integrating functional and related information, which reflects the overall structure, construction ideas, architectural patterns, and other guiding content of the system.

[0056] "Simplified datasets for learning support" refers to a group of data or code that has been simplified and optimized to facilitate learning and teaching while retaining the core functions of the original system.

[0057] "User queries in natural language form" refers to questions, needs, or instructions raised by users in a free-expression manner, using spoken or written natural language.

[0058] "Automatic knowledge generation device" refers to a software or hardware module that uses artificial intelligence technology to automatically generate answers, suggestions or other knowledge content based on input data.

[0059] "Interactive teaching information" refers to information used to communicate with users during the learning process, including operation guidance, feedback, demonstrations, and other content.

[0060] "Operation log information" refers to information that automatically records the user's various operations, steps, parameters, and results in the system.

[0061] "Feedback information" refers to the evaluations, suggestions, or opinions given by users based on their user experience regarding the system's functions, content, interface, etc.

[0062] "Information processing function" refers to the system's comprehensive ability to analyze, transform, generate, store, and output received data.

[0063] "Knowledge generation algorithm" refers to the software algorithms and processing rules used within a system to analyze input content, produce knowledge information, or generate answers.

[0064] The present invention can be implemented in the following specific ways.

[0065] This system includes an information processing device (server), a terminal providing an interactive interface, and users operating through the terminal. The server can utilize general-purpose server hardware, such as server equipment equipped with a multi-core CPU, high-speed memory, and large-capacity storage, and run a modern operating system, such as Linux. Code analysis and data processing can leverage existing software tools and platforms, such as static code analysis tools (e.g., SonarQube, CodeQL), structured database systems (e.g., PostgreSQL), and generative artificial intelligence models (e.g., GPT-4 API, Transformer-based native AI platforms, etc.). The terminal can be a personal computer, tablet, or smartphone, running software that supports modern front-end technologies (e.g., React, Vue.js, Android / iOS Apps).

[0066] The server automatically collects program data sets stored in the code repository and performs automated static parsing on all code files. The server can identify the functional and relational information of each file and store the results in a structured format. For example, the server can identify that a file defines a module for database access and has a dependency on the control layer.

[0067] Based on the parsed functional and relational information, the server further analyzes the design principles of the entire system. For example, the server can detect that the project uses the Model-View-Controller (MVC) design pattern and extract it as an important feature of the system structure. Using this information, the server automatically generates a simplified dataset to support learning, retaining the core functionality of the project, removing redundant elements, and optimizing it into a program example suitable for teaching and experimentation. The simplified code allows new members to quickly understand the overall structure and operation mechanism of the project.

[0068] The server also integrates a generative artificial intelligence model. Users can ask questions about the project by inputting prompts through their terminals. The server utilizes the acquired design principle and functional information, along with the generative AI model, to automatically generate natural language answers containing detailed code snippets, functional explanations, and implementation principles. These answers are returned to the user's terminal in real time, allowing new members to ask questions and receive answers instantly during the learning process.

[0069] The terminal is responsible for displaying the streamlined dataset of learning support pushed by the server and providing a visual and interactive teaching interface. Users can gradually experience the operation process of different functional modules on the terminal, such as manually entering database configuration information, trying CRUD operations, and viewing system feedback, thus deepening their understanding through practical operation. The terminal is also responsible for collecting user operation logs and feedback information and sending them back to the server to continuously optimize teaching content and system functions.

[0070] Users submit prompts through input fields on the terminal. Common prompts include: How is a database connection implemented? "Please explain the architectural design philosophy of this project and provide an example to illustrate the separation of the control layer and the data layer." How do I add a new API interface to an existing project? Please provide a code example. Where is the Model-View-Controller (MVC) pattern actually applied in this project? Please explain with code analysis. In summary, this invention automatically generates knowledge content through static analysis, knowledge extraction, code simplification, and generative artificial intelligence models. Combined with an interactive terminal teaching interface, it provides new members with efficient, automated, and intelligent learning and project adaptation support, greatly reducing the team's dependence on specific personnel and improving training efficiency and the overall project progress speed.

[0071] use Figure 11 The processing procedure is explained.

[0072] Step 1: The server automatically scans all program files in the project's codebase as input, uses static code analysis tools (such as SonarQube and CodeQL) to parse the content, structure, and inter-file relationships of each file, extracts the functional information and dependencies of each file, and generates a structured metadata report as output. For example, the server identifies the call patterns between the "database module" and the "business logic module" and stores the results as a JSON-formatted data document.

[0073] Step 2: The server takes the metadata report obtained in step 1 as input, uses a pattern recognition algorithm to analyze the functional distribution and module collaboration relationships, and automatically summarizes the overall system architecture and design principles. The server determines whether the project adopts mainstream design patterns such as MVC, and outputs the identified architectural features, data flow, and control flow paths as annotations to form a design principle document. For example, the server outputs "This system adopts a model-view-controller structure, and the control logic is concentrated in the controller module."

[0074] Step 3: The server takes the design principle document and metadata as input, filters out the most core functionalities of the project, and automatically generates a simplified learning program code using code extraction and trimming tools. The server removes auxiliary and redundant components, retaining only the core business processes and necessary interfaces, and outputs a streamlined code set suitable for teaching and demonstration. For example, the server generates a simple version of the program containing only key logic such as login and query, and saves it as a new project folder.

[0075] Step 4: The terminal takes a set of simplified code output from the server as input, loads this code on the front-end platform, and renders it into a modular, operable teaching interface. Users can practice operating the simplified program through the terminal interface. The terminal displays the operation results, feedback, and step prompts to the user in real time, providing a visual learning experience and data demonstrations. For example, the terminal displays a "database connection test" interface to the user. After the user inputs parameters, the system returns the connection result and log information.

[0076] Step 5: Users submit natural language prompts as input through the terminal's input fields, asking questions about the project. The terminal forwards the user input to the server, which reads design principle information and functional details from the project database. Based on this, the server calls a generative artificial intelligence model to generate a detailed answer to the question and returns it as output. For example, if a user inputs "How is the database connection implemented?", the server analyzes the code implementation details, the AI ​​model outputs a complete implementation description and relevant code snippets, and presents them to the user through the terminal.

[0077] Step 6: The terminal collects user logs and feedback during the learning and operation process as input, periodically summarizing this data and sending it back to the server. The server analyzes the collected user data and automatically optimizes teaching content, streamlines project procedures, and improves knowledge generation algorithms based on the analysis results, outputting optimized teaching resources and system parameter settings. For example, if the server analyzes that students make the most mistakes in the "data retrieval" step, it will optimize the teaching guidance and example code for that step.

[0078] Application Example 1 The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0079] Existing technologies for automatically parsing, understanding the structure, and transferring knowledge of software code in complex systems suffer from low efficiency, reliance on manual intervention, and high learning barriers. When learning new systems or modules, technical personnel often struggle to quickly grasp the full structure and functional relationships of the code, lacking efficient personalized training support. Furthermore, traditional training systems cannot dynamically adjust teaching content and pace based on users' emotional states, hindering the improvement of learning experience and mastery efficiency. In addition, these systems generally lack intelligent self-optimization capabilities based on user feedback and emotional information, failing to continuously adapt to the learning needs and operating styles of different users.

[0080] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0081] In this invention, the server includes: a device for automatically scanning program information in an electrical storage medium and identifying the functions and dependencies of each program; a device for extracting structural features and design concepts based on their attribute information; a device for automatically generating simplified learning aid code using machine learning methods; a device for analyzing user questions and outputting targeted structured solutions through a generative artificial intelligence model; an emotion inference device for analyzing user operations and identifying emotional states; and a device for dynamically adjusting teaching content and methods based on emotion results, and continuously accumulating user feedback and optimizing system functions. This enables automatic understanding and structural abstraction of complex code systems, generates personalized and emotionally adaptive interactive teaching content for users in real time, and achieves continuous system evolution and optimization through feedback-driven processes, fundamentally improving the learning efficiency and code management level of technical personnel.

[0082] "Program information" refers to a general representation stored in an electrical storage medium, including sets of instructions and code segments used to implement specific information processing functions.

[0083] "Attribute information" refers to data content about the characteristics and structure of the program information itself and its relationship with other programs, including functional descriptions, dependencies, etc.

[0084] "Function" refers to a specific operation, process, or service that a system or program module can perform.

[0085] "Dependency relationship" refers to the relationship between different program information that is related to or calls each other during the function implementation process.

[0086] "Structural features" refer to the abstract structure manifested by the organization, hierarchy, or connection relationships among multiple program units or modules.

[0087] "Design philosophy" refers to the core ideas or principles that support the overall architecture and functional allocation of an information processing system.

[0088] "Simplified program information for learning assistance" refers to code or program fragments that are simplified and refined after retaining the main functions in order to facilitate user understanding and learning.

[0089] "Generative artificial intelligence models" refer to artificial intelligence algorithm systems that can automatically generate text or code based on input information, including large language models.

[0090] "Output information" refers to the answers, annotations, or related support information generated and fed back by the system based on user input, structural features, and design concepts.

[0091] "Emotional information" refers to data inferred from user actions or feedback, indicating the user's emotions, feelings, or psychological state.

[0092] "Emotion estimation device" refers to a combination of hardware and software modules that can analyze user input and output emotional information.

[0093] "User feedback information" refers to data content that reflects the user's true experience, such as subjective evaluations, suggestions, and satisfaction during the system interaction process.

[0094] "Emotional history information" refers to the time-series data recorded by the system for the emotional state and feedback of the same user in multiple interactions.

[0095] "Continuous optimization" refers to the system automatically adjusting and intelligently evolving its functions, content, and presentation methods based on accumulated information and feedback to meet user needs.

[0096] "Remote information terminal" refers to various devices that interact with servers through electronic communication networks to display and input information, such as smart glasses, mobile terminals, or computers.

[0097] In the embodiments of the present invention, an information processing device (i.e., a server) is first used as the core component of the system, and it works in conjunction with remote information terminals (such as smart glasses, mobile terminals, computers, etc.) and users to achieve automatic parsing of the structure of complex code systems, knowledge extraction and learning assistance.

[0098] The server can be configured to include a high-performance general-purpose computer, data storage devices, and a software environment with static analysis tools (such as Python-based AST parsers and NetworkX libraries), database systems (such as relational or document databases), generative artificial intelligence models (such as large language model APIs), and sentiment recognition models (such as NLP sentiment analysis algorithms such as BERT and RoBERTa).

[0099] The server automatically scans all program information stored on electrical storage media (such as hard drives, SSDs, etc.), identifying the functional modules of each program and their dependencies, and further extracting the attribute information of the code. Using open-source static analysis tools such as Python AST and Ctags, the server automatically parses the code structure, function definitions, call relationships, and comments, and visualizes and structures the module relationships using libraries such as NetworkX.

[0100] The server is also equipped with a generative artificial intelligence model. This model can summarize, generalize, and transform extracted attribute information and design concepts based on specified prompts, generating concise teaching code suitable for new employees or technical staff. The teaching code includes core functionality implementations and rich comments, facilitating users' rapid understanding of the system structure and key logic.

[0101] Meanwhile, the server analyzes user input or queries in real time through a natural language processing module. When necessary, it invokes an emotion recognition model to determine the user's emotions and dynamically adjusts the level of detail and presentation of the pushed learning content. For example, when it detects user confusion or doubt, the server automatically pushes more detailed, step-by-step explanations.

[0102] The remote information terminal primarily serves as the interactive interface, responsible for receiving learning content or outputting information from the server and presenting it to the user in a phased, interactive manner. Users can operate the learning modules through selection, voice, gestures, etc., such as adjusting the explanation speed, requesting more cases, and entering specific questions. The terminal can dynamically switch between different display styles, such as "detailed explanation" and "simplified mode," according to the server's suggestions.

[0103] Users, as the main interactors in the system, have the right to proactively ask questions, operate the teaching process, and provide feedback on their learning experience, subjective feelings, or suggestions through their terminals. All user behavior information, ratings, and emotional feedback are automatically uploaded to the server for subsequent content optimization and system upgrades. The server regularly collects and analyzes all user operation logs, historical emotional sequences, and feedback data to automatically adjust teaching content generation strategies and optimize the order of knowledge delivery, enabling the system to self-evolve.

[0104] Specific examples include: When new technicians first encounter a factory robot control system and need to understand its start-up and stop functions, they simply request the relevant learning module through their terminal. The server automatically extracts key structures and implementation methods from the project code based on the "robot start-up-stop" design concept, and uses an AI model to generate simplified example code with step-by-step explanations and contextual annotations. This code, combined with a graphical representation of module dependencies, is then pushed to the user's terminal. If the user expresses confusion or submits specific questions, the system dynamically adds detailed explanations and allows the user to repeatedly operate and test the code snippets.

[0105] Generative artificial intelligence models can use the following example of prompt statements: "Please analyze the following code, briefly summarize the core control logic, and generate an interactive and simplified tutorial suitable for new employees to practice." "New user confusion with sensor interface detected, generating tutorial with detailed pseudocode and illustrations." "Please provide pseudocode and comments for the step-by-step explanation of the 'robot start-stop' process." The practice of this invention shows that, through the above-mentioned hardware and software configuration and processing flow, the code parsing efficiency of complex software projects, the intelligent adaptability of the learning experience, and the self-optimization capability of the overall system can be effectively improved, providing solid support for application scenarios such as industrial automation and intelligent manufacturing.

[0106] use Figure 12 The processing procedure is explained.

[0107] Step 1: The server automatically scans all program information in the electrical storage medium. The input is a collection of stored raw code files, and the output is preliminary parsed data containing functional descriptions of each program module, call relationships, file structure, etc. Specifically, the server uses tools such as Python AST and Ctags to traverse the project directory, parse the code, extract functions, classes, comments, and module dependencies, and then stores this information in a structured database.

[0108] Step 2: Based on the parsed data obtained in the first step, the server extracts attribute information, constructs a dependency graph between program modules, and further abstracts the system design philosophy and structural characteristics. The input is the structured metadata generated in step 1, and the output is an abstract description containing the design philosophy, module division, main functions, and dependency structure. Specific actions include: the server calling libraries such as NetworkX to model function and module dependencies, automatically analyzing the functional roles of each module, and forming metadata records for the overall system architecture.

[0109] Step 3: The server integrates design concepts and attribute information, invokes a generative artificial intelligence model, and automatically generates concise code and step-by-step tutorials suitable for new users based on specified prompts. The input is the design concepts and metadata generated in step 2, and the output is a teaching package containing simplified example code, detailed comments, and operation instructions. Specific actions include: the server sending the code structure and generation request to the large language model API, processing the returned content, and generating teaching examples that users can directly run and operate.

[0110] Step 4: The server detects and analyzes user input and query information on the terminal, and calls a sentiment recognition model to determine the user's emotional state. Input includes questions submitted by the user through the terminal, interaction logs, or voice recordings; output is categorized sentiment tags (such as confusion, satisfaction, anxiety, etc.). Specific actions include: the server performing NLP sentiment analysis on the input text or voice content, recording the user's state for each interaction, and automatically adjusting subsequent push content.

[0111] Step 5: The terminal receives instructional content and emotionally adaptive feedback generated by the server, presents it to the user in stages, and records user actions and feedback. Input consists of the instructional package and specific prompts pushed by the server; output includes the user's interaction log, feedback results, and operation commands. Specific actions include: displaying step-by-step instructional content on the terminal, allowing users to select learning modes and adjust the pace of instruction, and responding in real-time to server suggestions for adjusting content display.

[0112] Step 6: Users interact with the system through the terminal, completing exercises, submitting questions, and providing feedback. Input consists of the teaching content and interactive features displayed on the terminal, while output includes user feedback on system usage, subjective evaluations, and suggestions. Specific actions include clicking "Next," entering questions, rating satisfaction levels, and requesting example explanations. All input is uploaded to the server in real time.

[0113] Step 7: The server regularly collects and analyzes user behavior logs, emotional history, and feedback information uploaded from all terminals. Based on the aggregated data, it continuously optimizes the code parsing algorithm, teaching content generation rules, and interactive presentation methods. The input is a collection of interaction data and feedback from all users, and the output is the optimized system configuration and content version. Specific actions include: the server generating user profiles, identifying common needs and pain points, and dynamically adjusting lesson plan templates and AI prompts to achieve self-learning and iterative upgrades of system functions.

[0114] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0115] Example 2 The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0116] Existing information processing systems have shortcomings in program analysis, extraction of system design concepts, support for new user education, and dynamic interaction based on user emotions. Specifically, existing technologies struggle to automatically and accurately identify the structure and function of all programs, and to provide personalized responses and guidance based on users' psychological states. Furthermore, these systems often fail to enable new members to quickly understand and efficiently integrate into projects, or to automatically optimize and intelligently evolve based on user feedback.

[0117] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0118] In this invention, the server includes: a device for automatically parsing all information processing programs and identifying program functions and relevance; a device for extracting the overall system design principles; a device for identifying user psychological states and dynamically generating or adjusting personalized responses and guidance content using a generative artificial intelligence model; a device for providing users with diverse interactive guidance and visual displays; and a device for continuously collecting and analyzing user feedback and automatically optimizing the system structure and functions. This enables automated global system-wide parsing, precise design concept management, intelligent emotion recognition, and personalized interactive guidance, and allows for adaptive optimization and intelligent evolution of the system through user feedback, thereby significantly improving the responsiveness, flexibility, and user experience of the information processing system.

[0119] An "information processing program" refers to a set of computer instructions or codes used to perform specific functions or logical operations.

[0120] "Function" refers to the ability of an information processing program or system to complete a specific task or operation.

[0121] "Components" refers to the various modules, parts, or components that constitute the function of an information processing device or system.

[0122] "Relevance" refers to the interrelationships between various components or procedures in terms of function, data, control processes, etc.

[0123] "Attribute information" refers to a collection of data that describes the characteristics, state, and additional information of data, program, or system elements themselves.

[0124] "Design principles" refers to the basic concepts, principles, or guiding ideas upon which the architecture design of information processing devices or systems is based.

[0125] "User psychological state" refers to the user's emotional, cognitive, or psychological state during the operation process, such as confusion or satisfaction.

[0126] "Generative AI model" refers to an AI algorithm or system that can automatically generate content based on input through training.

[0127] "Dynamic adjustment" refers to the processing method that changes the response, operation, or display content in real time based on external input or status changes.

[0128] "Interactive guidance" refers to an auxiliary method in which the system and the user engage in multi-round, dynamic guidance through proactive responses and feedback.

[0129] "Prompt information" refers to explanatory content generated by the system and displayed to the user to explain, suggest, or guide operations.

[0130] "Feedback information" refers to the opinions, evaluations, or operational data that users actively or passively transmit to the system during use.

[0131] "Optimization" refers to the process of improving the performance or user experience of a system by adjusting or improving its structure, algorithms, or processes.

[0132] "Tacit knowledge explanation support data" refers to data used to help understand knowledge content, background, or logic that is difficult to obtain directly.

[0133] "Display" refers to the act of presenting information to users in a visual way through various forms such as graphics, text, and animation.

[0134] "Prompt statements" refer to text data that is input into a generative artificial intelligence model and used to guide it in generating target content.

[0135] The embodiments of the present invention can be implemented through the following specific structures and operations.

[0136] The server is deployed in a computer network environment and is responsible for core background processing. It is configured with a standard operating system (such as Linux or Windows Server) and a database management system (such as MySQL or PostgreSQL). To enable automatic parsing of information processing programs, the server integrates static code analysis tools (such as SonarQube) to traverse and analyze all relevant program files, identifying their respective functions and the relationships between their components. The server also deploys attribute information management software to store and manage metadata information obtained from code parsing.

[0137] On the user end, typically a smart terminal device such as a laptop, smartphone, or tablet, the terminal interacts with the server through an application (such as client software developed based on Flutter or React Native). Users can initiate actions and ask questions on the terminal interface, such as entering text, clicking on function modules, and submitting task requests. The terminal application can collect user behavioral and emotion-related data and send it to the server in real time.

[0138] After receiving user input, the server uses a built-in sentiment recognition engine (such as a machine learning-based text sentiment analysis engine, typically represented by a sentiment analysis API built on a natural language processing framework) to analyze the user's psychological state. Based on the analyzed psychological state, the server combines generative artificial intelligence models (such as large language models, GPT-4, general generative AI platforms, etc.) to automatically generate response content and personalized guidance plans tailored to the user's current needs and emotions.

[0139] The server further integrates all the aforementioned relevant information (functions, design principles, user status, etc.) to automatically generate prompts for generative artificial intelligence models, i.e., "prompt statements." The server automatically organizes and outputs the most suitable prompt content for the current scenario and dynamically adjusts it based on AI model feedback to ensure information accuracy and user-friendliness. The server can also collect user feedback and behavioral data from terminals and regularly optimize and upgrade the system structure and functions.

[0140] The terminal displays step-by-step operation instructions and visual materials returned by the server, including structural diagrams, mixed text and graphics explanations, and animated prompts. If the user encounters confusion during the operation, the terminal will intelligently pop up more detailed guidance and supports voice reading and slow-motion demonstrations. Through the front-end interface, users can provide real-time feedback and evaluate the usefulness of the system's responses; this data is used to further optimize the overall system.

[0141] This invention can adapt to diverse software development and information processing environments, providing fully automated information parsing and organization, intelligent emotion recognition, interactive customized guidance, and adaptive evolution mechanisms. It offers new users a low-threshold, high-efficiency learning and operation experience, while improving the overall intelligence and flexibility of the system.

[0142] Specific examples: A user, using a project management platform for the first time, wants to understand the "batch data upload" function. The user enters "How to import data in batches?" into the terminal application and clicks the corresponding module. The server automatically analyzes the relevant code to determine the module's function, dependencies, and design principles. Simultaneously, the server identifies the user's emotion as "confused" and automatically generates step-by-step instructions based on a generative artificial intelligence model: "1. Please click the 'Import Data' button in the upper left corner; 2. Select your Excel or CSV file and upload it; 3. The system will automatically verify the data format…". Upon receiving this information, the terminal dynamically displays and guides the user through each step using text, images, and audio.

[0143] Example of prompts for generative artificial intelligence models: "Please combine the functional modules that users are currently confused about, the code attribute information, and the system design principles to generate detailed step-by-step operation guides for beginners, and add visual demonstration suggestions." "The user is currently having questions about the 'Data Import' module. Please generate engaging and well-structured instructional materials and accompanying image descriptions for them." use Figure 13 The processing procedure is explained.

[0144] Step 1: Users submit operation requests or ask questions to the system through the terminal. This input can be text, clicks, selection of function modules, etc. The input includes user action data and text data. The terminal collects the user's input information and related context, and sends it as a data packet to the server. The output is request data containing details of the user's operation.

[0145] Step 2: The server receives request data from the terminal. The input is user action details. The server first processes the user's text and behavioral data using a sentiment analysis engine, determining the user's current psychological state (such as confusion, satisfaction, etc.) through algorithms. Then, the server generates structured sentiment data as output, based on the parsed sentiment state, action content, and other information.

[0146] Step 3: Based on the acquired user psychological state and request content, the server automatically invokes static code analysis tools to parse the relevant information processing programs. The input consists of the user's specified requirements and all program files stored in the system. The server analyzes the function, components, and interrelationships of each program file, extracting the attribute information and metadata of each module, and outputting functional descriptions and related data.

[0147] Step 4: The server utilizes the module information, metadata, design principles, and user psychological state obtained from the above analysis to organize this information into detailed prompt data. The input includes function descriptions, relevant data, design principles, and sentiment analysis results. The server automatically generates structured prompt statements for the generative AI model and submits them as input. The AI ​​model outputs personalized step-by-step operation guidance, visual display suggestions, and question-and-answer content.

[0148] Step 5: The server organizes the step-by-step instructions and visual materials output by the AI ​​model, transforming them into a multimedia content package suitable for terminal display. The input is the content output by the generative AI model. The server performs format conversion, integration, and optimization of the content, outputting standardized guidance information (including text, images, voice links, etc.).

[0149] Step 6: The terminal receives step-by-step instructions and visual materials from the server. The input is the multimedia guidance content sent by the server. The terminal dynamically presents the content, such as split-screen graphic displays, animations, and voice prompts from a helpful assistant. Based on user feedback and user actions, the terminal automatically adjusts the guidance pace and provides functions for asking questions again or reviewing past content. The output is a user-friendly, phased interactive guidance interface.

[0150] Step 7: Users follow the on-screen instructions and provide feedback after each step. The input is the interactive guidance content pushed by the terminal. Users can select feedback options such as "Understood" or "Needs help." The terminal collects user feedback and operation logs and sends them back to the server. The output consists of user interaction data and feedback information.

[0151] Step 8: The server periodically analyzes user feedback and behavioral data. Inputs include user feedback data and operation logs. Based on the feedback, the server optimizes the backend database, adjusts the AI ​​prompt generation logic, and enriches the visualization resource library, enabling adaptive evolution of functionality. Outputs include an optimized knowledge base and subsequent improvement plans.

[0152] Application Example 2 The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0153] Existing information processing systems in fields such as education or project management lack the ability to dynamically adjust content and interaction based on users' real-time emotional states. This makes it difficult to provide targeted support for users with different comprehension abilities and emotional states, resulting in high learning barriers and poor interactive experiences for new participants. Furthermore, the system's collection of user feedback and personalized responses are limited, affecting learning efficiency and user satisfaction.

[0154] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0155] In this invention, the server includes a device for automatically scanning all information processing instruction sequences and identifying their attributes and interrelationships; a device for generating an overall structural design principle based on parsed and extracted additional information; a device for generating simplified educational instruction sequences; a device for answering user queries based on the overall structural design principle; a device for acquiring user image and audio data and identifying user emotional states; a device for dynamically adjusting processing content and output information according to emotional states; a device for outputting visual and audio information on the user terminal; a device for continuously collecting user reaction data and continuously optimizing the system accordingly; and a device for generating generative information processing model prompts based on emotions and user reactions. This enables dynamic adjustment of content and interaction based on user emotions, lowering the learning threshold, improving learning efficiency and user satisfaction, and continuously optimizing the system experience.

[0156] "Information processing instruction sequence" refers to a set of operational instructions or codes that a computing device can interpret and execute to achieve a specific function or business logic.

[0157] "Attributes" refer to the characteristics, properties, or descriptive information of each element in an information processing instruction sequence, including but not limited to type, structure, and dependencies.

[0158] "Interrelationship" refers to the logical connections, dependencies, or interactions between elements within an information processing instruction sequence.

[0159] "Additional information" refers to extra data related to the information processing instruction sequence, including information such as structure, author, calling relationships, and comments used to describe the characteristics of the instruction sequence.

[0160] "Overall structural design principle" refers to the system structural framework and design concept abstracted and summarized based on the information processing instruction sequence and its additional information, providing an overall guiding ideology for the realization of system functions.

[0161] "Simplified instruction sequences for education" refers to a set of instructions that simplify and refine the original information processing instruction sequences without affecting the core functions, so as to facilitate knowledge transfer and learning.

[0162] "User query" refers to a question or request made by a user to the system regarding the content, structure, or implementation of information processing.

[0163] "Image data" refers to visual data information collected through devices such as cameras that can show a user's current expression, state, etc.

[0164] "Audio data" refers to sound information such as a user's voice, tone, and content of speech, collected through devices such as microphones.

[0165] "Emotional state" refers to the psychological and emotional state of a user as identified through analysis of user image and audio data, such as anxiety, happiness, or confusion.

[0166] "Dynamic adjustment" refers to the process of automatically modifying and optimizing information processing content, interaction methods, or output content based on real-time information such as the user's emotional state.

[0167] "User terminal" refers to an electronic device that can communicate with a server and display information to the user and collect relevant user data, such as smartphones, tablets, and smart glasses.

[0168] "Visual and audio information" refers to multimedia content such as images, videos, and sounds presented to users through output devices such as displays and speakers.

[0169] "User response data" refers to the information that can be collected during user interaction with the system, including but not limited to facial expressions, voice, gestures, and operational behaviors.

[0170] "Generative information processing model" refers to an artificial intelligence or machine learning model that can automatically generate content, answers, or suggestions based on input prompts.

[0171] "Prompt instructions" refer to text or data descriptions input into a generative information processing model to guide it in generating specific content or performing specific actions.

[0172] This invention can be implemented in the following ways.

[0173] The system of the present invention includes the collaboration of a server, a user terminal, and a user. Through information processing and data exchange, it achieves dynamic optimization of content and interaction based on the user's emotional state.

[0174] The server can use general-purpose high-performance computer equipment, run an operating system such as Linux, and install database management software (such as a relational database management system), as well as information processing and artificial intelligence-related software environments (such as Python, machine learning frameworks, computer vision libraries, etc.). The server structure includes the following modules: automatic code scanning and analysis module, design principle extraction module, simplified code generation module, user query response module, sentiment recognition module, content adjustment and distribution module, and prompt statement generation module.

[0175] User terminals can include smartphones, tablets, or smart wearable devices, and are generally equipped with cameras, microphones, display devices, and audio output devices. The terminals run mobile or embedded operating systems, can communicate bidirectionally with servers, and have the ability to collect and upload data including user images, audio, and user actions, as well as receive and display various types of information sent by the server.

[0176] When users use this system, such as in a learning environment or project management scenario, the terminal captures the user's facial expressions through a camera, collects voice information through a microphone, and gathers user interaction data. This data is then uploaded to the server via the terminal.

[0177] The server uses visual analysis libraries such as OpenCV to perform facial feature recognition on image data, and uses a general speech recognition API to transcribe audio data and analyze emotional features such as tone and speech rate. Then, it uses machine learning tools such as TensorFlow to identify the user's emotional state (such as confusion, anxiety, relaxation, etc.). The information processing program in the server automatically traverses and analyzes all engineering-related program instruction sequences, extracts metadata information, and constructs the overall system structure and design principles. Based on this, it retains the core structure and generates a simplified instruction sequence for educational use.

[0178] The server also automatically generates content responses based on user requests and the overall structural design principles. When it detects that a user is a beginner and appears confused, the server will automatically filter and synthesize multimedia explanatory materials of appropriate difficulty and style to match the user's emotional state using a Python program. This may include adding animations, step-by-step diagrams, or playing soothing audio to cater to the user's mood. The server sends the optimized content to the terminal in JSON or similar formats, and the terminal adapts the display according to different user states.

[0179] The server will also automatically generate prompts for generative artificial intelligence models based on changes in user emotions and interactive feedback, guiding them to generate content that is more relevant to the current user state.

[0180] In a specific embodiment, a user is learning a programming course on a mobile device and shows confusion and stress when encountering intermediate-difficulty problems. After the facial expressions and voice data collected by the terminal are uploaded to the server, the server identifies them as a "confused" state. Based on this, the server pushes detailed step-by-step explanations, illustrated tutorials, and soft music to help the user alleviate their emotions and gradually solve the current problem.

[0181] Generative AI models can provide prompts such as, "The user is confused by the current intermediate-level problem. Please provide a detailed step-by-step explanation and recommend playing relaxing music," or "The user has clicked to view hints multiple times. Please add more examples and illustrations to aid understanding." Based on these prompts, the server can automatically generate or adjust content to optimize the user experience.

[0182] Through the above methods, the present invention realizes an intelligent information processing system that integrates emotion perception, automatic content adjustment and generative artificial intelligence content generation, effectively reducing the threshold for users to learn or participate in projects, and improving interaction efficiency and experience satisfaction.

[0183] use Figure 14 The processing procedure is explained.

[0184] Step 1: The terminal captures the user's facial image data via a camera and audio data via a microphone, while also recording the user's actions (such as clicks and requests for help). The input consists of the user's current raw image, audio data, and interaction logs. The terminal uploads this data to the server over the network. The output is a structured raw user data packet (containing images, audio, and action records).

[0185] Step 2: After receiving data uploaded by the terminal, the server analyzes the image data using computer vision libraries (such as OpenCV), extracts feature points and recognizes facial expressions, and automatically transcribes the audio content and analyzes tone and emotional features using a speech recognition API. The input consists of uploaded user image and audio data. The server performs emotion recognition and data processing, and the output is a judgment result (such as "confused," "anxious," "focused," etc.), indicating the user's current emotional state.

[0186] Step 3: The server automatically scans and analyzes all information processing instruction sequences, including educational content and project code, identifying the attributes and relationships between each instruction sequence and extracting metadata. Simultaneously, based on this metadata, the server constructs the overall system architecture and design principles, preparing for the generation of subsequent content. The input is a complete instruction sequence file, and the output is system metadata information and a summary of the design principles.

[0187] Step 4: Based on the identified user emotional state and system metadata, the server dynamically selects and processes educational or project management content that matches the current user's state, utilizing content generation algorithms and a multimedia resource library. For example, when a user is identified as "confused," the system synthesizes detailed step-by-step guidance, selects appropriate text / image and video explanations, and specifies the playback of soothing music. Inputs include user emotional tags, system metadata, and the content resource library. Data processing includes content filtering, difficulty adjustment, and multimedia integration. Outputs are personalized content packages and interaction schemes.

[0188] Step 5: The server sends the aforementioned personalized content package to the terminal. Upon receiving it, the terminal performs the following actions: adjusting the interface display order, content display duration, and automatically playing appropriate background music according to the instructions in the content package. The terminal outputs visual and auditory content to the user and continuously monitors the user's subsequent interactions and emotional changes based on settings. The input is the content package pushed by the server, and the output is a personalized visual and auditory multimedia content display for the user.

[0189] Step 6: Users interact with content, such as watching video tutorials, clicking on operation guides, listening to music, and answering questions. The user's latest facial expressions, voice, and actions are then collected by the terminal and uploaded to the server. The input consists of the terminal's content display and function operations; the output is new raw interaction data used for subsequent emotion recognition and content optimization.

[0190] Step 7: The server uses an automated generative AI model to analyze the latest user reactions and changes in emotional state. Based on the analysis results, it adjusts subsequent content delivery strategies and automatically generates prompts to input into the content generation model, such as "The user is still confused about the current knowledge point; please continue to simplify the explanation and add more examples." The input consists of user feedback data and historical interaction information; the data processing results in the generation of prompts and content adjustment instructions; the output is the optimized content strategy and a new round of content pushes.

[0191] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0192] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be performed partially or entirely by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0193] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0194] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0195] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0196] Second Implementation Method Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0197] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0198] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0199] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0200] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0201] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0202] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0203] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0204] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0205] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0206] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0207] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0208] Example 1 The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0209] Application Example 1 The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0210] Example 2 The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0211] Application Example 2 The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0212] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0213] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be performed partially or entirely by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0214] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0215] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0216] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0217] Third Implementation Method Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0218] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0219] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0220] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0221] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0222] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0223] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0224] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0225] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0226] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0227] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0228] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0229] Example 1 The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0230] Application Example 1 The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0231] Example 2 The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0232] Application Example 2 The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0233] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0234] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 includes prompt words containing instructions, as well as inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be performed partially or entirely by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0235] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0236] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0237] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0238] Fourth Implementation Method Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0239] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0240] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0241] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0242] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0243] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0244] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0245] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0246] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0247] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0248] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0249] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0250] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0251] Example 1 The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0252] Application Example 1 The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0253] Example 2 The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0254] Application Example 2 The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0255] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0256] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be performed partially or entirely by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0257] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0258] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0259] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0260] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The emotion-specific model 59 can also determine the robot's emotion, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0261] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0262] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0263] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0264] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0265] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0266] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0267] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0268] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0269] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0270] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0271] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0272] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0273] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0274] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0275] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0276] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0277] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0278] In addition, the following notes are provided in response to the above explanation.

[0279] Example 1 (Note 1) An information processing system, comprising: A device for automatically parsing a data set containing program information on an information processing device, and determining the functional information and association information of the data set; A device for generating extracted design principle information based on the functional information and associated information; A device for automatically generating a concise dataset that maintains core functionality for learning support based on the design principle information and functional information; A device for parsing query information in natural language form obtained by an input device, and generating and presenting answer information through an automatic knowledge generation device based on the design principle information and functional information; A device for outputting actionable interactive instructional information using a streamlined dataset with the learning support described above.

[0280] (Note 2) According to the information processing system described in Appendix 1, It also includes a device that provides interactive learning support functions for new users based on the parsed and extracted information and teaching interaction information.

[0281] (Note 3) According to the information processing system described in Appendix 1, It also includes a device for collecting and analyzing operation log information and feedback information from users, and for automatically and continuously improving information processing functions and knowledge generation algorithms based on the analysis results.

[0282] Application Example 1 (Note 1) An information processing system, comprising: A device for automatically scanning program information stored in an electrical storage medium and identifying the function and interdependencies of each piece of program information. A device for identifying the design concept of an information processing structure based on the attribute information extracted from the above identification results, and extracting the structural features and design patterns among the constituent elements. A device for automatically generating simplified learning aid information that is easy to understand, based on the extracted design concepts and attribute information and combined with learning attributes, while maintaining the main functions. A device for parsing user-input query information and generating output information using a generative artificial intelligence model based on the aforementioned design concept and identified attribute information; A sentiment estimation device used to analyze user operation or response information and estimate sentiment information; A device for dynamically adjusting the content or presentation of the aforementioned output information or simplified program information for learning assistance based on the inferred emotional information; A device for accumulating user feedback and emotional history information, and for continuously optimizing system functions or learning aids.

[0283] (Note 2) According to the information processing system described in Appendix 1, The automatically generated simplified learning aid program information and output information are presented in stages and interactively through a remote information terminal, and the content and presentation speed can be dynamically adjusted according to the user's interaction and operation.

[0284] (Note 3) According to the information processing system described in Appendix 1, The system automatically and continuously updates and improves its components, information content, and presentation methods based on user evaluations, operation records, and emotional responses.

[0285] Example 2 (Note 1) An information processing system, comprising: A device for automatically analyzing all information processing programs and identifying the functions of each information processing program and the relationships between its components; A device for extracting the overall design principle of an information processing device based on the attribute information obtained from the parsed information. A device for automatically generating response content to user queries based on the design principles of information processing devices; A device for judging a user's psychological state by using the user's operation information and input information; A device for dynamically adjusting response and guidance content based on the psychological state determined by the assessment using a generative artificial intelligence model; A device for generating prompts that enable interactive guidance between the user and the information processing device; A device for collecting user usage data and feedback, and for optimizing the structure or function of the information processing device; A device for generating supporting data that includes tacit knowledge explanations of the functional descriptions or design principles of a parsed information processing program. A device for displaying the various components of an information processing device in multiple forms of expression, based on their function, relevance, design principles, and user status. A device for integrating user status, design principles, and functional information to construct prompt statements that serve as input for generative artificial intelligence models.

[0286] (Note 2) According to the information processing system described in Appendix 1, Its key feature is that it provides dynamic and interactive educational support to new users based on the parsed and extracted information and the user's psychological state.

[0287] (Note 3) According to the information processing system described in Appendix 1, Its features include: continuously collecting and analyzing user feedback information and behavior records, and automatically improving the structure or function of the information processing device accordingly.

[0288] Application Example 2 (Note 1) An information processing system, comprising: A device for automatically scanning all information processing instruction sequences and identifying the attributes and interrelationships of each instruction sequence; A device for extracting the design principles of the overall structure based on additional information related to the parsed instruction sequence; A device for generating simplified educational instruction sequences that retain core components to facilitate knowledge transfer; A device for generating response content based on user queries and the design principles of the overall structure; A device for acquiring a user’s image and audio data and identifying emotional states based on information obtained from an input device; A device for dynamically adjusting the content of information processing or the presentation of information based on the identified emotional state; A device for sending adjusted presentation information to a user terminal and outputting visual and auditory information on that terminal; A device for continuously acquiring user reaction data and updating processing content or output information based on the reaction data; A device for automatically generating prompts for generative information processing models based on changes in emotional state and user responses.

[0289] (Note 2) According to the information processing system described in Appendix 1, Based on the parsed and extracted information and the identified emotional states, interactive knowledge acquisition support is provided for new participants.

[0290] (Note 3) According to the information processing system described in Appendix 1, Continuously collect user reaction data and changes in emotional state, and optimize system structure or processing methods.

Claims

1. An information processing system, characterized in that, include: A device for automatically scanning all software code and identifying the function and relationships of each code file; A device for extracting the overall design concept of a system based on the parsed code metadata; A device for generating easily understandable, simplified instructional code while maintaining the core functionality of a project; and A device for generating corresponding answers to user-submitted questions based on the overall project design concept.

2. The information processing system according to claim 1, characterized in that, It also includes a device for providing interactive tutorials to new participants based on the parsed and extracted information.

3. The information processing system according to claim 1, characterized in that, It also includes devices for continuously collecting user feedback and using it to improve system functionality.

Images