Information processing system

Abstract

The present invention provides an information processing system comprising: means for collecting a series of operational data in a user computer device; means for analyzing the collected operational data and identifying repetitive operations; means for generating automation suggestions for the identified repetitive operations; the device is used for learning by utilizing the operation data and constructing a business-oriented generative model; and means for presenting automation suggestions to the user and performing automation operations upon approval.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] In existing business operation processes, users need to manually repeat a large number of similar or identical operations, which is not only inefficient but also prone to data anomalies or omissions of important processes due to human error. Furthermore, there is a lack of systems capable of automatically analyzing user operating habits, proactively identifying and generating targeted automation suggestions, and a lack of mechanisms for personalized optimization and self-learning for different business scenarios. In addition, information security and user data privacy protection are difficult to balance in existing automation solutions. Therefore, how to achieve an intelligent automation system that can automatically identify repetitive user operations, generate efficient automation suggestions, ensure data security, and possess self-learning and optimization capabilities is a pressing problem that needs to be solved in this field. Summary of the Invention

[0004] This invention provides an information processing system, comprising: means for collecting a series of operational data in a user's computer device; means for analyzing the collected operational data and identifying repetitive operations; means for generating automation suggestions for the identified repetitive operations; means for learning from the operational data and constructing a business-oriented generative model; and means for presenting the automation suggestions to the user and executing the automation operation after approval. The system may further include means for updating the generative model based on user feedback, and means for encrypting the operational data during communication. Through these means, this invention can automatically learn and mine users' repetitive operations, providing users with efficient and convenient automation suggestions, effectively improving business process efficiency, and achieving intelligent optimization and continuous improvement of the system while ensuring data security.

[0005] "User computer equipment" refers to computers, laptops, tablets, or other information terminal devices used by users for various information processing and operations.

[0006] "Operational data" refers to the collection of information generated by users on computer devices, including various inputs, software operations, file processing, window switching, and other operational behaviors.

[0007] "Analysis" refers to the process of processing collected operational data, such as pattern recognition and frequency detection, in order to extract useful information or behavioral patterns.

[0008] "Repetitive operations" refer to the same or highly similar sequences of operations or behavioral patterns that users repeatedly perform within a specific period.

[0009] "Automation suggestions" refer to automated execution plans or recommendations that the system proactively generates to improve efficiency and reduce human intervention for identified repetitive operations.

[0010] "Generative model" refers to a data model that is trained using machine learning or other algorithms based on collected operational data, and that can be adapted to specific business scenarios and generate optimization suggestions.

[0011] "User feedback" refers to the evaluation, modification opinions, or suggestions made by users regarding the automation suggestions provided by the system or the results of the system's execution.

[0012] "Encryption" refers to the process of encoding operational data using specific algorithms so that the data cannot be deciphered by unauthorized parties during transmission or storage.

[0013] "Performing automated operations" refers to the process by which the system automatically completes business steps or tasks that originally required manual operation by the user, based on user-approved automation suggestions. Attached Figure Description

[0014] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0015] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0016] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0017] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0018] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0019] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0020] Figure 7This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0021] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0022] Figure 9 This represents an emotion map that maps multiple emotions.

[0023] Figure 10 This represents an emotion map that maps multiple emotions.

[0024] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of Embodiment 1.

[0025] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0026] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system in Embodiment 2.

[0027] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0028] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0029] First, let me explain the terminology used in the following instructions.

[0030] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0031] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0032] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0033] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0034] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" is used to connect and express more than three items, the same interpretation as "A and / or B" applies.

[0035] First Implementation Method Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0036] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0037] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0038] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0039] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0040] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0041] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0042] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0043] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0044] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0045] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0046] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0047] Example 1 The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0048] With the development of the information society, users' daily business operations on computing devices are becoming increasingly complex and repetitive. Traditional manual operation methods lead to low work efficiency and waste a lot of time and human resources. At the same time, existing technologies lack efficient methods that can automatically identify repetitive user operations, generate personalized automation suggestions, and implement automated processes. In addition, how to ensure the security and privacy of user operation information during data transmission is also a major problem faced by existing systems. Therefore, there is an urgent need for a system and method that can effectively collect, analyze, identify, and automate repetitive user business operations while ensuring data security.

[0049] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0050] In this invention, the server includes multiple devices for receiving encrypted user operation log data, decrypting and analyzing the log data, identifying repetitive operations based on machine learning algorithms and generative artificial intelligence models, generating automated suggestions by combining prompts, pushing the suggestions to the user terminal and receiving user feedback and approval, executing automated tasks based on user approval, and continuously updating and optimizing the artificial intelligence model by collecting task execution results and user feedback. This enables the automatic identification and intelligent automated processing of repetitive user operations, improving work efficiency, reducing manual workload, and ensuring the security and privacy of user information through end-to-end data encryption and secure transmission.

[0051] "Information processing device" refers to an electronic device with the ability to collect, analyze and process data, including but not limited to servers, personal computers or mobile terminals.

[0052] "User" refers to an individual or organization that operates a computing device and generates business operation data.

[0053] "Log data" refers to a collection of data obtained by monitoring and recording different behaviors or operations of users on information processing devices. It typically includes information such as operation type, event time, and related applications.

[0054] "Encryption processing" refers to the technical means of converting raw data into secure data that can only be deciphered by authorized parties according to a specific algorithm, in order to prevent information leakage and unauthorized access.

[0055] "Data processing device" refers to a device that can receive, store, decrypt and process external data, typically including servers or big data processing platforms.

[0056] "Machine learning algorithms" refer to computational methods that learn and optimize themselves by analyzing large amounts of data to discover patterns and rules, and generate decisions or predictions.

[0057] Pattern recognition technology refers to the technology of automatically identifying and classifying similarities or repetitions from data.

[0058] "Repetitive operations" refer to the same or similar operations performed by a user multiple times within a certain period of time.

[0059] "Generative AI models" refer to AI systems that can automatically generate text, suggestions, processes, and other content based on input data, such as models built using deep learning algorithms.

[0060] "Prompt statements" refer to input statements or instructions used to guide generative artificial intelligence models to output specific content and results.

[0061] "Automated recommendations" refer to system recommendation schemes that can improve efficiency, automatically generated based on user behavior.

[0062] "Automated tasks" refer to operational processes that are automatically executed by the system based on automation suggestions without human intervention.

[0063] "Secure communication path" refers to a network channel that uses encryption and authentication mechanisms to ensure the integrity and confidentiality of data during transmission.

[0064] "Training data" refers to the set of data used to learn and optimize artificial intelligence models and improve the quality of their output.

[0065] A "user terminal" refers to a computing device that provides an operating interface for users and can receive and execute information or tasks sent by a server.

[0066] "Feedback information" refers to information such as user experience, suggestions, or results of automated task execution that users actively or passively provide during the use of the system.

[0067] This invention provides a system and method for automatically identifying and processing repetitive user operations based on a generative artificial intelligence model, involving multiple technical aspects such as information collection, data encryption, machine learning, automated suggestion generation, user interaction, and data security. This system can be widely applied to various business scenarios requiring process automation, such as office automation, enterprise management, data reporting, and human resource operations.

[0068] The system consists of an information processing unit (server), user terminals, and supporting software. The server can be a high-performance general-purpose server, with an operating system such as Linux, a database such as MySQL or MongoDB, and an AI training platform such as TensorFlow or PyTorch. User terminals include, but are not limited to, personal computers, laptops, and other smart devices, with operating systems such as Windows 10 and macOS. Client-side data acquisition programs are installed on the terminals and can be developed using Python or C#.

[0069] Specifically, the terminal monitors user actions in the operating system and various applications (such as application launch, file editing, window switching, text input, email sending, etc.) through client programs, and converts the monitored actions into structured log data (including application name, operation time, content summary, etc.). To protect data security, the terminal encrypts the log data using a symmetric encryption algorithm (such as AES, implemented by libraries such as PyCryptodome) before uploading it, and transmits it to the server via secure protocols such as HTTPS.

[0070] After receiving the log data, the server uses the corresponding decryption module to restore the data and employs machine learning algorithms (such as clustering analysis in scikit-learn and deep learning models in TensorFlow) and pattern recognition technology to automatically analyze the log data and identify repetitive, highly similar work operation sequences. By calling a generative artificial intelligence model and combining it with prompts written for the business scenario, intelligent automation suggestions suitable for users are automatically generated, giving high-frequency repetitive tasks the ability to be executed automatically.

[0071] The server pushes the generated automation suggestions to the user's terminal via the network. Users can view, review, modify, or approve the automation suggestions on the terminal interface, and set parameters such as the trigger time and specific scope of the automation tasks. After user approval, the terminal automatically generates the corresponding automation script (such as using Windows Task Scheduler, AutoHotkey scripts, PowerShell scripts, etc.) according to the suggestion content and runs it automatically, enabling unattended execution of actions such as data import / export, automatic email sending, and file backup.

[0072] In addition, the server continuously collects the execution results of automated tasks and user feedback. All feedback data is used as training data and is regularly used to optimize and retrain the generative AI model to continuously improve the accuracy and personalization of the recommendations.

[0073] Specific example: Suppose a user needs to collect sales data from the CRM system weekly, compile it into a specified Excel template, and then send it to the relevant management email addresses. The system can automatically recognize this workflow and generate an automation suggestion: "We suggest setting up a scheduled task to automatically extract sales data, automatically populate the template, and automatically send emails. Do you agree to enable this?" After the user approves, the terminal automatically completes the entire process every Monday at 8:00 AM without manual intervention.

[0074] Examples of prompts that combine generative artificial intelligence models include: "Weekly emails need to be sent automatically. Please generate optimization suggestions for this process." "Based on the following operation log, please propose an automation solution to improve efficiency: Export report — Edit Excel — Send email — Repeat execution." How can we automate the flow of information between Outlook and Excel, and combine this with an automated solution for sending group emails? "Help me create a script that automatically summarizes sales data weekly and sends it to management." The above system can effectively solve practical problems such as automating repetitive operations, improving business efficiency through personalization, and ensuring data transmission security, providing users with intelligent, automated, and highly secure data processing and business execution solutions.

[0075] use Figure 11 The processing procedure is explained.

[0076] Step 1: The terminal runs a monitoring program in the background on the operating system, automatically collecting various user operations, such as application launches, file editing, window switching, and text input. The input is a real-time record of user actions, and the output is structured operation log data. Based on the operation events, the terminal writes the application name, window name, operation type, timestamp, and other information to a local log file and temporarily stores the logs in local storage.

[0077] Step 2: The terminal periodically (e.g., every 5 minutes) calls the encryption module to encrypt local log data using a symmetric encryption algorithm (e.g., AES) to ensure data security during transmission. The input is an unencrypted local log file, and the output is the encrypted log data file. After encryption, the terminal uploads the encrypted log data to the designated interface on the server via HTTPS.

[0078] Step 3: After receiving the encrypted log data, the server decrypts it using the corresponding key to recover the original operation log. The input is an encrypted log file, and the output is decrypted structured log data. After decryption, the server imports the log data into the database and performs further formatting processing, such as unifying data uploaded from different terminals into a standard data structure.

[0079] Step 4: The server uses machine learning algorithms (such as cluster analysis and time series analysis) and pattern recognition technology to analyze log data and identify highly repetitive operation sequences. The input is standardized operation logs, and the output is a list of identified repetitive operation patterns. The server performs statistical analysis and similarity comparisons based on features such as time and operation content to mark which tasks are high-frequency repetitive operations.

[0080] Step 5: The server takes identified repetitive tasks as input, combines them with a generative artificial intelligence model and pre-written prompts to generate customized automation suggestions. The input consists of a list of repetitive task patterns and corresponding business context prompts; the output is the text content of the automation suggestions. The server calls the AI ​​model to analyze actual business needs and automatically generates efficiency improvement solutions suitable for the user's current scenario, such as automatically sending scheduled emails or batch file processing.

[0081] Step 6: Users receive automated suggestions generated by the server on the terminal interface and can view and approve them. The input is the content of the automated suggestion, and the output is the user's feedback decision (such as agree, modify, or reject). Users can adjust the suggestion content, set specific automatic execution time and scope on the terminal interface, and then submit the approval result back to the server.

[0082] Step 7: Based on user-approved automation suggestions, the terminal automatically invokes local automation tools (such as Windows Task Scheduler, script execution engine, etc.) to generate automation scripts and execute tasks according to set times or conditions. The input is the user-approved automation suggestion content, and the output is the execution result of the automation task. The terminal actually initiates automated operations such as file processing and mass email sending, and generates execution logs locally.

[0083] Step 8: The server periodically receives automated task execution results and user feedback uploaded by terminals, including task completion status, anomaly records, and user suggestions. The input is task execution feedback data, and the output is updated AI model training data and execution reports. The server incorporates the collected information into the learning system to optimize the generative artificial intelligence model, improving the accuracy and personalization of subsequent automated suggestions.

[0084] Application Example 1 The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0085] In work environments where repetitive tasks are frequent, users struggle to efficiently identify and implement automation solutions, leading to decreased work efficiency and increased human error. Existing technologies often fail to generate optimal automation proposals based on users' actual operational behavior and emotional state, and lack mechanisms for continuously optimizing automation suggestions based on user feedback and emotions. Therefore, developing methods and systems that can automatically generate and dynamically adjust efficient automation solutions based on user operational data and emotional states is a pressing technical challenge.

[0086] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0087] In this invention, the server includes a device for collecting operation and status information from a user information processing device; a device for processing the aforementioned information using data analysis and machine learning to identify repetitive work patterns; a device for generating automation suggestions based on the identification results and user status information using a generative artificial intelligence model; a device for pushing automation suggestions to a user information terminal device and receiving user feedback; a device for controlling the actions of the work equipment according to the approved automation suggestions; a device for estimating the user status and dynamically adjusting the automation suggestions; and a device for continuously learning and optimizing the generative artificial intelligence model based on operation information and feedback data. It also includes an encrypted communication device to ensure data security. This enables comprehensive perception, analysis, and response to user operations and emotional states, dynamically generating automation proposals that best meet actual needs, and continuously optimizing the automation processing flow based on user feedback, thereby significantly improving work efficiency and reducing human error.

[0088] "Information processing device" refers to various electronic devices capable of information processing, control, and data exchange, including but not limited to computers, smartphones, and tablet terminals.

[0089] "Operation information" refers to the data records related to various inputs, clicks, interface operations, application switching, and other actions performed by users on information processing devices.

[0090] "Status information" refers to data that reflects a user's physiological, behavioral, psychological, or emotional state, including typing speed, mouse movement, eye movement, and physiological signal collection results.

[0091] A "data analysis device" refers to a software or hardware module that performs algorithms such as data preprocessing, statistics, mining, and modeling to effectively process and analyze the collected data.

[0092] "Machine learning device" refers to a computing module that can train models through data, identify patterns, and achieve self-optimization, including hardware and software systems implemented using traditional machine learning and deep learning methods.

[0093] "Repetitive task mode" refers to a sequence or process of user operations that appears repeatedly within a specific time period and has a similar structure.

[0094] "Generative artificial intelligence models" refer to artificial intelligence methods and systems that are trained on a large amount of data and can generate highly relevant suggestions, solutions, or text outputs based on input content.

[0095] "Automation suggestions" refer to specific operational plans generated by artificial intelligence and provided to users to simplify or automate tasks based on detected repetitive work patterns and user status.

[0096] "User information terminal device" refers to portable or desktop user interaction devices used to receive, display and provide feedback on information, including smartphones, tablets, computers, etc.

[0097] "Operating equipment" refers to mechanical equipment, robots, or control systems that can automatically or semi-automatically complete specific tasks according to instructions in industrial, office, or production sites.

[0098] "Encrypted communication device" refers to a software or hardware module that uses cryptographic algorithms to encrypt communication content in order to ensure data transmission security.

[0099] "Feedback data" refers to information such as user evaluations, suggestions for improvement, and confirmation of effects regarding automation suggestions or system operation processes.

[0100] "User state estimation" refers to the process of comprehensively analyzing user operation behavior data and related signals, and using algorithms to infer the user's current psychological or physiological state, such as emotions, stress, and attention.

[0101] To better understand and implement this invention, the specific embodiments of this invention are described in detail below in conjunction with specific hardware and software environments.

[0102] This invention can be implemented based on a system architecture that includes servers, terminals (such as smartphones, tablets, computers, and other user information terminal devices), and operating equipment. The various components within the system have clearly defined functions, and information interaction and control are achieved through a network.

[0103] The server is equipped with high-performance computing units and large-capacity storage devices, employs machine learning frameworks such as TensorFlow or PyTorch, and has a runtime environment for generative artificial intelligence models. The server also includes data analysis and user state estimation modules. The terminal can be various intelligent mobile or desktop hardware configured with a processor and operating system, capable of running dedicated client programs (e.g., developed using Python, Swift, Kotlin), supporting real-time data acquisition and interactive interface display. The operating equipment includes industrial control machinery, robots, etc., capable of accepting external commands to complete automated operations.

[0104] During use, the terminal collects and records user operation information (such as application launch, window switching, input content, mouse and keyboard events, eye movement, etc.) and status information (such as operation speed, behavior patterns, and physiological signals) in real time through client programs. The terminal can also integrate an emotion engine to estimate the user's emotional state, such as stress and satisfaction. To ensure security, the terminal encrypts all logs, status, and feedback data before uploading them to the server in batches via secure protocols such as SSL.

[0105] After receiving data uploaded from the terminal, the server first preprocesses and cleans the data. Then, it uses machine learning algorithms (such as sequence pattern mining and frequent itemset mining) to analyze the data, identifying and extracting repetitive job patterns. Next, the server combines a generative artificial intelligence model to automatically generate the most suitable automation suggestions for the current scenario based on historical operation behavior and the current user status. Subsequently, the server transmits these automation suggestions to the user terminal via the communication network, displaying them in a user interface and allowing users to view, modify, approve, or reject them. If the user approves the suggestion, the server issues an instruction to control the work equipment to run automatically according to the new process. The server also continuously collects user feedback and new status data, dynamically updating and optimizing the generation model and control strategy, enabling the entire system to continuously evolve and meet the user's personalized and contextualized needs.

[0106] For example, on a factory production line, an operator needs to manually assemble and record product components every day. The terminal continuously records the operator's input behavior, including typing and clicking details, and also monitors the operator's high stress score due to overtime work. During the analysis, the server found that the operation sequence "assembly step A - input sequence - save - confirm" was repeated multiple times in the same time period. Combining this with sentiment estimation results, the server automatically generated the following suggestion: "Merge the number entry and assembly start steps into batch processing, and automatically remind the user to take a break every two hours during busy periods." After the user approves this suggestion on the terminal, the robot receives the instruction and automatically operates according to the optimized execution process, effectively improving production efficiency while significantly reducing human error and operational stress.

[0107] This invention also supports the following generative artificial intelligence prompt examples: "Based on the factory robot operation logs, identify repetitive patterns and propose efficient process optimization solutions that can be automated for each pattern." "By analyzing recent usage records of office automation software and combining them with users' stress scores, we can generate automation suggestions to reduce the frequency of manual operations in high-stress scenarios." "By monitoring repetitive behaviors of production line workers and combining the results of eye tracking, we recommend a robot motion automation solution that can simultaneously reduce fatigue and improve efficiency." Through the above methods, this invention can ensure that various users in complex working environments can continuously obtain fast, accurate and dynamically adjustable automated operation solutions through automatic sensing, data analysis and continuous optimization, effectively improving overall work efficiency and user experience, and is applicable to various industrial, office and other application scenarios.

[0108] use Figure 12 The processing procedure is explained.

[0109] Step 1: The terminal collects real-time user operation and status information on the information processing device through a client program, including application launches, window switching, input content, mouse and keyboard actions, and eye movements. The terminal can also use an emotion engine to estimate user emotional states such as stress and satisfaction. Input consists of the user's actual operational behavior and related physiological and psychological signal data, which are processed into structured operation logs and emotional parameters. Output is a data packet containing fields such as timestamps, operation sequences, and emotional scores.

[0110] Step 2: The terminal encrypts the collected operation logs and status information, employs secure communication protocols such as SSL, and periodically uploads data packets in batches to the server. The input is structured raw data packets, which are processed into a secure encrypted format. The output is an encrypted data stream or file.

[0111] Step 3: The server receives encrypted data streams from the terminal, first decrypts and verifies the data, and then uses a data analysis module to perform data cleaning and preprocessing, including format conversion, missing value replacement, and anomaly detection. The input is encrypted uploaded data, and the output is a standardized dataset of operations and statuses available for analysis.

[0112] Step 4: The server uses machine learning tools (such as TensorFlow and PyTorch) to analyze standardized datasets and identify repetitive task patterns that frequently occur in users over a period of time. The input consists of user action sequences and emotional state data, processed using pattern recognition algorithms and frequent itemset mining. The output is a report containing a list of all identified repetitive task patterns and related statistical information.

[0113] Step 5: The server combines a generative AI model to automatically generate personalized automated suggestions based on identified repetitive task patterns and the user's current state information. Inputs include pattern reports and sentiment parameters; data processing involves calling on the generative AI to generate optimized suggestion text based on prompts. Outputs are structured automated suggestions and their applicability instructions.

[0114] Step 6: The server pushes automated suggestions to the user's terminal via the communication network and prompts the user to view them on the terminal interface. The input is the generated automated suggestion data, which is encapsulated and transmitted over the network, and the output is the suggestion content displayed on the terminal interface.

[0115] Step 7: Users browse automation suggestions on their devices and can choose to confirm, modify, or reject them. Input consists of automation suggestions pushed by the server and the user's own operation commands, which are then processed into a decision. Output includes the user-approved automation plan and feedback data.

[0116] Step 8: The server receives the user-approved or revised automation plan, transforms it into control commands recognizable by the working equipment, and distributes them to the specific working equipment via the network. The input is the user-approved suggestion, which, combined with the equipment control protocol, generates equipment commands. The output is the automated operation commands received and executed by the working equipment.

[0117] Step 9: The equipment automatically executes optimized repetitive work processes based on instructions from the server. Inputs are equipment control commands, and data processing involves mechanical actions and process execution. Outputs include the actual work completion results and related status feedback.

[0118] Step 10: Users provide feedback on the automation's effectiveness via the terminal, such as ratings, opinions, and suggestions. The input consists of the user's subjective experience and the results of the automated operation, which is then documented as feedback data and uploaded to the server. The server collects this feedback data and uses it as parameters for AI model optimization and system self-learning. The output is a dataset used for subsequent optimization.

[0119] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0120] Example 2 The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0121] Existing automated suggestion systems typically rely solely on pattern recognition based on user action data, lacking effective consideration of users' emotional states. This prevents them from achieving efficient, personalized automated task recommendations tailored to users' real-time psychological burdens and individual differences. Furthermore, existing solutions suffer from data security risks during information collection and processing, failing to guarantee the confidentiality and integrity of sensitive action data and biometric information. Therefore, there is an urgent need for a system that can comprehensively collect user action data and emotional state data, utilize artificial intelligence models for intelligent automated suggestions, and possess a highly secure data processing mechanism to improve the accuracy of automated suggestions and user experience.

[0122] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0123] In this invention, the server includes means for acquiring user information, including operational data and biometric information; means for determining the user's behavior logs and emotional state on the information processing device based on the acquired information; means for generating natural language input data for inputting into the artificial intelligence model based on the determined behavior logs and emotional state, and for creating business process automation suggestions; means for displaying the automation suggestions on the display device of the information processing device and receiving user approval or correction; means for automatically executing the work process according to the approved automation suggestions; and means for encrypting the operational data and emotional state and conducting data communication. Thus, while ensuring data security, personalized and intelligent automated task recommendations can be achieved based on user operational behavior and real-time emotions, and the artificial intelligence model can be dynamically optimized to improve automation efficiency and user satisfaction.

[0124] "Operational data" refers to the behavioral records generated by users' various interactions on information processing devices, such as input, clicking, swiping, and interface switching.

[0125] "Bioinformation" refers to data acquired through bioinformation sensors that can reflect a user's physiological or psychological state, including but not limited to information such as heart rate, gaze trajectory, and facial expressions.

[0126] "User information" refers to a set of data, consisting of operational data and biometric information, used to describe a user's current behavior and state.

[0127] "Behavior log" refers to an ordered dataset that continuously records a user's actions over a period of time based on the actual operation data of the user information processing device.

[0128] "Emotional state" refers to the evaluation of a user's psychological or physiological state, such as stress, satisfaction, and pleasure, determined by analyzing the user's operational data and biometric information.

[0129] "Artificial intelligence model" refers to an algorithmic model that uses machine learning or deep learning technology and is trained on data to automatically reason, analyze, and generate business suggestions.

[0130] "Natural language input data" refers to text information expressed in human language and formatted for use as input to artificial intelligence models to generate automated suggestions.

[0131] "Business process automation suggestions" refer to automated execution plans that are automatically generated based on users' actual behavior and emotional state to improve productivity or optimize user experience.

[0132] "Approval or correction action" refers to the user's feedback action such as confirming, modifying, or supplementing the automated suggestions automatically generated by the system.

[0133] "Workflow" refers to a series of operational steps required for a user to complete a specific goal or task on an information processing device.

[0134] "Data encryption" refers to the process of converting sensitive information into a ciphertext state that cannot be directly recognized through specific algorithms in order to ensure the security of its transmission and storage.

[0135] "Data communication" refers to the process of transmitting and sharing data information between different devices or systems using a network.

[0136] "Bioinformation sensors" refer to terminal devices used to detect and collect data related to human physiology or behavior, such as eye trackers, cameras, or heart rate detectors.

[0137] "User interface" refers to the interface in an information processing device used for information interaction, display, and input operations with the user, including screens, display areas, or function panels.

[0138] The system of this invention includes a server, a terminal, and a human-computer interface for user interaction. Specifically, it can acquire user operation data and biological information through a terminal with data acquisition capabilities (such as a smartphone, tablet, personal computer, or other information processing device) combined with bio-information sensors (such as a camera, heart rate sensor, eye tracker, etc.).

[0139] The terminal is equipped with a client application that has the following functions: real-time monitoring and recording of user input (including keyboard typing, mouse clicks, interface swiping, window switching, etc.) and collection of user biometric parameters (such as heart rate, facial expressions, and eye movements). The collected data is processed to form a user behavior log and an estimate of their emotional state. The terminal also integrates an emotion recognition module, which can perform pattern recognition and analysis on operational data and biometric information based on artificial intelligence algorithms such as neural networks and support vector machines to achieve a quantitative assessment of the user's emotions (such as stress, pleasure, etc.). For example, when the terminal detects a significant decrease in the user's input speed, slower mouse movement, and a tense facial expression, the system can determine that the user is in a state of high stress.

[0140] All collected operational data and emotional state information are encrypted on the terminal side using common encryption algorithms such as AES (Advanced Encryption Standard) to ensure data security during transmission and storage. The encrypted data is then periodically and automatically sent to the server via communication protocols such as HTTPS.

[0141] The server possesses large-scale data processing and analysis capabilities, enabling it to decrypt and mine patterns in received log and sentiment data. Employing algorithms such as clustering and frequent itemset analysis, the server identifies frequently repeated user actions and their correlation with emotional states. For example, the server might detect that a user manually exports multiple reports daily after 5 PM, and that their stress score is high during this period.

[0142] The server also deploys generative AI models (such as natural language generation models based on the Transformer architecture). The server inputs user behavior logs, sentiment data, and business context information into the generative AI model in the form of natural language prompts to generate personalized automated business process suggestions. A specific example is: "It has been detected that the user frequently exports reports between 5 PM and 6 PM daily and experiences increased emotional stress. Please generate automated task suggestions for this operation." The generated suggestion could be: "Automate the report export process and add a rest reminder for the user after the operation is completed." After receiving automated business suggestions from the server, the terminal displays them to the user in a visual interface. Users can directly choose to accept, modify, or reject the suggestions. For example, users can select "Automatically export reports in batches" and set a specific execution time, or adjust the suggestion to "Only execute automatically under high pressure." User feedback and the final implementation status of the automated tasks are uploaded back to the server as feedback data, which the server uses to optimize and update the artificial intelligence model.

[0143] Typical examples of prompt statements include: "Users repeatedly perform file archiving operations and show obvious dissatisfaction and stress. Please generate automated suggestions to reduce the user's burden." "The system detected that the user was in a good mood while editing the report, but their mood declined during the data entry phase. Please develop an automated strategy to improve the efficiency of the data entry process." "The user frequently fills out order forms between 2:00 PM and 4:00 PM, and their typing speed slows down. Their emotional score is high stress. Please generate automated suggestions to improve their efficiency." Through the above-mentioned combination of hardware and software, the system of the present invention can achieve comprehensive perception of user operations and emotions, intelligent generation of personalized automated suggestions, secure and reliable data protection, and dynamic continuous optimization, which greatly improves the automation level of information processing equipment and user experience.

[0144] use Figure 13 The processing procedure is explained.

[0145] Step 1: The terminal initiates a real-time monitoring module for user behavior and biometrics via an installed client program. Specifically, the terminal collects data from user actions including keyboard input, mouse clicks, screen touch input, and application switching. Simultaneously, it uses biometric sensors (such as cameras and heart rate monitors) to collect data on the user's heart rate, facial expressions, and gaze patterns. The input consists of user interactions and real-time biometric parameters; after data collection and standardization, the output is structured behavior logs and biometric data.

[0146] Step 2: The terminal uses a built-in emotion discrimination algorithm module to analyze the behavior logs and biometric information output in step 1 to identify the current user's emotional state (such as stress level, satisfaction level, or fatigue index). The input is structured behavior logs and biometric information. The data processing includes behavioral feature extraction, emotion model inference, etc., and the output is quantitative emotional state data, such as "stress index: 7 points, fatigue index: 5 points".

[0147] Step 3: The terminal encrypts the collected behavior logs and emotional state data by calling encryption algorithms such as AES. The input consists of behavior logs and emotional states; after being converted by the encryption algorithm, the output is an encrypted data packet, which is temporarily stored in a local secure area.

[0148] Step 4: The terminal uses secure communication protocols such as HTTPS to automatically send encrypted data packets to the server at set intervals or according to a data volume threshold. The input is an encrypted data packet, which, after passing through the network communication process, outputs the data successfully transmitted to the server.

[0149] Step 5: After receiving data packets from the terminal, the server first decrypts the data. The input is an encrypted data packet, which is decrypted using AES, and the output is usable behavior logs and sentiment data. Subsequently, the server uses algorithms such as clustering analysis and pattern recognition to analyze the behavior logs, identify high-frequency repetitive operations of the user on the information processing device, and combine this with sentiment data to make correlation judgments. The output is a dataset containing repetitive operations and associated sentiment states.

[0150] Step 6: Based on the analysis results, the server generates natural language prompts and inputs them into a generative artificial intelligence model (such as GPT). The AI ​​model then automatically generates personalized business process automation suggestions based on the prompts. The input consists of a summary of user behavior and sentiment data, along with the prompts. The data processing includes NLP transformation and strategy generation, and the output is automated suggestions in natural language format.

[0151] Step 7: Users receive automated suggestions from the server via a terminal, and then browse, confirm, or modify the suggestions in the user interface. The input is the content of the automated suggestions; after the user views and interacts with them, the output is feedback indicating whether the user chooses to accept, modify, or reject them.

[0152] Step 8: Based on the user's final confirmed automation suggestions, the terminal automatically invokes the corresponding business process execution module, schedules or triggers relevant operations automatically, and continuously monitors the user's status during execution. The input is the user-confirmed automation task content, data processing includes automatic script invocation and process integration, and the output is the automatically executed business process results and process logs. User feedback after execution is also used as input for a new round of data collection.

[0153] Application Example 2 The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0154] Current technologies cannot comprehensively collect and analyze workers' operational behaviors and emotional states while operating information terminals in real time. This makes it difficult to promptly identify and automatically recognize repetitive labor processes, and also fails to effectively combine workers' psychological or physiological workload to provide accurate automation and environmental optimization suggestions. Consequently, business efficiency cannot be improved, workers are overburdened, and work quality and safety decline. Most existing automation proposals rely solely on operation logs, lacking personalized adjustment and continuous learning mechanisms, and cannot respond promptly to users' actual conditions and provide accurate continuous improvement suggestions.

[0155] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0156] In this invention, the server includes: a device for collecting a series of operational data from an information terminal connected to an information processing device; a device for parsing the collected data and identifying high-frequency repetitive work patterns; a device for generating processing suggestions based on automated identification results; a device for collecting and analyzing the operator's movement speed and audio information, and evaluating emotional state through a generative information processing model; a device for generating business automation and work environment adjustment suggestions in a timely manner based on operational data and emotional state, and pushing them to the operator; and a device for continuously optimizing the model and suggestions based on operator feedback. This enables comprehensive perception of operator operations and emotions, automated identification to optimize processes, and the generation of personalized, adaptive business automation and environment adjustment suggestions, thereby significantly improving the system's intelligence level and operational efficiency, reducing the burden on operators, and continuously improving the system's practical application effectiveness and security.

[0157] "Information processing device" refers to electronic equipment capable of processing, analyzing, or storing input data, including but not limited to servers, computers, and dedicated data processing units.

[0158] "Information terminal" refers to a device used to interact with operators and collect, display or send data, including but not limited to personal computers, industrial terminals, mobile devices, etc.

[0159] "A series of operation data" refers to the record of all operations (such as mouse, keyboard, touch, etc.) performed by operators on the information terminal, including information such as operation type, sequence, and timestamp.

[0160] "High-frequency repetitive operation mode" refers to an operation process or task combination that is executed multiple times and in a regular manner within a specified time interval.

[0161] "Automation suggestions" refer to the automatic execution proposals and corresponding implementation details generated by the system for operations or processes that can be automated, based on the analysis results of operational data and work patterns.

[0162] "Generative information processing model" refers to a data processing model that uses artificial intelligence or machine learning technology to automatically reason and generate conclusions or suggestions by analyzing historical and real-time data.

[0163] "Action speed" refers to the rate and rhythm parameters of physical actions performed by operators when operating a terminal, including clicking speed, typing speed, etc.

[0164] "Audio data" refers to the voice or other sound signals of workers collected through terminals, which are used for subsequent emotion analysis and state judgment.

[0165] "Emotional state assessment" refers to the process of judging the current psychological or physiological state (such as stress, fatigue, concentration, etc.) of workers by comprehensively analyzing multimodal data such as movement speed and audio data.

[0166] "Business environment adjustment suggestions" refer to environmental improvement proposals generated based on operational and emotional data analysis results to optimize work efficiency and employee health, such as rest reminders and adjustments to work rhythm.

[0167] "Feedback information" refers to the response data of operators to the automation suggestions or environmental adjustment suggestions given by the system, such as confirmation, modification, or rejection.

[0168] This invention relates to an artificial intelligence-based business automation and operational environment optimization system, which mainly includes three-way interaction between a server, a terminal (information terminal), and operators (users). The following describes specific implementation methods and typical application scenarios of this invention.

[0169] The server is configured as a high-performance data processing device, capable of deploying and running generative artificial intelligence models (such as deep learning-based machine learning platforms). Examples include using TensorFlow on a general-purpose server, OpenCV which integrates image and speech analysis, and automated model training platforms (such as cloud-based machine learning services). The terminal is an intelligent device with a user interface and data acquisition module. It may use ordinary computing devices, tablets, or industrial human-machine interface devices, equipped with sensors such as microphones and cameras, and capable of running log collection and data uploading software (such as local logging systems and data synchronization modules).

[0170] During actual operation, the terminal collects various user operation data in real time, including keyboard input, mouse actions, and touch screen operations, and automatically records this data. The terminal also works with built-in or external sensors to capture user movement speed and voice data, achieving multimodal data acquisition. All collected data undergoes preliminary local processing before being synchronously uploaded to the server via an encrypted channel.

[0171] The server performs in-depth analysis of the collected operational data. First, statistical analysis and machine learning methods (such as Pandas, Scikit-learn, and TensorFlow) are used to automatically detect frequently occurring operational patterns and repetitive tasks. Then, the server further invokes a generative artificial intelligence model to automatically infer which processes are suitable for automation based on the system's built-in prompts and the detected real-world scenarios, generating corresponding automation suggestions and scripts. Simultaneously, the server utilizes multimodal data to assess user sentiment and work status, combining action-based and voice characteristics to output targeted business environment adjustment suggestions.

[0172] The server then pushes automation proposals and environmental optimization suggestions to the user via the terminal. The user can confirm, adjust, or reject the suggestions through the terminal interface. All user feedback is instantly transmitted back to the server and used as the basis for further training and continuous optimization of the generative artificial intelligence model, thereby realizing an adaptive and continuously improving business automation and work environment improvement system.

[0173] Here's a specific example: User A repeatedly performs the series of actions—"open report program—enter data—save—upload"—on their industrial workstation. The system automatically collects these operation records. After analysis, the server determines that the process can be automated and sends a notification to the user suggesting configuring an automatic report generation script for daily scheduled automatic uploads. Simultaneously, by analyzing the user's voice and actions, the server detects a slower operating pace and a lower voice, indicating "mild fatigue," and sends a notification suggesting a 10-minute rest. After user confirmation, the terminal automatically executes the script and records the operation results and feedback, sending them back to the server.

[0174] A typical prompt for generative AI models would look like this: "Based on the user's current operation logs, action speed, and voice sentiment analysis results, please assess the user's current workload and recommend repetitive processes that can be automated and the best time to rest." use Figure 14 The processing procedure is explained.

[0175] Step 1: The terminal collects user operation data in real time. It initiates an operation log collection program to monitor all user input actions on the computing device, including keyboard input, mouse movement, clicks, and touchscreen operations. The input represents the user's real-time actions; the terminal appends event timestamps to this data and stores it in a local log database. The terminal periodically packages the logs into batches and transmits them to the server in encrypted form. The output is a structured operation log data file.

[0176] Step 2: The terminal analyzes operation data to initially filter for frequently recurring operations. The input is the collected local operation logs. The terminal calls its built-in statistical analysis module to perform frequency statistics on different operation types and combinations, marking frequently occurring operation sequences. The output is a candidate list of frequently recurring processes, which is then prepared for uploading to the server.

[0177] Step 3: The server aggregates and analyzes data from multiple devices to accurately identify typical repetitive tasks. Input consists of high-frequency operation candidates and detailed logs uploaded from multiple terminals. The server uses machine learning algorithms (such as cluster analysis and time-series pattern mining) to perform deep analysis of the operation logs, comprehensively determining which processes are most representative and best suited for automation. Output is a list of typical operation processes that can be automated.

[0178] Step 4: The server generates automation suggestions based on a generative artificial intelligence model. Input consists of identified typical repetitive processes and historical operation data. The server invokes the generative AI model, using customized prompts to analyze which steps can be replaced by automated scripts, and automatically generates the operation script text and suggestion explanations. Output consists of the automation script and suggestion text.

[0179] Step 5: The server pushes automation suggestions to the terminal, which then prompts the user. The server transmits the automation suggestions and scripts to the terminal via a secure interface. The terminal displays a pop-up window or message notification to the user, showing the suggested content and selectable actions. The input is the automation suggestion package, and the output is a user-visualized interface prompt.

[0180] Step 6: The terminal collects the operator's movement speed and audio data to assess their emotional state. Inputs include the operator's real-time movements (such as mouse click speed and typing rhythm) and speech data (such as speech content and tone). The terminal uses image recognition and speech analysis algorithms (such as OpenCV and TensorFlow) to identify emotions such as stress, fatigue, and focus. Outputs are structured emotional data and a state score.

[0181] Step 7: The terminal uploads emotion and status data, and the server generates environmental adjustment suggestions after comprehensive evaluation. The input consists of structured emotion data and the latest operation logs. The server combines historical data with a generative AI model to analyze the current workload and work status of personnel, generating environmental optimization proposals such as "suggest rest" or "adjust work speed." The output is an optimization proposal and its rationale, which is pushed to the terminal.

[0182] Step 8: The terminal displays optimization suggestions to the user, who then responds. After receiving the proposal from the server, the terminal notifies the user via interface pop-ups, voice announcements, or other means. The user selects "accept," "adjust," or "reject" the suggestion based on their feedback and can also fill out a simple feedback form. The input is the optimization proposal, and the user's action is the output.

[0183] Step 9: The terminal executes user-confirmed automated scripts or adjustment measures and provides feedback on the execution results. Inputs include user confirmation items and automated scripts. The terminal automatically invokes relevant scripts or performs timed breaks based on instructions, and generates an execution result report upon completion. Outputs include the execution results and user satisfaction feedback.

[0184] Step 10: The server collects all feedback and new data, continuously training and optimizing the generative AI model. Inputs include user feedback, sentiment ratings, and execution results uploaded from the terminal. The server uses this data for incremental model training, improving the accuracy and personalization of future automation and environmental adjustment suggestions. Outputs include optimized model parameters and improved system performance.

[0185] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0186] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be performed partially or entirely by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0187] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0188] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0189] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0190] Second Implementation Method Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0191] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0192] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0193] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0194] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0195] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0196] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0197] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0198] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0199] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0200] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0201] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0202] Example 1 The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0203] Application Example 1 The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0204] Example 2 The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0205] Application Example 2 The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0206] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0207] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be performed partially or entirely by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0208] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0209] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0210] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0211] Third Implementation Method Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0212] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0213] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0214] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0215] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0216] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0217] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0218] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0219] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0220] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0221] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0222] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0223] Example 1 The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0224] Application Example 1 The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0225] Example 2 The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0226] Application Example 2 The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0227] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0228] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 includes prompt words containing instructions, as well as inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be performed partially or entirely by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0229] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0230] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0231] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0232] Fourth Implementation Method Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0233] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0234] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0235] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0236] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0237] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0238] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0239] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0240] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0241] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0242] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0243] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0244] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0245] Example 1 The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0246] Application Example 1 The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0247] Example 2 The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0248] Application Example 2 The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0249] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0250] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be performed partially or entirely by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0251] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0252] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0253] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0254] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The emotion-specific model 59 can also determine the robot's emotion, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0255] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0256] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0257] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0258] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0259] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0260] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0261] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0262] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0263] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0264] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0265] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0266] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0267] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0268] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0269] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0270] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0271] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0272] In addition, the following notes are provided in response to the above explanation.

[0273] Example 1 (Note 1) An information processing system includes: a device for acquiring action information related to a series of operations performed by a user in an information processing device and generating log data; a device for periodically encrypting the log data and sending the encrypted log data to the data processing device via a secure communication path; a device for decrypting the received encrypted log data in the data processing device and identifying multiple repetitive operational behaviors using machine learning algorithms and pattern recognition technology; a device for generating automated suggestions for improving business efficiency by taking the identified repetitive behaviors as input, combining a generative artificial intelligence model and prompt statements; a device for pushing the automated suggestions to a user terminal and receiving user approval or modification; a device for executing automated tasks on the user terminal at specific times or under specific conditions based on the user-approved automated suggestions; and a device for collecting the execution results of the automated tasks and user feedback, and storing them as training data in the information processing device for learning by the generative artificial intelligence model.

[0274] (Note 2) The information processing system according to Appendix 1 further includes: an apparatus for relearning and updating the generative artificial intelligence model based on the collected execution results and user feedback information.

[0275] (Note 3) The information processing system according to Appendix 1 further includes: as a security information protection measure, means for encrypting and decrypting the action information and log data, and ensuring the security of the communication path and the confidentiality of the data.

[0276] Application Example 1 (Note 1) An information processing system includes: a device for collecting a series of operation information and status information from a user information processing device; a device for processing the collected operation information and status information using a data analysis device and a machine learning device to identify repetitive work patterns; a device for generating automation suggestions using a generative artificial intelligence model based on the identified repetitive work patterns and user status information; a device for sending the generated automation suggestions to a user information terminal device via a communication device and receiving user modifications or approvals; a device for controlling the execution of actions of work equipment according to user-approved automation suggestions; a device for estimating user status information and dynamically adjusting automation suggestions according to user status; a device for continuously learning and optimizing the generative artificial intelligence model using operation information and feedback data; and a device for encrypting the communication of the operation information, status information, and feedback data for security purposes.

[0277] (Note 2) The information processing system according to Appendix 1 is characterized in that it automatically updates the parameters or structure of the generative artificial intelligence model based on feedback information and operational data obtained from users, so as to improve the accuracy of subsequent automated suggestion generation.

[0278] (Note 3) The information processing system according to Appendix 1 is characterized in that, as a user state estimation device, it utilizes physiological information, behavioral information, input device information, etc., to perform multivariate analysis or deep learning to achieve emotion recognition processing.

[0279] Example 2 (Note 1) An information processing system includes: a device for acquiring user information, including operational data and biometric information; a device for determining a user's behavior log and emotional state on an information processing device based on the acquired information; a device for generating natural language input data for inputting into an artificial intelligence model based on the determined behavior log and emotional state, and creating business process automation suggestions; a device for displaying the automation suggestions on a display device of the information processing device and receiving user approval or correction; a device for automatically executing work processes according to the approved automation suggestions; and a device for encrypting the operational data and emotional state and performing data communication.

[0280] (Note 2) The information processing system according to Note 1 further includes means for learning and updating the artificial intelligence model based on feedback from user operation results and emotional state.

[0281] (Note 3) The information processing system according to Appendix 1 further includes a device for acquiring operational data and biological information using a bio-information sensor and a user interface.

[0282] Application Example 2 (Note 1) An information processing system includes: means for collecting a series of operational data from an information terminal connected to the information processing device; means for parsing the collected operational data and identifying frequently recurring work patterns; means for assessing the possibility of automation and generating automatic processing suggestions based on the identified repetitive work patterns; means for training a generative information processing model based on the operational data and work patterns; means for collecting the movement speed and audio data of operators, and using the information processing model to assess their emotional state and generate structured data; means for generating business environment adjustment suggestions or business automation suggestions to improve work efficiency or reduce workload based on the assessed emotional state and operational data; and means for displaying the automation suggestions and business environment adjustment suggestions to operators through the information terminal, and executing automated processing or environmental adjustments according to the operators' responses.

[0283] (Note 2) The information processing system according to Appendix 1 further includes means for updating the generative information processing model and continuously optimizing its accuracy and suggested content based on the responses and feedback from operators or users.

[0284] (Note 3) The information processing system according to Appendix 1 further includes a device for encrypting operation data, action data, and audio data during transmission to ensure information security.

Claims

1. An information processing system, characterized in that, include: A device for collecting a series of operational data in a user's computer device; A device for analyzing collected operational data and identifying repetitive operations; A device for generating automated recommendations for identified repetitive tasks; A device for learning and building business-oriented generative models using the operational data; and A device for presenting automation suggestions to users and performing automated operations upon approval.

2. The information processing system according to claim 1, characterized in that, Also includes: A device for updating the generated model based on user feedback.

3. The information processing system according to claim 1, characterized in that, Also includes: A means for encrypting the operational data during communication.

Images