Information processing system

Abstract

The present invention provides an information processing system, comprising: a model generation unit for analyzing video data acquired from a monitoring device; the unit is used for generating a job log according to an analysis result of the model generation unit; a unit for generating the job log into a non-homogenized token; and a unit for providing the non-homogenized tokens to a user.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] In existing food manufacturing processes, it is difficult to achieve comprehensive visualization and highly reliable information traceability. Traditional systems are inadequate in detecting errors or anomalies in the production process, ensuring transparency of work records, and facilitating product traceability and certification. This makes it difficult for consumers to fully trust the safety and quality of their products, and for manufacturers to promptly grasp production line anomalies and achieve fair and open information disclosure. Summary of the Invention

[0004] This invention, by setting up a generative model unit for parsing video data acquired by monitoring devices, can analyze and identify operations and product status during the manufacturing process in real time, and automatically detect errors or anomalies in production. The generated work logs are further generated as non-fungible tokens (NFTs), which contain production information and country of origin information, and are provided to users through a dedicated unit. In this way, users can verify detailed information throughout the entire product lifecycle at any time, achieving transparency and reliable traceability of the entire production process, thereby effectively improving product quality management and consumer trust.

[0005] "Generative model unit" refers to a processing module that uses technologies such as artificial intelligence or deep learning to analyze and identify video data obtained from monitoring devices, thereby extracting operational details in the manufacturing process or detecting anomalies.

[0006] "Work log" refers to a detailed operation record automatically generated based on video data analysis results during the product manufacturing process, including each step of the process, operation information, and abnormal situations.

[0007] "Non-fungible tokens" refer to non-interchangeable electronic identifiers and certificates generated by digitizing and uniquely representing manufacturing-related information such as work logs based on blockchain technology.

[0008] "Origin information" refers to various types of information that can identify and trace the origin of a product, including data such as the name of the company, factory, farm, and geographical location.

[0009] "Manufacturing process information" refers to information that describes the details of each process, operation, inspection record, and abnormal handling throughout the entire process of a product's production, from raw material input to finished product delivery.

[0010] "Monitoring device" refers to video acquisition hardware or sensing equipment installed on the manufacturing site to capture and transmit images or video data of the production site in real time.

[0011] "User" refers to a consumer or user who obtains non-fungible tokens through this system and can access product-related information. Attached Figure Description

[0012] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0013] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0014] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0015] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0016] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0017] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0018] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0019] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0020] Figure 9 This represents an emotion map that maps multiple emotions.

[0021] Figure 10 This represents an emotion map that maps multiple emotions.

[0022] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0023] Figure 12This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0024] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0025] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0026] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0027] First, let me explain the terminology used in the following instructions.

[0028] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0029] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0030] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0031] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0033] First Implementation Method

[0034] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0035] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0036] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0038] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0041] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0043] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0044] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0045] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0046] Example 1

[0047] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0048] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0049] In this invention, the server includes a preprocessing device for image quality correction and noise removal of time-series image information acquired from a monitoring device; a parsing device for determining the work status and object status of the preprocessed image information using a generative artificial intelligence model; a recording device for generating structured work records based on the determination results and storing them in a management information storage device; and a device for publishing the work records as non-homogeneous electronic proof information via distributed ledger technology and making them available to information users. This enables automated, intelligent, and fully transparent management of the manufacturing process, effectively improving the real-time performance and accuracy of anomaly detection, while ensuring data traceability throughout the product lifecycle and enhancing consumer trust in the product.

[0050] "Monitoring information acquisition device" refers to hardware equipment that can collect real-time image information related to the production environment and objects during the manufacturing process, such as cameras and visual sensors.

[0051] "Temporal image information" refers to visual image data acquired continuously in chronological order, used to reflect the implementation status of each stage of the manufacturing process.

[0052] "Image quality correction" refers to the automatic adjustment of parameters such as brightness, contrast, and color of the acquired raw image data to improve the accuracy of subsequent analysis.

[0053] "Noise removal" refers to a data processing method that filters out interfering signals or useless information in an image to improve image clarity and analysis efficiency.

[0054] "Generative artificial intelligence models" refer to artificial intelligence technologies based on deep learning algorithms that can automatically analyze, identify, and judge large-scale data, including but not limited to neural network models.

[0055] "Operation status determination" refers to the process of judging and determining whether on-site operations conform to standard operating procedures.

[0056] "Object status determination" refers to the process of automatically identifying and judging the status parameters of monitored objects such as products and parts on the production line.

[0057] "Structured data format" refers to saving system analysis results in a predefined data structure (such as JSON, XML, etc.), making information easy to store, retrieve, and transmit.

[0058] "Management information storage device" refers to data storage equipment used to store and manage various manufacturing process record information, including physical hard drives, database servers, etc.

[0059] "Distributed ledger technology" refers to the technological foundation for recording, sharing, and tracking information using decentralized electronic data structures, including implementation methods such as blockchain.

[0060] "Non-homogeneous electronic certification information" refers to electronic data certificates generated based on distributed ledger technology that uniquely identify the product manufacturing process or status, and are immutable and uniquely corresponding.

[0061] "Information users" refers to end users who obtain product manufacturing and quality information through the system, including consumers, partners, and managers.

[0062] This invention relates to a production process monitoring and data traceability system based on a combination of generative artificial intelligence models and distributed ledger technology. The system includes a server, terminals, and user-side devices, and achieves visualized, intelligent, automated recording and reliable traceability of the entire manufacturing process of food and other products through the following methods.

[0063] The server connects to multiple monitoring information acquisition devices (such as industrial IP cameras or high-resolution visual sensors) via a network interface to achieve real-time multi-point video acquisition of the production site. The server first stores all the acquired time-series image information in local or distributed storage devices, and then calls automated modules based on image processing software such as OpenCV to preprocess the raw video data (such as noise reduction, brightness and color correction) to ensure the accuracy and stability of subsequent artificial intelligence recognition.

[0064] After image preprocessing, the server inputs video frame data chronologically into a generative artificial intelligence model deployed on the server. This model uses mainstream deep learning frameworks such as TensorFlow or PyTorch to automatically determine whether worker behavior conforms to standard operating procedures based on large-scale training data, and monitors the status of production objects (such as product components, packaging processes, etc.). When the model determines that a process has deviations, product defects, or other non-conforming events, the server automatically extracts information such as the time of occurrence, content, and responsible parties, records it promptly in a structured data format (e.g., JSON), and securely stores all records on the database server.

[0065] The server integrates the work process records corresponding to each production batch and uses distributed ledger technology (such as blockchain) to generate a unique non-fungible electronic certificate (NFT certificate) for each piece of production process data. This certificate is then stored on the blockchain via smart contracts, ensuring its immutability and uniqueness. This electronic certificate contains key fields such as product origin attributes, the complete production process, video images of key time points, and various quality inspection records, facilitating subsequent traceability and supervision.

[0066] The terminal is primarily used by factory managers or operators. It receives processed video streams pushed in real-time from the server via wired or wireless networks and, in conjunction with mainstream streaming media playback software such as VLC Media Player, displays the actual production line footage in real time. Once the server detects an anomaly, the terminal automatically receives a notification and sends an alarm to operators and managers via pop-up windows or message alerts. Operators and managers can directly click to view the video playback of the anomaly, assisting in rapid on-site response.

[0067] Users, i.e., end consumers, scan the QR code on the product packaging using an application on their mobile device (such as a smartphone) after receiving the product. This automatically retrieves and displays the corresponding electronic verification information from a server or blockchain. Users can intuitively view all details of the product's production, distribution, inspection, and any anomaly records, effectively enhancing their trust in the product.

[0068] This invention can be specifically applied to various manufacturing and logistics industries, including food, pharmaceuticals, electronic components, and high-end agricultural products. For example, in the production of Hami melons, video surveillance and AI analysis of the entire process from harvesting to packaging, inspection, and delivery are recorded, extracted, and uploaded to the blockchain. End users can easily check the Hami melon's production farm, harvesting time, compliance of personnel operations, and test results by simply scanning a code, achieving full transparency throughout the process.

[0069] Generative artificial intelligence models can be used in conjunction with the following prompts:

[0070] "Please check for any abnormal operations in the camera video during this period and output a detailed log."

[0071] "This displays detailed records of every process and quality inspection for this batch of products."

[0072] "Please show the product traceability information, related videos, and test results to the end user."

[0073] This system enables high automation and transparency in the production process while ensuring production safety and high quality. It supports enterprises in strengthening brand credibility and helps consumers choose and use products with peace of mind.

[0074] use Figure 11 The processing flow is explained.

[0075] Step 1:

[0076] The server receives real-time time-series image information from the monitoring information acquisition device as input data. The server temporarily stores this raw image data in local storage and assigns a timestamp to each frame. Based on this, the server performs acquisition status checks on each frame to ensure smooth and complete data transmission. The output is a sequence of raw image data with timestamps.

[0077] Step 2:

[0078] The server preprocesses the raw image data obtained in step 1. The input is a sequence of raw images with timestamps. The server uses image processing libraries such as OpenCV to automatically correct brightness and contrast, adjust color balance, and remove noise to facilitate subsequent analysis. At this stage, the server also performs image size normalization. The output is a sequence of image data that has been improved in quality and standardized in format.

[0079] Step 3:

[0080] The server takes preprocessed image data sequences as input and feeds the data frame by frame into a generative artificial intelligence model deployed on the server. The server calls models in deep learning frameworks such as TensorFlow or PyTorch to determine operator actions, the state of on-site items, etc. Based on the model's recognition results, the server automatically screens for product defects, process anomalies, and non-standard operations, and provides detailed annotations for each detected anomaly. The output is analysis result data marked with normal and abnormal states.

[0081] Step 4:

[0082] The server processes the detected anomalies and operational procedures according to the information output, organizing them into structured data. The input is analysis results data with discriminant labels. For each anomaly, the server extracts key information such as occurrence time, involved personnel, anomaly type, and recommended measures, recording it as a structured log (e.g., JSON) and storing it in a management database. The output is a structured operational process log and anomaly records.

[0083] Step 5:

[0084] The server collects all structured log information related to a single batch of products as input and generates non-fungible electronic proof information (NFT) based on distributed ledger technology. The server calls the blockchain platform's API to upload the constructed structured production information, inspection data, key images, and traceability information to the blockchain via smart contracts. The output is a unique, traceable electronic certificate address and content.

[0085] Step 6:

[0086] The terminal receives real-time video data from the server and displays it. The input is streaming video data pushed by the server, and the terminal runs media players such as VLC Media Player to display real-time images of the production line. If the server detects an anomaly, the terminal will also immediately send an anomaly alert via a pop-up window. The output is an operable video screen and an anomaly message interface.

[0087] Step 7:

[0088] Users scan the QR code on the product packaging using their mobile devices, using the QR code information as input to access a server or blockchain server. The server, upon request, returns the product's NFT certificate and detailed manufacturing, inspection, and anomaly log information. The mobile application integrates this information and displays it to the user, ultimately outputting a visual, transparent, and reliable product information interface.

[0089] Application Example 1

[0090] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0091] In modern manufacturing and production sites, efficient product quality and safety management remains a crucial issue. However, existing systems struggle to achieve real-time monitoring and automatic anomaly detection in the production process, lack sufficient data traceability and tamper-proofing, and make it difficult for users to obtain transparent and reliable product production information. Furthermore, existing systems typically lack timely notification mechanisms for operational anomalies and the ability to automatically identify and utilize user feedback and emotions. Therefore, there is an urgent need for a new system that integrates automated monitoring, intelligent analysis, tamper-proof data traceability, and enhanced user experience to improve factory management efficiency and increase user trust and satisfaction.

[0092] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0093] In this invention, the server includes a device for preprocessing information data from an information acquisition device, a device for inputting the preprocessed information data into a generative artificial intelligence model and parsing it, a device for automatically generating time-series job record data based on the parsing results, a device for generating digital data with irreversible identifiers from the job record data and related information, a device for recording the digital data with irreversible identifiers into a distributed ledger, a device for transferring ownership of the digital data with irreversible identifiers recorded in the distributed ledger to the user, a device for generating and managing input prompt statements for the generative artificial intelligence model, a device for sending notification information to a display terminal based on the parsing results or the generated digital data with irreversible identifiers when an anomaly is detected, and a device for identifying and evaluating emotional information from the user. This enables efficient acquisition and intelligent analysis of real-time production data, automatic generation and tamper-proof detailed job history uploaded to the blockchain, and closed-loop management of anomaly detection, alerts, and user feedback, greatly improving factory management efficiency and information transparency, enhancing product traceability reliability, and increasing user participation and satisfaction.

[0094] "Information acquisition device" refers to hardware equipment used to acquire information data from the target environment in real time, including but not limited to cameras, sensors, scanners, etc.

[0095] "Preprocessing" refers to the process of denoising, format conversion, and feature enhancement of raw information data to facilitate subsequent analysis and optimization.

[0096] "Generative artificial intelligence models" refer to mathematical models that are based on deep learning or machine learning techniques and can intelligently analyze and automatically extract features from input data.

[0097] "Analysis" refers to the process of automatically analyzing, identifying, classifying, or detecting anomalies in input data.

[0098] "Time-sequenced operation record data" refers to a data set that records the production process, operation details, and event information in chronological order.

[0099] "Irreversible identifier" refers to a unique digital identity that cannot be tampered with or copied once it is generated.

[0100] "Digital data" refers to information data that is stored, transmitted, and processed electronically, including text, images, audio, or other formats.

[0101] "Distributed ledger" refers to a data ledger system implemented through a distributed network. Its characteristics include transparency, decentralization, and immutability of data, including but not limited to blockchain.

[0102] "Transfer of rights" refers to the act of formally transferring ownership, access rights, etc. of data or digital assets from one entity to another.

[0103] "Input prompt statements" refer to the instruction text or statements that guide generative artificial intelligence models to perform specific tasks or analyses.

[0104] "Notification information" refers to data content used to promptly convey events, alarms, prompts, and other information to terminals or users.

[0105] "Display terminal" refers to human-computer interaction devices used to display system output information, including computer terminals, tablets, mobile phones, or dedicated monitoring terminals.

[0106] "Emotional information" refers to data that reflects a user's current psychological state or attitude, including forms such as facial expressions, voice, and text feedback.

[0107] This invention can be implemented in the following ways.

[0108] The servers are deployed in the local area network environment of the manufacturing plant and include hardware resources such as high-performance processors (e.g., general-purpose server platforms), storage devices (e.g., solid-state drives or hard disk drives), and network interfaces. The servers come pre-installed with operating systems (e.g., Linux), data acquisition middleware (e.g., video streaming services), artificial intelligence platform software (e.g., TensorFlow, PyTorch), and blockchain interaction modules (e.g., web3.py).

[0109] The server first connects to the data acquisition device (surveillance cameras, sensors, etc.) via a network interface to receive data in real time. For example, the server can periodically pull video streams from cameras using the ONVIF or RTSP protocol, and all video data is temporarily stored on local storage devices. Then, the server uses open-source image processing software such as OpenCV to preprocess the video data, including noise removal, grayscale conversion, and edge enhancement, ensuring the data meets the requirements for subsequent analysis.

[0110] The preprocessed data is fed into generative artificial intelligence models, such as YOLO and ResNet models trained on TensorFlow. These AI models can automatically identify work behaviors in videos, recognize anomalies (such as personnel not wearing protective equipment or product surface defects), and output analytical results. The server connects to the deep learning models through service interfaces such as TensorFlowServing.

[0111] The parsed results are automatically organized into time-series job record data. The server writes information such as exception frames, event content, and occurrence time into a database (e.g., MySQL, PostgreSQL). Records for each batch of products or each key operation node are organized and archived. The server further calls a blockchain API (e.g., web3.py) to hash the job data and product-related information (e.g., product codes, quality inspection results), and uses this as metadata to generate digital data with irreversible identifiers. For example, the server can package this data into immutable NFTs via a contract API and save it on the blockchain to a distributed ledger (e.g., Ethereum, public blockchains).

[0112] When a product leaves the warehouse or is delivered to a user, the server transfers ownership of the corresponding NFT to the user's blockchain account via a smart contract mechanism. Users can access and view the NFT through mobile devices (smartphones, tablets) or a blockchain wallet on a PC (such as MetaMask). The server also centrally manages the "prompt statements" input to the AI ​​model to optimize data acquisition and model inference efficiency. For example, it might generate or invoke a prompt statement like: "Please preprocess the raw video stream captured by the above factory cameras (denoising, grayscale conversion, edge enhancement), then use a deep learning model to automatically detect all abnormal behaviors on the production line and output the time, location, and operator number of the abnormality."

[0113] When the AI ​​model detects abnormal behavior, the server can automatically send notifications to display terminals (such as operator tablets and management consoles) via push technologies like WebSocket, MQTT, or Firebase CloudMessaging. The notifications include the type of abnormality, the time of occurrence, and suggested handling measures, prompting on-site personnel to take timely action. Simultaneously, the server is also equipped with an emotion recognition module that can receive emotional information from users (such as facial expressions captured by the app, satisfaction ratings, etc.). After collection, it automatically assesses user satisfaction through an emotion recognition API (such as an open platform API) to optimize subsequent service experiences.

[0114] For example, the terminal can be a tablet, mobile phone, PC, etc., running a custom factory monitoring app. Administrators can view the production line status in real time through the app and take immediate action when they receive abnormal notifications. After purchasing a product, users receive the NFT through a related app or wallet and can access detailed product traceability information and production process data at any time. Satisfaction ratings and facial expression data generated by users while viewing the NFT can also serve as a basis for subsequent product and service improvements.

[0115] An example instruction for a generative artificial intelligence model is as follows: "Analyze the following factory monitoring video stream data, identify abnormal behaviors in the operation process, and output the timestamps and descriptions of the detected abnormal segments in real time. Output all detected abnormal results in text form and register them on the blockchain in the form of NFTs."

[0116] Through the above structure and implementation methods, the present invention can realize real-time production monitoring, automatic anomaly detection, traceability of the operation process, reliable on-chain storage of product information, and can comprehensively utilize user feedback and sentiment data to improve the overall production and service quality, and has a high degree of information transparency and data security.

[0117] use Figure 12 The processing flow is explained.

[0118] Step 1:

[0119] The server receives raw data in real time from information acquisition devices (such as surveillance cameras and sensors). The input is the raw video stream data generated by the camera. The server performs preliminary storage and formatting of the input data. The output is a temporarily stored raw video data file. Specific actions include using network protocols (such as RTSP) to pull the video stream and saving it as an MP4 or H.264 format file.

[0120] Step 2:

[0121] The server preprocesses the temporarily stored raw video data. The input is the raw video data file. The server uses image processing software (such as OpenCV) to perform steps such as noise reduction, grayscale conversion, edge enhancement, and frame segmentation to enhance image features, thereby improving the accuracy of subsequent analysis. The output is a sequence of preprocessed image frames. The data processing includes Gaussian filtering, color space transformation, and edge detection using the Canny algorithm for each frame.

[0122] Step 3:

[0123] The server inputs preprocessed image frames into a generative artificial intelligence model (such as a model trained with TensorFlow) for intelligent analysis. The input is a sequence of preprocessed image frames. The server calls the model inference interface to perform feature extraction and classification operations on the data, enabling automatic detection of worker behavior, product status, and anomalies. The output is an analysis result containing classification results, anomaly types, detection timestamps, etc.

[0124] Step 4:

[0125] Based on the analysis results output by the AI ​​model, the server automatically generates job record data arranged chronologically. Inputs include the model's anomaly detection results and process-related metadata. Data processing involves organizing the content of abnormal events, time, and workstation association information to form a structured log. Outputs are detailed job history data entries, stored in the database. Specifically, this involves writing database records containing: anomaly content, time of occurrence, operator number, etc.

[0126] Step 5:

[0127] The server integrates job history data with product information, quality inspection records, etc., and generates digital data with irreversible identifiers through hash calculation. The input is the job history and related data. The server calls the blockchain API to upload this data to the blockchain, forging unique digital tokens (NFTs). The output is the NFT address recorded in the distributed ledger and its metadata.

[0128] Step 6:

[0129] When the server detects an anomaly, it generates a notification and sends it to the terminal in real time via WebSocket or push service. The input is the analyzed anomaly event data. The output is the anomaly notification content received by the terminal. Specifically, the server sends an alarm to the specified terminal app, such as a pop-up window or sound alert.

[0130] Step 7:

[0131] The terminal receives and displays notification information from the server, while simultaneously presenting relevant videos and anomaly handling suggestions to operators or managers in real time. Input consists of notification messages and video streams pushed by the server. The terminal decodes, displays, and provides feedback on the received data. Output includes real-time images, anomaly warnings, and operational suggestions on the terminal interface.

[0132] Step 8:

[0133] During the product purchase process, users receive the corresponding NFT information via smart devices or PCs. The input is the NFT ownership transfer transaction data. Users can view and verify the NFT through an app or web wallet. The output is user-accessible product history information and blockchain data proof.

[0134] Step 9:

[0135] When users browse product NFT information, the app automatically collects emotional information such as user facial expressions and ratings. Inputs include user interaction feedback and data captured by the camera. The data is encrypted and uploaded to the server, where it is analyzed by an emotion recognition engine. Outputs include the user's emotional evaluation and satisfaction rating.

[0136] Step 10:

[0137] The server analyzes all users' emotional feedback to improve subsequent service experience and product quality. Input is user emotional data. The server performs statistical analysis on big data, generating improvement suggestions and evaluation reports. Output includes actionable management recommendations and product optimization strategies.

[0138] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0139] Example 2

[0140] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0141] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0142] In this invention, the server includes: an information parsing device for parsing time-series data from a monitoring information acquisition device using a generative artificial intelligence model; a device for generating time-series record data containing operation information, process information, and detection result information based on the parsing results of the information parsing device; a device for generating the time-series record data as a unique digital asset into a non-homogeneous recording medium; a device for providing information about the non-homogeneous recording medium to the user; a sentiment analysis device for acquiring and parsing the user's emotion-related information and generating user evaluation data; and a device for providing additional information or reward information based on the evaluation data. This enables the full digitization and tamper-proof recording of production process information, and allows for personalized optimization of product information services based on user feedback, thereby improving information transparency, product reliability, and user satisfaction in the manufacturing process.

[0143] "Monitoring information acquisition device" refers to information acquisition equipment used to acquire on-site images, videos, sounds or other sensor data in real time during the manufacturing or service process.

[0144] "Time series data" refers to a collection of data arranged in chronological order that reflects the dynamic changes of a process, including image sequences, sensor records, operation logs, etc.

[0145] "Generative artificial intelligence models" refer to artificial intelligence models built using technologies such as machine learning and deep learning, which are capable of analyzing input data and generating features. Typical examples include deep neural networks and generative adversarial networks.

[0146] "Information analysis device" refers to an electronic or software device that uses generative artificial intelligence models and other methods to process collected time-series data by identification, detection, classification, and analysis to obtain target information.

[0147] "Time-series data" refers to the time sequence data recorded based on data analysis results, which includes operations, procedures, test results, and related events.

[0148] "Non-fungible recording media" refers to a recording medium that is unique, irreplaceable, and digitally storable, such as non-fungible tokens (NFTs) on the blockchain, used to identify unique objects and their associated information.

[0149] "User" refers to the person or entity that ultimately obtains the non-homogeneous recording medium and can query detailed information about the product or service, such as users, consumers, and customers.

[0150] "Emotion-related information" refers to feedback or comment information related to the user's subjective emotions, feelings, satisfaction, and other psychological states, which can be expressed in the form of text, ratings, voice, etc.

[0151] "Sentiment analysis device" refers to a software or hardware device that uses natural language processing or machine learning algorithms to calculate and analyze emotion-related information in order to evaluate the user's emotional state and satisfaction.

[0152] "Evaluation data" refers to the data results obtained after processing emotion-related information through emotion analysis devices, reflecting the user's subjective judgment, satisfaction, or emotional state.

[0153] "Additional information" refers to the rich information provided to users based on evaluation data, such as product descriptions, technical parameters, and special services.

[0154] "Reward information" refers to information that provides incentives such as promotions, points, coupons, and gifts to users based on their evaluation data.

[0155] To facilitate the implementation of this invention, the following will describe in detail the system implementation methods involved in this invention, in conjunction with the system structure, hardware devices, software tools, and operation procedures.

[0156] The system of this invention mainly includes a server, a terminal, and user-side devices. The server is responsible for core functions such as data collection, information parsing, data storage, token generation, sentiment analysis, and information distribution. The terminal is used for information display and anomaly alerts. The user devices are used for product purchase, acquisition of non-homogeneous recording media, information query, and sentiment feedback.

[0157] In terms of hardware, the system can utilize various general-purpose computers, network servers, surveillance cameras, smartphones, and display devices. For example, a high-performance general-purpose server with sufficient CPU and GPU resources is recommended to meet the inference needs of the artificial intelligence model. Surveillance cameras are installed at key processes in the production site to capture real-time footage of the production process and are connected to the server via a local area network (LAN) or wide area network (WAN).

[0158] The server receives time-series video data captured by surveillance cameras. It can use the RTSP protocol to acquire images or video streams from the cameras in real time, temporarily storing the video data in a cloud storage platform (such as a general cloud storage service). Subsequently, the server calls a data processing script written in Python, combined with general open-source libraries such as OpenCV, to perform preprocessing operations on the raw video, including noise reduction and frame extraction.

[0159] After data preprocessing, the server feeds the processed time-series data into a generative artificial intelligence model built on deep learning technology. This model can be trained and deployed using mainstream open-source frameworks such as TensorFlow or PyTorch. The AI ​​model analyzes the image sequences and can automatically detect worker operations, product status, process anomalies, and potential defects in production.

[0160] Based on the model analysis results, the server automatically organizes time-series record data containing key information such as operations, procedures, and inspections, and stores it using a NoSQL database (such as a common distributed database). This record data is used for subsequent traceability, verification, and data asset generation.

[0161] The server further utilizes Python's Web3 library to generate non-fungible recording media (such as NFTs on the blockchain) from time-series recorded data in the form of unique and tamper-proof digital assets. The server implements contract calls to register and write relevant attribute information, process details, and test results into a general blockchain platform, establishing a unique correspondence between them and the product.

[0162] When a user purchases a product through a smartphone app or website, the server transfers ownership of the corresponding NFT to the user's wallet address. Users can then query information via a blockchain explorer (such as a general blockchain visualization platform) or wallet applications (such as blockchain wallet plugins) to view complete product traceability, process details, and quality inspection data.

[0163] Users can submit feedback or reviews about the product through the app or online platform. The server uses NLP libraries (such as spaCy and transformers) to perform sentiment analysis on the natural language of user feedback and automatically assess user satisfaction. If the server determines that the satisfaction level is high, it can automatically push more product information, technical materials, or issue incentives such as coupons; if the satisfaction level is low, the server will collect user opinions and provide feedback to the production or management departments for improvement.

[0164] Terminal devices, serving as auxiliary tools for production managers or operators, can consist of PCs, tablets, or customized displays. Using real-time communication protocols such as WebSocket, the terminal displays real-time video streams transmitted from the server and issues audible alarms or illuminates warning lights in real time when anomalies occur during production, prompting operators to address the issues promptly.

[0165] The system of this invention uses general-purpose open-source and commercial software components, which facilitates batch deployment, maintenance and upgrades. Specific implementation examples:

[0167] For example, on an automated packaging line in a food factory, several surveillance cameras are installed to capture key processes such as dispensing, weighing, and sealing. A server receives the video streams from the cameras in real time via the network. First, it uses OpenCV for noise filtering and keyframe extraction. Then, an AI model built using the PyTorch framework analyzes these frames, automatically recording process steps and any anomalies detected. The generated production records are registered on the blockchain as digital assets. Users receive a unique NFT after purchasing the product and can query detailed information through a blockchain explorer. If a user rates the product as "fresh" and "transparent" in the app, and the server's analysis indicates positive sentiment, a discount coupon can be automatically issued.

[0168] Example of prompts for generative artificial intelligence models:

[0169] Please design a system that can analyze video data from cameras in a factory, automatically record key operations, and monitor abnormal events.

[0170] How can we use Python and deep learning frameworks to achieve intelligent detection and log generation of production process videos?

[0171] Please provide an example of how to generate unique digital credentials from operation logs using blockchain and make them available for customer query.

[0172] use Figure 13 The processing flow is explained.

[0173] Step 1:

[0174] The server receives video data from surveillance cameras in real time. The input is the raw video stream sent by the surveillance cameras via the RTSP protocol, and the output is the raw video file stored locally on the server or in cloud storage. The server periodically splits the stream into individual video segments every 5 minutes and backs them up to cloud storage.

[0175] Step 2:

[0176] The server preprocesses the stored video data. The input is the original video file, and the output is the denoised and segmented image data. The server calls Python and OpenCV to read the video file frame by frame, remove noise through Gaussian filtering and other methods, extract keyframes from the video, and save each frame as an image file.

[0177] Step 3:

[0178] The server uses a generative artificial intelligence model to analyze image frames. The input is a processed keyframe image, and the output is the process detection results and anomaly identification information. The server batches image frames into a deep learning model trained on TensorFlow or PyTorch to determine the operator's actions, equipment status, and potential anomalies in each frame, such as whether the worker is wearing protective gloves or whether the equipment is operating abnormally.

[0179] Step 4:

[0180] The server generates job logs based on the model analysis results. The input is the process and anomaly detection data output from step 3, and the output is detailed job log text. The server automatically organizes the operation time, detection results, and anomaly content for each process, generates standardized logs according to the timeline, and writes them to a NoSQL database (such as MongoDB).

[0181] Step 5:

[0182] The server generates non-fungible tokens (NFTs) based on the job logs. The input is the job log data, and the output is the NFT registration information on the blockchain. The server calls a Web3 library to mint unique data containing information such as production traceability, origin, and inspection reports into NFTs according to digital asset specifications, and registers them on the blockchain platform via smart contracts.

[0183] Step 6:

[0184] The terminal receives video data and abnormal alarms from the server and displays them in real time. Input consists of the video stream and abnormal messages pushed by the server; output is the real-time image and warning prompts on the terminal screen. The terminal uses WebSocket to receive data and VLC player to display the image. Upon encountering an abnormal signal from the server, it automatically activates sound or light alarms.

[0185] Step 7:

[0186] When a user acquires a product through a smart device, they receive the corresponding NFT. The input consists of an NFT identifier generated by the server and user information; the output is the NFT linked to the user's blockchain wallet. When purchasing a product, the user completes identity verification via a mobile app or website, and the server automatically transfers NFT ownership to the user's wallet.

[0187] Step 8:

[0188] Users can query NFTs and view detailed information. The input is the NFT identifier or the user's wallet address, and the output is product traceability details displayed on the user's device. Using a blockchain explorer or wallet tool, users can enter the product's serial number to query information such as the production process, manufacturing details, and quality inspection data.

[0189] Step 9:

[0190] Users submit product reviews, and the server performs sentiment analysis. The input is the user's text review or feedback, and the output is a satisfaction score and personalized service. After a user comments in the app, the server uses an NLP sentiment analysis model to process the data, assess the sentiment tone, and push rewards or information to the user based on the analysis results, while also collecting suggestions for improvement.

[0191] Application Example 2

[0192] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0193] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0194] In this invention, the server includes a generative information processing device for parsing video data acquired by monitoring equipment, an information recording device for recording process information based on the parsing results of the generative information processing device, an information asset generation device for generating unique identification information assets based on the process information, an information providing device for providing the unique identification information assets to users, an emotional state recognition device for inferring emotional information, an evaluation value calculation device for calculating the user's evaluation value based on the emotional information acquired by the emotional state recognition device, and an assignment device for assigning additional information or special features based on the evaluation value. This enables high-precision acquisition, real-time analysis, blockchain traceability, and unique identifier generation of process data. Combined with user emotional states, it dynamically assesses user satisfaction and provides automatic feedback, thereby significantly improving product transparency, user trust, and the company's service responsiveness.

[0195] "Monitoring equipment" refers to various monitoring camera devices used to collect real-time image data of the environment or objects in production sites, distribution venues, or terminal environments.

[0196] "Generative information processing device" refers to a computer processing module that can automatically identify, extract features, and detect anomalies in collected video data based on artificial intelligence or deep learning models.

[0197] "Process information" refers to the collection of data related to each specific work step, its corresponding operation content, time, participating personnel, and test results during the product manufacturing or processing process.

[0198] "Information recording device" refers to a data processing and storage module that can store and manage process information and other related data in a specified format.

[0199] "Unique identification information assets" refer to digital unique identifiers or data assets generated by uniquely binding attributes such as process information of a specific product or object using distributed ledger technologies such as blockchain.

[0200] "Information asset generation device" refers to a computer device that generates unique identification information assets based on process information and other relevant data, and realizes registration, authentication and management.

[0201] "Information providing device" refers to an information output module used to electronically push or display the content of unique identification information assets or their access methods to end users.

[0202] "Emotional state recognition device" refers to an artificial intelligence processing module that automatically infers a user's current emotional state by analyzing multimodal data such as facial expressions, actions, and voice.

[0203] "Evaluation value calculation device" refers to a data processing device that can score or quantify user satisfaction, approval, etc., based on emotional information obtained from the emotional state recognition device.

[0204] "Awarding device" refers to the reward processing module that distributes additional information or benefits such as points, coupons, and customized push notifications to users based on the evaluation results.

[0205] To better understand the present invention, the embodiments of the present invention will be described in detail below with reference to specific hardware and software examples.

[0206] This invention relates to a comprehensive information processing system applicable to manufacturing, distribution, and service scenarios. The system mainly includes a server, terminals, and intelligent devices for user use. The server employs high-performance computing hardware, such as servers equipped with graphics processing units (GPUs) and large-capacity solid-state drives; monitoring equipment can include network cameras, industrial cameras, etc. The server integrates an operating system (such as Linux), an efficient data storage system, computer vision software (such as OpenCV), artificial intelligence deep learning frameworks (such as TensorFlow and Keras), a blockchain access module, and sentiment analysis APIs (such as facial expression analysis APIs).

[0207] The server is responsible for receiving and storing real-time video data collected by the monitoring equipment, and performing a series of data processing steps. First, the server receives video streams from various monitoring devices over the network and saves the data to its local storage system. Then, the server uses image processing software such as OpenCV to preprocess the video content, including noise reduction, enhancement, and still frame segmentation. The preprocessed images are then subjected to feature extraction and automatic parsing by an integrated generative artificial intelligence model (built on deep learning frameworks such as TensorFlow and Keras) to identify key information such as personnel actions, production status, or on-site anomalies.

[0208] The above analysis results are used to generate a work log, which records in detail each operation step, time, and related results. Then, the server calls the blockchain module to package the work log, process information, and other data to generate a unique identification information asset (i.e., a non-fungible digital identifier), and registers it through blockchain technology to achieve immutability and traceability.

[0209] After purchasing products or services, users can receive unique identification information assets pushed by the server through smart terminals (such as smartphones, tablets, personal computers, etc.), and access all traceability details of the information assets through relevant applications or web browsers, including the origin of the item, production process history, testing process, and qualification status, thereby enhancing users' trust in products and services.

[0210] This invention also integrates emotion state recognition functionality. When a user views a digital icon using a mobile application or computer, the terminal device can utilize its front-facing camera to capture the user's facial expressions in real time. The server analyzes the user's current emotional state using emotion recognition APIs (such as Affectiva, emotion recognition cloud services, etc.) and automatically calculates a satisfaction score accordingly. If the user is satisfied, a high score will trigger the server to automatically push additional services such as points, coupons, and secondary marketing; if dissatisfaction is detected, the server will automatically push questionnaires and feedback portals to collect user opinions.

[0211] This invention can be widely applied to manufacturing enterprises, logistics enterprises, retail industries, and even high-end service industries to achieve integrated intelligent management of production processes, product traceability, and user satisfaction feedback.

[0212] Specific examples:

[0213] When a user purchases an electronic product, the server automatically generates a unique identification information asset and pushes it to the user. After scanning the QR code with their mobile phone, the user can not only view the entire production process and key milestones, but also capture facial expressions through the phone's front-facing camera. If the server determines the user is "happy" after emotion analysis, the system immediately pushes a personalized points reward; if the analysis result is "confused," a pop-up window automatically invites the user to fill in improvement suggestions to enhance after-sales service and customer experience.

[0214] Example of prompts for generative artificial intelligence models:

[0215] Please analyze the video frames captured by the camera, detect the operator's actions in the manufacturing process, automatically identify the type of abnormality or error, and provide the detection type and the time of occurrence.

[0216] Please perform an emotion (e.g., happiness, confusion, dissatisfaction) analysis on the facial photos captured by the camera when users browse digital icons, and return the analysis probability and results description.

[0217] use Figure 14 The processing flow is explained.

[0218] Step 1:

[0219] The server receives video data in real time from surveillance equipment deployed within the factory or store. The input is the raw video stream captured by the surveillance cameras. The server acquires the data via network protocols (such as RTSP) and stores the video stream locally. Specifically, the server checks the camera connection status, establishes a data channel, and receives the video data.

[0220] Step 2:

[0221] The server preprocesses the received raw video data. The input is the stored raw video stream, and the output is preprocessed image frames. The server uses image processing tools such as OpenCV to perform operations such as noise reduction, brightness equalization, and frame extraction. The server processes the video data frame by frame, saving each frame as an independent image for subsequent analysis by artificial intelligence models.

[0222] Step 3:

[0223] The server inputs the processed image frames into a generative artificial intelligence model for analysis. The input is the pre-processed image frames, and the output is a list of detected specific action types and abnormal events. The server calls a model trained on a deep learning framework (such as TensorFlow or Keras) to perform action recognition and anomaly detection on the images. For example, the model analysis might result in "Operator not wearing a safety helmet, probability 0.92".

[0224] Step 4:

[0225] The server generates and records job logs based on the analysis results of the artificial intelligence model. The input is the action type and exception information output by the model, and the output is a structured job log file. The server appends and saves each detection result (such as process ID, operator ID, exception type, timestamp) in real time in formats such as JSON, facilitating full traceability.

[0226] Step 5:

[0227] The server generates unique identification information assets based on work logs and process information, and writes them to the blockchain. The input is a structured work log, and the output is the on-chain digital identifier. The server packages the process flow, quality results, and traceability information, uploads it to the blockchain network using the blockchain API, and obtains a unique asset identification code.

[0228] Step 6:

[0229] The server pushes unique identification information assets to users. The input consists of an on-chain digital identifier and user identification information; the output is a blockchain access link or QR code received by the user's terminal. The server automatically sends the digital identifier information to the user's smart device via message notification or email.

[0230] Step 7:

[0231] Users can view and access digital identifiers through the terminal. The input is a digital identifier link or QR code, and the output is a terminal page displaying the entire product manufacturing process, traceability information, and detailed steps. Users can scan the QR code or click the link on a mobile app or computer to browse the detailed content of the digital asset.

[0232] Step 8:

[0233] When a user accesses a digital identifier, the server automatically initiates the emotion state recognition process. The input is a real-time facial expression image captured during the user's browsing, and the output is the emotion classification result and satisfaction score. The server calls the emotion recognition API to analyze the user's photo captured by the front-facing camera, identifying states such as happiness and confusion, and records the analysis probability and category.

[0234] Step 9:

[0235] The server distributes personalized content to users based on their satisfaction scores. The input is the satisfaction score obtained from sentiment analysis, and the output is points, coupons, or intention surveys pushed through the app via pop-up notifications. If the server detects high satisfaction, it automatically sends out rewards; if it detects dissatisfaction, it displays a feedback page to encourage continuous improvement.

[0236] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0237] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0238] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0239] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0240] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0241] Second Implementation Method

[0242] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0243] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0244] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0245] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0246] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0247] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0248] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0249] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0250] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0251] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0252] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0253] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0254] Example 1

[0255] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0256] Application Example 1

[0257] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0258] Example 2

[0259] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0260] Application Example 2

[0261] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0262] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0263] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0264] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0265] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0266] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0267] Third Implementation Method

[0268] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0269] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0270] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0271] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0272] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0273] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0274] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0275] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0276] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0277] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0278] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0279] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0280] Example 1

[0281] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0282] Application Example 1

[0283] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0284] Example 2

[0285] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0286] Application Example 2

[0287] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0288] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0289] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0290] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0291] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0292] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0293] Fourth Implementation Method

[0294] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0295] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0296] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0297] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0298] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0299] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0300] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0301] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0302] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0303] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0304] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0305] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0306] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0307] Example 1

[0308] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0309] Application Example 1

[0310] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0311] Example 2

[0312] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0313] Application Example 2

[0314] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0315] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0316] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0317] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0318] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0319] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0320] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0321] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0322] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0323] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0324] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0325] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0326] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0327] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0328] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0329] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0330] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0331] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0332] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0333] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0334] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0335] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0336] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0337] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0338] In addition, the following notes are provided in response to the above explanation.

[0339] Example 1

[0340] (Note 1)

[0341] An information processing system includes: a device for preprocessing time-series image information obtained from a monitoring information acquisition device, including image quality correction and noise removal; a device for parsing the preprocessed time-series image information, possessing generative artificial intelligence model functionality, and performing job status determination or object status determination; a device for generating structured data format job process record information based on the determination result of the generative artificial intelligence model and storing it in a management information storage device; a device for publishing the job process record information as non-homogeneous electronic proof information generated using distributed ledger technology; and a device for enabling information users to access the non-homogeneous electronic proof information.

[0342] (Note 2)

[0343] According to the information processing system described in Appendix 1, the generative artificial intelligence model has the function of automatically detecting deviations in work steps, product defects, or non-conforming events during the manufacturing process, and generating immediate warnings or notifications based on the detection results.

[0344] (Note 3)

[0345] According to the information processing system described in Appendix 1, the non-homogeneous electronic certification information includes product origin attribute information, complete manufacturing process information, quality inspection records, and image evidence information at a specific point in time.

[0346] Application Example 1

[0347] (Note 1)

[0348] An information processing system includes: a device for preprocessing information data from an information acquisition device; a device for inputting the preprocessed information data into a generative artificial intelligence model and parsing it; a device for automatically generating time-series job record data based on the parsing results; a device for generating digital data with irreversible identification from the job record data and related information; a device for recording the digital data with irreversible identification into a distributed ledger; a device for transferring ownership of the digital data with irreversible identification recorded in the distributed ledger to a user; a device for generating and managing input prompt statements for the generative artificial intelligence model; a device for sending notification information to a display terminal based on the parsing results or the generated digital data with irreversible identification when an anomaly is detected; and a device for identifying and evaluating emotional information from the user.

[0349] (Note 2)

[0350] According to the information processing system described in Appendix 1, the generative artificial intelligence model has the function of automatically detecting errors or anomalies in the process and generating job record data associated with the detected anomaly content and the time of occurrence.

[0351] (Note 3)

[0352] According to the information processing system described in Appendix 1, the digital data with irreversible identification includes information about the item's history, process, attributes, quality inspection, and user reviews.

[0353] Example 2

[0354] (Note 1)

[0355] An information processing system includes: an information parsing device for parsing time-series data from a monitoring information acquisition device using a generative artificial intelligence model; a device for generating time-series record data containing operation information, process information, and detection result information based on the parsing results of the information parsing device; a device for generating a non-homogeneous recording medium from the time-series record data as a unique digital asset; a device for providing information about the non-homogeneous recording medium to a user; a sentiment analysis device for acquiring and parsing emotion-related information of the user and generating user evaluation data; and a device for providing additional information or reward information based on the evaluation data.

[0356] (Note 2)

[0357] According to the information processing system described in Appendix 1, the information analysis device has the function of detecting operational abnormalities or malfunctions during the process.

[0358] (Note 3)

[0359] According to the information processing system described in Appendix 1, the non-homogeneous recording medium includes the object's origin attribute information, process details, and quality information.

[0360] Application Example 2

[0361] (Note 1)

[0362] An information processing system includes: a generative information processing device for parsing video data acquired by a monitoring device; an information recording device for recording process information based on the parsing results of the generative information processing device; an information asset generation device for generating unique identification information assets based on the process information; an information providing device for providing the unique identification information assets to users; an emotion state recognition device for inferring emotional information; an evaluation value calculation device for calculating the user's evaluation value based on the emotion information acquired by the emotion state recognition device; and an assignment device for assigning additional information or special features based on the evaluation value.

[0363] (Note 2)

[0364] According to the information processing system described in Appendix 1, the generative information processing device has the function of automatically extracting and detecting abnormal states or errors in the work process.

[0365] (Note 3)

[0366] According to the information processing system described in Appendix 1, the unique identification information asset includes item origin information, process history information, quality inspection result information, and related attribute information.

Claims

1. An information processing system, characterized in that, include: Generative model unit used to parse video data acquired from monitoring devices; A unit used to generate job logs based on the parsing results of the generation model unit; Unit used to generate non-fungible tokens from the job logs; and Units used to provide the non-fungible tokens to users.

2. The information processing system according to claim 1, characterized in that, The generated model unit has the function of detecting errors or anomalies in the manufacturing process.

3. The information processing system according to claim 1, characterized in that, The non-fungible token contains information about the product's country of origin and manufacturing process.

Images