Information processing system

Abstract

The invention provides an information processing system. The information processing system comprises a device for monitoring the garden boarding condition of a child entering a conservation facility through a face recognition camera; the device is used for automatically sending contact to a child guardian under the condition that the child does not climb the park within the specified time; a device for receiving the feedback of the guardian and updating the system based on the feedback; and the device is used for sending a notification to the nursing personnel when the feedback of the guardian is not received or an abnormal condition is detected.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] Existing childcare facilities and shuttle buses primarily rely on manual roll call and visual verification for managing children's arrival and boarding. This not only increases the workload of childcare staff but also increases the risk of safety hazards due to human error, such as children being forgotten, late, or not being detected in time. Furthermore, information transmission is untimely, communication between parents and caregivers is inefficient, and it is difficult to respond promptly to abnormal situations. Therefore, there is an urgent need for a system that can monitor children's arrival and boarding status in real time, automatically notify parents of any abnormalities, and improve information management efficiency to ensure children's safety. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides an information processing system comprising: monitoring children's arrival at childcare facilities via facial recognition cameras; automatically sending notifications to parents when children fail to arrive within a specified time and automatically updating system data upon receiving parental feedback; and proactively sending alerts to childcare staff if no parental response is received or an anomaly is detected. Furthermore, the system includes real-time recording of children's boarding time and behavior via cameras installed on the buses used for boarding; automatically notifying caregivers if prolonged inactivity or other abnormal situations are detected. The system integrates artificial intelligence technology to analyze the data collected by the cameras, achieving automated management and anomaly response for boarding and bus safety, thereby effectively reducing management burden and improving childcare safety levels.

[0005] A "facial recognition camera" is a specialized camera device that can collect and analyze facial images of people and automatically identify their identities by comparing them with facial features registered in a database.

[0006] "Child attendance status" refers to the status information of whether a child has arrived at the childcare facility at the designated time and completed the check-in on the day.

[0007] "Automatic communication" refers to a mechanism in which the system automatically sends notifications, reminders, or confirmation requests to parents or guardians according to set rules without requiring manual operation.

[0008] "Parent feedback" refers to the actions taken by parents or guardians after receiving system notifications, such as replying with information about their child's attendance at kindergarten or absence requests via the app or SMS.

[0009] "System update" refers to the process of modifying and synchronizing information such as children's status and time records within the system based on new data or feedback received.

[0010] "Anomaly detection" refers to the technical means by which the system automatically detects cases of missing attendance, prolonged inactivity, or other abnormal situations by analyzing children's behavioral data.

[0011] "Notifying childcare staff" refers to the action of the system sending a warning or reminder message to childcare staff through the terminal device when an abnormality is detected or when parents do not provide feedback.

[0012] "Bus camera" refers to a camera device installed inside or at the entrance of a childcare bus to film and record children boarding the bus and their behavior inside the bus.

[0013] "Boarding time" refers to the specific time when a child actually boards the childcare bus and is recorded by the system.

[0014] "Behavioral monitoring" refers to the process of continuously monitoring the actions and activities of children inside buses or childcare facilities through cameras and AI analysis technology.

[0015] "Artificial intelligence technology" refers to intelligent technologies that use computer algorithms to analyze, identify, and judge anomalies in large-scale image or behavioral data. Attached Figure Description

[0016] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0017] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0018] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0019] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0020] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0021] Figure 6This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0022] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0023] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0024] Figure 9 This represents an emotion map that maps multiple emotions.

[0025] Figure 10 This represents an emotion map that maps multiple emotions.

[0026] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0027] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0028] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0029] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0030] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0031] First, let me explain the terminology used in the following instructions.

[0032] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0033] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0034] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0035] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0036] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0037] First Implementation Method

[0038] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0039] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0040] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0041] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0042] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0043] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0044] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0045] Figure 2The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0046] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0047] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0048] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0049] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0050] Example 1

[0051] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0052] Existing venue entry and vehicle access management systems primarily rely on manual monitoring and information transmission, which is inefficient, prone to oversights, and unable to provide real-time updates on entry and safety status. Furthermore, in cases of abnormal situations or prolonged inactivity, information cannot be promptly relayed to management personnel, posing security risks. Additionally, traditional notifications rely on fixed templates, failing to adapt to specific scenarios and thus hindering personalized and intelligent notification capabilities. Therefore, achieving efficient, automated, and intelligent monitoring of venue entry and behavior, as well as abnormal notifications, has become a pressing issue.

[0053] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0054] In this invention, the server includes: acquiring image and behavioral information of people entering and boarding a venue through an image acquisition device; performing individual identification and anomaly detection using image recognition technology and machine learning algorithms; automatically filtering out people who have not confirmed their entry or have not provided timely feedback based on a set time; generating prompt statements using a generative artificial intelligence model; and automatically generating notification information based on the model's output, which is then promptly pushed to guardians or management terminals via a communication device; and automatically updating the object status in the information storage device upon receiving feedback from guardians. This significantly improves the automation and intelligence level of object entry and behavior monitoring, enables efficient notification of abnormal information, reduces manual workload, and ensures object safety.

[0055] "Image acquisition device" refers to hardware equipment that can acquire images of objects at a designated location or on a moving body, including cameras, monitoring equipment, etc., for real-time acquisition of relevant image information such as entry and vehicle travel.

[0056] Image recognition technology refers to algorithms based on artificial intelligence and computer vision that process and analyze acquired images to achieve automatic identification and differentiation of individual objects.

[0057] "Individual identification" refers to the process of determining the identity of each individual by comparing and analyzing the collected object feature information.

[0058] "Communication device" refers to electronic equipment used to send or receive notifications, information and feedback, including mobile phones, terminals, computers, servers, etc.

[0059] "Guardian" refers to the parent or legal person responsible for the care and management of a minor.

[0060] "Information storage device" refers to a data management device or database used to store data information such as identity, status and historical records related to an object.

[0061] "Management terminal" refers to the operating equipment used by managers to receive, process, and view various system notifications and monitoring information, including computers, smartphones, tablets, etc.

[0062] "Generative artificial intelligence models" refer to information processing models that use artificial intelligence methods to automatically generate response content based on input prompts, such as natural language processing generative models.

[0063] "Prompt statements" refer to the text content input into a generative artificial intelligence model, used to guide the model to generate specific types of notifications, inquiries, and other messages.

[0064] "Machine learning algorithm" refers to a computational method that enables a system to learn by training it on a large amount of data, thereby automatically identifying objects, detecting anomalies, and updating data.

[0065] "Anomaly detection" refers to the process of identifying situations that deviate from normal patterns, may pose risks, or require attention by analyzing the behavior or state data of an object.

[0066] "Behavioral information" refers to data obtained through image acquisition devices that can reflect the activity status of an object, such as actions, dwell time, and activity trajectory.

[0067] "Notification information" refers to relevant reminders or notifications that the system automatically generates and sends to guardians or management terminals based on the actual situation.

[0068] This invention relates to an object entry and behavior management system based on image acquisition, image recognition, generative artificial intelligence, and communication linkage. The following describes the implementation of this invention in conjunction with its system structure, hardware and software selection, and data processing scheme.

[0069] First, this system consists of three main parts: a server, terminals, and users. The server, as the central control unit, is responsible for data processing, storage, and message push. Terminals include image acquisition devices deployed in relevant locations and on mobile vehicles (such as school buses and commuter vehicles), such as network cameras (multiple models of high-definition cameras from mainstream manufacturers can be selected), and are equipped with image recognition software modules (such as those using deep learning image recognition models like OpenCV, TensorFlow, PyTorch, and ResNet / FaceNet). Users include administrators and guardians of the objects, who primarily interact with the system through management terminals such as smartphones, tablets, or computers, via a dedicated app or web page.

[0070] In practice, the terminal is responsible for collecting image information of people entering and boarding the vehicle in real time. The collected image data is uploaded to the server through an encrypted network channel. The server calls an image recognition algorithm to extract the feature values ​​of the objects and automatically compares them with the feature information of registered individuals in the database to automatically confirm the identity of the objects. When an object fails to have a check-in or boarding record within the specified time, the server will automatically filter out a list of abnormal individuals.

[0071] For situations requiring notifications, the server invokes a generative artificial intelligence model (such as an integrated NLP generation system like GPT or BERT) to automatically generate personalized notification content based on the specific scenario. Specifically, the server first constructs a prompt statement, inputs it into the generative AI model, and automatically generates a notification message suitable for the current situation based on the model's output text. For example, if someone fails to sign in, the server generates a prompt statement like: "Student XXX has not yet arrived. Please generate a friendly notification for the guardian inquiring about the reason." After the model returns the result, the server automatically sends the notification to the guardian via communication devices (such as SMS, email, or a dedicated app push notification).

[0072] After receiving the notification, the guardian can provide feedback on the actual situation through the mobile terminal's interactive interface (e.g., "Requesting a day off due to illness"). The server receives and parses this feedback and automatically updates the object's status data in the database. If the server does not receive a feedback response within the specified time limit, or if the terminal behavior analysis module (e.g., based on YOLO or action recognition algorithms) determines that an object has been inactive for an extended period, the server will automatically push an anomaly alert to the management system. Management personnel can then view the object's location or the vehicle's camera footage in real time to determine whether further action is necessary.

[0073] The system's hardware includes: a general-purpose server host, network storage devices, image acquisition cameras (such as network cameras), a terminal management computer, and user smart terminals. The software includes: backend data processing programs (such as Python and Node.js), image recognition and behavior analysis modules (OpenCV, TensorFlow, PyTorch, etc.), a generative artificial intelligence text generation engine (OpenAI GPT / local NLP model), a database management system (such as MySQL), and a communication push platform (such as Twilio SMS API, email server, and mobile push module).

[0074] Specific examples: In daily management, after the system is activated at 8:00 AM, all attendees must walk under the surveillance cameras. The image recognition algorithm automatically verifies their identity and completes the sign-in. If Wang is not on the sign-in list at 9:00 AM, the server automatically generates a message: "Your child Wang has not arrived today. Please inform us of the reason, thank you." If a parent reports "My child is absent today" via the app at 8:55 AM, the server updates the status in real time, requiring no further manual intervention. If the monitoring shows that a student has not been active for 20 minutes after boarding the bus, the server automatically pushes a reminder: "Student XXX has been inactive on the bus for an extended period. Please pay attention."

[0075] Examples of prompts for generative artificial intelligence models include the following:

[0076] "Student XXX has not yet arrived. Please generate a friendly notification for the guardian to inquire about the reason."

[0077] "Your child YYY did not take the bus on time today. Is there any special reason? Please let us know as soon as possible."

[0078] "Student ZZZ has been inactive in the vehicle for an extended period of time. Please generate an emergency reminder text for the administrator."

[0079] Through the integration of the aforementioned hardware and software and the collaboration of intelligent algorithms, this invention can achieve efficient and real-time object recognition, information push, anomaly detection and emergency response, greatly improving the level of management automation and intelligence.

[0080] use Figure 11 The processing flow is explained.

[0081] Step 1:

[0082] At the start of each daily management cycle, the server is activated by a user (e.g., an administrator) to launch the system application, load the entry and boarding schedules for all objects for the day, and initialize the database.

[0083] Input: Management operation instructions, object list in the database, and entry / ride plan.

[0084] Data processing / data operation: The server retrieves data from the database, filters all objects to be monitored that day, and loads the corresponding information into memory.

[0085] Output: The management terminal interface displays a complete list of objects and plans, and the system is ready to enter monitoring mode.

[0086] Specific actions: Administrators click "Start Monitoring" on the management terminal, and the server displays the complete list of devices to be monitored.

[0087] Step 2:

[0088] The terminal (image acquisition device) collects real-time image information such as the face of the person entering the venue and captures images of the person entering the venue or moving.

[0089] Input: A live stream of images captured by the camera.

[0090] Data processing / data operation: The terminal extracts key feature points through image recognition software algorithms, generates feature vectors, and automatically labels timestamps.

[0091] Output: Object feature vector and original image frame.

[0092] Specific actions: The camera detects someone entering the monitored area and automatically captures a facial image.

[0093] Step 3:

[0094] The terminal transmits the processed facial feature data to the server via an encrypted network.

[0095] Input: Object feature vector and time information output by terminal image recognition.

[0096] Data processing / data operation: Local image data is uploaded to the server's designated interface, along with a unique identifier and timestamp.

[0097] Output: The server receives a set of data on the objects to be identified via the receiving port.

[0098] Specific action: The terminal calls a secure communication protocol (such as HTTPS) to upload data to the server.

[0099] Step 4:

[0100] After receiving the feature data uploaded by the terminal, the server uses an image recognition model (such as ResNet / FaceNet) to compare the features with those of registered objects in the database to verify the identity.

[0101] Input: Feature vector of the newly acquired object, and feature vector set of registered objects in the database.

[0102] Data processing / data operation: The server compares various feature vectors to determine the identity of the current object; if the comparison is successful, entry / boarding check-in registration is performed.

[0103] Output: Check-in / boarding status data is updated to the database, and the recognition results are returned to the management terminal interface.

[0104] Specific actions: The server automatically compares and records "XX object has officially checked in" in the background.

[0105] Step 5:

[0106] The server periodically (e.g., at 9:00 AM daily) scans the database and automatically filters out the list of those who have not completed the sign-in or boarding process.

[0107] Input: Real-time updated entry / boarding status data from the database, and check-in deadline settings.

[0108] Data processing / data operation: The server compares the list with the sign-in records and generates a list of unsigned individuals.

[0109] Output: List of those who did not sign in.

[0110] Specific actions: The system automatically performs background screening at regular intervals and pushes the list to the next task.

[0111] Step 6:

[0112] For those who have not checked in, the server calls a generative artificial intelligence model to generate notification content.

[0113] Input: List of those who did not sign in, and contact information of the guardian for each person.

[0114] Data processing / data computation: The server constructs a prompt statement for each object and inputs it into a generative artificial intelligence model (e.g., "Student XXX has not yet arrived. Please generate a friendly notification asking why.") to obtain personalized notification text.

[0115] Output: The generated notification text.

[0116] Specific actions: The server automatically sends requests to the generative artificial intelligence model in batches and receives all returned information.

[0117] Step 7:

[0118] The server sends the notification text to the guardian via a communication device.

[0119] Input: Guardian's contact number, email address, or App account, and the generated notification content.

[0120] Data processing / data operation: The server calls the SMS API, email service or App push to send the notification text to the guardian's terminal.

[0121] Output: The guardian's terminal receives a notification message.

[0122] Specific actions: The server automatically sends a text message or app push notification, and the guardian's mobile phone displays a warning or reminder.

[0123] Step 8:

[0124] After receiving the notification, the user (guardian) should provide feedback on the actual situation via the App, SMS, or other means.

[0125] Input: The leave request or feedback entered by the user.

[0126] Data processing / data calculation: After filling in the specific details, selecting the leave type or explaining the reason on the App, the user clicks send, and the data is transmitted back to the server via the network.

[0127] Output: Feedback data arrives at the server.

[0128] Specific steps: Parents fill out and submit feedback on their mobile phones, such as "Requesting leave for one day due to fever".

[0129] Step 9:

[0130] After receiving the feedback, the server automatically parses the feedback content and updates the object status in the database accordingly.

[0131] Input: User feedback information.

[0132] Data processing / data operation: The server identifies key information in the feedback and updates the object status from "not checked in" to "on leave" or "reason explained".

[0133] Output: Database status updated, management terminal page synchronizes status changes.

[0134] Specific actions: The server backend automatically modifies the status and records it.

[0135] Step 10:

[0136] If the server determines that there are still objects that have not responded, or if the terminal monitors that the objects have not shown any activity for an extended period of time, the server will send an abnormal alert to the management terminal.

[0137] Input: Database comparison results, abnormal results from behavioral analysis algorithms.

[0138] Data processing / data operation: The server runs anomaly detection algorithms (such as scoring behavioral segments through YOLO action recognition). If an anomaly is detected, an alert text is generated and pushed to the management terminal.

[0139] Output: The management terminal or monitoring interface receives an anomaly alert.

[0140] Specific action: A pop-up window in the administrator interface displays "Object XX has been inactive for an extended period of time. Please confirm its safety as soon as possible."

[0141] Application Example 1

[0142] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0143] Existing access control and monitoring systems cannot achieve multi-dimensional, real-time, and intelligent data collection and analysis when monitored personnel (such as children and passengers) enter or exit facilities or moving objects (such as parks and vehicles). Traditional systems lack automated notification and accurate identification methods when personnel fail to arrive on time, exhibit abnormal behavior, or display unusual emotions. This results in managers and monitoring personnel not being able to promptly detect abnormal situations and respond in a timely manner, leading to security risks and low management efficiency.

[0144] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0145] In this invention, the server includes a device for receiving data collected by the imaging device, performing identity recognition, entry / exit status judgment, and abnormal behavior and emotion detection based on an artificial intelligence model, and a device for automatically generating notification content based on a generative artificial intelligence model and combining it with prompt statements to communicate with users (such as managers, monitoring personnel, etc.). This enables real-time intelligent monitoring, automatic identification, and response to the entry, exit, and activities of personnel, timely notification of relevant responsible persons, ensuring the safety of personnel, and significantly improving the automation and intelligence level of management.

[0146] "Imaging device" refers to a device that can collect and record image information of a person or object, including but not limited to cameras, infrared cameras, etc.

[0147] "Facilities" refers to places where personnel are managed or activities are carried out, including various fixed areas such as kindergartens, schools, hospitals, and office buildings.

[0148] "Mobile vehicle" refers to a vehicle or device that can move, including buses, school buses, taxis, and autonomous vehicles.

[0149] "Target personnel" refers to people who are monitored or managed, such as children, passengers, and employees.

[0150] "Manager" refers to the personnel or organization responsible for the overall management and supervision of the personnel's entry, exit, activities, and safety.

[0151] "Responsible person" refers to the user who is notified and responds to the system when an anomaly is detected or when manual confirmation is required, including monitoring personnel, teachers, childcare workers, etc.

[0152] A "status database" refers to a database system used to store data sets related to the identity information, entry and exit records, behavioral data, emotional states, etc. of the target personnel.

[0153] An "anomaly detection algorithm" refers to an algorithm that uses statistical analysis or artificial intelligence to analyze the behavior or emotions of individuals and determine whether there are any abnormal situations.

[0154] "Generative artificial intelligence models" refer to algorithms or models based on artificial intelligence technologies such as deep learning that can automatically generate text content, notification information, etc., based on prompts.

[0155] "Prompt statements" refer to instructions or input information used to guide generative artificial intelligence models in generating specific types of text content.

[0156] "Behavioral data" refers to data about the actions and activities of individuals collected through imaging devices and other means.

[0157] "Emotional presumption" refers to the process of using algorithms or artificial intelligence models to analyze the facial expressions, text content, and other information of individuals to infer their psychological or emotional state.

[0158] "Notification content" refers to the message text that is automatically generated by the system and sent to the administrator or person in charge when a specific situation is detected.

[0159] To better understand and implement this invention, the following describes the embodiments of this invention in detail with reference to the specific system structure of this invention.

[0160] This invention relates to a security management system for personnel access and activities based on imaging devices, artificial intelligence technology, and generative artificial intelligence models. The system mainly includes several components such as a server, a terminal (imaging device), and user terminals.

[0161] The server, as the core processing unit of the system, uses general-purpose data processing hardware (such as high-performance processors and storage units) and is either installed locally or provided as a cloud service (such as a public cloud or private cloud platform). The server runs various software modules, including facial recognition algorithms (such as those using artificial intelligence frameworks such as OpenCV and TensorFlow), behavior analysis modules, anomaly detection algorithms, sentiment inference algorithms, and generative artificial intelligence models (such as automatic text generation APIs based on large-scale pre-trained language models).

[0162] The terminals employ high-performance imaging devices (such as cameras with facial recognition capabilities and infrared cameras) and are deployed in locations such as facility entrance areas and inside moving objects (such as vehicles and school buses). These terminals can automatically detect individuals entering or leaving the facility or moving object, collect their images and behavioral data, and transmit this data to a server in real time via wired or wireless networks. The terminals can also integrate edge computing nodes for preliminary data filtering and preprocessing.

[0163] Users can interact with the system through a dedicated application (which can be installed on mobile devices such as smartphones or tablets). For example, monitoring personnel can receive automatic notifications pushed by the system in real time through the app, view the status of individuals, historical entry and exit records, and real-time monitoring footage, and respond accordingly based on system feedback. Administrators or parents can interact with the server through the app or SMS to submit reasons, receive notifications, and provide feedback.

[0164] After receiving image data uploaded by the client, the server first verifies the identity data using a facial recognition module, and then compares the target object's information and status with the database. When it detects that an individual has not entered the facility or moved within a specified time, or detects specific abnormal behavior or emotions, the server automatically triggers a notification process. At this time, the generative artificial intelligence model automatically generates a corresponding notification text based on a preset prompt statement (e.g., "Please generate a friendly reminder to parents that their child has not arrived at the kindergarten and inquire about the reason."), and notifies relevant personnel (such as administrators, monitoring personnel, etc.) through an SMS gateway or information push interface.

[0165] Furthermore, the server can perform sentiment analysis and cause classification on user response data, and synchronize the analysis results and the latest status of the target personnel to the database, continuously improving the accuracy of the system's early warning.

[0166] This invention is adaptable to various facilities and mobile scenarios, such as kindergartens, schools, hospitals, office areas, buses, school buses, taxis, and autonomous vehicles. Through the combination of the aforementioned hardware and software, comprehensive, real-time, and intelligent management of personnel access security can be achieved, effectively reducing management costs and improving response efficiency and personnel safety.

[0167] The specific prompt statement (example) is as follows:

[0168] "Please apply the facial recognition algorithm to the camera video stream to identify the person in question and send the result to the server."

[0169] "Please use the anomaly detection algorithm in the video stream and automatically push notifications after an anomaly is detected."

[0170] "Please analyze the emotional state of the following message and provide emotional keywords."

[0171] "Please generate a text message to notify the person being managed that they did not appear on time and inquire about the reason."

[0172] use Figure 12 The processing flow is explained.

[0173] Step 1:

[0174] The terminal acquires real-time image data of individuals entering the facility or moving objects via an imaging device. The input is a video stream or image, and the output is data containing facial regions. The terminal uses a local face detection algorithm (such as OpenCV) to quickly locate faces and transmits the facial image data to the server over the network. Specific actions include automatic image capture, image preprocessing, and encrypted data upload.

[0175] Step 2:

[0176] The server receives facial image data from the client. The input is the facial image data uploaded by the client, and the output is the identification result of the person being identified. The server calls a facial recognition algorithm (such as TensorFlow) to extract features from the data and compare them with the database to confirm the identity of the person. Specific actions include identity verification, database cross-query, and result recording.

[0177] Step 3:

[0178] The server compares the incoming and outgoing databases with the reservation list to determine if any individuals have failed to arrive on time. Inputs include the identity verification results and database entries / exits; output is a list of absent individuals. The server periodically verifies the current time against the check-in requirements and generates a list of those who have not arrived. Specific actions include database retrieval, time matching, and list generation.

[0179] Step 4:

[0180] The server uses a generative artificial intelligence model to automatically generate notification content for missing persons based on prompts. The input is a list of missing persons and prompts; the output is the automatically generated notification text. The server feeds the list and instructions into the generative AI model to obtain text that matches the actual scenario. Specific actions include model invocation, text generation, and content formatting.

[0181] Step 5:

[0182] The server automatically sends notifications to relevant users (such as administrators, parents, and monitoring personnel) via a communication interface. Input includes the notification text and user contact information; output is a record of successfully sent notifications. The server calls an SMS gateway or push notification API to send the notification to the target terminal. Specific actions include interface integration, content delivery, and receipt monitoring.

[0183] Step 6:

[0184] Users receive notifications and reply with the reason or feedback via an app or SMS. The input is the notification message, and the output is the user's response text. Users open the notification on their mobile devices (such as phones or tablets), fill in their reply, and submit. The specific actions include message reception, interface input, and response submission.

[0185] Step 7:

[0186] The server receives user responses and parses and performs sentiment analysis on the reply text. The input is the user's response text, and the output is a status update and sentiment analysis results. The server uses natural language processing algorithms to extract keywords, employs a sentiment analysis model to determine tone and psychological state, and synchronizes the results to the object state database. Specific actions include text parsing, sentiment classification, and database updates.

[0187] Step 8:

[0188] The server combines behavioral data and sentiment analysis results to monitor for prolonged inactivity or abnormal emotions. Inputs are the latest behavioral data and analysis results, and outputs are anomaly detection and alarm decisions. The server runs anomaly detection algorithms to comprehensively assess the activity frequency and emotional changes of individuals, generating anomaly notifications when necessary. Specific actions include data aggregation, algorithm analysis, and automatic alarm activation.

[0189] Step 9:

[0190] The terminal continuously collects on-site behavioral data and periodically uploads it to the server. The input is real-time on-site behavioral images, and the output is real-time behavioral data packets. The terminal captures the actions of on-site objects in real time through timed snapshots or video streams and automatically uploads the data. Specific actions include behavior detection, data packaging, and network transmission.

[0191] Step 10:

[0192] When the server detects an anomaly, it uses a generative artificial intelligence model to generate an anomaly notification and sends it to the responsible person. The input includes anomaly event information and prompts; the output is an anomaly warning message. The server inputs the anomaly content and template prompts into the generative AI to generate a realistic warning message, which is then pushed to the responsible person via the communication interface. Specific actions include anomaly attribution, text generation, and emergency push notification.

[0193] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0194] Example 2

[0195] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0196] Existing security monitoring systems for managed individuals typically only implement basic entry / exit registration and behavioral monitoring, failing to effectively analyze and respond to the emotional state of those under their management and their guardians. The detection accuracy for abnormal behavior and emergencies is low, and notifications and feedback rely heavily on manual processing, leading to response delays, increased staff workload, and difficulty in timely and accurately grasping various dynamic information, resulting in security management loopholes. Furthermore, notification texts often lack personalization and a natural communicative feel, failing to enhance guardians' willingness to respond and improve communication efficiency.

[0197] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0198] In this invention, the server includes: modules for monitoring the entry and vehicle travel status of managed individuals through image acquisition devices with human recognition capabilities; modules for automatically generating and sending customized notifications; modules for receiving and updating feedback information from guardians; a sentiment analysis module for analyzing the emotional state of image data and text information; a module for automatically pushing alarms to staff based on abnormal states; and a module for generating natural language notification and feedback text based on input prompts using a generative artificial intelligence model. This enables multi-dimensional monitoring and management of the safety of managed individuals, improves the efficiency of identifying emotions and abnormal events, automates notification and feedback, significantly reduces the workload of staff, and improves the timeliness and accuracy of safety management.

[0199] "Image acquisition device" refers to an electronic device that can acquire image information of the managed object and transmit the image data to a server for identity recognition and behavior monitoring.

[0200] "Person recognition function" refers to the technical capability that can automatically identify and determine the identity of people in an image through algorithms.

[0201] "Managed premises" refers to the spaces where personnel safety monitoring and management are implemented, including but not limited to schools, kindergartens, childcare institutions, and factories.

[0202] "The managed objects" refer to the target personnel who need to undergo identity verification, behavior and security status monitoring within the managed premises, such as children, students, and employees.

[0203] "Guardian" refers to an individual who has the responsibility of guardianship, supervision and contact with the person being managed, usually a parent or legal guardian.

[0204] "System recorded information" refers to the data stored in the management system regarding the identity, entry and exit, behavioral status, and emotional state of the managed object.

[0205] "Staff" refers to service personnel responsible for management, monitoring, information processing, and emergency response.

[0206] "Emotional analysis device" refers to a component or module that can automatically analyze and judge the emotional state of the managed object and its guardian based on image data or text data.

[0207] "Generative artificial intelligence models" refer to information processing technologies based on artificial intelligence algorithms that can automatically generate natural language text such as notifications, replies, and prompts.

[0208] "Prompt statements" refer to keywords or phrases input to guide the direction of text generation when using generative artificial intelligence models to generate text.

[0209] "Means of transport" refers to vehicles used to pick up and drop off the managed entities, including but not limited to school buses, public buses, and shuttle buses.

[0210] "Travel time and behavior" refers to the specific time when the managed object gets on and off the vehicle, as well as its dynamic activity status within the means of transportation.

[0211] "Abnormal state" refers to a situation that differs from the normal activity or emotional state of the managed object, including prolonged inactivity, absence from the venue, and abnormal emotions.

[0212] "Machine learning algorithms" refer to artificial intelligence technologies that automatically learn patterns based on historical data and are used for data analysis, identification, and judgment.

[0213] Natural Language Processing (NLP) refers to human-computer interaction technology that uses computers to understand, analyze, and generate text or language data.

[0214] "Mobile information terminal" refers to a portable electronic device that can be used to receive, send and process information, such as smartphones and tablets.

[0215] The embodiments of the present invention are described below.

[0216] The system mainly includes a server, terminals (such as image acquisition devices with human recognition capabilities), and user terminals for information interaction. The server exchanges data with the terminals and user terminals via a local area network or the Internet to achieve multi-dimensional monitoring and management of the managed objects' entry, transportation process, and emotional state.

[0217] The server can utilize general-purpose server hardware platforms, such as multi-core processor servers, and run operating systems like Linux. Software modules for face recognition and emotion recognition can leverage open-source frameworks such as OpenCV (for image signal processing), TensorFlow or PyTorch (for face recognition feature operators in artificial neural networks), and natural language processing tools (such as NLTK, spaCy, or similar products). Data management can be based on database systems such as SQL Server or MongoDB. Push notification and feedback reception functionality can be implemented using push messaging platforms, SMS gateways, and dedicated mobile applications.

[0218] The terminal is a video camera or image acquisition device with human recognition capabilities. For example, network cameras installed in access control systems have real-time shooting, data encryption, and network transmission functions, and can synchronously transmit the captured images and timestamp information to the server. Video surveillance terminals can also be installed inside transportation vehicles (buses, shuttle buses) to collect boarding times and dynamic behavior inside the vehicle.

[0219] The user terminals are mainly smartphones or tablets, running dedicated management and communication apps, allowing staff and guardians to communicate efficiently.

[0220] In actual system operation, the server first collects real-time images of managed individuals entering or boarding the vehicle via the terminal, uses facial recognition algorithms to determine identity and register time, and then analyzes the emotional state in the images and user text information through the sentiment analysis module. If an absence is detected or abnormal behavior is detected, the server automatically calls a generative artificial intelligence model (such as a natural language generation platform based on GPT-4) to automatically generate a message (notification text or feedback text) suitable for notifying the guardian based on preset prompts.

[0221] The server pushes the generated notification to the guardian, who can directly reply with relevant information (such as the reason for absence) via a smart terminal. The server automatically parses the text entered by the guardian, performs sentiment analysis, and records it in the system database. If no feedback is received within the specified time, or if anomalies are detected in the behavioral / sentiment analysis, the server automatically generates a warning message and pushes it to relevant staff, realizing an unmanned, automated, and intelligent anomaly handling and security response process.

[0222] Specific examples:

[0223] For example, one morning, the terminal recognized Xiaoming's facial image at the entrance of the after-school care facility and uploaded it to the server. The server used OpenCV and neural network libraries to perform real-time identity recognition and determined that Xiaoming's expression at the time was "nervous" through an emotion analysis API. Data showed that one child had not arrived before 9:00 AM. The server automatically used a generative AI model to generate a natural language notification text based on the prompt, "Your child is not present. Please explain the reason," and pushed it to the parent's management app. The parent replied, "Due to a cold, I am taking the day off." The server received and stored this information, and simultaneously used natural language processing to identify that the parent expressed "concern" and recorded it. If no reply was received or if the vehicle detected prolonged stillness, the server immediately sent a warning message to the staff: "No reply received from the guardian or suspected abnormality. Please check promptly."

[0224] Example of prompts for generative artificial intelligence models:

[0225] "Please generate a simple and gentle text message reminder that reads, 'Your child has not yet arrived at the kindergarten. Please explain why.'"

[0226] "Based on the following response text, please determine the parent's main emotional attitude, such as 'worried,' 'reassured,' or 'dissatisfied': 'My child is not feeling well today and cannot come to school.'"

[0227] use Figure 13 The processing flow is explained.

[0228] Step 1:

[0229] The server receives the user's (staff's) "Start Monitoring" command in the management app and automatically retrieves the current date.

[0230] Input: User operation command (start monitoring), date information.

[0231] Output: Status of the appearance monitoring function being activated.

[0232] Data processing: The server reads the personnel list and schedule from the database based on the date to initialize data collection and analysis.

[0233] Specific steps: The server displays that monitoring is ready.

[0234] Step 2:

[0235] The terminal (image acquisition device) captures real-time facial images of the managed objects entering the management area, adds a timestamp, and then uploads the data to the server.

[0236] Input: Face image of the managed object, timestamp.

[0237] Output: A time-stamped stream of face image data uploaded to the server.

[0238] Data processing: The terminal automatically detects personnel entering and exiting and encapsulates data packets to ensure encrypted transmission.

[0239] Specific operation: The camera sends data immediately upon detecting a new object.

[0240] Step 3:

[0241] The server uses a facial recognition algorithm to verify the identity of the received facial image and timestamp, and then registers the results in the database.

[0242] Input: Face image data, timestamp.

[0243] Output: Entry registration records of the managed objects.

[0244] Data processing: The server uses image processing and feature extraction to compare the face with the features already archived in the database. If the comparison is successful, the entry time and identity information are written into the database.

[0245] Specific operation: The server generates a record of "A person successfully entered the venue at a certain time".

[0246] Step 4:

[0247] The server calls the sentiment analysis module to perform sentiment analysis on the facial images and related behavioral data of the managed objects, and registers the results in the database.

[0248] Input: Facial images, behavioral data.

[0249] Output: sentiment status labels, sentiment analysis records.

[0250] Data processing: The server outputs emotion classifications such as "happy" and "anxious" based on facial expression recognition and behavior analysis models.

[0251] Specific operation: The system will automatically pop up a special emotion warning prompt.

[0252] Step 5:

[0253] The server compiles a list of those who did not enter before a specified time.

[0254] Input: All registration information, current time.

[0255] Output: List of participants who did not enter the venue.

[0256] Data processing: The server compares the time with database records to filter out personnel who have not clocked in.

[0257] Specific operation: The server automatically generates a list of those who have not entered the venue in the background.

[0258] Step 6:

[0259] The server uses a generative artificial intelligence model to automatically generate natural language text suitable for notifying guardians based on the "prompt statement" and pushes the notification to the relevant guardians.

[0260] Input: List of those who did not enter, and prompt message.

[0261] Output: Customized notification text, push message.

[0262] Data processing: The server inputs the list and prompts into the AI ​​model to generate multiple personalized SMS messages or APP push content.

[0263] Specific steps: Associate the server with the push ID to achieve precise notifications.

[0264] Step 7:

[0265] Users (guardians) receive notifications through a mobile app, fill in or select their attendance status and reasons for feedback, and send a reply to the server.

[0266] Input: Notification content, user feedback.

[0267] Output: Feedback message uploaded to the server.

[0268] Data processing: The user terminal formats the content entered by the parent, adds the user ID and uploads it synchronously with the time.

[0269] Specific steps: After clicking the notification, parents can directly fill in the reason information and send it.

[0270] Step 8:

[0271] The server receives and archives user feedback, and calls the natural language processing module to perform sentiment analysis on the text, identify the guardian's emotional attitude, and register it in the database.

[0272] Input: Guardian's feedback text.

[0273] Output: Feedback results, sentiment tags.

[0274] Data processing: The server uses NLP algorithms to analyze the content, extract keywords and sentiment categories, and categorize them into relevant database fields.

[0275] Specific operation: The server dynamically updates the state history of each object.

[0276] Step 9:

[0277] The server continuously monitors the system. If no user feedback is received within a certain period of time or abnormal behavior is detected by the terminal, the server calls the AI ​​model to generate a warning message and pushes it to the staff.

[0278] Input: Feedback time limit, abnormal behavior judgment result.

[0279] Output: Warning notification text.

[0280] Data processing: The server monitors and responds to feedback and input from various devices in real time. When an anomaly is detected, a generative artificial intelligence model outputs a warning message and pushes it to relevant staff.

[0281] Specific operation: The system will automatically send a "Please check in time" or "Data abnormality warning" notification in the management APP.

[0282] Application Example 2

[0283] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0284] Existing access control and health monitoring systems struggle to track the entry and exit status and health status of personnel in real time, hindering managers from efficiently and accurately monitoring personnel safety and health. In the event of abnormal conditions, response is often delayed, posing safety hazards, and there is a lack of mechanisms for dynamic detection and immediate feedback on changes in individuals' emotional states (such as fatigue and stress). Therefore, a new system is urgently needed that can automatically identify, monitor in real time, intelligently analyze data, and promptly notify managers to improve overall safety and health management efficiency.

[0285] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0286] In this invention, the server includes: a device for real-time monitoring of the entry and exit status of individuals using a biometric acquisition device; a device for automatically sending a notification to the guardian when an individual's attendance or departure within a specified time is not confirmed; a device for receiving a response from the guardian and updating system information promptly; a device for automatically notifying the administrator when no response is received from the guardian or when an anomaly is detected; a device for analyzing the individual's emotional and health status based on image and behavioral data using a parsing device; and an automatic notification device for immediately sending an alarm to the administrator when an anomaly is detected. This enables real-time, automated, and high-precision management of individual entry and exit and health status, and allows for rapid notification to the administrator in case of anomalies, further enhancing the system's intelligence and security capabilities.

[0287] "Bioinformation acquisition device" refers to a device used to collect biometric information (such as facial images, behavioral data, etc.) of an individual and convert it into a data format that can be processed by the system.

[0288] "Facilities" refers to physical locations where personnel conduct activities, access management, and monitoring, including but not limited to factories, schools, and office buildings.

[0289] "Objects" refers to the personnel monitored and managed by the system, including employees, students, and other groups that require access records and health management.

[0290] "Real-time monitoring" refers to a management approach that involves continuous and dynamic data collection and real-time analysis of the entry, exit, or status of an object, and the ability to respond promptly to changes.

[0291] "Guardian" refers to an individual who has a supervisory responsibility over the person in question, such as a parent or person in charge.

[0292] "Manager" refers to the person in charge of facility safety, staff attendance, and health management.

[0293] "Analysis device" refers to software or hardware modules used to analyze and process collected image information and behavioral data, and can output analysis results such as emotional state and health status.

[0294] "Automatic notification device" refers to a communication device or system that can automatically generate and send notification information to a designated recipient when set conditions are met or an anomaly is detected.

[0295] "Response information" refers to the content of the guardian's response or feedback to the notification sent by the system.

[0296] "Abnormal state" refers to a situation where the entry or exit, health or emotional state of the person being observed does not conform to the preset normal range, which may pose a safety risk or health hazard.

[0297] "Artificial intelligence technology" refers to the methods and technologies that use intelligent algorithms such as machine learning and deep learning to autonomously analyze and judge acquired data.

[0298] This invention can be implemented in the following specific ways.

[0299] This system consists of three parts: a server, terminals, and users. Terminals refer to biometric acquisition devices installed at facility entrances and exits and on mobile devices (such as vehicles), such as general-purpose digital cameras (e.g., facial recognition cameras), used to collect facial images and behavioral data of people entering and exiting. The server serves as the core processing unit, including a processor, memory, and network module. Administrators and guardians interact with data and process information through user-end devices (such as smartphones and tablets).

[0300] In practical applications, terminals (such as cameras) collect biometric information of employees or students in real time, transmitting the raw data to a server via wired or wireless networks. The server is equipped with facial recognition algorithms (e.g., implemented using the OpenCV open-source library), emotion and health status analysis software (such as the EmotionEngine custom analysis module), a database management system (such as SQLite or PostgreSQL for information storage), and an automatic notification system (such as push APIs and SMS APIs). The server first performs noise reduction, formatting, feature point extraction, and comparison on the collected data to achieve rapid identification of the individual and automatically record entry / exit times and status.

[0301] The server also uses emotion analysis software to analyze facial features and behavioral changes to determine the subject's emotional and health status, such as whether they are fatigued or stressed. If a subject is found to be absent at the scheduled time or an abnormal health condition is detected, the server can automatically send a message to the guardian or administrator through the notification system and wait for a response. If no response is received from the guardian within the set time, or if the abnormal condition persists, the server will issue a warning to the administrator through the management terminal, prompting them to intervene in a timely manner.

[0302] For example, if employee A enters the factory at 8:00 AM, a terminal camera captures their facial image, the server uses OpenCV algorithms for recognition, and EmotionEngine analyzes their expression. The database records that A has arrived on time and is in a normal mood. If EmotionEngine detects obvious signs of fatigue in A at 10:00 AM, the server automatically pushes an alert to the manager's mobile phone, allowing the manager to arrange for A to rest. All records and feedback information are stored in the database in real time for subsequent tracking, management, and optimization.

[0303] The system's hardware and software components include, but are not limited to: a general-purpose camera, a smart terminal, a facial recognition algorithm (OpenCV), an emotion and health analysis engine (EmotionEngine), a database management system (SQLite or PostgreSQL), and automatic notification services (such as email, app push, and SMS API).

[0304] Generative AI models can be used to improve the accuracy of emotion and health analysis and can also support further data-driven decision-making.

[0305] Example of a prompt statement:

[0306] "Please generate a complete Python program for employee attendance and emotion analysis. It should take photos with the camera, recognize faces using OpenCV, analyze emotions using EmotionEngine, and automatically notify managers via WeChat and update the database upon detecting anomalies."

[0307] use Figure 14 The processing flow is explained.

[0308] Step 1:

[0309] The terminal automatically collects facial images and behavioral data of individuals at facility entrances / exits or mobile device locations using biometric acquisition devices (such as cameras). The input is real-time images and behavioral information of the individual, and the output is structured image data packets. The terminal periodically captures images or automatically sends this data to the server upon detecting someone passing by.

[0310] Step 2:

[0311] After receiving image data from the terminal, the server first performs image preprocessing, including denoising, grayscale conversion, and normalization. The input is the raw image data uploaded by the terminal, and the output is a usable, clear facial image. Specifically, the server calls the image processing module to convert the data into a format suitable for subsequent feature extraction.

[0312] Step 3:

[0313] The server uses facial recognition algorithms (such as the OpenCV module) to process the preprocessed image, extract facial feature points, and compares them with registered object information in the database. The input is a standardized image, and the output is the object ID. Based on the input image features, the server obtains the object's identity information through algorithmic matching.

[0314] Step 4:

[0315] The server writes the identified object ID and current time to the database, updating the entry and exit records. The input is the object ID and timestamp; the output is the real-time updated database record. The server performs database insert or update operations to ensure that the entry and exit times and status of all objects are accurately recorded.

[0316] Step 5:

[0317] The server uses an emotion and health status analysis engine (such as EmotionEngine) to analyze image and behavioral data to determine the subject's emotional state and health level (e.g., whether there are signs of fatigue, stress, etc.). The input is the subject's facial image and behavioral sequence data, and the output is the emotion analysis result (e.g., "normal," "fatigued," "stressed"). The server analyzes facial expressions, eye movements, etc., using algorithms to output the corresponding health status.

[0318] Step 6:

[0319] Based on the sentiment analysis results, the server determines whether any abnormal conditions exist. If an individual fails to register for attendance within the specified time or is detected to have abnormalities such as fatigue or stress, the server immediately generates and pushes an alert via the automatic notification system. The input is the sentiment and health status analysis conclusion, and the output is the alert information. The server automatically assembles the notification message and pushes it to the terminal device of the administrator or guardian.

[0320] Step 7:

[0321] Users (administrators or guardians) receive alert notifications from the server on their terminal devices (such as mobile phones and tablets), view abnormal content, and can provide feedback or take appropriate measures. Input is the server-pushed notification, and output is processing feedback information or action instructions. Users respond to the system or arrange for personnel to respond based on the server information.

[0322] Step 8:

[0323] The server receives feedback from users, stores the responses from administrators or guardians, along with subsequent processing results, in the database, and adjusts monitoring strategies as needed. Input is user feedback, and output is updated system logs or management records. Based on this information, the server improves object management processes and optimizes system response mechanisms.

[0324] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0325] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0326] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0327] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0328] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0329] Second Implementation Method

[0330] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0331] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0332] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0333] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0334] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0335] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0336] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0337] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0338] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0339] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0340] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0341] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0342] Example 1

[0343] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0344] Application Example 1

[0345] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0346] Example 2

[0347] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0348] Application Example 2

[0349] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0350] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0351] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0352] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0353] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0354] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0355] Third Implementation Method

[0356] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0357] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0358] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0359] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0360] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0361] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0362] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0363] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0364] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0365] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0366] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0367] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0368] Example 1

[0369] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0370] Application Example 1

[0371] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0372] Example 2

[0373] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0374] Application Example 2

[0375] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0376] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0377] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0378] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0379] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0380] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0381] Fourth Implementation Method

[0382] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0383] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0384] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0385] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0386] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0387] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0388] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0389] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0390] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0391] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0392] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0393] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0394] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0395] Example 1

[0396] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0397] Application Example 1

[0398] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0399] Example 2

[0400] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0401] Application Example 2

[0402] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0403] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0404] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0405] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0406] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0407] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0408] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0409] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0410] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0411] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0412] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0413] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0414] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0415] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0416] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0417] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0418] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0419] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0420] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0421] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0422] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0423] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0424] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0425] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0426] In addition, the following notes are provided in response to the above explanation.

[0427] Example 1

[0428] (Note 1)

[0429] An information processing system includes: a device for monitoring the entry status of individuals entering a venue using an image acquisition device and for performing individual identification on the acquired image information using image recognition technology; a device for automatically receiving notifications from the guardians of individuals who have not been confirmed to have entered within a specified time via a communication device; a device for receiving feedback information from guardians and updating information in an information storage device based on the received information; a device for sending notifications to a management terminal when no feedback is received from guardians within a specified time or when an abnormal status is detected; and a device for generating prompt statements based on a generative artificial intelligence model while processing the acquired information, and for generating notification information based on the output content of the generative artificial intelligence model by inputting the prompt statements into the generative artificial intelligence model.

[0430] (Note 2)

[0431] According to the information processing system described in Appendix 1, the system acquires the passenger's travel time and behavior information through an image acquisition device installed on the mobile body, and sends a notification to the management terminal when analyzing the acquired behavior information if it detects that the passenger has not been active within a specified time or an abnormal state is detected.

[0432] (Note 3)

[0433] According to the information processing system described in Appendix 1, when parsing image information and behavioral information from the image acquisition device in the information processing device, machine learning algorithms are used for individual identification, anomaly detection, and automatic data updates.

[0434] Application Example 1

[0435] (Note 1)

[0436] An information processing system includes: a device for monitoring the entry and exit status of objects and personnel in facilities and mobile bodies via an imaging device; a device for automatically sending a notification to a manager if the entry and exit of an object or personnel is not confirmed within a specified time; a device for receiving the manager's response, parsing the response content and the information contained therein, and updating the object or personnel status database accordingly; a device for processing behavioral data and response data and performing status analysis including emotional inference; a device for sending a notification to the person in charge (monitoring personnel) when the manager does not respond or when an abnormal behavior or emotion is detected by an anomaly detection algorithm; and a device for automatically generating notification content and message content based on a generative artificial intelligence model and prompt statements.

[0437] (Note 2)

[0438] The information processing system according to Appendix 1 further includes: a device for monitoring the behavior and activities of the subject personnel through an imaging device installed on the moving body, and for sending a notification to the person in charge when an activity is detected to have not occurred for more than a set time or when an abnormality is detected.

[0439] (Note 3)

[0440] The information processing system according to Appendix 1 further includes: a device for processing image data and behavioral data acquired from multiple imaging devices, and for performing recognition, anomaly detection and sentiment inference through artificial intelligence.

[0441] Example 2

[0442] (Note 1)

[0443] An information processing system includes: a device for monitoring the entry status of managed objects in a managed area using an image acquisition device with person recognition function; a device for automatically generating and sending notification content to the guardian of a managed object who has not entered before a specified time; a device for receiving the guardian's response information and updating the system record information accordingly; a device for sending warning information to staff when the guardian does not respond within a specified time limit or when an abnormal state is detected; an emotion analysis device for analyzing the image data and response information of the managed object and its guardian to determine the emotional state; a device for recording the emotion analysis results and notifying staff when abnormal emotions are detected; a device for staff and guardians to exchange information via a mobile information terminal; and a device for generating notification text and response text for the guardian using a generative artificial intelligence model, and allowing input of prompt statements during generation.

[0444] (Note 2)

[0445] The information processing system according to Appendix 1 further includes: a device for monitoring the travel time and behavior of the managed object through an image acquisition device installed in the means of transport, and for sending a warning message to staff when a specified behavior is detected not to occur within a specified time or an abnormal state occurs.

[0446] (Note 3)

[0447] The information processing system according to Appendix 1 further includes: a device for analyzing data acquired by an image acquisition device with human recognition function and an image acquisition device inside a vehicle, and for monitoring the safety status of the managed object using artificial intelligence technology including machine learning algorithms and natural language processing.

[0448] Application Example 2

[0449] (Note 1)

[0450] An information processing system includes: a device for real-time monitoring of the entry and exit status of individuals within a facility using a biometric acquisition device; a device for automatically sending a notification to a guardian if attendance or exit is not confirmed before a predetermined time; a device for receiving a reply from the guardian and updating the content of an information recording device based on the reply; a device for automatically sending a warning to the administrator if a reply from the guardian is not received within a specified time or if the system detects an abnormal state; a device for analyzing the emotional and health status of individuals based on acquired image and behavioral information using an analysis device, and detecting signs of fatigue or stress; and an automatic notification device for immediately sending a notification to the administrator based on the detected abnormal state.

[0451] (Note 2)

[0452] According to the information processing system described in Appendix 1, the biometric information acquisition device is installed on the mobile device to acquire the boarding and alighting times and behavioral information of the subject, and automatically sends a notification to the administrator when no activity is detected for more than a predetermined time or when the analysis device detects an abnormal state.

[0453] (Note 3)

[0454] According to the information processing system described in Appendix 1, the parsing device uses artificial intelligence technology to parse information from the bio-information acquisition device and the bio-information acquisition device installed on the mobile device, so as to realize real-time monitoring of the object's safety and health status.

Claims

1. An information processing system, characterized in that, include: A device that monitors children's entry into childcare facilities using facial recognition cameras; A device that automatically sends a contact message to the child's guardian if the child does not arrive at the park within the specified time; A device for receiving feedback from guardians and updating the system based on that feedback; and A device that sends a notification to caregivers when no feedback is received from the guardian or when an abnormality is detected.

2. The information processing system according to claim 1, characterized in that, Further includes: Devices that monitor children's boarding time and behavior through cameras installed on buses used for boarding at childcare facilities; A device that sends a notification to caregivers when prolonged inactivity or an abnormal situation is detected.

3. The information processing system according to claim 1, characterized in that, Further includes: A device that processes data from facial recognition cameras and cameras on school buses, and uses artificial intelligence technology to monitor children's safety.

Images