system

Abstract

We provide the system. [Solution] The present invention relates to a system that integrates and manages information such as children's school and extracurricular activity schedules, traffic information, weather information, and parents' work schedules for full-time working parents, and automatically generates an optimal transportation plan using generative AI. The system includes a schedule management unit, a schedule optimization unit, a vehicle dispatch coordination unit, a video monitoring unit, etc., and realizes automatic dispatch of transportation vehicles, real-time monitoring of in-vehicle video, and immediate notification in the event of an anomaly detection. This significantly reduces the burden on parents while comprehensively realizing safe and efficient transportation and schedule management for children.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method executed by at least one processor, the method comprising the steps of: receiving a user utterance; adding the user utterance to a prompt including an instruction sentence associated with a description related to the character of the chatbot; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to comprehensively and efficiently solve the complex challenges faced by child-rearing households in dual-income families and a low birthrate society, in the current situation where full-time working parents are forced to bear enormous time and physical burdens to drop off and pick up their children from extracurricular lessons and school, making it difficult to balance work and childcare, including problems such as children's overcrowded schedules, situations where parents have to adjust their working hours or leave work early, housework and personal time being squeezed due to transportation duties, anxiety about safety during transportation and emergency response, financial and psychological hurdles for transportation means such as taxis, delay risks caused by traffic congestion and weather changes, and lack of peace of mind due to difficulty in grasping the child's situation during movement, so as to provide an environment that enables parents to concentrate on their work while realizing safe and smooth transportation and schedule management for their children.

Means for Solving the Problems

[0005] This invention provides a system comprising: a schedule management unit that acquires schedule information for children's extracurricular activities and school; a schedule optimization unit that comprehensively analyzes multiple external data such as the acquired schedule information, traffic information, weather information, and parents' work schedules, and optimizes the child's travel route and pick-up / drop-off times using generative AI; and a video monitoring unit that automatically selects a safe means of transportation based on the optimized schedule, arranges a vehicle in cooperation with a ride-hailing service, and delivers real-time video from a camera mounted on the vehicle to the parent's terminal. The system includes: a schedule management unit that has the function of acquiring information from the child's calendar information registered by the parent and from an external calendar app; a schedule optimization unit that automatically generates an optimal pick-up / drop-off plan using generative AI, taking into account multiple constraints such as traffic congestion, weather changes, start and end times of extracurricular activities, the parent's work end time, and emergency response requirements; and a video monitoring unit that provides the parent's terminal with the vehicle's current location information, video data, and emergency notification functions in real time. This system is a means of achieving safe and efficient transportation and schedule management for children while reducing the burden on parents. [Brief explanation of the drawing]

[0006] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0007] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0008] First, let's explain the terminology used in the following explanation.

[0009] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0010] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0011] In the following embodiments, the referenced storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include a flash memory (SSD (Solid State Drive)), a magnetic disk (for example, a hard disk), or a magnetic tape and the like.

[0012] In the following embodiments, the referenced communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F governs communication between a plurality of computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark) and the like.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, may be only B, or may be a combination of A and B. Further, in the present specification, when three or more matters are expressed by being connected with "and / or", the same concept as that for "A and / or B" is applied.

[0014] [First Embodiment]

[0015] Figure 1 illustrates an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] As shown in Figure 1, the data processing system 10 includes a data processing apparatus 12 and a smart device 14. An example of the data processing apparatus 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of the "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network), and the like.

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, the camera 42, and the communication I / F 44 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A receives user input from contact of an indicator by detecting contact of the indicator (for example, a pen, a finger, or the like). The microphone 38B receives voice-based user input by detecting the user's voice. A control unit 46A in the processor 46 transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, a specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0022] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0024] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0026] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0027] (Example 1)

[0028] The flow of a specific process in Example 1 will now be described. Each part of the system described below is implemented by a data processing device 12 and a smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0029] (Embodiments for carrying out the present invention)

[0030] Embodiments of the present invention are implemented as a system consisting of a server and terminals (smartphones, tablets, personal computers, etc.), and include three main functional units: a schedule management unit, a schedule optimization unit, and a video monitoring unit, each of which is appropriately distributed and implemented on the server side and the terminal side.

[0031] First, while the schedule management function is primarily implemented on the server side, a user interface also exists on the terminal side. Parents can input their child's school and extracurricular activity schedules, desired pick-up / drop-off times and locations, their own work schedules, and special family circumstances (e.g., schedules for siblings, family events, unexpected illnesses or school events, parental business trips, grandparent visits, etc.) from their terminals. Furthermore, it is possible to integrate with external calendar apps, social media, school communication systems, and extracurricular activity reservation systems, and automatically retrieve schedule information via APIs, etc. For example, if a parent enters their child's schedule into Google Calendar® or iOS Calendar® on their terminal, the server-side schedule management function automatically retrieves information from these external calendars and integrates it into the system's schedule database. In addition, it can integrate with reservation management systems provided by the organizers of extracurricular activities, and changes to the start time or location of activities are automatically reflected. For example, if a piano lesson is suddenly canceled or the venue for a soccer lesson changes, the server can automatically retrieve the information and reflect it in the schedule. Furthermore, by linking with the school's communication network system, it can automatically retrieve information such as changes to dismissal times and temporary school closures, and reflect this in the schedule.

[0032] The schedule management section allows for the simultaneous management of multiple schedules for each child, and if there are siblings, each child's schedule can be registered and managed individually. For example, if the eldest son attends soccer lessons, the eldest daughter attends piano lessons, and the second son attends cram school, each schedule can be managed centrally and then integrated and optimized in the schedule optimization section described below. In addition, parents' work schedules can be registered in detail, and information such as "working until 6 PM today," "working from home on Wednesdays," and "business trip on Fridays" can be registered and reflected in the pick-up and drop-off plan. Furthermore, family events and special circumstances (for example, "no pick-up and drop-off needed on this day because grandparents are visiting," or "child is unwell, so lesson is cancelled") can also be registered flexibly.

[0033] Next, while the schedule optimization unit is primarily implemented on the server side, some user interface processing is also performed on the terminal side. The schedule optimization unit uses generative AI to comprehensively analyze a variety of data, including acquired schedule information, traffic information, weather information, vehicle availability, road safety information, parents' work schedules, children's health and preferences, past travel history, vehicle driver evaluation information, road construction information, event information, disaster information, local security information, and vehicle fuel and battery levels. For example, if a child has multiple extracurricular activities, the generative AI considers all factors such as the start and end times, locations, travel time, traffic congestion, weather forecasts, road safety, vehicle availability, parents' work end times, sibling schedules, vehicle driver evaluations, road closures due to construction or events, local security information, disaster information, and vehicle fuel and battery levels, to automatically generate the most efficient and safest pick-up and drop-off sequence, times, routes, and means of transportation.

[0034] For example, for a series of trips from school to soccer practice, from soccer practice to cram school, and from cram school to home, the generative AI obtains real-time traffic congestion information, information on sudden weather changes such as torrential downpours, road closures due to road construction or events, local safety information, disaster occurrence information, and the fuel and battery levels of the transport vehicle, and selects the shortest route or a safe detour route. For example, if the soccer practice venue is by the Tama River and torrential downpours are forecast for that day, the generative AI will suggest canceling or postponing the outdoor activity and change the transport route. For example, if a large event is being held near the cram school, the generative AI will suggest leaving earlier or taking an alternative route to avoid traffic congestion. For example, if the driver of the transport vehicle has a low rating or the vehicle's fuel level is low, the generative AI will prioritize assigning another vehicle or driver. For example, considering local safety information and disaster occurrence information, it will select a route that avoids busy areas and dangerous areas at night.

[0035] Furthermore, the generative AI can refer to a child's past travel history and health data (for example, if they have a history of leaving school early due to illness the previous day, or if they tend to get tired easily on certain days of the week) to suggest a manageable schedule. For example, if a child left school early due to illness the previous day, the AI ​​will automatically cancel the lesson for that day and restructure the transportation plan. For example, if there are siblings, the AI ​​will optimize the schedules of multiple children simultaneously and efficiently allocate transportation vehicles. For example, even if siblings participate in different lessons, the generative AI will automatically determine the optimal transportation order and vehicle allocation. For example, if the transportation of siblings overlaps at the same time, the generative AI will efficiently allocate multiple transportation vehicles.

[0036] The results of the schedule optimization are sent to the application on the device, and parents can review, modify, and approve the optimized pick-up and drop-off plan on their device. For example, if a parent has to work extra hours unexpectedly or if their child becomes ill, they can send a schedule change request from their device, and the server-side generative AI will perform the optimization process again and immediately present a new pick-up and drop-off plan. For example, if a parent requests from their device, "I have to work overtime unexpectedly today, so I would like to automate all of my child's pick-up and drop-off," the server-side generative AI will immediately recalculate the schedule, arrange the most suitable vehicle, and provide the parent with real-time progress and video.

[0037] The video monitoring system functions on both the server and terminal sides. Vehicles are equipped with onboard cameras, and in-vehicle video is transmitted to the server in real time. On the server side, generative AI is used to analyze video data and location information, automatically detecting abnormal behavior and signs of danger (e.g., a child crying in the vehicle, the vehicle traveling an unscheduled route, the vehicle stopped for an extended period, the driver acting suspiciously, or a suspicious person being present in the vehicle). For example, the generative AI comprehensively analyzes the child's facial expressions and movements in the video, the vehicle's driving pattern, and surrounding environmental sounds to extract unusual situations in real time. For instance, if the vehicle travels an unscheduled route or a child complains of feeling unwell in the vehicle, the generative AI automatically detects the anomaly and displays a warning on the parent's terminal. Similarly, if the vehicle is stopped for an extended period or the driver acts suspiciously, the generative AI immediately detects the anomaly and notifies the parent's terminal.

[0038] The video monitoring system also provides a function that allows parents to constantly check the current location, route, estimated arrival time, and in-vehicle video feed of the transport vehicle on their device. For example, if a parent opens the app on their device while at work, they can check in real time where their child is, what route they are taking, and what the situation is like inside the vehicle. Furthermore, video data and location information are encrypted and communicated between the server and the device, ensuring privacy and security. For example, if a parent requests to "check the video now" from their device, the server will immediately stream the video. For example, if a parent presses the "emergency call" button on their device, the server will immediately and automatically notify the police and emergency contacts.

[0039] Furthermore, the system integrates with the vehicle dispatch service to automatically arrange transportation vehicles based on an optimized schedule. For example, it automatically assigns the most suitable vehicle and driver by considering factors such as the driver, vehicle type (sedan, minivan, electric vehicle, etc.), vehicle amenities (child seats, wheelchair accessibility, air purifier, etc.), driver ratings, vehicle fuel level and battery level, and driver's work status. For instance, if a child has allergies, a vehicle with an air purifier will be prioritized. If siblings are being transported simultaneously, a larger vehicle such as a minivan will be arranged. During nighttime or bad weather, vehicles with high driver ratings will be prioritized.

[0040] The system also notifies parents' devices in real time of the progress of the transportation plan. For example, it will send updates such as, "We left school at 3 PM," "We arrived at soccer practice at 4 PM," "We left for cram school at 6 PM," and "We arrived home at 8 PM." Furthermore, if delays or problems occur, it will immediately notify parents' devices and suggest alternative measures as needed (e.g., arranging another vehicle, canceling extracurricular activities, making emergency contact with parents, etc.).

[0041] As described above, in the embodiments of the present invention, the functions of the schedule management unit, schedule optimization unit, and video monitoring unit are distributed and implemented on the server and terminal, and the generative AI is responsible for core processing such as schedule optimization, anomaly detection, data analysis, and vehicle dispatch, thereby significantly reducing the burden on parents and realizing safe and efficient transportation and schedule management for children. For example, if a parent requests from the terminal, "I have to work overtime unexpectedly today, so I would like to automate all of my child's transportation," the generative AI on the server side immediately recalculates the schedule, arranges the optimal transportation vehicle, and provides the parent with real-time progress and video. For example, in the event of a sudden downpour, the generative AI immediately acquires weather information and automatically decides to cancel or postpone outdoor lessons and change the transportation route. For example, even if siblings have lessons at different locations at the same time, the generative AI efficiently allocates multiple transportation vehicles, minimizing the burden on parents. For example, if a parent requests from the terminal, "I want to check the video now," the server immediately delivers the video stream. For example, if a parent presses the "Emergency Call" button on their device, the server will immediately and automatically notify the police and emergency contacts.

[0042] Thus, the embodiments of the present invention have a specific and comprehensive configuration that addresses the diverse needs of modern dual-income families and low-birthrate societies, including cooperative operation between servers and terminals, advanced data analysis and optimization by generative AI, real-time video monitoring, integration with external services, user interface flexibility, and security and privacy protection.

[0043] (System Configuration)

[0044] The system according to this embodiment comprises a schedule management unit, a schedule optimization unit, a vehicle dispatch coordination unit, a video monitoring unit, a user interface unit, a database unit, a security management unit, and a notification control unit. The schedule management unit receives various schedule information from parents via a terminal, such as their child's school and extracurricular activity schedules, desired pick-up / drop-off times, pick-up / drop-off locations, the parents' own work schedules, family events and sudden schedule changes, sibling schedules, and absence information due to illness. It also integrates with external calendar apps, social media, school communication systems, and extracurricular activity reservation systems via API, enabling features such as automatic schedule acquisition from Google Calendar® and iOS Calendar®, automatic reflection of temporary school closure information from schools, automatic acquisition of venue change information from extracurricular activity organizers, and schedule sharing among parents via social media. The schedule management unit centrally manages the schedules of multiple children, allowing for individual registration of, for example, the eldest son's soccer lessons, the eldest daughter's piano lessons, and the second son's cram school, while simultaneously managing pick-up / drop-off plans for siblings.

[0045] The schedule optimization unit is primarily implemented on the server side and utilizes generative AI to comprehensively analyze diverse data received from the schedule management unit, including schedule information, traffic information, weather information, vehicle availability, road safety information, parents' work schedules, children's health and preferences, past travel history, driver evaluation information for transport vehicles, road construction information, event information, disaster information, local security information, and the fuel and battery levels of transport vehicles. For example, if a child has multiple extracurricular activities, the generative AI considers all factors such as the start and end times, locations, travel time, traffic congestion, weather forecasts, road safety, vehicle availability, parents' work end times, sibling schedules, driver evaluations, road closures due to construction or events, local security information, disaster information, and the fuel and battery levels of transport vehicles, and automatically generates the most efficient and safest transport order, times, routes, and means of transport. For example, if a soccer class is held outdoors and a sudden downpour is forecast for that day, the generative AI will suggest canceling or postponing the outdoor activity and changing the pick-up / drop-off route. For example, if a large event is being held near a cram school, it will suggest leaving earlier or taking an alternative route to avoid traffic congestion. For example, if the driver of the pick-up / drop-off vehicle has a low rating or the vehicle's fuel level is low, it will prioritize assigning a different vehicle or driver. For example, at night or in bad weather, it will prioritize arranging a vehicle with a high driver rating. Specific examples of prompts to be fed into the generative AI include: "Consider the following child's schedule, parent's work schedule, traffic information, weather information, availability of pick-up / drop-off vehicles, and road safety information, and generate the most efficient and safe pick-up / drop-off plan. If siblings have overlapping schedules, assign multiple vehicles, and prioritize safety at night or in bad weather. Also consider the driver's rating and fuel level of the pick-up / drop-off vehicles."

[0046] The dispatch coordination unit automatically arranges transportation vehicles in cooperation with the dispatch service based on the transportation plan generated by the schedule optimization unit. For example, it automatically assigns the optimal vehicle and driver by considering factors such as the driver, vehicle type (sedan, minivan, electric vehicle, etc.), vehicle equipment (child seats, wheelchair accessibility, air purifier, etc.), driver evaluation information, vehicle fuel level and battery level, and driver's work status. For instance, if a child has allergies, a vehicle with an air purifier will be prioritized; if siblings are being transported at the same time, a larger vehicle such as a minivan will be arranged; and during nighttime or bad weather, a vehicle with a highly-rated driver will be prioritized. The dispatch coordination unit also works with the dispatch service's API to obtain the vehicle's location and estimated arrival time in real time and manage the progress of the transportation plan.

[0047] The video monitoring unit transmits real-time video from cameras mounted on the transport vehicles to a server, where it uses generative AI to analyze the video data and location information to automatically detect abnormal behavior and signs of danger. For example, it can detect anomalies such as a child crying in the car, the vehicle traveling on an unscheduled route, the vehicle being stopped for an extended period, the driver behaving suspiciously, or a suspicious person being a passenger in the vehicle. The video monitoring unit also provides a function that allows parents to constantly check the current location, route, estimated arrival time, and in-vehicle video of the transport vehicle on their devices. For example, if a parent opens the app on their device while at work, they can check in real time where their child is, what route the vehicle is traveling on, and what the situation inside the vehicle is like. Video data and location information are encrypted and communicated between the server and the device, ensuring privacy and security. Specific examples of prompts to be fed into the generative AI include: "Analyze the real-time video and location information of the transport vehicle and detect abnormal behavior of children or abnormal vehicle operation. If an anomaly is detected, immediately notify the parent's device and make an emergency call if necessary."

[0048] The user interface unit provides an operation screen that allows parents to input, modify, and confirm schedules, approve pick-up and drop-off plans, view real-time video, receive progress notifications, and send emergency contacts from their devices. For example, if a parent requests from their device, "I have to work overtime unexpectedly today, so I want to automate all of my child's pick-up and drop-off," the user interface unit sends that information to the server, and the schedule optimization unit immediately recalculates. For example, if a parent requests from their device, "I want to check the video now," the user interface unit sends a request to the server, and the video monitoring unit immediately delivers the video stream. For example, if a parent presses the "emergency call" button from their device, the user interface unit notifies the server, and the notification control unit immediately and automatically notifies the police and emergency contacts.

[0049] The database unit centrally manages data for the entire system, including schedule information, transportation plans, traffic information, weather information, vehicle information, driver evaluation information, video data, location information, notification history, and user settings information. For example, it can store past transportation history and anomaly detection history, which can then be used by generative AI to optimize future trips and improve the accuracy of anomaly detection. For instance, parents can review past transportation history and specify and arrange for a particular driver or vehicle.

[0050] The Security Management Department is responsible for encrypting data communications and video data across the entire system, user authentication, access control, privacy protection, and preventing information leaks. For example, it encrypts communication between parental devices and servers using SSL to prevent unauthorized access to video data and location information. For example, it sets access permissions for each user to prevent anyone other than parents from accessing video data. For example, if unauthorized access to the system or information leakage is detected, it immediately notifies the administrator and automatically implements the necessary countermeasures.

[0051] The notification control unit notifies parents' devices in real time of various events, anomaly detections, progress updates, schedule changes, and emergencies that occur within the system. For example, it provides real-time progress updates such as "Left school at 3 PM," "Arrived at soccer practice at 4 PM," "Left for cram school at 6 PM," and "Arrived home at 8 PM." For example, if a delay or problem occurs, it immediately notifies parents' devices and suggests alternative measures as needed (e.g., arranging another vehicle, canceling the lesson, making an emergency call to the parent). For example, when an anomaly is detected, it displays a warning on the parent's device and automatically notifies the police or emergency contacts as needed.

[0052] As described above, the system according to this embodiment consists of a schedule management unit, a schedule optimization unit, a vehicle dispatch coordination unit, a video monitoring unit, a user interface unit, a database unit, a security management unit, and a notification control unit, all working in cooperation with each other. It has a specific and comprehensive configuration to meet the diverse needs of modern dual-income families and a society with a declining birthrate, including advanced data analysis and optimization centered on generative AI, real-time video monitoring, integration with external services, a flexible user interface, security and privacy protection, and real-time notifications.

[0053] (Implementation Steps)

[0054] Step 1: Collect and register schedule information Parents can input a variety of schedule information from their devices, including their child's school and extracurricular activity schedules, preferred pick-up / drop-off times and locations, their own work schedules, family events and sudden schedule changes, sibling schedules, and absence information due to illness. Furthermore, it integrates with external calendar apps, social media, school communication systems, and extracurricular activity reservation systems via APIs, enabling automatic retrieval of schedules from Google Calendar® and iOS Calendar®, automatic reflection of temporary school closure information from schools, automatic retrieval of venue change information from extracurricular activity organizers, and schedule sharing among parents via social media. This allows for centralized management of schedules for multiple children and parents' work schedules.

[0055] Step 2: Integrate and retrieve external data The system acquires real-time traffic information, weather information, vehicle operating status, road safety information, driver evaluation information, road construction information, event information, disaster information, local security information, and vehicle fuel and battery levels on the server side, and integrates this with schedule information. For example, traffic congestion information, sudden heavy rain, road closures due to road construction, deterioration of local security, and low battery levels of vehicle transport are automatically reflected.

[0056] Step 3: Optimizing transportation plans using generated AI The server-side schedule optimization unit uses generative AI to comprehensively analyze schedule information, external data, past travel history, the child's health and preferences, siblings' schedules, driver evaluations, and vehicle equipment to automatically generate the most efficient and safe pick-up / drop-off sequence, pick-up / drop-off times, pick-up / drop-off routes, and means of transportation. For example, if an outdoor lesson is canceled due to rain, or if pick-up / drop-off for siblings overlaps at the same time, it automatically decides whether to assign multiple vehicles or change the route. Specific examples of prompts to be fed into the generating AI include: "Consider the following: the child's schedule, the parent's work schedule, traffic information, weather information, the availability of the pick-up / drop-off vehicles, and road safety information, and generate the most efficient and safe pick-up / drop-off plan. If siblings' schedules overlap, assign multiple vehicles, and prioritize safety at night or in bad weather. Also consider the driver evaluation and fuel level of the pick-up / drop-off vehicles."

[0057] Step 4: Coordination with ride-hailing services and vehicle arrangement. Based on the transportation plan generated by the Schedule Optimization Department, the Dispatch Coordination Department integrates with the dispatch service via API to automatically arrange the most suitable vehicle and driver. For example, priority is given to assigning vehicles equipped with child seats, air purifiers, large vehicles such as minivans, and vehicles with high driver ratings. The Dispatch Coordination Department obtains the vehicle's location and estimated arrival time in real time and manages the progress.

[0058] Step 5: Real-time video monitoring and anomaly detection Real-time video from cameras mounted on the transport vehicles is transmitted to a server, and the video monitoring unit uses generative AI to analyze the video data and location information to automatically detect abnormal behavior in children, abnormal vehicle operation, suspicious behavior by the driver, suspicious persons in the vehicle, and prolonged stops. For example, it can immediately detect if a child is crying in the vehicle or if the vehicle is traveling on an unscheduled route. Specific examples of prompts to be fed into the generative AI include: "Analyze the real-time video and location information of the transport vehicle to detect abnormal behavior in children and abnormal vehicle operation. If an abnormality is detected, immediately notify the parent's terminal and make an emergency call if necessary."

[0059] Step 6: Provide notifications and interfaces to parental devices. The user interface section provides a screen for parents to input, modify, and confirm schedules from their devices, approve transportation plans, view real-time video, receive progress notifications, and make emergency contacts. For example, it provides real-time progress notifications such as "Left school at 3 PM," "Arrived at soccer practice at 4 PM," "Left for cram school at 6 PM," and "Arrived home at 8 PM." It also immediately notifies parents of delays or problems and suggests alternative means or emergency calls as needed.

[0060] Step 7: Database Recording and Security Management The database department centrally records and manages information such as schedules, transportation plans, traffic information, weather information, vehicle information, driver evaluation information, video data, location information, notification history, and user settings. The security management department is responsible for data communication and video data encryption, user authentication, access control, privacy protection, and information leakage prevention, such as SSL encryption of communications, setting access rights for each user, and automatically executing countermeasures when unauthorized access is detected.

[0061] (Specific use cases)

[0062] For example, consider a dual-income household where parents work full-time while their children participate in multiple extracurricular activities on weekdays. Parents input information from their device, including their child's school dismissal time, the start time and location of soccer lessons, the start time and location of tutoring, desired return time, siblings' extracurricular activity schedules, their own expected work end time, family events or sudden schedule changes, and the child's health and allergy information. Furthermore, the system integrates with external calendar apps, school communication systems, and extracurricular activity reservation systems via APIs to automatically retrieve information such as temporary dismissal time changes from schools, venue changes from activity organizers, and schedule sharing among parents via social media, all of which are centrally managed by the schedule management department.

[0063] The scheduling department manages individual schedules simultaneously, even when siblings participate in different extracurricular activities. For example, if the eldest son attends soccer lessons, the eldest daughter takes piano lessons, and the second son attends cram school, the department manages all of their schedules in a unified manner. External data such as traffic information, weather information, vehicle availability, road safety information, driver evaluation information, road construction information, event information, disaster information, local security information, and vehicle fuel and battery levels are acquired in real time and integrated with the schedule information. For example, traffic congestion information, sudden heavy rain, road closures due to construction, deterioration of local security, and low vehicle battery levels are automatically reflected.

[0064] The schedule optimization unit uses generative AI to comprehensively analyze schedule information, external data, past travel history, children's health and preferences, siblings' schedules, driver ratings, and vehicle equipment to automatically generate the most efficient and safe pick-up / drop-off order, pick-up / drop-off times, pick-up / drop-off routes, and pick-up / drop-off methods. For example, if a soccer lesson is held outdoors and a sudden downpour is forecast for that day, the generative AI will suggest canceling or postponing the outdoor activity and change the pick-up / drop-off route. If pick-up / drop-off for siblings overlaps at the same time, the system will automatically decide whether to assign multiple vehicles or change the route. At night or in bad weather, the system will prioritize arranging vehicles with high driver ratings and select a safe route considering local safety information. A specific example of a prompt message to be fed into the generative AI is: "Consider the following children's schedules, parents' work schedules, traffic information, weather information, vehicle availability, and road safety information, and generate the most efficient and safe pick-up / drop-off plan. If siblings' schedules overlap, assign multiple vehicles, and at night or in bad weather, prioritize safety. Also consider the driver ratings and fuel levels of the vehicles."

[0065] The dispatch coordination unit, based on the transportation plans generated by the schedule optimization unit, integrates with the dispatch service via API to automatically arrange the most suitable vehicle and driver. For example, it prioritizes the allocation of vehicles equipped with child seats, air purifiers, large vehicles such as minivans, and vehicles with high driver ratings. The dispatch coordination unit obtains the vehicle's location and estimated arrival time in real time and manages the progress.

[0066] The video monitoring unit transmits real-time video from cameras mounted on the transport vehicles to a server. Using generative AI, it analyzes the video data and location information to automatically detect abnormal behavior in children, abnormal vehicle operation, suspicious driver behavior, suspicious persons in the vehicle, and prolonged stops. For example, it can immediately detect if a child is crying in the vehicle or if the vehicle is traveling on an unscheduled route. A specific example of a prompt message to be fed into the generative AI is: "Analyze the real-time video and location information of the transport vehicle to detect abnormal behavior in children and abnormal vehicle operation. If an abnormality is detected, immediately notify the parent's terminal and make an emergency call if necessary."

[0067] The user interface provides a screen for parents to input, modify, and confirm schedules, approve transportation plans, view real-time video, receive progress notifications, and make emergency contacts from their devices. For example, it provides real-time notifications of progress such as "Leave school at 3 PM," "Arrive at soccer practice at 4 PM," "Leave for cram school at 6 PM," and "Arrive home at 8 PM," and immediately notifies parents of delays or problems, suggesting alternative means or emergency calls as needed.

[0068] The database department centrally records and manages schedule information, transportation plans, traffic information, weather information, vehicle information, driver evaluation information, video data, location information, notification history, and user settings information. The security management department is responsible for data communication and video data encryption, user authentication, access control, privacy protection, and information leakage prevention. For example, it performs communication encryption using SSL, sets access rights for each user, and automatically executes countermeasures when unauthorized access is detected.

[0069] In such use cases, the system of the present invention can significantly reduce the burden on parents while enabling safe and efficient transportation and schedule management for children.

[0070] (Application Example 1) The flow of a specific process in Application Example 1 will be explained below. Each part of the system described below is implemented by a data processing device 12 and a smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0071] (Embodiments for carrying out the present invention)

[0072] Embodiments of the present invention are systems that integrate multi-layered and complex elements, such as the work status of family members and care staff, traffic, weather, and local safety information, vehicle dispatch services, monitoring cameras, optimization by generative AI, real-time video monitoring, user interface, and security management, to realize all processes related to outings and daily life support for those requiring care. Each component will be described in more specific and detailed terms below.

[0073] The schedule management system allows family members and care staff to individually input a wide variety of outings and daily living support schedules for those requiring care, including hospital visits, rehabilitation, shopping, hobbies, day services, outings with friends, weddings and funerals, administrative procedures, picking up prescriptions at pharmacies, walks, participation in community events, visits to hair salons, home nursing care, home rehabilitation, home bathing, home dental care, home beauty services, receiving meal delivery services, taking out the trash, receiving mail, and participating in community volunteer activities, all from their own devices. For example, a family member can register a schedule such as "June 15th, 10am: Clinic visit," "1pm: Rehabilitation at rehabilitation facility," "3pm: Shopping at the supermarket," and "5pm: Return home from day services" from their smartphone or tablet. Furthermore, it can automatically retrieve and reflect the schedules of those requiring care by integrating with external calendar apps, medical institution reservation systems, day service attendance management systems, community comprehensive support center communication systems, long-term care insurance provider schedule management systems, pharmacy reservation systems, and home service reservation systems via API. For example, if there is a change in a medical appointment, a temporary closure of a day care service, a change in the date and time of medication pickup from the pharmacy, an additional home nursing appointment, or a change in the date and time of a local event, the system will automatically retrieve the information and reflect it in the schedule. When siblings or multiple family members are involved in supporting the person requiring care, each person's schedule and assigned days can be managed simultaneously, and responsibilities can be flexibly assigned by day of the week and time of day, for example, "Family A will provide transportation to and from medical appointments on Monday, care staff B will provide transportation to and from rehabilitation on Tuesday, and Family C will accompany the person shopping on Wednesday."

[0074] The system meticulously manages information such as the availability of family members and care staff, their expertise, qualifications, work history, past evaluations, emergency contact information, health status, preferred working days, preferred duties, means of transportation (private car, bicycle, walking, public transport, etc.), the level of care they can handle, gender, age, language skills, religious considerations, allergy considerations, infection control history, and vaccination history. For example, it can register detailed information such as: Family A is available from 9 am to 6 pm; care staff B is available from 12 pm to 5 pm; staff C is qualified to assist with wheelchairs; staff D has received specialized training in dementia care; staff E has received high ratings from past care recipients; staff F is male and good at physical work; staff G is female and skilled in dementia care; staff H can speak English; staff I can accommodate clients with religious considerations; staff J has received infection control training; and staff K is vaccinated. This allows for the automatic assignment of staff with the appropriate expertise and availability for each planned outing. For example, family member A can be assigned to morning appointments, while care staff member B can be assigned to afternoon rehabilitation or shopping. If the person requiring care is female, female staff members will be given priority; if the person with dementia is a female, staff trained in dementia care will be given priority; if a wheelchair user is a wheelchair user, a qualified wheelchair assistant will be assigned; if heavy lifting is required, male staff members will be assigned; if English communication is needed, English-speaking staff members will be assigned; if religious considerations are required, appropriate staff members will be assigned; and during infectious disease outbreaks, vaccinated staff members will be given priority. Extremely meticulous considerations are possible.

[0075] External data such as traffic information, weather information, local safety information, ride-hailing service operational status, public transport operation status, road construction information, accident information, disaster information, local event information, local public safety information, police / fire / ambulance dispatch information, nearby crime information, Japan Meteorological Agency warnings and advisories, heatstroke warnings, snow / freezing information, pollen information, yellow dust / PM2.5 information, and infectious disease outbreak information are acquired and integrated in real time on the server side. For example, traffic congestion, road construction, accidents, public transport delays, torrential downpours and typhoons, sudden weather changes such as extreme heat, snow, and strong winds, worsening local public safety and accident-prone areas, disaster occurrences, ride-hailing service vehicle shortages and low battery levels, pollen and yellow dust dispersion, and infectious disease outbreaks are automatically reflected. For example, if there is traffic congestion when traveling from a clinic to a rehabilitation facility, the system will automatically select an alternative route. If rain is forecast when traveling to the rehabilitation facility, warnings such as "bring an umbrella," "wear rain gear," and "be careful of slippery areas" will be added. If the area around the supermarket is a high-risk area for safety, staff will be notified with instructions such as "strengthen patrols," "avoid going out at night," and "choose well-trafficked routes." If there is a shortage of vehicles for the ride-hailing service, it will be suggested to switch to public transport or walking. On days with high levels of pollen or yellow dust, it will be reminded to "wear a mask" and "reduce the time spent outdoors." During infectious disease outbreaks, instructions such as "wear a mask," "sanitize hands," and "avoid crowded places" will be automatically added.

[0076] The Schedule Optimization Department uses generative AI to comprehensively analyze factors such as the care recipient's planned outings, the availability of family members and care staff, traffic congestion, weather forecasts, local safety information, the operational status of ride-hailing services, vehicle equipment, staff expertise, the care recipient's health and preferences, past travel history, care level, medical history, medication status, dietary restrictions, allergies, presence of dementia, walking ability, presence of visual or hearing impairments, the wishes of the care recipient and family, local medical and welfare resources, and emergency response plans. Based on this analysis, it dynamically determines the optimal pick-up and drop-off order, pick-up and drop-off times, pick-up and drop-off routes, assigned staff, vehicle arrangements, monitoring camera arrangements, and precautions. For example, if a person requiring care has multiple outings planned, the system automatically generates the most efficient and safest transportation plan by considering all factors, including the start and end times, locations, travel time, traffic congestion, weather forecast, local safety information, staff availability, vehicle equipment, staff expertise, the person requiring care's health and preferences, past travel history, care level, medical history, medication status, dietary restrictions, allergies, presence of dementia, walking ability, presence of visual or hearing impairments, the wishes of the person and their family, local medical and welfare resources, and emergency response plans. For instance, if the plan involves a morning doctor's appointment followed by rehabilitation and then shopping, the system proposes a manageable schedule considering travel distance and time, weather and traffic conditions, staff changeover timing, the person requiring care's fatigue level, medication timing, dietary restrictions, need for toilet breaks, availability of walking aids, wheelchair use, consideration for visual and hearing impairments, the wishes of the person and their family, local medical and welfare resources, and emergency response plans. For example, if a person requiring care cancels an outing due to illness the previous day, the system automatically readjusts the day's schedule and notifies medical institutions and family members as needed. When siblings or multiple family members are involved, the system optimizes the schedules of multiple individuals simultaneously and efficiently allocates transportation vehicles and staff. For example, even if siblings attend different facilities, the generative AI automatically determines the optimal transportation order and vehicle allocation.

[0077] The dispatch coordination department, based on the transportation plan generated by the schedule optimization department, integrates with the dispatch service via API to automatically arrange the most suitable vehicle and driver. For example, it prioritizes the allocation of vehicles based on factors such as wheelchair accessibility, large vehicles, vehicles with air purifiers, vehicles with child seats, vehicles with highly-rated drivers, female drivers, vehicles with infection control measures in place, barrier-free vehicles, pet-friendly vehicles, electric vehicles, vehicles with sufficient fuel and battery levels, driver expertise and ratings, driver health status, driver work preferences, driver language skills, driver religious considerations, and driver vaccination history. The dispatch coordination department obtains the vehicle's location and estimated arrival time in real time and manages the progress. For example, if a vehicle is delayed, it immediately notifies the terminals of family members and care staff and suggests arranging an alternative vehicle or rescheduling as needed. If a vehicle's battery level is low, it automatically switches to a fully charged vehicle.

[0078] The video monitoring unit transmits real-time video from cameras mounted on the transport vehicles to a server. Using generative AI, it analyzes the video data and location information to automatically detect abnormal behavior of the care recipient, abnormal vehicle operation, suspicious driver behavior, suspicious person in the vehicle, prolonged stopping, falls, sudden illness, verbal abuse / violence, trouble inside the vehicle, vehicle lane departure, sudden illness of the driver, vehicle accidents, vehicle theft, abandonment inside the vehicle, abnormal temperature inside the vehicle, poor ventilation inside the vehicle, and increased risk of infectious diseases inside the vehicle. For example, it can immediately detect if a care recipient complains of feeling unwell inside the vehicle, if the vehicle is traveling on an unscheduled route, if the driver is behaving suspiciously, if a suspicious person is in the vehicle, if the vehicle is stopped for a long time, if a fall or sudden illness occurs, if verbal abuse / violence occurs, if trouble occurs inside the vehicle, if the vehicle veers off course or an accident occurs, if the vehicle is stolen or abandoned, if the temperature inside the vehicle is abnormal or ventilation is poor, or if the risk of infectious diseases increases, and immediately notify family members and care staff on their terminals. If necessary, it can also automatically notify emergency contacts and medical institutions. Video data and location information are encrypted and transmitted between the server and the device, ensuring privacy and security. Family members and care staff can check the vehicle's status and movement in real time from their devices.

[0079] The user interface section provides an operation screen that allows family members and care staff to input, modify, and confirm schedules from their terminals, approve transportation plans, view real-time video, receive progress notifications, make emergency contacts, issue instructions for responding to detected anomalies, change assigned personnel, specify vehicles and staff, add notes, reflect the wishes of the individual and family, utilize local resources, confirm emergency response plans, refer to past history, and input evaluations and feedback. For example, it provides real-time notifications of progress such as "Depart for clinic at 10:00," "Arrive at rehabilitation facility at 11:00," "Depart for supermarket at 13:00," and "Arrive home at 15:30," immediately notifying of delays or problems and suggesting alternative means or emergency calls as needed. When an anomaly is detected, a warning is displayed on the terminals of family members and care staff, and automatic notifications are made to the police or medical institutions as needed.

[0080] The Database Department centrally records and manages schedule information, transportation plans, traffic information, weather information, vehicle information, driver evaluation information, video data, location information, notification history, user settings information, personal and family preferences, local resource information, emergency response plans, past history, evaluations and feedback, etc. The Security Management Department is responsible for data communication and video data encryption, user authentication, access control, privacy protection, information leakage prevention, unauthorized access detection, cyberattack countermeasures, data backup in the event of a disaster, and automatic recovery in the event of a system failure. For example, it performs communication encryption using SSL, sets access rights for each user, executes automatic countermeasures when unauthorized access is detected, backs up data in the event of a disaster, automatically recovers in the event of a system failure, and takes measures against cyberattacks.

[0081] Thus, the embodiments of the present invention have an extremely specific and comprehensive configuration to address the diverse challenges and needs in modern care settings, including the diverse outing and life support needs of those requiring care, the working conditions of family members and care staff, traffic, weather, and local safety information, ride-hailing services, monitoring cameras, optimization by generative AI, real-time video monitoring, user interface flexibility, security and privacy protection, emergency response, utilization of local resources, reflection of the wishes of the individual and their family, and utilization of evaluation and feedback.

[0082] (System Configuration)

[0083] The system according to this embodiment comprises a schedule management unit, a staff management unit, an external information acquisition unit, a schedule optimization unit, a vehicle dispatch coordination unit, a video monitoring unit, a user interface unit, a database unit, a security management unit, and a notification control unit. The schedule management unit allows family members and care staff to individually input a wide variety of outing and life support schedules from their terminals, such as hospital visits, rehabilitation, shopping, hobby activities, day services, outings with friends, weddings and funerals, administrative procedures, picking up medication at pharmacies, walks, participation in local events, use of hair salons and beauty salons, home nursing and home rehabilitation, receiving meal delivery services, taking out the trash, receiving mail, and participating in local volunteer activities. Furthermore, it can integrate via API with external calendar apps, medical institution reservation systems, day service attendance management systems, community comprehensive support center communication systems, pharmacy reservation systems, and home service reservation systems, enabling it to automatically acquire information such as changes to medical institution reservations, temporary closures of day services, changes to medication pick-up dates and times from pharmacies, additions to home nursing appointments, and changes to local event dates and times, and reflect them in the schedule.

[0084] The staff management department meticulously manages information such as the availability of family members and care staff, their expertise, qualifications, work history, past performance evaluations, emergency contact information, health status, preferred working days, preferred duties, means of transportation, the level of care they can handle, gender, age, language skills, religious considerations, allergy considerations, infection control history, and vaccination history. For example, it manages information such as: Family A is available from 9 am to 6 pm; care staff B is available from 12 pm to 5 pm; staff C is qualified to assist with wheelchairs; staff D has received specialized training in dementia care; staff E has received high ratings from care recipients in the past; staff F is a male who is good at physical work; staff G is a female who is skilled in dementia care; staff H can speak English; staff I can handle clients with religious considerations; staff J has received infection control training; and staff K is vaccinated. The department centrally manages this extremely diverse personnel information and automatically assigns the appropriate staff member for each outing.

[0085] The external information acquisition unit acquires information such as traffic information, weather information, local safety information, ride-hailing service operational status, public transportation operation status, road construction information, accident information, disaster information, local event information, local public safety information, police, fire, and ambulance dispatch information, nearby crime information, Japan Meteorological Agency warnings and advisories, heatstroke warning information, snow and freezing information, pollen information, yellow dust and PM2.5 information, and infectious disease outbreak information in real time. For example, it automatically reflects traffic congestion, road construction, accidents, public transportation delays, torrential downpours and typhoons, extreme heat, snow and strong winds, deteriorating local public safety and accident-prone areas, disaster occurrences, ride-hailing service vehicle shortages and low battery levels, pollen and yellow dust dispersion, and infectious disease outbreaks in schedules and optimization processes.

[0086] The Schedule Optimization Department uses generative AI to comprehensively analyze factors such as the care recipient's planned outings, the availability of family members and care staff, traffic congestion, weather forecasts, local safety information, ride-hailing service availability, vehicle equipment, staff expertise, the care recipient's health and preferences, past travel history, care level, medical history, medication status, dietary restrictions, allergies, presence of dementia, walking ability, presence of visual or hearing impairments, the wishes of the care recipient and family, local medical and welfare resources, and emergency response plans. Based on this analysis, it dynamically determines the optimal pick-up and drop-off order, times, routes, assigned staff, vehicle arrangements, monitoring camera arrangements, and precautions. For example, in a scenario involving a morning doctor's appointment followed by rehabilitation and then shopping, the system proposes a manageable schedule considering factors such as travel distance and time, weather and traffic conditions, staff change timing, the care recipient's fatigue level, medication timing, dietary restrictions, need for toilet breaks, availability of walking aids, wheelchair use, consideration for visual and hearing impairments, the wishes of the care recipient and family, local medical and welfare resources, and emergency response plans. Specific examples of prompts to be fed into the generating AI include: "Consider the following: the care recipient's planned outing, the availability of family and care staff, traffic information, weather information, local safety information, availability of ride-hailing services, the care recipient's health and preferences, the staff's expertise, past travel history, care level, medical history, medication status, dietary restrictions, allergies, presence of dementia, walking ability, presence of visual or hearing impairment, the wishes of the care recipient and family, local medical and welfare resources, and emergency response plans, and generate the safest and most efficient transportation and monitoring plan. Bring an umbrella in case of rain, enhance monitoring in areas requiring special attention, and automatically switch staff if staff availability exceeds their hours. Clearly indicate the departure point, destination, departure time, arrival time, route, precautions, vehicle, camera, and assigned staff for the entire trip."

[0087] The dispatch coordination department, based on the transportation plans generated by the schedule optimization department, integrates with the dispatch service via API to prioritize the allocation of vehicles based on factors such as wheelchair accessibility, large vehicles, vehicles with air purifiers, vehicles with child seats, vehicles with high driver ratings, female drivers, vehicles with infection control measures in place, barrier-free vehicles, pet-friendly vehicles, electric vehicles, vehicles with sufficient fuel and battery levels, driver expertise and ratings, driver health status, driver work preferences, driver language skills, driver religious considerations, and driver vaccination history. It also obtains the vehicle's location and estimated arrival time in real time and manages the progress. For example, if a vehicle is delayed, it immediately notifies the terminals of family members and care staff and proposes alternative vehicles or schedule changes as needed.

[0088] The video monitoring unit transmits real-time video from cameras mounted on the transport vehicles to a server. Using generative AI, it analyzes the video data and location information to automatically detect abnormal behavior of the care recipient, abnormal vehicle operation, suspicious driver behavior, suspicious person in the vehicle, prolonged stopping, falls, sudden illness, verbal abuse / violence, trouble inside the vehicle, vehicle lane departure, sudden illness of the driver, vehicle accident, vehicle theft, abandonment inside the vehicle, abnormal temperature inside the vehicle, poor ventilation inside the vehicle, and increased risk of infectious diseases inside the vehicle. For example, if a care recipient complains of feeling unwell inside the vehicle, if the vehicle is traveling on an unscheduled route, if the driver is behaving suspiciously, if a suspicious person is in the vehicle, if the vehicle is stopped for a long time, if a fall or sudden illness occurs, if verbal abuse / violence occurs, if trouble occurs inside the vehicle, if the vehicle veers off course or an accident occurs, if the vehicle is stolen or abandoned, if the temperature inside the vehicle is abnormal or ventilation is poor, or if the risk of infectious diseases increases, it will immediately detect these issues and notify family members and care staff on their terminals, and, if necessary, automatically notify emergency contacts and medical institutions. Video data and location information are encrypted and communicated between the server and the terminal, allowing family members and care staff to check the vehicle's condition and movement in real time from their terminals.

[0089] The user interface provides a screen for family members and care staff to input, modify, and confirm schedules from their terminals, approve transportation plans, view real-time video, receive progress notifications, make emergency contacts, issue response instructions in case of anomalies, change assigned personnel, specify vehicles and staff, add notes, reflect the wishes of the individual and family, utilize local resources, confirm emergency response plans, refer to past history, and input evaluations and feedback. For example, it provides real-time notifications of progress such as "Depart for clinic at 10:00," "Arrive at rehabilitation facility at 11:00," "Depart for supermarket at 13:00," and "Arrive home at 15:30," immediately notifying of delays or problems, and suggesting alternative means or emergency calls as needed.

[0090] The Database Department centrally records and manages schedule information, transportation plans, traffic information, weather information, vehicle information, driver evaluation information, video data, location information, notification history, user settings information, personal and family preferences, local resource information, emergency response plans, past history, evaluations and feedback, etc. The Security Management Department is responsible for data communication and video data encryption, user authentication, access control, privacy protection, information leakage prevention, unauthorized access detection, cyberattack countermeasures, data backup in case of disaster, and automatic recovery in case of system failure. For example, it performs communication encryption using SSL, setting access rights for each user, executing automatic countermeasures when unauthorized access is detected, data backup in case of disaster, automatic recovery in case of system failure, and cyberattack countermeasures.

[0091] The notification control unit notifies family members and care staff terminals in real time of various events, anomaly detections, progress status, schedule changes, and emergencies that occur within the system. For example, it provides timely updates on progress such as "Departing for the clinic at 10:00," "Arriving at the rehabilitation facility at 11:00," "Departing for the supermarket at 13:00," and "Arriving home at 15:30." It also immediately notifies of any delays or problems and suggests alternative means or emergency calls as needed.

[0092] (Implementation Steps)

[0093] Step 1: Collect and register schedule information Family members and care staff can individually input diverse outings and daily living support schedules for those requiring care, such as hospital visits, rehabilitation, shopping, hobby activities, day services, outings with friends, weddings and funerals, administrative procedures, picking up prescriptions at pharmacies, walks, participating in community events, using hairdressers and beauticians, home nursing care, home rehabilitation, receiving meal delivery services, taking out the trash, receiving mail, and participating in community volunteer activities, using smartphones, tablets, etc. This information is also automatically retrieved via API integration with external calendar apps, medical institution reservation systems, day service attendance management systems, community comprehensive support center communication systems, pharmacy reservation systems, and home service reservation systems, and then consolidated in the schedule management department.

[0094] Step 2: Integrated management of staff and external information The staff management department meticulously manages information such as the available working hours, expertise, qualifications, work history, past performance evaluations, emergency contact information, health status, preferred working days, preferred work content, means of transportation, acceptable care level, gender, age, language proficiency, religious considerations, allergy considerations, infection control history, and vaccination history of family members and care staff. The external information acquisition department acquires information in real time, including traffic information, weather information, local safety information, ride-hailing service status, public transportation status, road construction information, accident information, disaster information, local event information, local public safety information, police / fire / ambulance dispatch information, nearby crime information, Japan Meteorological Agency warnings and advisories, heatstroke warnings, snow / freezing information, pollen information, yellow dust / PM2.5 information, and infectious disease outbreak information, and integrates all of this information into the database department.

[0095] Step 3: Optimizing transportation and monitoring plans using generated AI The Schedule Optimization Department uses generative AI to comprehensively analyze factors such as the care recipient's planned outings, the availability of family members and care staff, traffic congestion, weather forecasts, local safety information, the operational status of ride-hailing services, vehicle equipment, staff expertise, the care recipient's health and preferences, past travel history, care level, medical history, medication status, dietary restrictions, allergies, presence of dementia, walking ability, presence of visual or hearing impairments, the wishes of the care recipient and family, local medical and welfare resources, and emergency response plans. Based on this analysis, it dynamically determines the optimal pick-up and drop-off order, pick-up and drop-off times, pick-up and drop-off routes, assigned staff, vehicle arrangements, monitoring camera arrangements, and precautions. Specific examples of prompts to be fed into the generating AI include: "Consider the following: the care recipient's planned outing, the availability of family and care staff, traffic information, weather information, local safety information, availability of ride-hailing services, the care recipient's health and preferences, the staff's expertise, past travel history, care level, medical history, medication status, dietary restrictions, allergies, presence of dementia, walking ability, presence of visual or hearing impairment, the wishes of the care recipient and family, local medical and welfare resources, and emergency response plans, and generate the safest and most efficient transportation and monitoring plan. Bring an umbrella in case of rain, enhance monitoring in areas requiring special attention, and automatically switch staff if staff availability exceeds their hours. Clearly indicate the departure point, destination, departure time, arrival time, route, precautions, vehicle, camera, and assigned staff for the entire trip."

[0096] Step 4: Automated dispatch of vehicles, monitoring cameras, and staff. The dispatch coordination department, based on the transportation plans generated by the schedule optimization department, prioritizes the allocation of vehicles based on factors such as wheelchair accessibility, large vehicles, vehicles with air purifiers, vehicles with child seats, vehicles with highly-rated drivers, female drivers, vehicles with infection control measures in place, barrier-free vehicles, pet-friendly vehicles, electric vehicles, vehicles with sufficient fuel and battery levels, driver expertise and ratings, driver health status, driver work preferences, driver language skills, driver religious considerations, and driver vaccination history. It also obtains the vehicle's location and estimated arrival time in real time and manages the progress. Monitoring cameras are also arranged simultaneously, and the assignment of staff is automated.

[0097] Step 5: Real-time video monitoring and anomaly detection The video monitoring unit transmits real-time video from cameras mounted on the transport vehicles to a server. Using generative AI, it analyzes the video data and location information to automatically detect abnormal behavior of the care recipient, abnormal vehicle operation, suspicious driver behavior, suspicious person in the vehicle, prolonged stopping, falls, sudden illness, verbal abuse / violence, trouble inside the vehicle, vehicle lane departure, sudden illness of the driver, vehicle accident, vehicle theft, abandonment inside the vehicle, abnormal temperature inside the vehicle, poor ventilation inside the vehicle, and increased risk of infectious diseases inside the vehicle. For example, if a care recipient complains of feeling unwell inside the vehicle, if the vehicle is traveling on an unscheduled route, if the driver is behaving suspiciously, if a suspicious person is in the vehicle, if the vehicle is stopped for a long time, if a fall or sudden illness occurs, if verbal abuse / violence occurs, if trouble occurs inside the vehicle, if the vehicle veers off course or an accident occurs, if the vehicle is stolen or abandoned, if the temperature inside the vehicle is abnormal or ventilation is poor, or if the risk of infectious diseases increases, it will immediately detect these issues and notify family members and care staff on their terminals, and, if necessary, automatically notify emergency contacts and medical institutions. Specific examples of prompt messages to be fed into the generating AI include: "Analyze real-time video and location information of the transport vehicle to automatically detect abnormal behavior of the care recipient, abnormal vehicle operation, suspicious driver behavior, suspicious person in the vehicle, prolonged stopping, falls, sudden illness, verbal abuse / violence, trouble inside the vehicle, vehicle lane departure, sudden illness of the driver, vehicle accident, vehicle theft, being left unattended inside the vehicle, abnormal temperature inside the vehicle, poor ventilation inside the vehicle, and infection risk inside the vehicle. If an abnormality is detected, immediately notify the terminals of family members and care staff, and, if necessary, automatically notify emergency contacts and medical institutions."

[0098] Step 6: Progress / anomaly notifications and user interface response The notification control unit notifies family members and care staff in real time of various events, anomaly detections, progress status, schedule changes, and emergencies that occur within the system. The user interface unit provides operation screens for family members and care staff to input, modify, and confirm schedules, approve transportation plans, check real-time video, receive progress notifications, send emergency contacts, issue response instructions in case of anomaly detection, change assigned personnel, specify vehicles and staff, add precautions, reflect the wishes of the individual and family, utilize local resources, check emergency response plans, refer to past history, and input evaluations and feedback.

[0099] (Specific use cases)

[0100] For example, consider a case where a person requiring care has multiple outings scheduled for the day: a medical appointment at 10am, rehabilitation at a rehabilitation facility at noon, shopping at a supermarket at 2pm, and participation in a local event at 4pm. The family enters these appointments into the schedule management unit via a terminal, and also automatically retrieves schedule information through API integration with reservation systems for medical institutions and rehabilitation facilities, and information systems for local events. The staff management unit manages information such as whether family member A is available from 9am to 1pm, and care staff member B is available from 1pm to 5pm, and whether staff member B is qualified to assist with wheelchair use and has completed infectious disease control training. The external information acquisition unit acquires real-time information such as traffic congestion information for the day, weather information (sunny in the morning, rain in the afternoon), local safety information (area around the supermarket is a caution area), the operating status of ride-hailing services (vehicle congestion in the afternoon), the operating status of public transportation (some delays in the afternoon), road construction information (construction near the rehabilitation facility), disaster information (no warnings), pollen information (high in the afternoon), and infectious disease outbreak information (warnings issued in the area). The Schedule Optimization Department uses generative AI to comprehensively analyze the entire schedule of the person requiring care, the working hours of family and staff, traffic, weather, safety, vehicle allocation, health, qualifications, past travel history, the wishes of the person and family, medical history, medication status, dietary restrictions, walking ability, allergies, presence of dementia, presence of visual and hearing impairments, local resources, and emergency response plans. Based on this analysis, it assigns family member A to handle morning appointments, selects a route that bypasses construction zones for travel to rehabilitation facilities, assigns staff member B to afternoon shopping and local events, adds precautions such as bringing an umbrella in case of rain, increased monitoring around supermarkets, wearing a mask during pollen season, and avoiding crowded areas during infectious disease outbreaks. It also suggests switching to public transportation if there is congestion in afternoon vehicle allocation. The Vehicle Allocation Coordination Department arranges wheelchair-accessible vehicles in the morning and barrier-free vehicles in the afternoon, assigning them while considering staff member B's qualifications and health condition. The video monitoring unit operates surveillance cameras throughout the entire travel route, and a generative AI analyzes real-time video and location information to automatically detect falls of the person requiring care, sudden illness, vehicle lane departure, prolonged stops, abnormal temperatures inside the vehicle, infection risks, suspicious driver behavior, and other problems inside the vehicle. In the event of an abnormality, it immediately notifies family members and staff on their devices and, if necessary, automatically notifies emergency contacts and medical institutions.The user interface unit allows family members and staff to check progress and real-time video from their terminals, and to immediately issue response instructions, change personnel, select alternative methods, and make emergency calls in the event of an anomaly. The database unit centrally records and manages all schedules, movement history, video data, notification history, evaluations and feedback, the wishes of the individual and their family, local resources, and emergency response plans. The security management unit is responsible for encrypting all data, access control, detecting unauthorized access, disaster backup, and countermeasures against cyberattacks. The notification control unit notifies family members and staff terminals in real time of progress, anomalies, schedule changes, and emergencies. Specific examples of prompts to be fed into the generating AI include: "Consider the following: the care recipient's planned outing, the availability of family and care staff, traffic information, weather information, local safety information, availability of ride-hailing services, the care recipient's health and preferences, the staff's expertise, past travel history, care level, medical history, medication status, dietary restrictions, allergies, presence of dementia, walking ability, presence of visual or hearing impairment, the wishes of the care recipient and family, local medical and welfare resources, and emergency response plans, and generate the safest and most efficient transportation and monitoring plan. Bring an umbrella in case of rain, enhance monitoring in areas requiring special attention, and automatically switch staff if staff availability exceeds their hours. Clearly indicate the departure point, destination, departure time, arrival time, route, precautions, vehicle, camera, and assigned staff for the entire trip."

[0101] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0102] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0103] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0104] For example, the data collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the smart device 14 and processes it with the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using generation AI. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the devices and control units is not limited to the examples described above and can be modified in various ways.

[0105] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0106] [Second Embodiment]

[0107] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0108] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0109] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0110] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and communication interface 44 are also connected to the bus 52.

[0111] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0112] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0113] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0114] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0115] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0116] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0117] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the identification processing unit 290 using these models.

[0118] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is realized by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0119] (Example 1)

[0120] Since the process flow is the same as in Example 1 described in the First Embodiment above, the explanation will be omitted.

[0121] (Application Example 1) Since the process flow is the same as in Application Example 1 described in the first embodiment above, the explanation will be omitted.

[0122] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0123] Furthermore, the processing by the data processing system 10 described above is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0124] For example, the data collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the smart glasses 214 and processes it using the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using generation AI. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0125] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0126] [Third Embodiment]

[0127] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0128] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0129] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0130] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, display 343, and communication interface 44 are also connected to the bus 52.

[0131] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0132] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0133] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0134] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0135] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0136] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0137] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0138] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the headset terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0139] (Example 1) Since the process flow is the same as in Example 1 described in the First Embodiment above, the explanation will be omitted.

[0140] (Application Example 1) Since the process flow is the same as in Application Example 1 described in the first embodiment above, the explanation will be omitted.

[0141] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0142] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0143] For example, the data collection unit is implemented by the control unit 46A of the headset terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the headset terminal 314 and processes it using the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the headset terminal 314 or the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0144] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0145] [Fourth Embodiment]

[0146] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0147] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0148] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0149] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, controlled object 443, and communication interface 44 are also connected to the bus 52.

[0150] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0151] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0152] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0153] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0154] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0155] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0156] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0157] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0158] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0159] (Example 1) Since the process flow is the same as in Example 1 described in the First Embodiment above, the explanation will be omitted.

[0160] (Application Example 1) Since the process flow is the same as in Application Example 1 described in the first embodiment above, the explanation will be omitted.

[0161] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0162] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or external devices, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or external devices.

[0163] For example, the data collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the robot 414 and processes it using the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and controlled object 443 of the robot 414 and the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0164] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0165] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0166] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0167] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0168] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0169] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0170] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0171] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0172] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0173] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0174] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0175] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0176] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0177] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0178] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0179] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0180] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0181] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0182] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0183] Furthermore, the following additional information is disclosed regarding the above explanation.

[0184] (Claim 1)

[0185] The system comprises a schedule management unit that manages the outing schedules and daily living support schedules of care recipients; a schedule optimization unit that integrates and analyzes diverse data such as the working hours of family members and care staff, traffic information, weather information, local safety information, the operating status of the dispatch service, the health condition and individual attributes of the care recipient, and past travel history, and automatically generates optimal transportation and monitoring plans using generative AI; and a video monitoring unit that, based on the optimized plan, automatically arranges transportation vehicles and monitoring cameras in cooperation with the dispatch service, delivers in-vehicle video and location information in real time to the terminals of family members and care staff, and immediately notifies or makes emergency calls when an anomaly is detected. The system is characterized by its ability to automatically acquire various outing schedules and life support schedules, such as hospital visits, rehabilitation, shopping, and hobby activities, by inputting them from the terminals of family members and care staff or by linking with external calendars or medical institution reservation systems. The schedule optimization unit uses generative AI to automatically generate the optimal pick-up and drop-off order, pick-up and drop-off times, pick-up and drop-off routes, assigned staff, and precautions, taking into account multiple constraints such as traffic congestion, sudden weather changes, public safety information, available working hours for staff, changes in the health condition of the person requiring care, the operating status of the dispatch service, vehicle equipment, and the expertise of the staff. The video monitoring unit provides the terminals of family members and care staff in real time with the current location information of the pick-up and drop-off vehicles, in-vehicle video, and immediate notifications and emergency call functions in case of abnormality detection. (Claim 2)

[0186] The system according to claim 1, wherein the schedule optimization unit uses a generative AI to comprehensively analyze the care recipient's planned outings, the available working hours of family members and care staff, traffic congestion, weather forecasts, local security information, the operational status of the ride-hailing service, vehicle equipment, staff expertise, the care recipient's physical condition and preferences, and past travel history, and dynamically determines the optimal pick-up and drop-off order, pick-up and drop-off times, pick-up and drop-off routes, assigned staff, vehicle arrangements, monitoring camera arrangements, and precautions, thereby automatically generating a pick-up and drop-off and monitoring plan to maximize the safety of the care recipient and reduce the burden on family members and staff. (Claim 3)

[0187] The system according to claim 1, wherein the video monitoring unit encrypts real-time video data and location information from a camera mounted on the transport vehicle and distributes it to the terminals of family members and care staff, allowing family members and care staff to constantly check the condition and movement status of the person requiring care during transport via their terminals, and furthermore, using generative AI, it automatically detects abnormal behavior and signs of danger from the video data and location information, and if an abnormality is detected, it immediately notifies the terminals of family members and care staff, and automatically notifies emergency contacts, medical institutions, etc. as necessary. [Explanation of Symbols]

[0188] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. The system comprises a schedule management unit that acquires schedule information for children's extracurricular activities and school; a schedule optimization unit that comprehensively analyzes multiple external data such as the acquired schedule information, traffic information, weather information, and parents' work schedules, and uses generative AI to optimize the child's travel route and pick-up / drop-off times; and a video monitoring unit that automatically selects a safe means of transportation based on the optimized schedule, arranges a vehicle in cooperation with a ride-hailing service, and further delivers real-time video from a camera installed in the vehicle to the parent's terminal. The system is characterized in that the schedule management unit has the function of acquiring information from the child's calendar information registered by the parent and from an external calendar application; the schedule optimization unit uses generative AI to automatically generate an optimal pick-up / drop-off plan considering multiple constraints such as traffic congestion, weather changes, start and end times of extracurricular activities, end times of parents' work, and emergency response requirements; and the video monitoring unit provides the parent's terminal with real-time current location information of the vehicle, video data, and emergency notification functions.

2. The system according to claim 1, wherein the schedule optimization unit uses a generative AI to comprehensively analyze the child's extracurricular activities and school schedule, the parent's work schedule, traffic congestion, weather forecast, the operating status of the transport vehicle, road safety information, past travel history, and individual attribute information such as the child's physical condition and preferences, and dynamically determines the optimal order of transport, transport time, transport route, and transport method, thereby automatically generating a transport plan that maximizes the reduction of the parent's burden and the assurance of the child's safety.

3. The system according to claim 1, wherein the video monitoring unit encrypts real-time video data from a camera mounted on the transport vehicle and distributes it to the parent's terminal, allowing the parent to constantly check the child's condition during transport through an application, and further uses generative AI to detect anomalies from the video data and location information, immediately notifies the parent's terminal if an anomaly is detected, and automatically notifies emergency contacts or the police as necessary.

Images