System

Abstract

An object of a system according to an embodiment is to support an action of a visually handicapped person by voice in real time.SOLUTION: A system includes an object detection unit and a voice notification unit. The object detector analyzes surrounding environments using the object detection AI and detects an obstacle. The voice notification unit notifies information on the obstacle detected by the object detection unit by voice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technologies do not provide sufficient support for visually impaired people to move around safely, and there is room for improvement.

[0005] The system according to the embodiment aims to provide real-time audio support for the activities of visually impaired people. [Means for solving the problem]

[0006] The system according to the embodiment includes an object detection unit and a voice notification unit. The object detection unit analyzes the surrounding environment using object detection AI and detects obstacles. The voice notification unit notifies the user by voice of information about the obstacle detected by the object detection unit. [Effects of the Invention]

[0007] The system according to the embodiment can provide real-time audio support for the activities of visually impaired people. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) A support system according to an embodiment of the present invention uses object detection AI to provide real-time voice support for the activities of visually impaired people. This support system detects obstacles, provides information while traveling, guides users when shopping or out to restrooms, recognizes the names and facial expressions of people they are facing, and notifies users of items they are looking for or dirty areas for cleaning indoors. This allows the support system to provide multifaceted support for the lives of visually impaired people and improve their quality of daily life.

[0029] The support system according to the embodiment includes an object detection unit and an audio notification unit. The object detection unit analyzes the surrounding environment and detects obstacles using object detection AI. For example, the object detection unit analyzes video data from a camera and detects information about an obstacle ahead. The object detection unit can identify the type and location of the obstacle using an object detection AI algorithm (e.g., YOLO, SSD, Faster R-CNN, etc.). Furthermore, the object detection unit can analyze the environment in real time and handle dynamic obstacles. The audio notification unit notifies the user of the obstacle detected by the object detection unit via audio. For example, the audio notification unit generates an audio message such as, "There is an obstacle ahead. Please be careful." The audio notification unit can also adjust the type and volume of the audio to provide information in a format that is easy for the user to hear. Furthermore, the audio notification unit can update information in real time to provide the user with the latest information. As a result, the support system according to the embodiment can provide real-time audio support for the activities of visually impaired people. For example, the voice notification unit can provide information about obstacles to help the user move safely, increasing the sense of security when going out. In addition, the voice notification unit can provide appropriate voice messages depending on the user's situation, reducing the user's stress.

[0030] The object detection unit can analyze the type and size of an obstacle in detail and provide the user with audio instructions on specific ways to avoid it. For example, the object detection unit can analyze video data acquired from a camera and identify the type (e.g., bicycle, car, person) and size of the obstacle. For example, the object detection unit can provide the user with audio instructions on specific ways to avoid it, such as, "There is a bicycle ahead. Move to the right." The object detection unit can also measure the height and width of the obstacle and provide the user with detailed information, such as, "There is a one-meter-high obstacle ahead. Move to the left." Furthermore, the object detection unit can predict the movement of the obstacle and respond to dynamic obstacles in real time. For example, the object detection unit can provide instructions such as, "There is a moving car ahead. Please stop." This allows the user to know specific ways to avoid the obstacle and travel safely.

[0031] The object detection unit predicts the movement of obstacles and can respond to dynamic obstacles in real time. For example, the object detection unit analyzes the movement of dynamic obstacles (e.g., pedestrians or bicycles) from camera footage and instructs the user, such as, "There is a pedestrian moving ahead. Please stop." The object detection unit can also predict the speed and direction of a dynamic obstacle and provide audio instructions on specific avoidance methods, such as, "There is a bicycle moving ahead. Please move to the right." Furthermore, the object detection unit can track the movement of obstacles in real time and instruct the user, such as, "There is a car moving ahead. Please move to the left." This allows for real-time response to dynamic obstacles, ensuring the user's safety.

[0032] The object detection unit can be applied to a navigation system for autonomous vehicles for people without visual disabilities. For example, the object detection unit analyzes camera footage of the autonomous vehicle to identify the type and size of an obstacle. For example, the object detection unit may provide voice instructions on specific avoidance methods, such as, "There is a bicycle ahead. Move to the right." The object detection unit may also integrate an obstacle detection function into the navigation system of an autonomous vehicle to respond to dynamic obstacles in real time. For example, the object detection unit may provide instructions such as, "There is a moving vehicle ahead. Please stop." Furthermore, the object detection unit may integrate an emotion estimation function into the navigation system of an autonomous vehicle to provide relaxing voice guidance, such as, "It's okay. There is an obstacle ahead, but please proceed slowly," if the driver feels anxious. This allows the object detection unit to be applied to the navigation system of an autonomous vehicle to support safe driving for people without visual disabilities.

[0033] The object detection unit can be installed in an automated transport robot in a factory to improve work efficiency. For example, the object detection unit is equipped with a camera on the automated transport robot in a factory to identify the type and size of obstacles. For example, the object detection unit can provide voice instructions on specific avoidance methods, such as, "There is a pallet ahead. Move to the right." The object detection unit can also integrate an obstacle detection function into the automated transport robot's navigation system to respond to dynamic obstacles in real time. For example, the object detection unit can provide instructions such as, "There is a moving forklift ahead. Please stop." Furthermore, the object detection unit can integrate an emotion estimation function into the automated transport robot's navigation system to provide relaxing voice guidance, such as, "It's okay. There is an obstacle ahead, but please proceed slowly," if a worker feels anxious. This allows the object detection unit to be applied to automated transport robots in a factory to improve work efficiency.

[0034] When providing information while traveling, the object detection unit can learn the user's past travel history and provide individually customized information. For example, the object detection unit analyzes the user's past travel history and identifies frequently used routes and places. For example, the object detection unit provides customized information by voice, such as, "There is a cafe you always go to up ahead." The object detection unit also prioritizes providing information about places the user frequently visits based on the travel history. For example, the object detection unit can notify the user in real time of information such as, "There is a bus stop you always go to on the right." Furthermore, the object detection unit learns the user's travel patterns and provides individually customized information. For example, the object detection unit can provide audio information such as, "There is a convenience store you always go to up ahead." This makes it possible to provide individually customized information to the user, thereby improving convenience during travel.

[0035] The object detection unit can analyze the surrounding sound environment and automatically adjust the volume of the audio notification in noisy places. For example, the object detection unit can analyze the surrounding sound environment and automatically increase the volume of the audio notification in noisy places. For example, the object detection unit can provide information such as "There is a crosswalk ahead" at a volume appropriate to the noise level. Furthermore, the object detection unit can automatically lower the volume of the audio notification in quiet places. For example, the object detection unit can provide information such as "There are stairs on the right" at an appropriate volume. Furthermore, the object detection unit can analyze the surrounding sound environment in real time and dynamically adjust the volume of the audio notification. For example, the object detection unit can provide information such as "There is an obstacle ahead" at a volume appropriate to the noise level. This allows notifications to be provided at an appropriate volume even in noisy places, preventing the user from missing information.

[0036] The object detection unit can apply the on-the-move information follow-up function to a tourist guide system and provide information about tourist spots in real time. The object detection unit, for example, analyzes camera footage of a tourist spot and provides information about tourist spots in real time. For example, the object detection unit provides information such as, "There is a famous tourist spot on the right." by voice. The object detection unit also integrates the on-the-move information follow-up function into a tourist guide system and provides information about tourist spots in real time. For example, the object detection unit can notify information such as, "There is a historical building ahead." Furthermore, the object detection unit can analyze information about tourist spots in real time and provide tourists with information such as, "There is a beautiful view on the left." This allows information about tourist spots to be provided in real time, improving convenience for tourists.

[0037] The object detection unit can apply the on-the-move information follow-up function to a training support system for athletes to provide real-time training information. For example, the object detection unit analyzes camera footage while an athlete is training and provides real-time training information. For example, the object detection unit provides audio information such as, "There is an obstacle ahead. Please move to the right." The object detection unit can also integrate the on-the-move information follow-up function into a training support system to provide real-time training information. For example, the object detection unit can provide audio information such as, "There are stairs ahead. Please be careful." Furthermore, the object detection unit can analyze the environment during training in real time and provide audio information such as, "There is training equipment on the right." This allows for real-time training information to be provided to athletes, improving training efficiency.

[0038] The object detection unit can analyze the congestion status of the restrooms in real time and prioritize guidance to an empty restroom. The object detection unit, for example, analyzes camera footage to grasp the congestion status of the restrooms in real time. For example, the object detection unit provides information such as, "The restroom ahead is crowded, but the restroom on the left is empty." The object detection unit also analyzes the usage status of the restrooms in real time and prioritize guidance to an empty restroom. For example, the object detection unit can provide information such as, "The restroom on the right is empty." The object detection unit can also analyze the congestion status in real time and provide specific guidance to the user by voice, such as, "The restroom ahead is crowded, but the restroom on the left is empty." This allows priority guidance to an empty restroom, improving user convenience.

[0039] The object detection unit can analyze the cleanliness of the toilets and prioritize guidance to clean toilets. The object detection unit, for example, analyzes camera footage and grasps the cleanliness of the toilets in real time. For example, the object detection unit provides information such as "The toilet ahead is clean" by voice. The object detection unit also analyzes the cleaning status of the toilets in real time and prioritize guidance to clean toilets. For example, the object detection unit can notify information such as "The toilet on the right is clean" by voice. Furthermore, the object detection unit can analyze the cleanliness in real time and provide specific guidance to the user by voice, such as "The toilet ahead is clean." This allows priority guidance to clean toilets, improving user convenience.

[0040] The object detection unit can apply the restroom guidance function to providing guidance to other facilities within a shopping mall. For example, the object detection unit analyzes camera footage within the shopping mall to determine the locations of cafes and restaurants in real time. For example, the object detection unit provides audio information such as, "There is a cafe ahead." The object detection unit also integrates facility guidance functions within the shopping mall to provide real-time guidance to the locations of cafes and restaurants. For example, the object detection unit can provide audio information such as, "There is a restaurant on your right." Furthermore, the object detection unit can analyze facility information within the shopping mall in real time and provide specific audio guidance to the user such as, "There is a cafe ahead." This allows the restroom guidance function to be applied to providing guidance to other facilities within the shopping mall, improving user convenience.

[0041] The object detection unit can apply the restroom guidance function to large-scale facilities such as event venues and sports stadiums, thereby improving visitor convenience. For example, the object detection unit analyzes camera footage within the event venue or sports stadium to determine the location of restrooms in real time. For example, the object detection unit provides audio information such as, "There is a restroom ahead." The object detection unit also integrates the restroom guidance function within the large-scale facility to provide visitors with real-time guidance on the location of restrooms. For example, the object detection unit can provide audio information such as, "There is a restroom on your right." Furthermore, the object detection unit can analyze restroom information within the event venue or sports stadium in real time and provide users with specific audio guidance such as, "There is a restroom ahead." This allows the restroom guidance function to be applied to large-scale facilities such as event venues and sports stadiums, thereby improving visitor convenience.

[0042] The object detection unit can analyze not only the name and facial expression of the other party, but also the characteristics of their clothing and belongings, and provide detailed information. For example, the object detection unit analyzes camera footage to identify not only the name and facial expression of the other party, but also the characteristics of their clothing and belongings. For example, the object detection unit can provide detailed information by voice, such as "There is Mr. Tanaka in front of you. He is wearing a blue shirt." The object detection unit can also analyze the other party's belongings and provide the user with specific information by voice, such as "There is Mr. Tanaka in front of you. He is carrying a black bag." Furthermore, the object detection unit can analyze the characteristics of the other party's clothing and belongings in real time and provide the user with detailed information by voice, such as "There is Mr. Tanaka in front of you. He is wearing a red hat." This allows for providing detailed information about the other party, allowing the user to more accurately recognize the other party.

[0043] The object detection unit can learn the conversation history of the other party and provide related information in real time. The object detection unit can, for example, analyze the conversation history of the other party and provide related information in real time. For example, the object detection unit can provide information such as, "There's Tanaka ahead. Let's talk about the project we talked about in our last conversation." in a voice message. The object detection unit can also prioritize providing information related to the other party based on the conversation history. For example, the object detection unit can notify the other party in real time of information such as, "There's Tanaka ahead. Let's talk about the trip we talked about in our last conversation." The object detection unit can also learn the conversation history of the other party and provide related information in real time. For example, the object detection unit can provide information such as, "There's Tanaka ahead. Let's talk about the hobbies we talked about in our last conversation." in a voice message. This makes it possible to provide related information based on the other party's conversation history, thereby smoothly progressing the dialogue.

[0044] The object detection unit can apply the name and facial expression recognition function to a business meeting support system and provide participant information in real time. For example, the object detection unit analyzes camera footage during a business meeting to identify the names and facial expressions of participants in real time. For example, the object detection unit provides information such as, "There is Mr. Tanaka in front. He is smiling." via voice. Furthermore, the object detection unit can integrate the name and facial expression recognition function into a business meeting support system and provide participant information in real time. For example, the object detection unit can provide information such as, "There is Mr. Suzuki on your right. He looks a little nervous." Furthermore, the object detection unit can analyze participant information during a business meeting in real time and provide specific information to the user via voice, such as, "There is Mr. Sato in front. He looks a little tired." This allows participant information to be provided in real time during a business meeting, facilitating the smooth progress of the meeting.

[0045] The object detection unit's name and facial expression recognition functions can be applied to monitoring students' understanding and emotional states in educational settings. For example, the object detection unit analyzes camera footage in educational settings to identify students' names and facial expressions in real time. For example, the object detection unit provides audio information such as, "There's Yamada-san ahead. He looks a little worried." The object detection unit also integrates the name and facial expression recognition functions into educational support systems to monitor students' understanding and emotional states in real time. For example, the object detection unit can provide audio information such as, "There's Sato-san on your right. He looks a little tired." Furthermore, the object detection unit can analyze student information in educational settings in real time and provide teachers with specific audio information such as, "There's Tanaka-san ahead. He looks a little nervous." This allows students' understanding and emotional states to be monitored in educational settings, improving the quality of education.

[0046] The object detection unit can analyze not only the location of the object being searched for but also the condition of the object and provide detailed information. For example, the object detection unit analyzes camera footage to identify not only the location of the object being searched for but also the condition of the object (e.g., damage or dirt). For example, the object detection unit can provide detailed information by voice, such as, "There is a key on the table, but it is a little dirty." The object detection unit can also analyze the condition of the object in real time and provide the user with specific information by voice, such as, "There is a remote control on the sofa, but it is broken." Furthermore, the object detection unit can simultaneously analyze the location and condition of the object being searched for and provide the user with detailed information by voice, such as, "There is a wallet on the floor, but it is a little dirty." This allows the location and condition of the object to be provided in detail, allowing the user to find the object efficiently.

[0047] The object detection unit can learn the user's search history and prioritize notifications of frequently searched items. The object detection unit, for example, analyzes the user's search history and identifies frequently searched items. For example, the object detection unit provides audio information such as, "The keys I always look for are on the table." The object detection unit also prioritizes notifications of frequently searched items based on the user's search history. For example, the object detection unit can provide real-time notifications of, "The remote control I always look for is on the sofa." The object detection unit can also learn the user's search patterns and prioritize notifications of frequently searched items. For example, the object detection unit can provide audio information such as, "The wallet I always look for is on the floor." This allows frequently searched items to be prioritized, improving user convenience.

[0048] The object detection unit can apply the missing item and cleaning notification functions to the management of documents and equipment in an office environment. For example, the object detection unit analyzes camera footage in an office environment to determine the location of documents and equipment in real time. For example, the object detection unit provides audio information such as, "There is an important document on your desk." The object detection unit can also integrate the missing item and cleaning notification functions into an office environment management system to provide real-time guidance on the location of documents and equipment. For example, the object detection unit can provide audio information such as, "There is a file on the shelf on the right." Furthermore, the object detection unit can analyze information about documents and equipment in an office environment in real time and provide the user with specific audio guidance such as, "There is a pen on the desk in front." This allows the missing item and cleaning notification functions to be applied to the management of documents and equipment in an office environment, thereby improving work efficiency.

[0049] The object detection unit can apply the missing item and cleaning notification functions to improve the efficiency of cleaning work in hotels and accommodation facilities. The object detection unit, for example, analyzes camera footage within a hotel or accommodation facility to identify areas that need cleaning in real time. For example, the object detection unit provides audio information such as, "There is trash in the room ahead." The object detection unit also integrates the missing item and cleaning notification functions into the accommodation facility's cleaning management system to provide real-time guidance on areas that need cleaning. For example, the object detection unit can provide audio information such as, "The room on the right needs cleaning." Furthermore, the object detection unit can analyze cleaning information within a hotel or accommodation facility in real time and provide specific audio guidance to cleaning staff such as, "There is trash in the room ahead." This allows the missing item and cleaning notification functions to be applied to improve the efficiency of cleaning work in hotels and accommodation facilities, thereby improving the efficiency of cleaning work.

[0050] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0051] The object detection unit can analyze the user's moving speed and provide voice guidance to encourage the user to move at an appropriate speed. For example, if the user is walking too fast, the object detection unit can provide voice guidance such as "Please slow down a little." Also, if the user is walking too slowly, the object detection unit can provide voice guidance such as "Please walk a little faster." Furthermore, the object detection unit can monitor the user's moving speed in real time and provide voice guidance to encourage the user to move at an appropriate speed. This allows the user to move safely and efficiently.

[0052] The object detection unit can analyze the user's travel route and suggest the optimal route. For example, the object detection unit may provide voice guidance such as "Turn right" on the optimal route when the user is heading towards their destination. The object detection unit can also suggest a route that allows the user to avoid congestion. For example, the object detection unit may provide guidance such as "There is congestion ahead. Turn left." Furthermore, the object detection unit can learn the user's travel history and suggest an optimal route that is individually customized. This allows the user to reach their destination efficiently.

[0053] The object detection unit can monitor the user's health condition and provide appropriate advice. For example, the object detection unit can analyze the user's walking pattern and provide advice such as "Please take a short break" if it detects fatigue. The object detection unit can also monitor the user's heart rate and provide advice such as "Consult a doctor" if it detects an abnormality. Furthermore, the object detection unit can monitor the user's health condition in real time and provide appropriate advice. This allows the user to move safely while maintaining their health.

[0054] The object detection unit can provide surrounding environmental information to the user while the user is moving. For example, when the user passes through a park, the object detection unit can provide environmental information by voice, such as "There is a flower bed on the right." Also, when the user passes through a historical site, the object detection unit can provide information, such as "There is a historical building ahead." Furthermore, the object detection unit can provide surrounding environmental information in real time according to the user's movement route. This allows the user to obtain information about the surrounding environment while moving, allowing the user to enjoy their movement.

[0055] The object detection unit can provide weather information to a user while the user is traveling. For example, when the user goes out, the object detection unit can provide weather information by voice, such as "The weather is currently sunny." The object detection unit can also provide information to a user while the user is traveling, such as "It is raining ahead. Please prepare an umbrella." Furthermore, the object detection unit can provide weather information in real time according to the user's travel route. This allows the user to obtain information about the weather while traveling and take appropriate measures.

[0056] The object detection unit can provide traffic information to a user while the user is traveling. For example, the object detection unit can provide traffic information by voice, such as "There is a traffic jam ahead." When the user is using public transportation, the object detection unit can also provide information such as "The next bus will arrive in 5 minutes." Furthermore, the object detection unit can provide real-time traffic information according to the user's travel route. This allows the user to obtain information about traffic while traveling and travel efficiently.

[0057] The object detection unit can analyze the surrounding sound environment while the user is moving and provide appropriate audio notifications. For example, the object detection unit can increase the volume of information such as "There is a crosswalk ahead" when the user passes through a noisy area. The object detection unit can also provide information such as "There are stairs on the right" at an appropriate volume when the user passes through a quiet area. Furthermore, the object detection unit can analyze the surrounding sound environment in real time and dynamically adjust the volume of the audio notifications. This allows the user to receive appropriate audio notifications according to the surrounding sound environment and ensures that they do not miss any information.

[0058] The processing flow of the first embodiment will be briefly explained below.

[0059] Step 1: The object detection unit uses object detection AI to analyze the surrounding environment and detect obstacles. For example, the object detection unit analyzes video data from a camera and detects information about obstacles ahead. The object detection unit can also identify the type and location of obstacles using object detection AI algorithms (e.g., YOLO, SSD, Faster R-CNN, etc.). Furthermore, the object detection unit analyzes the environment in real time and can handle dynamic obstacles. Step 2: The audio notification unit notifies the user by voice of information about the obstacle detected by the object detection unit. For example, the audio notification unit generates an audio message such as "There is an obstacle ahead. Please be careful." The audio notification unit can also adjust the type and volume of the audio to provide information in a format that is easy for the user to hear. Furthermore, the audio notification unit can update information in real time to provide the user with the latest information.

[0060] (Example 2) A support system according to an embodiment of the present invention uses object detection AI to provide real-time voice support for the activities of visually impaired people. This support system detects obstacles, provides information while traveling, guides users when shopping or out to restrooms, recognizes the names and facial expressions of people they are facing, and notifies users of items they are looking for or dirty areas for cleaning indoors. This allows the support system to provide multifaceted support for the lives of visually impaired people and improve their quality of daily life.

[0061] The support system according to the embodiment includes an object detection unit and an audio notification unit. The object detection unit analyzes the surrounding environment and detects obstacles using object detection AI. For example, the object detection unit analyzes video data from a camera and detects information about an obstacle ahead. The object detection unit can identify the type and location of the obstacle using an object detection AI algorithm (e.g., YOLO, SSD, Faster R-CNN, etc.). Furthermore, the object detection unit can analyze the environment in real time and handle dynamic obstacles. The audio notification unit notifies the user of the obstacle detected by the object detection unit via audio. For example, the audio notification unit generates an audio message such as, "There is an obstacle ahead. Please be careful." The audio notification unit can also adjust the type and volume of the audio to provide information in a format that is easy for the user to hear. Furthermore, the audio notification unit can update information in real time to provide the user with the latest information. As a result, the support system according to the embodiment can provide real-time audio support for the activities of visually impaired people. For example, the voice notification unit can provide information about obstacles to help the user move safely, increasing the sense of security when going out. In addition, the voice notification unit can provide appropriate voice messages depending on the user's situation, reducing the user's stress.

[0062] The object detection unit can analyze the type and size of an obstacle in detail and provide the user with audio instructions on specific ways to avoid it. For example, the object detection unit can analyze video data acquired from a camera and identify the type (e.g., bicycle, car, person) and size of the obstacle. For example, the object detection unit can provide the user with audio instructions on specific ways to avoid it, such as, "There is a bicycle ahead. Move to the right." The object detection unit can also measure the height and width of the obstacle and provide the user with detailed information, such as, "There is a one-meter-high obstacle ahead. Move to the left." Furthermore, the object detection unit can predict the movement of the obstacle and respond to dynamic obstacles in real time. For example, the object detection unit can provide instructions such as, "There is a moving car ahead. Please stop." This allows the user to know specific ways to avoid the obstacle and travel safely.

[0063] The object detection unit predicts the movement of obstacles and can respond to dynamic obstacles in real time. For example, the object detection unit analyzes the movement of dynamic obstacles (e.g., pedestrians or bicycles) from camera footage and instructs the user, such as, "There is a pedestrian moving ahead. Please stop." The object detection unit can also predict the speed and direction of a dynamic obstacle and provide audio instructions on specific avoidance methods, such as, "There is a bicycle moving ahead. Please move to the right." Furthermore, the object detection unit can track the movement of obstacles in real time and instruct the user, such as, "There is a car moving ahead. Please move to the left." This allows for real-time response to dynamic obstacles, ensuring the user's safety.

[0064] The object detection unit can use its emotion estimation function to provide voice guidance to reduce the anxiety and stress a user feels about obstacles. The object detection unit, for example, analyzes the user's voice and facial expressions to detect anxiety and stress about obstacles. For example, the object detection unit can provide reassuring voice guidance such as, "There is an obstacle ahead. Please proceed slowly." Furthermore, the object detection unit can use its emotion estimation function to provide relaxing voice guidance such as, "It's okay. There is an obstacle ahead. Please move to the right." Furthermore, the object detection unit can monitor the user's emotional state in real time and provide voice guidance to reduce stress. For example, the object detection unit can provide instructions such as, "There is an obstacle ahead. Please proceed slowly." This can reduce the user's anxiety and stress, allowing them to travel with peace of mind.

[0065] The object detection unit can be applied to a navigation system for autonomous vehicles for people without visual disabilities. For example, the object detection unit analyzes camera footage of the autonomous vehicle to identify the type and size of an obstacle. For example, the object detection unit may provide voice instructions on specific avoidance methods, such as, "There is a bicycle ahead. Move to the right." The object detection unit may also integrate an obstacle detection function into the navigation system of an autonomous vehicle to respond to dynamic obstacles in real time. For example, the object detection unit may provide instructions such as, "There is a moving vehicle ahead. Please stop." Furthermore, the object detection unit may integrate an emotion estimation function into the navigation system of an autonomous vehicle to provide relaxing voice guidance, such as, "It's okay. There is an obstacle ahead, but please proceed slowly," if the driver feels anxious. This allows the object detection unit to be applied to the navigation system of an autonomous vehicle to support safe driving for people without visual disabilities.

[0066] The object detection unit can be installed in an automated transport robot in a factory to improve work efficiency. For example, the object detection unit is equipped with a camera on the automated transport robot in a factory to identify the type and size of obstacles. For example, the object detection unit can provide voice instructions on specific avoidance methods, such as, "There is a pallet ahead. Move to the right." The object detection unit can also integrate an obstacle detection function into the automated transport robot's navigation system to respond to dynamic obstacles in real time. For example, the object detection unit can provide instructions such as, "There is a moving forklift ahead. Please stop." Furthermore, the object detection unit can integrate an emotion estimation function into the automated transport robot's navigation system to provide relaxing voice guidance, such as, "It's okay. There is an obstacle ahead, but please proceed slowly," if a worker feels anxious. This allows the object detection unit to be applied to automated transport robots in a factory to improve work efficiency.

[0067] The object detection unit can use the emotion estimation function to monitor the user's emotional state when an obstacle is detected and play relaxing music at an appropriate time. The object detection unit, for example, analyzes the user's voice and facial expressions to detect anxiety or stress regarding the obstacle. For example, the object detection unit can play relaxing music along with audio guidance such as, "There is an obstacle ahead, please relax." Furthermore, the object detection unit can use the emotion estimation function to play music along with relaxing audio guidance such as, "It's okay. There is an obstacle ahead, but please proceed slowly" when the user is feeling anxious. Furthermore, the object detection unit can monitor the user's emotional state in real time and provide audio guidance and music to reduce stress. For example, the object detection unit can play music along with instructions such as, "There is an obstacle ahead, please relax." This allows relaxing music to be played according to the user's emotional state, thereby reducing the user's stress.

[0068] When providing information while traveling, the object detection unit can learn the user's past travel history and provide individually customized information. For example, the object detection unit analyzes the user's past travel history and identifies frequently used routes and places. For example, the object detection unit provides customized information by voice, such as, "There is a cafe you always go to up ahead." The object detection unit also prioritizes providing information about places the user frequently visits based on the travel history. For example, the object detection unit can notify the user in real time of information such as, "There is a bus stop you always go to on the right." Furthermore, the object detection unit learns the user's travel patterns and provides individually customized information. For example, the object detection unit can provide audio information such as, "There is a convenience store you always go to up ahead." This makes it possible to provide individually customized information to the user, thereby improving convenience during travel.

[0069] The object detection unit can analyze the surrounding sound environment and automatically adjust the volume of the audio notification in noisy places. For example, the object detection unit can analyze the surrounding sound environment and automatically increase the volume of the audio notification in noisy places. For example, the object detection unit can provide information such as "There is a crosswalk ahead" at a volume appropriate to the noise level. Furthermore, the object detection unit can automatically lower the volume of the audio notification in quiet places. For example, the object detection unit can provide information such as "There are stairs on the right" at an appropriate volume. Furthermore, the object detection unit can analyze the surrounding sound environment in real time and dynamically adjust the volume of the audio notification. For example, the object detection unit can provide information such as "There is an obstacle ahead" at a volume appropriate to the noise level. This allows notifications to be provided at an appropriate volume even in noisy places, preventing the user from missing information.

[0070] The object detection unit can use its emotion estimation function to provide voice messages that elicit positive emotions so that the user can travel with peace of mind. The object detection unit, for example, analyzes the user's voice and facial expressions to detect anxiety or stress during travel. For example, the object detection unit can provide a positive voice message such as, "There is a crosswalk ahead, but please proceed safely." Furthermore, the object detection unit can use its emotion estimation function to provide a positive voice message such as, "It's okay. There are stairs ahead, but please proceed slowly" when the user is feeling anxious. Furthermore, the object detection unit can monitor the user's emotional state in real time and provide a voice message that gives a sense of security. For example, the object detection unit can provide instructions such as, "There is an obstacle ahead, but please relax." This can support the user in traveling with peace of mind and reduce stress during travel.

[0071] The object detection unit can apply the on-the-move information follow-up function to a tourist guide system and provide information about tourist spots in real time. The object detection unit, for example, analyzes camera footage of a tourist spot and provides information about tourist spots in real time. For example, the object detection unit provides information such as, "There is a famous tourist spot on the right." by voice. The object detection unit also integrates the on-the-move information follow-up function into a tourist guide system and provides information about tourist spots in real time. For example, the object detection unit can notify information such as, "There is a historical building ahead." Furthermore, the object detection unit can analyze information about tourist spots in real time and provide tourists with information such as, "There is a beautiful view on the left." This allows information about tourist spots to be provided in real time, improving convenience for tourists.

[0072] The object detection unit can apply the on-the-move information follow-up function to a training support system for athletes to provide real-time training information. For example, the object detection unit analyzes camera footage while an athlete is training and provides real-time training information. For example, the object detection unit provides audio information such as, "There is an obstacle ahead. Please move to the right." The object detection unit can also integrate the on-the-move information follow-up function into a training support system to provide real-time training information. For example, the object detection unit can provide audio information such as, "There are stairs ahead. Please be careful." Furthermore, the object detection unit can analyze the environment during training in real time and provide audio information such as, "There is training equipment on the right." This allows for real-time training information to be provided to athletes, improving training efficiency.

[0073] The object detection unit can use the emotion estimation function to monitor the user's emotional state when providing information during travel and provide relaxation guidance to reduce stress. The object detection unit can, for example, analyze the user's voice and facial expressions to detect anxiety or stress during travel. For example, the object detection unit can provide relaxation guidance such as, "There is a crosswalk ahead. Please relax." If the user feels anxious, the object detection unit can use the emotion estimation function to provide relaxation guidance such as, "It's okay. There are stairs ahead. Please proceed slowly." Furthermore, the object detection unit can monitor the user's emotional state in real time and provide relaxation guidance to reduce stress. For example, the object detection unit can issue instructions such as, "There is an obstacle ahead. Please relax." This allows the user to receive relaxation guidance to reduce stress and travel with peace of mind.

[0074] The object detection unit can analyze the congestion status of the restrooms in real time and prioritize guidance to an empty restroom. The object detection unit, for example, analyzes camera footage to grasp the congestion status of the restrooms in real time. For example, the object detection unit provides information such as, "The restroom ahead is crowded, but the restroom on the left is empty." The object detection unit also analyzes the usage status of the restrooms in real time and prioritize guidance to an empty restroom. For example, the object detection unit can provide information such as, "The restroom on the right is empty." The object detection unit can also analyze the congestion status in real time and provide specific guidance to the user by voice, such as, "The restroom ahead is crowded, but the restroom on the left is empty." This allows priority guidance to an empty restroom, improving user convenience.

[0075] The object detection unit can analyze the cleanliness of the toilets and prioritize guidance to clean toilets. The object detection unit, for example, analyzes camera footage and grasps the cleanliness of the toilets in real time. For example, the object detection unit provides information such as "The toilet ahead is clean" by voice. The object detection unit also analyzes the cleaning status of the toilets in real time and prioritize guidance to clean toilets. For example, the object detection unit can notify information such as "The toilet on the right is clean" by voice. Furthermore, the object detection unit can analyze the cleanliness in real time and provide specific guidance to the user by voice, such as "The toilet ahead is clean." This allows priority guidance to clean toilets, improving user convenience.

[0076] The object detection unit can use the emotion estimation function to provide voice guidance to reduce stress when a user searches for a restroom. The object detection unit, for example, analyzes the user's voice and facial expressions to detect anxiety or stress when searching for a restroom. For example, the object detection unit can provide voice guidance such as, "There is a restroom ahead, but please relax." Furthermore, the object detection unit can use the emotion estimation function to provide relaxing voice guidance such as, "It's okay. There is a restroom ahead," when the user is feeling anxious. Furthermore, the object detection unit monitors the user's emotional state in real time and provides voice guidance to reduce stress. For example, the object detection unit can provide instructions such as, "There is a restroom ahead, but please relax." This reduces stress when a user searches for a restroom and allows the user to use the restroom with peace of mind.

[0077] The object detection unit can apply the restroom guidance function to providing guidance to other facilities within a shopping mall. For example, the object detection unit analyzes camera footage within the shopping mall to determine the locations of cafes and restaurants in real time. For example, the object detection unit provides audio information such as, "There is a cafe ahead." The object detection unit also integrates facility guidance functions within the shopping mall to provide real-time guidance to the locations of cafes and restaurants. For example, the object detection unit can provide audio information such as, "There is a restaurant on your right." Furthermore, the object detection unit can analyze facility information within the shopping mall in real time and provide specific audio guidance to the user such as, "There is a cafe ahead." This allows the restroom guidance function to be applied to providing guidance to other facilities within the shopping mall, improving user convenience.

[0078] The object detection unit can apply the restroom guidance function to large-scale facilities such as event venues and sports stadiums, thereby improving visitor convenience. For example, the object detection unit analyzes camera footage within the event venue or sports stadium to determine the location of restrooms in real time. For example, the object detection unit provides audio information such as, "There is a restroom ahead." The object detection unit also integrates the restroom guidance function within the large-scale facility to provide visitors with real-time guidance on the location of restrooms. For example, the object detection unit can provide audio information such as, "There is a restroom on your right." Furthermore, the object detection unit can analyze restroom information within the event venue or sports stadium in real time and provide users with specific audio guidance such as, "There is a restroom ahead." This allows the restroom guidance function to be applied to large-scale facilities such as event venues and sports stadiums, thereby improving visitor convenience.

[0079] The object detection unit can use the emotion estimation function to monitor the user's emotional state when providing restroom guidance and provide a reassuring voice message. The object detection unit, for example, analyzes the user's voice and facial expressions to detect anxiety or stress when searching for a restroom. For example, the object detection unit can provide a voice message such as, "There is a restroom ahead, but please relax." Furthermore, the object detection unit can use the emotion estimation function to provide a relaxing voice message such as, "Don't worry. There is a restroom ahead," when the user is feeling anxious. Furthermore, the object detection unit can monitor the user's emotional state in real time and provide a reassuring voice message. For example, the object detection unit can provide instructions such as, "There is a restroom ahead, but please relax." This allows the user to receive a reassuring voice message, thereby reducing stress when searching for a restroom.

[0080] The object detection unit can analyze not only the name and facial expression of the other party, but also the characteristics of their clothing and belongings, and provide detailed information. For example, the object detection unit analyzes camera footage to identify not only the name and facial expression of the other party, but also the characteristics of their clothing and belongings. For example, the object detection unit can provide detailed information by voice, such as "There is Mr. Tanaka in front of you. He is wearing a blue shirt." The object detection unit can also analyze the other party's belongings and provide the user with specific information by voice, such as "There is Mr. Tanaka in front of you. He is carrying a black bag." Furthermore, the object detection unit can analyze the characteristics of the other party's clothing and belongings in real time and provide the user with detailed information by voice, such as "There is Mr. Tanaka in front of you. He is wearing a red hat." This allows for providing detailed information about the other party, allowing the user to more accurately recognize the other party.

[0081] The object detection unit can learn the conversation history of the other party and provide related information in real time. The object detection unit can, for example, analyze the conversation history of the other party and provide related information in real time. For example, the object detection unit can provide information such as, "There's Tanaka ahead. Let's talk about the project we talked about in our last conversation." in a voice message. The object detection unit can also prioritize providing information related to the other party based on the conversation history. For example, the object detection unit can notify the other party in real time of information such as, "There's Tanaka ahead. Let's talk about the trip we talked about in our last conversation." The object detection unit can also learn the conversation history of the other party and provide related information in real time. For example, the object detection unit can provide information such as, "There's Tanaka ahead. Let's talk about the hobbies we talked about in our last conversation." in a voice message. This makes it possible to provide related information based on the other party's conversation history, thereby smoothly progressing the dialogue.

[0082] The object detection unit can use the emotion estimation function to analyze the emotional state of the other party and suggest an appropriate dialogue method. The object detection unit, for example, analyzes the other party's voice and facial expressions to grasp the other party's emotional state in real time. For example, the object detection unit can suggest a dialogue method by voice, such as, "There's Tanaka ahead. He's smiling. Let's talk about positive topics." Furthermore, the object detection unit can use the emotion estimation function to suggest a dialogue method when the other party is feeling anxious, such as, "There's Tanaka ahead. He's a little nervous. Let's talk about relaxing topics." Furthermore, the object detection unit can monitor the other party's emotional state in real time and suggest an appropriate dialogue method. For example, the object detection unit can issue instructions such as, "There's Tanaka ahead. He's a little tired. Let's talk about relaxing topics." This allows an appropriate dialogue method to be suggested based on the other party's emotional state, thereby smoothly progressing the dialogue.

[0083] The object detection unit can apply the name and facial expression recognition function to a business meeting support system and provide participant information in real time. For example, the object detection unit analyzes camera footage during a business meeting to identify the names and facial expressions of participants in real time. For example, the object detection unit provides information such as, "There is Mr. Tanaka in front. He is smiling." via voice. Furthermore, the object detection unit can integrate the name and facial expression recognition function into a business meeting support system and provide participant information in real time. For example, the object detection unit can provide information such as, "There is Mr. Suzuki on your right. He looks a little nervous." Furthermore, the object detection unit can analyze participant information during a business meeting in real time and provide specific information to the user via voice, such as, "There is Mr. Sato in front. He looks a little tired." This allows participant information to be provided in real time during a business meeting, facilitating the smooth progress of the meeting.

[0084] The object detection unit's name and facial expression recognition functions can be applied to monitoring students' understanding and emotional states in educational settings. For example, the object detection unit analyzes camera footage in educational settings to identify students' names and facial expressions in real time. For example, the object detection unit provides audio information such as, "There's Yamada-san ahead. He looks a little worried." The object detection unit also integrates the name and facial expression recognition functions into educational support systems to monitor students' understanding and emotional states in real time. For example, the object detection unit can provide audio information such as, "There's Sato-san on your right. He looks a little tired." Furthermore, the object detection unit can analyze student information in educational settings in real time and provide teachers with specific audio information such as, "There's Tanaka-san ahead. He looks a little nervous." This allows students' understanding and emotional states to be monitored in educational settings, improving the quality of education.

[0085] The object detection unit can use the emotion estimation function to monitor the user's emotional state during a face-to-face encounter and provide advice to facilitate a smooth dialogue. The object detection unit, for example, analyzes the user's voice and facial expressions to grasp the user's emotional state during a face-to-face encounter in real time. For example, the object detection unit can provide advice by voice, such as "There's Tanaka ahead. Please relax." Furthermore, the object detection unit can use the emotion estimation function to provide advice such as "It's okay. There's Tanaka ahead. Please relax." when the user is feeling anxious. Furthermore, the object detection unit can monitor the user's emotional state in real time and provide advice to facilitate a smooth dialogue. For example, the object detection unit can issue instructions such as "There's Tanaka ahead. Please relax." This makes it possible to provide advice to facilitate a smooth dialogue during a face-to-face encounter and support user communication.

[0086] The object detection unit can analyze not only the location of the object being searched for but also the condition of the object and provide detailed information. For example, the object detection unit analyzes camera footage to identify not only the location of the object being searched for but also the condition of the object (e.g., damage or dirt). For example, the object detection unit can provide detailed information by voice, such as, "There is a key on the table, but it is a little dirty." The object detection unit can also analyze the condition of the object in real time and provide the user with specific information by voice, such as, "There is a remote control on the sofa, but it is broken." Furthermore, the object detection unit can simultaneously analyze the location and condition of the object being searched for and provide the user with detailed information by voice, such as, "There is a wallet on the floor, but it is a little dirty." This allows the location and condition of the object to be provided in detail, allowing the user to find the object efficiently.

[0087] The object detection unit can learn the user's search history and prioritize notifications of frequently searched items. The object detection unit, for example, analyzes the user's search history and identifies frequently searched items. For example, the object detection unit provides audio information such as, "The keys I always look for are on the table." The object detection unit also prioritizes notifications of frequently searched items based on the user's search history. For example, the object detection unit can provide real-time notifications of, "The remote control I always look for is on the sofa." The object detection unit can also learn the user's search patterns and prioritize notifications of frequently searched items. For example, the object detection unit can provide audio information such as, "The wallet I always look for is on the floor." This allows frequently searched items to be prioritized, improving user convenience.

[0088] The object detection unit can use its emotion estimation function to provide voice guidance to reduce stress when searching for things or cleaning. The object detection unit, for example, analyzes the user's voice and facial expressions to detect anxiety or stress when searching for things or cleaning. For example, the object detection unit can provide voice guidance such as, "Your keys are on the table. Please relax." Furthermore, the object detection unit can use its emotion estimation function to provide relaxing voice guidance such as, "It's okay. The remote control is on the sofa" when the user is feeling anxious. Furthermore, the object detection unit can monitor the user's emotional state in real time and provide voice guidance to reduce stress. For example, the object detection unit can provide instructions such as, "Your wallet is on the floor. Please relax." This allows voice guidance to be provided to reduce stress when searching for things or cleaning, allowing the user to work efficiently.

[0089] The object detection unit can apply the missing item and cleaning notification functions to the management of documents and equipment in an office environment. For example, the object detection unit analyzes camera footage in an office environment to determine the location of documents and equipment in real time. For example, the object detection unit provides audio information such as, "There is an important document on your desk." The object detection unit can also integrate the missing item and cleaning notification functions into an office environment management system to provide real-time guidance on the location of documents and equipment. For example, the object detection unit can provide audio information such as, "There is a file on the shelf on the right." Furthermore, the object detection unit can analyze information about documents and equipment in an office environment in real time and provide the user with specific audio guidance such as, "There is a pen on the desk in front." This allows the missing item and cleaning notification functions to be applied to the management of documents and equipment in an office environment, thereby improving work efficiency.

[0090] The object detection unit can apply the missing item and cleaning notification functions to improve the efficiency of cleaning work in hotels and accommodation facilities. The object detection unit, for example, analyzes camera footage within a hotel or accommodation facility to identify areas that need cleaning in real time. For example, the object detection unit provides audio information such as, "There is trash in the room ahead." The object detection unit also integrates the missing item and cleaning notification functions into the accommodation facility's cleaning management system to provide real-time guidance on areas that need cleaning. For example, the object detection unit can provide audio information such as, "The room on the right needs cleaning." Furthermore, the object detection unit can analyze cleaning information within a hotel or accommodation facility in real time and provide specific audio guidance to cleaning staff such as, "There is trash in the room ahead." This allows the missing item and cleaning notification functions to be applied to improve the efficiency of cleaning work in hotels and accommodation facilities, thereby improving the efficiency of cleaning work.

[0091] The object detection unit can use the emotion estimation function to monitor the user's emotional state while searching for something or cleaning and play music to relax them. The object detection unit, for example, analyzes the user's voice and facial expressions to detect anxiety or stress while searching for something or cleaning. For example, the object detection unit can play relaxing music along with voice guidance such as, "Your keys are on the table. Relax." Furthermore, the object detection unit can use the emotion estimation function to play music along with relaxing voice guidance such as, "It's okay. The remote control is on the sofa" when the user is feeling anxious. Furthermore, the object detection unit can monitor the user's emotional state in real time and provide voice guidance and music to reduce stress. For example, the object detection unit can play music along with instructions such as, "Your wallet is on the floor. Relax." This allows relaxing music to be played according to the user's emotional state while searching for something or cleaning, thereby reducing the user's stress.

[0092] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0093] The object detection unit can analyze the user's moving speed and provide voice guidance to encourage the user to move at an appropriate speed. For example, if the user is walking too fast, the object detection unit can provide voice guidance such as "Please slow down a little." Also, if the user is walking too slowly, the object detection unit can provide voice guidance such as "Please walk a little faster." Furthermore, the object detection unit can monitor the user's moving speed in real time and provide voice guidance to encourage the user to move at an appropriate speed. This allows the user to move safely and efficiently.

[0094] The object detection unit can analyze the user's travel route and suggest the optimal route. For example, the object detection unit may provide voice guidance such as "Turn right" on the optimal route when the user is heading towards their destination. The object detection unit can also suggest a route that allows the user to avoid congestion. For example, the object detection unit may provide guidance such as "There is congestion ahead. Turn left." Furthermore, the object detection unit can learn the user's travel history and suggest an optimal route that is individually customized. This allows the user to reach their destination efficiently.

[0095] The object detection unit can monitor the user's health condition and provide appropriate advice. For example, the object detection unit can analyze the user's walking pattern and provide advice such as "Please take a short break" if it detects fatigue. The object detection unit can also monitor the user's heart rate and provide advice such as "Consult a doctor" if it detects an abnormality. Furthermore, the object detection unit can monitor the user's health condition in real time and provide appropriate advice. This allows the user to move safely while maintaining their health.

[0096] The object detection unit can monitor the user's emotional state and play relaxing music at an appropriate time. For example, if the user is feeling anxious, the object detection unit can play relaxing music along with a voice guidance such as "Relax." Also, if the user is feeling stressed, the object detection unit can play music along with a voice guidance such as "It's okay." Furthermore, the object detection unit can monitor the user's emotional state in real time and provide voice guidance and music to reduce stress. This allows relaxing music to be played according to the user's emotional state, thereby reducing the user's stress.

[0097] The object detection unit can monitor the user's emotional state and provide positive messages at appropriate times. For example, if the user is feeling anxious, the object detection unit can provide a positive message such as "There is an obstacle ahead, but please proceed with confidence." Also, if the user is feeling stressed, the object detection unit can provide a message such as "It's okay. There is an obstacle ahead, but please proceed slowly." Furthermore, the object detection unit monitors the user's emotional state in real time and provides messages that give a sense of security. This can support the user in traveling with peace of mind and reduce stress during travel.

[0098] The object detection unit can provide surrounding environmental information to the user while the user is moving. For example, when the user passes through a park, the object detection unit can provide environmental information by voice, such as "There is a flower bed on the right." Also, when the user passes through a historical site, the object detection unit can provide information, such as "There is a historical building ahead." Furthermore, the object detection unit can provide surrounding environmental information in real time according to the user's movement route. This allows the user to obtain information about the surrounding environment while moving, allowing the user to enjoy their movement.

[0099] The object detection unit can provide weather information to a user while the user is traveling. For example, when the user goes out, the object detection unit can provide weather information by voice, such as "The weather is currently sunny." The object detection unit can also provide information to a user while the user is traveling, such as "It is raining ahead. Please prepare an umbrella." Furthermore, the object detection unit can provide weather information in real time according to the user's travel route. This allows the user to obtain information about the weather while traveling and take appropriate measures.

[0100] The object detection unit can provide traffic information to a user while the user is traveling. For example, the object detection unit can provide traffic information by voice, such as "There is a traffic jam ahead." When the user is using public transportation, the object detection unit can also provide information such as "The next bus will arrive in 5 minutes." Furthermore, the object detection unit can provide real-time traffic information according to the user's travel route. This allows the user to obtain information about traffic while traveling and travel efficiently.

[0101] The object detection unit can analyze the surrounding sound environment while the user is moving and provide appropriate audio notifications. For example, the object detection unit can increase the volume of information such as "There is a crosswalk ahead" when the user passes through a noisy area. The object detection unit can also provide information such as "There are stairs on the right" at an appropriate volume when the user passes through a quiet area. Furthermore, the object detection unit can analyze the surrounding sound environment in real time and dynamically adjust the volume of the audio notifications. This allows the user to receive appropriate audio notifications according to the surrounding sound environment and ensures that they do not miss any information.

[0102] The object detection unit can monitor the user's emotional state and provide relaxation guidance at appropriate times. For example, if the user feels anxious, the object detection unit can provide relaxation guidance such as "There is an obstacle ahead, but please relax." Also, if the user feels stressed, the object detection unit can provide guidance such as "It's okay. There are stairs ahead, but please proceed slowly." Furthermore, the object detection unit monitors the user's emotional state in real time and provides relaxation guidance to reduce stress. This allows the user to receive relaxation guidance to reduce stress, allowing them to travel with peace of mind.

[0103] The processing flow of the second embodiment will be briefly explained below.

[0104] Step 1: The object detection unit uses object detection AI to analyze the surrounding environment and detect obstacles. For example, the object detection unit analyzes video data from a camera and detects information about obstacles ahead. The object detection unit can also identify the type and location of obstacles using object detection AI algorithms (e.g., YOLO, SSD, Faster R-CNN, etc.). Furthermore, the object detection unit analyzes the environment in real time and can handle dynamic obstacles. Step 2: The audio notification unit notifies the user by voice of information about the obstacle detected by the object detection unit. For example, the audio notification unit generates an audio message such as "There is an obstacle ahead. Please be careful." The audio notification unit can also adjust the type and volume of the audio to provide information in a format that is easy for the user to hear. Furthermore, the audio notification unit can update information in real time to provide the user with the latest information.

[0105] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0106] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> Examples of generative AIs include the data generation model 58, such as a neural network model (e.g., a neural network model), and a neural network model (e.g., a neural network model). The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input to the data generation model 58. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0107] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0108] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0109] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0110] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0111] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0112] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0113] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0114] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0115] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0116] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0117] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0118] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0119] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0120] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0121] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0122] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0123] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0124] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0125] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0126] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0127] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0128] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0129] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0130] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0131] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0132] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0133] In the headset type terminal 314, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0134] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0135] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0136] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0137] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0138] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0139] 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0140] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0141] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0142] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0143] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0144] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0145] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0146] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0147] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0148] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0149] In the robot 414, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The robot 414 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0150] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0151] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0152] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0153] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0154] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0155] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0156] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0157] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0158] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0159] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0160] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0161] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0162] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0163] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0164] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0165] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0166] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0167] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0168] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0169] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0170] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0171] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0172] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. This system uses object detection AI to provide real-time voice support for people with visual disabilities. An object detection unit that uses object detection AI to analyze the surrounding environment and detect obstacles; a voice notification unit that notifies information about an obstacle detected by the object detection unit by voice. A system characterized by:

2. The object detection unit The type and size of the obstacle are analyzed in detail, and specific avoidance methods are given to the user via voice.

2. The system of claim 1.

3. The object detection unit 2. The system according to claim 1, wherein the system predicts the movement of the obstacle and responds to dynamic obstacles in real time.

4. The object detection unit Providing audio guidance to reduce anxiety and stress felt by the user regarding the obstacle 2. The system of claim 1.

5. The object detection unit 10. The system of claim 1, wherein the system is applied to a navigation system for self-driving cars for people who are not visually impaired.

6. The object detection unit 2. The system according to claim 1, which is mounted on an automatic transport robot in a factory to improve work efficiency.

Images