System

Abstract

An object of the system according to the embodiment is to replace an object in the real world with a favorite character or entertainer and display the character or entertainer.SOLUTION: A system according to an embodiment includes an object recognition unit, a posture estimation unit, a replacement display unit, and an AR support unit. The object recognition unit recognizes passersby and vehicles. The posture estimation unit estimates a motion of the pedestrian or the vehicle recognized by the object recognition unit. The replacement display unit replaces the passerby or the vehicle with a designated character on the basis of the motion estimated by the posture estimation unit. The AR handling unit displays the character replaced by the replacement display unit on the AR glasses or the AR contact lenses.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technology does not allow users to replace real-world objects with their favorite characters or celebrities, leaving room for improving the user experience.

[0005] The system according to the embodiment aims to display real-world objects by replacing them with characters or celebrities of a user's choice. [Means for solving the problem]

[0006] The system according to the embodiment includes an object recognition unit, a posture estimation unit, a replacement display unit, and an AR support unit. The object recognition unit recognizes passersby and vehicles. The posture estimation unit estimates the movement of the passersby and vehicles recognized by the object recognition unit. The replacement display unit replaces the passersby and vehicles with a designated character based on the movement estimated by the posture estimation unit. The AR support unit displays the character replaced by the replacement display unit on AR glasses or AR contact lenses. [Effects of the Invention]

[0007] The system according to the embodiment can display real-world objects by replacing them with characters or celebrities of a user's choice. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) The AR display system according to an embodiment of the present invention replaces passersby and vehicles in the city with celebrities or favorite characters. This system uses object recognition and pose estimation technology to track and replace objects. The system also allows users to customize the appearance and interior of room furniture to their liking, and to freely specify the view from the window. Furthermore, the system is compatible with not only AR glasses but also AR contact lenses, which will likely be developed in the future. This allows users to customize the real world to their liking. For example, users can decorate the cityscape with their favorite characters during their commute, and then relax in their room with the interior décor they like. Furthermore, changing the view from their window to a beautiful natural landscape can add a new level of enjoyment to everyday life.

[0029] An AR display system according to an embodiment includes an object recognition unit, a posture estimation unit, a replacement display unit, and an AR support unit. The object recognition unit recognizes passersby and vehicles. For example, the object recognition unit detects passersby and vehicles using an image recognition algorithm. The object recognition unit can also recognize passersby and vehicles using sensor technology. The object recognition unit analyzes video data acquired from a camera to recognize passersby and vehicles. The posture estimation unit estimates the movement of passersby and vehicles recognized by the object recognition unit. For example, the posture estimation unit estimates the movement of passersby and vehicles using a motion analysis algorithm. The posture estimation unit can also estimate movement using a speed and direction calculation method. The posture estimation unit analyzes video data acquired from a camera to estimate the movement of passersby and vehicles. The replacement display unit replaces passersby and vehicles with designated characters based on the movement estimated by the posture estimation unit. For example, the replacement display unit replaces them with anime characters, animal characters, robot characters, etc. The replacement display unit can also replace them with characters designated by a user. The replacement display unit can also generate and replace a character using a generation AI. The AR support unit displays the character replaced by the replacement display unit on AR glasses or AR contact lenses. For example, the AR support unit displays a character on AR glasses. The AR support unit can also display a character on AR contact lenses. The AR support unit can also display a character according to the user's line of sight and movement. This allows the AR display system according to the embodiment to recognize passersby and vehicles in real time and replace them with designated characters for display. For example, a user can enjoy replacing passersby and vehicles in town with their favorite characters. A user can also customize the real world to their liking using AR glasses or AR contact lenses.

[0030] The replacement display unit can change the character's facial expression and movements in real time in accordance with the movements of passersby and vehicles. For example, the replacement display unit makes the replaced character walk at the same speed and direction as the passersby. For example, if a passerby is in a hurry, the character also appears to be in a hurry. The replacement display unit also makes the replaced character imitate the movement of the vehicle in accordance with the movement of the vehicle. For example, when the vehicle turns, the character also turns in the same direction. The replacement display unit also changes the character's facial expression in accordance with the movements of passersby and vehicles. For example, if a passerby is walking with a smile, the character also appears to be smiling. This makes it possible to change the character's facial expression and movements in real time in accordance with the movements of passersby and vehicles.

[0031] The replacement display unit can automatically select the most suitable character using the user's past preference data. For example, the replacement display unit preferentially displays the character most frequently selected by the user based on data on characters selected by the user in the past. For example, if the user frequently selects a particular anime character, the replacement display unit preferentially displays that character. The replacement display unit also analyzes the user's preference data and automatically selects the most suitable character according to a specific time period or location. For example, a businessman character is displayed during commuting time. The replacement display unit also automatically selects a character suited to a season or event based on the user's preference data. For example, a Santa Claus character is displayed during the Christmas season. This makes it possible to automatically select the most suitable character using the user's past preference data.

[0032] The replacement display unit can replace not only passersby and vehicles, but also buildings and signs in the city with a preferred design. For example, the replacement display unit changes the appearance of buildings in the city to suit the user's preferences. For example, old buildings are replaced with modern designs. The replacement display unit also changes signs in the city to designs and messages that the user likes. For example, advertising billboards are replaced with artwork that the user likes. The replacement display unit also customizes the overall scenery in the city to suit the user's preferences. For example, an urban landscape is changed to a landscape rich in nature. This allows not only passersby and vehicles, but also buildings and signs in the city to be replaced with a preferred design.

[0033] The replacement display unit can customize the replacement display of passersby and vehicles to suit a specific event or season. For example, the replacement display unit changes the characters of passersby and vehicles according to the season. For example, in spring, it displays a character with cherry blossom petals fluttering in the air. The replacement display unit also customizes the characters to suit a specific event. For example, it displays a character in costume for Halloween. The replacement display unit also changes the characters to suit a local event. For example, it displays a character wearing festival costume during a local festival. In this way, the replacement display of passersby and vehicles can be customized to suit a specific event or season.

[0034] The object recognition unit and the posture estimation unit track the movements of animals and drones, and the replacement display unit can perform replacement display in accordance with the movements. The object recognition unit and the posture estimation unit, for example, track the movements of animals and display a character in accordance with the movements. For example, the running movement of a dog is tracked and replaced with an animated dog character. The object recognition unit and the posture estimation unit also track the flight pattern of a drone and display a character in accordance with the movements. For example, the flying motion of a drone is replaced with a bird character. The object recognition unit and the posture estimation unit also track the movements of animals and drones in real time and adjust the behavior of the character in accordance with the movements. For example, the jumping movement of an animal is reflected in the character. This makes it possible to track the movements of animals and drones and perform replacement display in accordance with the movements.

[0035] The object recognition unit and the posture estimation unit can dynamically change the background and the surrounding environment according to the movement of an object. The object recognition unit and the posture estimation unit dynamically change the background scenery according to, for example, the movement of a vehicle. For example, when a vehicle is moving at high speed, the background scenery is displayed as if it is flowing at high speed. The object recognition unit and the posture estimation unit also dynamically change the surrounding environment according to the movement of a passerby. For example, buildings and signs in the background are made to move according to the direction in which the passerby is walking. The object recognition unit and the posture estimation unit also dynamically change the natural background environment according to the movement of an animal. For example, a grassland or forest in the background is displayed as if it is moving when an animal runs. In this way, the background and the surrounding environment can be dynamically changed according to the movement of an object.

[0036] The object recognition unit and posture estimation unit can perform real-time replacement display even in live video of a sporting event, concert, or the like. The object recognition unit and posture estimation unit, for example, track the movements of players at a sporting event and display characters in accordance with those movements. For example, a soccer player is replaced with an animated character. The object recognition unit and posture estimation unit can also track the movements of performers at a concert and display characters in accordance with those movements. For example, a singer is replaced with an animated character. The object recognition unit and posture estimation unit can also track objects moving in live video in real time and display characters in accordance with their movements. For example, the movements of spectators are replaced with animated characters. This allows real-time replacement display even in live video of a sporting event, concert, or the like.

[0037] The object recognition unit and pose estimation unit can recreate historical figures and animals in educational applications to support learning. For example, the object recognition unit and pose estimation unit track the movements of historical figures and display characters in accordance with those movements. For example, Napoleon can be recreated in a history class. The object recognition unit and pose estimation unit can also track the movements of animals and display characters in accordance with those movements. For example, dinosaurs can be recreated in a biology class. The object recognition unit and pose estimation unit can also use object recognition and pose estimation techniques in educational applications to visually recreate learning content. For example, the movement of molecules can be recreated in a science class. This allows historical figures and animals to be recreated in educational applications to support learning.

[0038] The replacement display unit can automatically suggest the optimal design using the user's past preference data when changing the furniture or interior of a room. For example, the replacement display unit preferentially suggests the most frequently selected design based on data on furniture and interior that the user has previously selected. For example, it reflects the user's preferred colors and styles. The replacement display unit also analyzes the user's preference data and automatically suggests the optimal design according to a specific time of day or season. For example, it suggests a warm design in winter. The replacement display unit also automatically suggests a design that matches a specific event or theme based on the user's preference data. For example, it suggests Christmas decorations during the Christmas season. In this way, the optimal design can be automatically suggested using the user's past preference data.

[0039] The replacement display unit can dynamically adjust the lighting and acoustic environment of a room in addition to changing the furniture and interior. For example, the replacement display unit dynamically adjusts the color and brightness of the lighting to match changes to the room's furniture and interior. For example, warm-colored lighting can be used to create a relaxing atmosphere. The replacement display unit also dynamically adjusts the acoustic environment to match the room's interior. For example, classical music can be played to match the interior of a luxurious hotel. The replacement display unit also builds a system that automatically adjusts the lighting and acoustic environment based on user preference data. For example, the system reflects the user's preferred music and lighting settings. This allows the lighting and acoustic environment of a room to be dynamically adjusted in addition to changes to the furniture and interior.

[0040] The replacement display unit can change not only the furniture and interior of the room, but also the design of the garden and balcony. For example, the replacement display unit changes the design of the garden and balcony to match the interior of the room. For example, if the room has a modern design, the garden and balcony will also have a modern design. The replacement display unit also automatically suggests designs for the garden and balcony based on the user's preference data. For example, it reflects the plants and decorations that the user likes. The replacement display unit also changes the design of the garden and balcony to match the season or an event. For example, it displays a design with blooming flowers in spring and illuminations at Christmas. This makes it possible to change not only the furniture and interior of the room, but also the design of the garden and balcony.

[0041] The replacement display unit can customize changes to furniture and interior décor to suit specific events or seasons. The replacement display unit, for example, changes the interior décor of a room according to the season. For example, it suggests cool designs in summer and warm designs in winter. The replacement display unit also customizes the interior décor of a room to suit specific events. For example, it suggests Halloween decorations for Halloween. The replacement display unit also automatically suggests interior décor to suit events or seasons based on the user's preference data. For example, it displays designs that match the user's favorite theme. This makes it possible to customize changes to furniture and interior décor to suit specific events or seasons.

[0042] The replacement display unit can add dynamic elements (for example, moving clouds or a flowing river) to the view seen through the window, providing a realistic experience. For example, the replacement display unit can add moving clouds to the view seen through the window to express realistic changes in the sky. For example, it can display clouds moving in the wind. The replacement display unit can also add a flowing river to the view seen through the window to express realistic water movement. For example, it can also play the sound of water flowing in time with the flow of the river. The replacement display unit can also add dynamic elements to the view seen through the window, changing it according to the season or time of day. For example, it can display a sunrise in the morning and a starry sky at night. In this way, dynamic elements can be added to the view seen through the window, providing a realistic experience.

[0043] The replacement display unit can customize the view from the window based on the user's past travel destinations or memorable places. The replacement display unit, for example, recreates the view of a travel destination that the user has visited in the past. For example, it displays a view of a beach that the user has visited. The replacement display unit also customizes the view from the window based on the user's memorable places. For example, it displays a view of the town where the user lived as a child. The replacement display unit also suggests the view of the most memorable place based on the user's travel history data. For example, it displays a view of a place where the user has taken many photographs. This makes it possible to customize the view from the window based on the user's past travel destinations or memorable places.

[0044] The replacement display unit can freely change not only the view seen through the window but also the shape and size of the window. For example, the replacement display unit changes the shape of the window to suit the user's preferences. For example, it changes a square window to a round window. The replacement display unit also freely changes the size of the window to display a wider view that the user desires. For example, it changes a small window to a large panoramic window. The replacement display unit also changes the shape and size of the window to suit the season or an event. For example, it changes the window to the shape of a snowflake at Christmas. This makes it possible to freely change not only the view seen through the window but also the shape and size of the window.

[0045] The replacement display unit can customize the view from the window to suit a specific event or season. The replacement display unit changes the view from the window according to the season, for example. For example, it displays a view of cherry blossoms in bloom in spring and a view of autumn leaves in autumn. The replacement display unit also customizes the view from the window to suit a specific event. For example, it displays a view decorated with Halloween decorations for Halloween. The replacement display unit also automatically suggests views that match the event or season based on the user's preference data. For example, it displays a view that matches a theme that the user likes. In this way, the view from the window can be customized to suit a specific event or season.

[0046] The AR support unit can monitor the usage status of the AR glasses or AR contact lenses in real time and automatically adjust the optimal display method. For example, the AR support unit can monitor the usage status of the AR glasses in real time and adjust the display content according to the user's line of sight and movements. For example, it can display a character according to the direction the user is looking. The AR support unit can also monitor the usage status of the AR contact lenses in real time and adjust the display content according to the user's line of sight and movements. For example, it can display a character according to the object the user is gazing at. The AR support unit can also build a system that automatically adjusts the optimal display method based on user usage data. For example, if the user is moving, it can display content that matches their movement. This makes it possible to monitor the usage status of the AR glasses or AR contact lenses in real time and automatically adjust the optimal display method.

[0047] The AR support unit can track the user's line of sight and point of gaze while using AR glasses or AR contact lenses, and prioritize displaying the object the user is looking at. For example, the AR support unit can track the user's line of sight and prioritize displaying the object the user is looking at. For example, it can display a character in the direction the user is looking. The AR support unit can also track the user's point of gaze and highlight the object the user is looking at. For example, it can prominently display the vehicle the user is looking at. The AR support unit can also build a system that displays the object the user is looking at in real time based on the user's gaze data. For example, it can dynamically adjust the display content according to the direction the user is looking. This makes it possible to track the user's line of sight and point of gaze while using AR glasses or AR contact lenses, and prioritize displaying the object the user is looking at.

[0048] The AR support unit can be compatible with not only AR glasses and AR contact lenses, but also other devices such as smartphones and tablets. The AR support unit can, for example, display AR images not only on AR glasses and AR contact lenses, but also on smartphones and tablets. For example, it can replace passersby and vehicles with characters through the smartphone camera. The AR support unit can also use a smartphone or tablet to change the furniture and interior of a room. For example, it can change the interior of a room through the smartphone screen. The AR support unit can also use a smartphone or tablet to change the view seen from a window. For example, it can change the view outside the window through the smartphone screen. This makes it possible to support not only AR glasses and AR contact lenses, but also other devices such as smartphones and tablets.

[0049] The AR support unit adds a voice assistant function to the AR glasses or AR contact lenses while they are in use, allowing them to receive user instructions via voice. The AR support unit adds a voice assistant function to the AR glasses or AR contact lenses, for example, and receives user instructions via voice. For example, when a user says, "Change the character," the displayed character changes. The AR support unit also uses the voice assistant function to change the room's interior and the view from the window. For example, when a user says, "Make the room look like a luxury hotel," the interior changes. The AR support unit also uses the voice assistant function to dynamically adjust the AR display content. For example, when a user says, "Change the scenery to mountains," the view from the window changes to mountains. This allows the voice assistant function to be added to the AR glasses or AR contact lenses while they are in use, allowing them to receive user instructions via voice.

[0050] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0051] The AR display system may further include a voice recognition unit. The voice recognition unit can analyze the user's voice commands and operate the system. For example, if the user says, "Change the character," the replacement display unit changes to the specified character. The voice recognition unit can also change the interior of the room or the view from the window based on the user's voice command. For example, if the user says, "Make the room more relaxing," the interior will be changed. Furthermore, the voice recognition unit can adjust the settings of the AR glasses or AR contact lenses according to the user's voice command. For example, if the user says, "Make the view brighter," the brightness of the display will be adjusted.

[0052] The AR display system may further include a haptic feedback unit. The haptic feedback unit can provide haptic feedback when the user touches the AR display content. For example, when the user touches a character seen through the AR glasses, the haptic feedback unit makes the user feel vibrations or pressure. The haptic feedback unit can also provide haptic feedback when the user touches the interior or furniture of a room. For example, when the user sits on a sofa displayed in AR, the user feels as if they are actually sitting there. Furthermore, the haptic feedback unit can dynamically adjust feedback according to the user's movements. For example, the haptic feedback unit can reproduce the feeling of touching the ground as the user walks.

[0053] The AR display system can further include an environmental sensor unit. The environmental sensor unit can acquire information about the surrounding environment and dynamically adjust the display content. For example, it can detect the ambient brightness and temperature and adjust the brightness and color of the display content. The environmental sensor unit can also detect ambient sounds and reflect them in the display content. For example, if the surroundings are quiet, it can provide display content that helps you relax, and if the surroundings are noisy, it can provide display content that helps you concentrate. Furthermore, the environmental sensor unit can acquire weather information and change the display content to suit the weather. For example, on a rainy day, it can change the indoor display content to something warmer.

[0054] The processing flow of the first embodiment will be briefly explained below.

[0055] Step 1: The object recognition unit recognizes passersby and vehicles. For example, the object recognition unit detects passersby and vehicles using an image recognition algorithm. The object recognition unit can also recognize passersby and vehicles using sensor technology. Furthermore, the object recognition unit analyzes video data acquired from a camera to recognize passersby and vehicles. Step 2: The posture estimation unit estimates the movement of passersby and vehicles recognized by the object recognition unit. For example, the posture estimation unit estimates the movement of passersby and vehicles using a motion analysis algorithm. The posture estimation unit can also estimate movement using a speed and direction calculation method. Furthermore, the posture estimation unit analyzes the video data acquired from the camera and estimates the movement of passersby and vehicles. Step 3: The replacement display unit replaces passersby or vehicles with designated characters based on the movements estimated by the posture estimation unit. For example, the replacement display unit may replace them with anime characters, animal characters, robot characters, etc. The replacement display unit may also replace them with characters designated by the user. Furthermore, the replacement display unit may generate and replace characters using a generation AI. Step 4: The AR support unit displays the character replaced by the replacement display unit on the AR glasses or AR contact lenses. For example, the AR support unit displays the character on the AR glasses. The AR support unit can also display the character on the AR contact lenses. Furthermore, the AR support unit can also display the character according to the user's line of sight or movement.

[0056] (Example 2) The AR display system according to an embodiment of the present invention replaces passersby and vehicles in the city with celebrities or favorite characters. This system uses object recognition and pose estimation technology to track and replace objects. The system also allows users to customize the appearance and interior of room furniture to their liking, and to freely specify the view from the window. Furthermore, the system is compatible with not only AR glasses but also AR contact lenses, which will likely be developed in the future. This allows users to customize the real world to their liking. For example, users can decorate the cityscape with their favorite characters during their commute, and then relax in their room with the interior décor they like. Furthermore, changing the view from their window to a beautiful natural landscape can add a new level of enjoyment to everyday life.

[0057] An AR display system according to an embodiment includes an object recognition unit, a posture estimation unit, a replacement display unit, and an AR support unit. The object recognition unit recognizes passersby and vehicles. For example, the object recognition unit detects passersby and vehicles using an image recognition algorithm. The object recognition unit can also recognize passersby and vehicles using sensor technology. The object recognition unit analyzes video data acquired from a camera to recognize passersby and vehicles. The posture estimation unit estimates the movement of passersby and vehicles recognized by the object recognition unit. For example, the posture estimation unit estimates the movement of passersby and vehicles using a motion analysis algorithm. The posture estimation unit can also estimate movement using a speed and direction calculation method. The posture estimation unit analyzes video data acquired from a camera to estimate the movement of passersby and vehicles. The replacement display unit replaces passersby and vehicles with designated characters based on the movement estimated by the posture estimation unit. For example, the replacement display unit replaces them with anime characters, animal characters, robot characters, etc. The replacement display unit can also replace them with characters designated by a user. The replacement display unit can also generate and replace a character using a generation AI. The AR support unit displays the character replaced by the replacement display unit on AR glasses or AR contact lenses. For example, the AR support unit displays a character on AR glasses. The AR support unit can also display a character on AR contact lenses. The AR support unit can also display a character according to the user's line of sight and movement. This allows the AR display system according to the embodiment to recognize passersby and vehicles in real time and replace them with designated characters for display. For example, a user can enjoy replacing passersby and vehicles in town with their favorite characters. A user can also customize the real world to their liking using AR glasses or AR contact lenses.

[0058] The replacement display unit can change the character's facial expression and movements in real time in accordance with the movements of passersby and vehicles. For example, the replacement display unit makes the replaced character walk at the same speed and direction as the passersby. For example, if a passerby is in a hurry, the character also appears to be in a hurry. The replacement display unit also makes the replaced character imitate the movement of the vehicle in accordance with the movement of the vehicle. For example, when the vehicle turns, the character also turns in the same direction. The replacement display unit also changes the character's facial expression in accordance with the movements of passersby and vehicles. For example, if a passerby is walking with a smile, the character also appears to be smiling. This makes it possible to change the character's facial expression and movements in real time in accordance with the movements of passersby and vehicles.

[0059] The replacement display unit can automatically select the most suitable character using the user's past preference data. For example, the replacement display unit preferentially displays the character most frequently selected by the user based on data on characters selected by the user in the past. For example, if the user frequently selects a particular anime character, the replacement display unit preferentially displays that character. The replacement display unit also analyzes the user's preference data and automatically selects the most suitable character according to a specific time period or location. For example, a businessman character is displayed during commuting time. The replacement display unit also automatically selects a character suited to a season or event based on the user's preference data. For example, a Santa Claus character is displayed during the Christmas season. This makes it possible to automatically select the most suitable character using the user's past preference data.

[0060] The replacement display unit uses an emotion estimation function to change the character according to the user's emotional state, thereby improving the user's mood. The replacement display unit, for example, analyzes the user's facial expressions and voice to estimate the user's emotional state. For example, if the user is smiling, a happy character is displayed. The replacement display unit also changes the character's movements and facial expressions according to the user's emotional state. For example, if the user is tired, a relaxed character is displayed. The replacement display unit also monitors the user's emotional state in real time and dynamically changes the character whenever the emotion changes. For example, if the user is surprised, a character with a surprised expression is displayed. In this way, the character displayed can be changed according to the user's emotional state, thereby improving the user's mood.

[0061] The replacement display unit can replace not only passersby and vehicles, but also buildings and signs in the city with a preferred design. For example, the replacement display unit changes the appearance of buildings in the city to suit the user's preferences. For example, old buildings are replaced with modern designs. The replacement display unit also changes signs in the city to designs and messages that the user likes. For example, advertising billboards are replaced with artwork that the user likes. The replacement display unit also customizes the overall scenery in the city to suit the user's preferences. For example, an urban landscape is changed to a landscape rich in nature. This allows not only passersby and vehicles, but also buildings and signs in the city to be replaced with a preferred design.

[0062] The replacement display unit can customize the replacement display of passersby and vehicles to suit a specific event or season. For example, the replacement display unit changes the characters of passersby and vehicles according to the season. For example, in spring, it displays a character with cherry blossom petals fluttering in the air. The replacement display unit also customizes the characters to suit a specific event. For example, it displays a character in costume for Halloween. The replacement display unit also changes the characters to suit a local event. For example, it displays a character wearing festival costume during a local festival. In this way, the replacement display of passersby and vehicles can be customized to suit a specific event or season.

[0063] The replacement display unit uses the emotion estimation function to analyze how the user feels about a specific character and can adjust the display content of the character based on the results. For example, if the user feels positive emotions about a specific character, the replacement display unit will preferentially display that character. For example, it will display many characters that make the user smile. The replacement display unit also adjusts the character's movements and facial expressions based on the user's emotional response. For example, if the user is relaxed, it will display a character that moves calmly. The replacement display unit also analyzes the user's emotional data and tracks changes in emotions toward a specific character. For example, if the user loses interest in a specific character, it will suggest a new character. This makes it possible to analyze how the user feels about a specific character and adjust the display content based on the results.

[0064] The object recognition unit and the posture estimation unit track the movements of animals and drones, and the replacement display unit can perform replacement display in accordance with the movements. The object recognition unit and the posture estimation unit, for example, track the movements of animals and display a character in accordance with the movements. For example, the running movement of a dog is tracked and replaced with an animated dog character. The object recognition unit and the posture estimation unit also track the flight pattern of a drone and display a character in accordance with the movements. For example, the flying motion of a drone is replaced with a bird character. The object recognition unit and the posture estimation unit also track the movements of animals and drones in real time and adjust the behavior of the character in accordance with the movements. For example, the jumping movement of an animal is reflected in the character. This makes it possible to track the movements of animals and drones and perform replacement display in accordance with the movements.

[0065] The object recognition unit and the posture estimation unit can dynamically change the background and the surrounding environment according to the movement of an object. The object recognition unit and the posture estimation unit dynamically change the background scenery according to, for example, the movement of a vehicle. For example, when a vehicle is moving at high speed, the background scenery is displayed as if it is flowing at high speed. The object recognition unit and the posture estimation unit also dynamically change the surrounding environment according to the movement of a passerby. For example, buildings and signs in the background are made to move according to the direction in which the passerby is walking. The object recognition unit and the posture estimation unit also dynamically change the natural background environment according to the movement of an animal. For example, a grassland or forest in the background is displayed as if it is moving when an animal runs. In this way, the background and the surrounding environment can be dynamically changed according to the movement of an object.

[0066] The object recognition unit and posture estimation unit can use the emotion estimation function to analyze the emotion a user has toward a specific object and adjust the display method of the object based on that emotion. For example, if a user has positive emotions toward a specific object, the object recognition unit and posture estimation unit can display the object more attractively. For example, a vehicle that the user likes can be displayed beautifully. The object recognition unit and posture estimation unit can also adjust the display method of the object based on the user's emotion data. For example, if the user is relaxed, the object's movement can be made gentler. The object recognition unit and posture estimation unit can also monitor the user's emotional response in real time and dynamically adjust the display method of the object. For example, if the user is surprised, the object's movement can be made faster. This makes it possible to analyze the emotion a user has toward a specific object and adjust the display method of the object based on that emotion.

[0067] The object recognition unit and posture estimation unit can perform real-time replacement display even in live video of a sporting event, concert, or the like. The object recognition unit and posture estimation unit, for example, track the movements of players at a sporting event and display characters in accordance with those movements. For example, a soccer player is replaced with an animated character. The object recognition unit and posture estimation unit can also track the movements of performers at a concert and display characters in accordance with those movements. For example, a singer is replaced with an animated character. The object recognition unit and posture estimation unit can also track objects moving in live video in real time and display characters in accordance with their movements. For example, the movements of spectators are replaced with animated characters. This allows real-time replacement display even in live video of a sporting event, concert, or the like.

[0068] The object recognition unit and pose estimation unit can recreate historical figures and animals in educational applications to support learning. For example, the object recognition unit and pose estimation unit track the movements of historical figures and display characters in accordance with those movements. For example, Napoleon can be recreated in a history class. The object recognition unit and pose estimation unit can also track the movements of animals and display characters in accordance with those movements. For example, dinosaurs can be recreated in a biology class. The object recognition unit and pose estimation unit can also use object recognition and pose estimation techniques in educational applications to visually recreate learning content. For example, the movement of molecules can be recreated in a science class. This allows historical figures and animals to be recreated in educational applications to support learning.

[0069] The object recognition unit and posture estimation unit use the emotion estimation function to analyze the emotion a user has toward a specific object and can adjust the display content of the object based on the results. For example, if a user has positive emotions toward a specific object, the object recognition unit and posture estimation unit display the object more attractively. For example, a vehicle that the user likes can be displayed beautifully. The object recognition unit and posture estimation unit also adjust the display method of the object based on the user's emotion data. For example, if the user is relaxed, the object's movement is made gentler. The object recognition unit and posture estimation unit also monitor the user's emotional response in real time and dynamically adjust the display method of the object. For example, if the user is surprised, the object's movement is made faster. This makes it possible to analyze the emotion a user has toward a specific object and adjust the display content of the object based on the results.

[0070] The replacement display unit can automatically suggest the optimal design using the user's past preference data when changing the furniture or interior of a room. For example, the replacement display unit preferentially suggests the most frequently selected design based on data on furniture and interior that the user has previously selected. For example, it reflects the user's preferred colors and styles. The replacement display unit also analyzes the user's preference data and automatically suggests the optimal design according to a specific time of day or season. For example, it suggests a warm design in winter. The replacement display unit also automatically suggests a design that matches a specific event or theme based on the user's preference data. For example, it suggests Christmas decorations during the Christmas season. In this way, the optimal design can be automatically suggested using the user's past preference data.

[0071] The replacement display unit can dynamically adjust the lighting and acoustic environment of a room in addition to changing the furniture and interior. For example, the replacement display unit dynamically adjusts the color and brightness of the lighting to match changes to the room's furniture and interior. For example, warm-colored lighting can be used to create a relaxing atmosphere. The replacement display unit also dynamically adjusts the acoustic environment to match the room's interior. For example, classical music can be played to match the interior of a luxurious hotel. The replacement display unit also builds a system that automatically adjusts the lighting and acoustic environment based on user preference data. For example, the system reflects the user's preferred music and lighting settings. This allows the lighting and acoustic environment of a room to be dynamically adjusted in addition to changes to the furniture and interior.

[0072] The replacement display unit uses the emotion estimation function to change the room design according to the user's emotional state, thereby enhancing the relaxation effect. The replacement display unit, for example, analyzes the user's facial expressions and voice to estimate the user's emotional state. For example, if the user is feeling stressed, it suggests a design that will help them relax. The replacement display unit also dynamically changes the room design according to the user's emotional state. For example, if the user is relaxed, it displays a design with calming colors. The replacement display unit also monitors the user's emotional state in real time and dynamically adjusts the room design each time the user's emotion changes. For example, if the user is tired, it provides lighting and music that have a relaxing effect. In this way, the room design can be changed according to the user's emotional state, enhancing the relaxation effect.

[0073] The replacement display unit can change not only the furniture and interior of the room, but also the design of the garden and balcony. For example, the replacement display unit changes the design of the garden and balcony to match the interior of the room. For example, if the room has a modern design, the garden and balcony will also have a modern design. The replacement display unit also automatically suggests designs for the garden and balcony based on the user's preference data. For example, it reflects the plants and decorations that the user likes. The replacement display unit also changes the design of the garden and balcony to match the season or an event. For example, it displays a design with blooming flowers in spring and illuminations at Christmas. This makes it possible to change not only the furniture and interior of the room, but also the design of the garden and balcony.

[0074] The replacement display unit can customize changes to furniture and interior décor to suit specific events or seasons. The replacement display unit, for example, changes the interior décor of a room according to the season. For example, it suggests cool designs in summer and warm designs in winter. The replacement display unit also customizes the interior décor of a room to suit specific events. For example, it suggests Halloween decorations for Halloween. The replacement display unit also automatically suggests interior décor to suit events or seasons based on the user's preference data. For example, it displays designs that match the user's favorite theme. This makes it possible to customize changes to furniture and interior décor to suit specific events or seasons.

[0075] The replacement display unit uses the emotion estimation function to analyze how a user feels about a particular design and can adjust the design based on the results. For example, if a user feels positive about a particular design, the replacement display unit will preferentially suggest that design. For example, it will display many designs that make the user feel relaxed. The replacement display unit also adjusts the color and placement of the design based on the user's emotion data. For example, if the user is feeling stressed, it will suggest a design with calming colors. The replacement display unit also monitors the user's emotional response in real time and dynamically adjusts the design. For example, if the user is surprised, it will suggest a design that will lessen the surprise. This makes it possible to analyze how a user feels about a particular design and adjust the design based on the results.

[0076] The replacement display unit can add dynamic elements (for example, moving clouds or a flowing river) to the view seen through the window, providing a realistic experience. For example, the replacement display unit can add moving clouds to the view seen through the window to express realistic changes in the sky. For example, it can display clouds moving in the wind. The replacement display unit can also add a flowing river to the view seen through the window to express realistic water movement. For example, it can also play the sound of water flowing in time with the flow of the river. The replacement display unit can also add dynamic elements to the view seen through the window, changing it according to the season or time of day. For example, it can display a sunrise in the morning and a starry sky at night. In this way, dynamic elements can be added to the view seen through the window, providing a realistic experience.

[0077] The replacement display unit can customize the view from the window based on the user's past travel destinations or memorable places. The replacement display unit, for example, recreates the view of a travel destination that the user has visited in the past. For example, it displays a view of a beach that the user has visited. The replacement display unit also customizes the view from the window based on the user's memorable places. For example, it displays a view of the town where the user lived as a child. The replacement display unit also suggests the view of the most memorable place based on the user's travel history data. For example, it displays a view of a place where the user has taken many photographs. This makes it possible to customize the view from the window based on the user's past travel destinations or memorable places.

[0078] The replacement display unit uses an emotion estimation function to change the scenery according to the user's emotional state, thereby enhancing the relaxation effect. The replacement display unit, for example, analyzes the user's facial expressions and voice to estimate the user's emotional state. For example, if the user is feeling stressed, it displays a relaxing natural scenery. The replacement display unit also dynamically changes the scenery seen from the window according to the user's emotional state. For example, if the user is relaxed, it displays a calm ocean view. The replacement display unit also monitors the user's emotional state in real time and dynamically adjusts the scenery each time the emotion changes. For example, if the user is tired, it displays a relaxing mountain view. In this way, the scenery can be changed according to the user's emotional state, enhancing the relaxation effect.

[0079] The replacement display unit can freely change not only the view seen through the window but also the shape and size of the window. For example, the replacement display unit changes the shape of the window to suit the user's preferences. For example, it changes a square window to a round window. The replacement display unit also freely changes the size of the window to display a wider view that the user desires. For example, it changes a small window to a large panoramic window. The replacement display unit also changes the shape and size of the window to suit the season or an event. For example, it changes the window to the shape of a snowflake at Christmas. This makes it possible to freely change not only the view seen through the window but also the shape and size of the window.

[0080] The replacement display unit can customize the view from the window to suit a specific event or season. The replacement display unit changes the view from the window according to the season, for example. For example, it displays a view of cherry blossoms in bloom in spring and a view of autumn leaves in autumn. The replacement display unit also customizes the view from the window to suit a specific event. For example, it displays a view decorated with Halloween decorations for Halloween. The replacement display unit also automatically suggests views that match the event or season based on the user's preference data. For example, it displays a view that matches a theme that the user likes. In this way, the view from the window can be customized to suit a specific event or season.

[0081] The replacement display unit uses the emotion estimation function to analyze how the user feels about a particular scene and can adjust the scene based on the results. For example, if the user feels positive about a particular scene, the replacement display unit will preferentially display that scene. For example, it will display many scenes that the user finds relaxing. The replacement display unit also adjusts the color and placement of the scene based on the user's emotion data. For example, if the user is feeling stressed, it will display a scene with calming colors. The replacement display unit also monitors the user's emotional response in real time and dynamically adjusts the scene. For example, if the user is surprised, it will display a scene that soothes the surprise. This makes it possible to analyze how the user feels about a particular scene and adjust the scene based on the results.

[0082] The AR support unit can monitor the usage status of the AR glasses or AR contact lenses in real time and automatically adjust the optimal display method. For example, the AR support unit can monitor the usage status of the AR glasses in real time and adjust the display content according to the user's line of sight and movements. For example, it can display a character according to the direction the user is looking. The AR support unit can also monitor the usage status of the AR contact lenses in real time and adjust the display content according to the user's line of sight and movements. For example, it can display a character according to the object the user is gazing at. The AR support unit can also build a system that automatically adjusts the optimal display method based on user usage data. For example, if the user is moving, it can display content that matches their movement. This makes it possible to monitor the usage status of the AR glasses or AR contact lenses in real time and automatically adjust the optimal display method.

[0083] The AR support unit can track the user's line of sight and point of gaze while using AR glasses or AR contact lenses, and prioritize displaying the object the user is looking at. For example, the AR support unit can track the user's line of sight and prioritize displaying the object the user is looking at. For example, it can display a character in the direction the user is looking. The AR support unit can also track the user's point of gaze and highlight the object the user is looking at. For example, it can prominently display the vehicle the user is looking at. The AR support unit can also build a system that displays the object the user is looking at in real time based on the user's gaze data. For example, it can dynamically adjust the display content according to the direction the user is looking. This makes it possible to track the user's line of sight and point of gaze while using AR glasses or AR contact lenses, and prioritize displaying the object the user is looking at.

[0084] The AR support unit uses the emotion estimation function to change the AR display content according to the user's emotional state, thereby providing a more comfortable experience. The AR support unit, for example, analyzes the user's facial expressions and voice to estimate the user's emotional state. For example, if the user is relaxed, the display content is changed to a calming one. The AR support unit also dynamically changes the AR display content according to the user's emotional state. For example, if the user is feeling stressed, the display content is changed to one that has a relaxing effect. The AR support unit also monitors the user's emotional state in real time and dynamically adjusts the AR display content whenever the user's emotion changes. For example, if the user is surprised, the display content is changed to one that soothes the surprise. In this way, the emotion estimation function can be used to change the AR display content according to the user's emotional state, thereby providing a more comfortable experience.

[0085] The AR support unit can be compatible with not only AR glasses and AR contact lenses, but also other devices such as smartphones and tablets. The AR support unit can, for example, display AR images not only on AR glasses and AR contact lenses, but also on smartphones and tablets. For example, it can replace passersby and vehicles with characters through the smartphone camera. The AR support unit can also use a smartphone or tablet to change the furniture and interior of a room. For example, it can change the interior of a room through the smartphone screen. The AR support unit can also use a smartphone or tablet to change the view seen from a window. For example, it can change the view outside the window through the smartphone screen. This makes it possible to support not only AR glasses and AR contact lenses, but also other devices such as smartphones and tablets.

[0086] The AR support unit adds a voice assistant function to the AR glasses or AR contact lenses while they are in use, allowing them to receive user instructions via voice. The AR support unit adds a voice assistant function to the AR glasses or AR contact lenses, for example, and receives user instructions via voice. For example, when a user says, "Change the character," the displayed character changes. The AR support unit also uses the voice assistant function to change the room's interior and the view from the window. For example, when a user says, "Make the room look like a luxury hotel," the interior changes. The AR support unit also uses the voice assistant function to dynamically adjust the AR display content. For example, when a user says, "Change the scenery to mountains," the view from the window changes to mountains. This allows the voice assistant function to be added to the AR glasses or AR contact lenses while they are in use, allowing them to receive user instructions via voice.

[0087] The AR support unit uses an emotion estimation function to analyze how a user feels about specific display content and can adjust the display content based on the results. For example, if a user feels positive emotions about specific display content, the AR support unit will prioritize displaying that content. For example, it will display more display content that helps the user relax. The AR support unit also adjusts the color and layout of the display content based on the user's emotion data. For example, if the user is feeling stressed, it will change the display content to more gentle colors. The AR support unit also monitors the user's emotional response in real time and dynamically adjusts the display content. For example, if the user is surprised, it will change the display content to one that lessens the surprise. This makes it possible to analyze how a user feels about specific display content and adjust the display content based on the results.

[0088] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0089] The AR display system may further include a voice recognition unit. The voice recognition unit can analyze the user's voice commands and operate the system. For example, if the user says, "Change the character," the replacement display unit changes to the specified character. The voice recognition unit can also change the interior of the room or the view from the window based on the user's voice command. For example, if the user says, "Make the room more relaxing," the interior will be changed. Furthermore, the voice recognition unit can adjust the settings of the AR glasses or AR contact lenses according to the user's voice command. For example, if the user says, "Make the view brighter," the brightness of the display will be adjusted.

[0090] The AR display system can further include a biometrics recognition unit. The biometrics recognition unit can analyze the user's biometric information and perform individual customization. For example, it can recognize the user's face and suggest characters and interiors that are best suited to each individual user. The biometrics recognition unit can also monitor the user's heart rate and body temperature and provide display content according to their health condition. For example, if the user is tired, it can display characters or scenery that will help them relax. Furthermore, the biometrics recognition unit can analyze the movement of the user's pupils and dynamically adjust the display content according to the direction of their gaze.

[0091] The AR display system may further include a haptic feedback unit. The haptic feedback unit can provide haptic feedback when the user touches the AR display content. For example, when the user touches a character seen through the AR glasses, the haptic feedback unit makes the user feel vibrations or pressure. The haptic feedback unit can also provide haptic feedback when the user touches the interior or furniture of a room. For example, when the user sits on a sofa displayed in AR, the user feels as if they are actually sitting there. Furthermore, the haptic feedback unit can dynamically adjust feedback according to the user's movements. For example, the haptic feedback unit can reproduce the feeling of touching the ground as the user walks.

[0092] The AR display system can further include an environmental sensor unit. The environmental sensor unit can acquire information about the surrounding environment and dynamically adjust the display content. For example, it can detect the ambient brightness and temperature and adjust the brightness and color of the display content. The environmental sensor unit can also detect ambient sounds and reflect them in the display content. For example, if the surroundings are quiet, it can provide display content that helps you relax, and if the surroundings are noisy, it can provide display content that helps you concentrate. Furthermore, the environmental sensor unit can acquire weather information and change the display content to suit the weather. For example, on a rainy day, it can change the indoor display content to something warmer.

[0093] The AR display system may further include a social interaction unit. The social interaction unit provides a function to promote interaction with other users. For example, a user may share a character in the same AR space with other users. The social interaction unit may also provide a function to support communication between users. For example, when a user converses with other users through the AR glasses, the user may share facial expressions and gestures in real time. Furthermore, the social interaction unit may analyze the user's emotional state and optimize interactions with other users. For example, when a user is relaxed, the social interaction unit may provide gentle interactions.

[0094] The AR display system can further include a learning support unit. The learning support unit provides functions specialized for educational purposes. For example, it can recreate historical figures in a history class to provide an interactive learning experience. The learning support unit can also visually recreate the movement of molecules in a science class to deepen understanding. The learning support unit can also monitor the user's learning progress and provide individual learning plans. For example, it can provide learning content that focuses on areas in which the user is weak. The learning support unit can also analyze the user's emotional state and provide display content to increase motivation to study.

[0095] The AR display system may further include an entertainment unit. The entertainment unit provides content for the user to enjoy. For example, it may provide an experience that combines the real world and the virtual world through an AR game. The entertainment unit may also display movies or music according to the user's preferences in AR. The entertainment unit may also analyze the user's emotional state and provide optimal entertainment content. For example, if the user wants to relax, it may display calming music or scenery. The entertainment unit may also provide interactive content according to the user's movements. For example, an AR character may dance along when the user dances.

[0096] The AR display system may further include a health management unit. The health management unit monitors the user's health status and provides appropriate advice. For example, it may measure the user's heart rate and number of steps and warn the user of lack of exercise. The health management unit may also analyze the user's diet and sleep data and suggest healthy lifestyle habits. Furthermore, the health management unit may analyze the user's emotional state and provide display content for stress management. For example, if the user is feeling stressed, it may display relaxing scenery or music. The health management unit may also provide an individual fitness plan based on the user's health goals.

[0097] The AR display system may further include a travel guide unit. The travel guide unit provides information about places the user will visit. For example, it may display the history and highlights of tourist spots in AR. The travel guide unit may also suggest nearby restaurants and shops based on the user's current location. The travel guide unit may also analyze the user's emotional state and provide display content to reduce stress during travel. For example, if the user is tired, it may suggest a cafe where the user can relax. The travel guide unit may also provide information related to places the user has visited in the past based on the user's travel history data. For example, it may suggest new tourist spots near places the user has previously visited.

[0098] The AR display system can further include a shopping support unit. The shopping support unit provides information that is useful when the user is shopping. For example, detailed product information and reviews can be displayed in AR. The shopping support unit can also suggest recommended products based on the user's preferences. Furthermore, the shopping support unit can analyze the user's emotional state and provide display content to reduce shopping stress. For example, if the user is unsure, the shopping support unit can suggest the best option. The shopping support unit can also provide information related to products purchased in the past based on the user's purchase history data. For example, it can suggest complementary items to products the user previously purchased.

[0099] The processing flow of the second embodiment will be briefly explained below.

[0100] Step 1: The object recognition unit recognizes passersby and vehicles. For example, the object recognition unit detects passersby and vehicles using an image recognition algorithm. The object recognition unit can also recognize passersby and vehicles using sensor technology. Furthermore, the object recognition unit analyzes video data acquired from a camera to recognize passersby and vehicles. Step 2: The posture estimation unit estimates the movement of passersby and vehicles recognized by the object recognition unit. For example, the posture estimation unit estimates the movement of passersby and vehicles using a motion analysis algorithm. The posture estimation unit can also estimate movement using a speed and direction calculation method. Furthermore, the posture estimation unit analyzes the video data acquired from the camera and estimates the movement of passersby and vehicles. Step 3: The replacement display unit replaces passersby or vehicles with designated characters based on the movements estimated by the posture estimation unit. For example, the replacement display unit may replace them with anime characters, animal characters, robot characters, etc. The replacement display unit may also replace them with characters designated by the user. Furthermore, the replacement display unit may generate and replace characters using a generation AI. Step 4: The AR support unit displays the character replaced by the replacement display unit on the AR glasses or AR contact lenses. For example, the AR support unit displays the character on the AR glasses. The AR support unit can also display the character on the AR contact lenses. Furthermore, the AR support unit can also display the character according to the user's line of sight or movement.

[0101] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0102] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> Examples of generative AIs include the data generation model 58, such as a neural network model (e.g., a neural network model), and a neural network model (e.g., a neural network model). The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input to the data generation model 58. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0103] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0104] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0105] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0106] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0107] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0108] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0109] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0110] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0111] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0112] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0113] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0114] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0115] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0116] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0117] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0118] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0119] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0120] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0121] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0122] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0123] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0124] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0125] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0126] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0127] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0128] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0129] In the headset type terminal 314, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0130] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0131] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0132] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0133] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0134] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0135] 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0136] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0137] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0138] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0139] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0140] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0141] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0142] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0143] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0144] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0145] In the robot 414, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The robot 414 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0146] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0147] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0148] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0149] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0150] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0151] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0152] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0153] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0154] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0155] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0156] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0157] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0158] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0159] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0160] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0161] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0162] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0163] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0164] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0165] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0166] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0167] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0168] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. an object recognition unit that recognizes passersby and vehicles; a posture estimation unit that estimates the movement of passersby and vehicles recognized by the object recognition unit; a replacement display unit that replaces passersby and vehicles with designated characters based on the movements estimated by the posture estimation unit; an AR support unit that displays the character replaced by the replacement display unit on AR glasses or AR contact lenses; A system characterized by:

2. The replacement display unit The facial expression and movement of the character are changed in real time in accordance with the movement of the passersby and the vehicle.

2. The system of claim 1.

3. The replacement display unit Automatically select the most suitable character using the user's past preference data 2. The system of claim 1.

4. The replacement display unit Changing the character depending on the user's emotional state to improve the user's mood 2. The system of claim 1.

5. The replacement display unit Not only the passersby and vehicles, but also buildings and signs around the city can be replaced with your favorite designs.

2. The system of claim 1.

6. The replacement display unit The replacement display of the passersby and the vehicle is customized according to events and seasons.

2. The system of claim 1.

Images