Information processing device

Abstract

We provide an information processing device that utilizes artificial intelligence (AI) to remain unobtrusive under normal circumstances, providing reliable visual information only when needed. [Solution] The information processing device 100 integrates and analyzes information acquired by the imaging unit 111, the sound collection unit 112, and the sensor unit 113, and performs an inference task to identify the context, including the user's current situation, emotions, and intentions. If the computing power to perform the inference task is insufficient, it delegates the inference task to an external private cloud computing unit. The inference unit 120 identifies the context based on the inference results returned from the private cloud computing unit 200. The action determination unit 130 autonomously determines the action to be taken based on the identified context and switches to a no-image mode or an image-enabled mode depending on the situation. The output unit 140 provides feedback to the user on the results of the determined action.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus utilizing artificial intelligence (AI).

Background Art

[0002] In conventional information processing apparatuses such as smartphones, in order for a user to obtain a desired result (e.g., taxi dispatch, restaurant reservation), it is necessary to visually search for and activate an appropriate one from among a number of application icons and operate the GUI (Graphical User Interface) specific to each application. This app-centric operation system forces the user to focus on the screen and remember complex procedures, causing social problems such as a decrease in safety while walking and an obstruction of face-to-face communication (so-called "immersing in the screen").

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Disclosure of the Invention

Problems to be Solved by the Invention

[0004] This disclosure aims to provide an information processing device that does not necessarily require the user to manually launch or operate a specific application. [Means for solving the problem]

[0005] One aspect of this disclosure is, An information processing device comprising an imaging unit for acquiring visual information, a sound collection unit for acquiring auditory information, and a sensor unit for acquiring the user's biometric information, An inference unit that integrates and analyzes the information acquired by the imaging unit, the sound receiving unit, and the sensor unit, and performs an inference task to identify the context, including the user's current situation, emotions, and intentions. If the computing power required to perform the inference task is insufficient, the inference unit delegates the task to an external cloud computing unit and identifies the context based on the inference results returned from the cloud computing unit. An action determination unit that autonomously determines the action to be taken based on the identified context, An output unit that provides feedback to the user regarding the results of the action that was determined, This is an information processing device characterized by having [a certain feature]. [Effects of the Invention]

[0006] According to this disclosure, it is possible to provide an information processing device that does not necessarily require the user to take an active action such as manually launching or operating a specific application. [Brief explanation of the drawing]

[0007] [Figure 1] This is a functional configuration diagram of an information processing device 100 according to one embodiment. [Figure 2] This is a hardware configuration diagram of the information processing device 100. [Figure 3] This is a flowchart showing an overview of the processing performed by the information processing device 100. [Modes for carrying out the invention]

[0008] The embodiments of this disclosure will be described in detail below. Figure 1 is a functional configuration diagram of an information processing device 100 according to one embodiment. (Information processing device 100) As shown in Figure 1, an information processing device 100 of one embodiment includes a housing 101, an imaging unit 111, a sound collection unit 112, a sensor unit 113, an inference unit 120, an action determination unit 130, an output unit 140, an input unit 150, a speech recognition processing unit 160, a communication unit 170, a control unit 180, and a storage unit 190.

[0009] (Cabinet 101) The housing 101 is an outer casing of the device with a shape and dimensions that can be grasped and carried by a user. Because the housing 101 has a shape and dimensions that can be grasped and carried by a user, the information processing device 100 is realized as a portable device.

[0010] (Image capture section 111) The imaging unit 111 is a functional unit for acquiring visual information. The imaging unit 111 is realized by a plurality of cameras 111c and an image processing circuit (a dedicated chip or part of the NPU described later) for processing the images captured by the cameras 111c.

[0011] (Sound collection unit 112) The sound pickup unit 112 is a functional unit for acquiring auditory information. The sound-collecting unit 112 has a microphone unit (acoustic-electrical conversion unit) 112m for collecting ambient sounds and the user's voice. The sound-collecting unit 112 separates ambient sounds picked up by the microphone unit 112m from the user's voice and can detect whispers that are below a predetermined volume threshold. The sound pickup unit 112 also functions as an input unit 150 for the user to actively give instructions to the information processing device 100 through speech.

[0012] (Sensor unit 113) The sensor unit 113 is a functional unit for acquiring the user's biometric information for the portable information processing device 100. The sensor unit 113 is realized by a heartbeat / respiration sensor, a voice / vibration analysis sensor (high-precision microphone), an electromyogram sensor (muscle activity detection means), a visual biological information sensor, an activity amount sensor (acceleration sensor), and the like.

[0013] The heartbeat / respiration sensor can detect the user's heart rate and respiratory irregularities. Based on the heart rate and respiratory irregularities, it is possible to measure the user's excitement state, stress level, and fatigue level. Thereby, it is possible to grasp physical conditions such as "the heart rate is increasing (anxious / angry)" or "the breathing is irregular (running / nervous)".

[0014] The voice / vibration analysis sensor can detect slight tremors and tone changes in the user's voice. From the slight tremors and tone changes in the voice, it is possible to detect not only verbalized commands but also "emotions" and "physical conditions" contained in the voice quality. Thereby, it is possible to read whether the user is feeling不安 or lacking confidence.

[0015] The electromyogram sensor can detect slight movements of the muscles in the user's throat and mouth area. From the movements of the muscles in the throat and mouth area, it is possible to realize silent speech (non-vocal input) that can read what the user is trying to say just from the behavior of the muscles moving the mouth without actually making a sound. Thereby, it is possible to operate without being noticed by others even in a train or a quiet place.

[0016] The visual biological information sensor can detect changes in the user's expression and complexion from the captured image by the camera 111c of the imaging unit 111. From the changes in the user's expression and complexion, it is possible to read visual biological information such as "showing a slightly tired expression". The visual biological information sensor can perform a comprehensive physical condition judgment in combination with changes in expression and complexion and the heart rate and respiratory irregularities obtained by the heartbeat / respiration sensor.

[0017] Activity trackers can measure a user's activity level, such as steps taken and movement speed. The information obtained from activity trackers is used not only to count steps, but also to understand the context of the user's actions, such as "I'm hurrying to the station (=I'm busy right now)."

[0018] (Inference part 120) The inference unit 120 is a functional unit that integrates and analyzes information acquired by the imaging unit 111, the sound collection unit 112, and the sensor unit 113, and performs an inference task to identify the context, including the user's current situation, emotions, and intentions. At least a portion of the processing in the inference unit 120 can be performed within the device itself without the need for external devices.

[0019] If the inference unit 120 lacks the computing power to perform the inference task, it delegates the task to an external private cloud computing unit 200 and identifies the context based on the inference results returned from the private cloud computing unit 200.

[0020] Communication between external devices such as the private cloud computing unit 200 and the inference unit 120 is performed via the communication unit 170.

[0021] If there are multiple private cloud computing units 200, and each private cloud computing unit 200 has different performance and characteristics, the inference unit 120 selects a private cloud computing unit 200 to which to delegate the inference task, according to the attributes of the inference task (task attributes).

[0022] The criteria for selecting the Private Cloud Computing Unit 200 include the cloud's area of ​​expertise, usage costs, performance and characteristics, and the type of model to be executed. The selection criterion for specialization is whether the cloud is specialized for a particular type of processing (e.g., advanced language understanding, image analysis, or execution of specific actions). The cost of use is a selection criterion that takes into account the expenses (such as monetary costs and the consumption of computing resources) incurred in running the inference task on that cloud. Performance and characteristics are selection criteria that take into account the processing speed and responsiveness of the cloud, or the characteristics of the hardware it uses (such as the type of TPU: Tensor Processing Unit or GPU: Graphics Processing Unit). The type of model to be executed is determined by selection criteria that consider whether the inference task requires thinking and dialogue using a Large Language Model (LLM) or operation and execution of external services using a Large Action Model (LAM). A Large Language Model is a software model specialized in understanding and generating text (e.g., document generation, image generation). A Large Action Model is a software model that, based on language understanding, can perform tasks involving actual actions (e.g., application operation, workflow execution). The required computing power is a selection criterion that takes into account cases where the inference task requires large-scale data analysis and necessitates a specific accelerator (TPU or GPU).

[0023] In this way, the inference unit 120 dynamically determines the optimal recipient of the inference task by comparing the content of the inference task (e.g., language processing, action execution, cost-consciousness) with the expertise, cost, and performance of each cloud.

[0024] (Action decision section 130) The action determination unit 130 is a functional unit that autonomously determines the action to be performed based on the context identified by the inference unit 120. An action refers not only to simply answering a question, but also to the AI ​​operating an application or service in the background on behalf of the user to complete a procedure.

[0025] For example, the action determination unit 130 automatically generates access and operation procedures for multiple different external services (such as ride-hailing, payment, reservation, and communication) based on the user's natural language utterances or gaze inputs, without requiring the user to launch or operate specific application software. It then performs processing to obtain the result of executing the determined action. Access to external services by the action determination unit 130 is performed via the communication unit 170.

[0026] Actions include processes such as booking and arranging physical services (e.g., dispatching a taxi, arranging travel, shopping on behalf of someone), acting as a facilitator for communication (e.g., processing large volumes of emails, replying to messages), proactive actions based on context (situation, physical condition) (e.g., suggesting and booking meals, filtering information, blocking notifications), and proxy viewing and extraction of visual information (e.g., extracting key points from videos, searching the real world, preparing for meetings).

[0027] (Output section 140) The output unit 140 is a functional unit that provides feedback to the user regarding the results of the determined action.

[0028] The output unit 140 can express the result of the determined action using at least one of the following: an image display visible from outside the housing 101, vibration of the housing 101, or a change in transmitted light from inside the housing 101.

[0029] The output unit 140 has an image display unit 141 for displaying images. The image display unit 141 is implemented, for example, by a touch display 141d. The touch display 141d functions as an output unit 140 and also functions as an input unit 150 for the user to actively give instructions to the portable information processing device 100 by touch operation.

[0030] The output unit 140 has a vibration generating unit 142 for generating vibrations in the housing 101. The vibration generating unit 142 is implemented, for example, by a vibration motor 142b.

[0031] The output unit 140 has a light-emitting unit 143 for generating transmitted light from inside the housing 101. The light-emitting unit 143 is implemented, for example, by a light-emitting diode 143p.

[0032] (Input section 150) The input unit 150 is a functional unit for the user to actively give instructions to the information processing device 100 through speech or touch operation. In particular, while the action decision unit 130 proposes an action to be performed, which is presented to the user by the output unit 140, user approval (user permission, or approval one-action) is performed by the function of this input unit 150. Specifically, the sound collection unit 112 functions as an instruction input via speech, while the touch display 141d, which constitutes the image display unit 141, functions as an active instruction input via touch operation.

[0033] (Speech recognition processing unit 160) The speech recognition processing unit 160 performs a process to convert the whisper voice detected by the sound collection unit 112 into linguistic information.

[0034] Furthermore, the speech recognition processing unit 160 can also process the movement of the muscles around the throat and mouth detected by the electromyography sensor of the sensor unit 113 into language information.

[0035] (Communications Section 170) The communications unit 170 is a functional unit responsible for external connections to the information processing device 100. Its function is primarily focused on establishing two important communication paths.

[0036] One feature is the delegation of advanced processing. If the inference unit 120 determines that it exceeds its own computing capacity, it securely entrusts the task to an external private cloud computing unit 200 via the communication unit 170 and receives the analysis results.

[0037] Another function is action execution support. When the action decision unit 130 completes a task on behalf of the user, the communication unit 170 connects with various external services such as payment, reservations, and vehicle dispatch, and executes automatically generated operation procedures.

[0038] (Control unit 180) The control unit 180 is a functional unit that provides overall control for the information processing device 100. Its main function is to direct the coordination between the inference unit 120 and the action decision unit 130, execute autonomous actions that take into account the user's intentions, and eliminate the need for conventional application-centric operation.

[0039] Specifically, the control unit 180 comprehensively manages the following main functions. One feature is the autonomous execution of actions (utilization of external services and presentation of information). This function allows a series of tasks, such as searching, booking, and payment, which were previously performed using multiple apps, to be completed with a single instruction, based on the user's natural language utterances and instructions. For example, it automatically arranges physical services on behalf of the user, such as dispatching a taxi, fully arranging a trip considering the user's preferences and schedule, and having groceries delivered to an online supermarket on their behalf.

[0040] Another function is communication proxy. This function understands the context and handles everything from creating replies to sending them on behalf of the user. For example, it can determine the importance of a large volume of incoming emails, summarize them, and create draft replies, completing the process with just a user's permission. It can also understand the content of messages and automatically generate and send appropriate replies even when the user is walking or in other situations.

[0041] Another key feature is context-based proactive behavior. This function anticipates user needs and takes action based on biometric data (heart rate, fatigue level, etc.) and the situation, even without explicit instructions from the user. For example, if it determines that the user is fatigued, it will suggest and book a restaurant with easily digestible food rather than a heavy meal. Also, if it determines that the user is in a conversation or fatigued, it will automatically block non-urgent notifications and report them all later, helping the user filter information and concentrate.

[0042] Another function is the proxy viewing and extraction of visual information. In this function, the AI ​​analyzes video and real-world visual information acquired by the imaging unit 111, and extracts and presents only the information the user needs. Specifically, this can be done by extracting key points from long explanatory videos, searching for objects (such as flowers) in the real world, and creating reports with the latest information about the person you are meeting with.

[0043] In addition, the control unit 180 implements adaptive output mode control. Specifically, the action determination unit 130 infers the user's activity state (walking, talking, standing still, etc.) and the type of information required by the user (presence or absence of visual information) based on the context identified by the inference unit 120, and autonomously determines the feedback format based on the inference result. Specifically, by switching between an "image-less mode" that uses only sound, vibration, or light modulation without using the image display unit 141, and an "image-enabled mode" that uses the image display unit and provides at least images, it is possible to always provide information to the user with the optimal modality.

[0044] (Storage section 190) The memory unit 190 has ROM (Read Only Memory) and RAM (Random Access Memory). Firmware and basic programs are stored in the ROM, while the currently running program and temporary data are placed in the RAM. This provides a foundation for cooperating with the control unit 180 to direct and manage all the functional units of the information processing device 100.

[0045] (Details of the hardware configuration of the information processing device 100) Referring to Figure 2, we will now explain the hardware aspects of the information processing device 100 shown in Figure 1 in more detail. The hardware configuration of the information processing device 100 is realized by a hybrid architecture that forms the core of the arithmetic processing and various sensing and feedback mechanisms.

[0046] The control unit 180, which provides overall control of the information processing device 100, is mainly composed of a processor 181, such as a CPU or MPU. This processor 181 executes the OS and control software stored in the memory unit 190. The memory unit 190 places firmware and basic programs in a read-only ROM and running programs and temporary data in a rewritable RAM. The processor 181 directs and manages all the functional units of the information processing device 100 through the cooperation of this ROM and RAM.

[0047] The inference unit 120 and the action decision unit 130 are implemented in a hybrid AI architecture. This is a mechanism in which dedicated on-device hardware and external high-performance computing resources work together. In this example, the inference unit 120 and the action decision unit 130 share a dedicated AI chip (NPU: Neural Processing Unit) 125 for inference.

[0048] Processes requiring immediate response (such as simple inference, speech recognition, and notification judgment) are executed as inference software on the AI ​​chip 125 dedicated to inference. This allows at least a portion of the processing of the inference unit 120 and the action decision unit 130 to be executed within the device itself without the need for external devices.

[0049] On the other hand, if complex inference tasks or large-scale data analysis are required and the on-device computing power is insufficient, the inference tasks are delegated to an external private cloud computing unit 200 via the communication unit 170.

[0050] This private cloud computing unit 200 is equipped with high-performance accelerators such as TPUs (Tensor Processing Units) and GPUs (Graphics Processing Units). Software such as Large-Scale Language Models (LLMs) and Large-Scale Action Models (LAMs) run on the TPUs and GPUs in the cloud, and are responsible for identifying inference results and automatically generating and executing procedures for accessing and operating external services.

[0051] In other words, the functions of the inference unit 120 and the action decision unit 130 are realized by the on-device NPU and the TPU / GPU on the cloud working together under the control of the CPU / MPU to coordinate the inference software and the large-scale action model (LAM) software.

[0052] The output unit 140 provides feedback to the user regarding the results of the determined action. This unit consists of multiple output hardware components, such as a touch display 141d, a vibration motor 142b, and a light-emitting element (LED 143p, etc.). The control unit 180 (processor 181) autonomously determines whether to use "no image mode" or "image mode" by executing software that drives these hardware components while switching between them, based on the context identified by the inference unit 120, and always presents information in the most optimal format.

[0053] The overall coordination of these processes is directed by the processor (CPU / MPU) 181 that constitutes the control unit 180.

[0054] (Processing performed by the information processing device 100) Referring to Figure 3, the processing performed by the information processing device 100 will be explained. The information processing device 100 first acquires multimodal information to understand the user's state and surrounding environment (step S1). Specifically, the imaging unit 111 acquires visual information, the sound acquisition unit 112 acquires auditory information, and the sensor unit 113 acquires biological information such as heart rate and activity level, and these diverse pieces of information are used as input for integrated analysis. In particular, the voice acquired by the sound acquisition unit 112 is converted from whispers and electromyographic signals into linguistic information in the speech recognition processing unit 160.

[0055] Next, the information processing device 100 integrates and analyzes the acquired information to identify the context, including the user's current situation, emotions, and intentions (step S2). This inference task is performed by the inference unit 120, but if the computing power is insufficient, it delegates the task to an external private cloud computing unit 200 via the communication unit 170, and adopts a hybrid configuration in which the context is identified based on the returned inference results.

[0056] Once the context is identified, the information processing device 100 (action determination unit 130) autonomously determines the action to be performed based on the identified context (step S3). This action is not merely to present information, but also to automatically generate access and operation procedures for multiple different external services (such as dispatch, payment, and reservation) on behalf of the user via the communication unit 170, without the need for specific application software, and to complete the task.

[0057] Prior to executing an action, the information processing device 100 (control unit 180) infers the user's activity state and the type of information required based on the identified context, and controls the adaptive output mode (step S4). As a result, the feedback mode is autonomously determined to be either an image-less mode (only sound, vibration, and light modulation) that does not use the image display unit 141, or an image-enabled mode that uses the image display unit and provides at least images.

[0058] Finally, the information processing device 100 executes the determined action and provides feedback on the result (step S5). The action determination unit 130 executes the access and operation procedure for the external service via the communication unit 170, and the result of the execution is fed back to the user by the output unit 140 using the mode determined in step S4 (at least one of image display, vibration, or change in transmitted light).

[0059] (Effects and mechanisms of the embodiment) According to this embodiment of the information processing device 100, it is possible to solve the social problems associated with conventional application-centric operating systems, such as forcing users to stare at the screen and memorize complex procedures, which in turn leads to reduced safety while walking and hinders face-to-face communication. This provides an information processing device that does not necessarily require users to actively launch or operate specific applications.

[0060] Specifically, the configuration and processing described in Figures 1 to 3 achieve the following actions and effects. Firstly, there is the autonomous execution of actions (use of external services and presentation of information). This is the function in which the action decision unit 130 automatically generates and executes access and operation procedures for multiple different external services (ride-hailing, payment, reservation, communication, etc.) based on the user's natural language utterances or gaze indications, without going through the procedure of launching and operating specific application software. As a result, a series of tasks that previously required opening multiple apps to search, book, and pay, such as dispatching a taxi, making complete travel arrangements that take into account the user's preferences and schedule, and having groceries purchased on behalf of the user, can now be completed with a single instruction, or with a single action of a suggestion from the action decision unit 130 and user approval.

[0061] Secondly, a communication proxy function is realized. Based on the context identified by the inference unit 120, the action decision unit 130 takes over the creation and sending of the reply on behalf of the user. For example, it can determine the importance of a large number of emails received at night, summarize them, and create a draft reply, completing the process with just the user's permission. It can also understand the content of messages and automatically generate and send appropriate reply messages even when the user is walking, significantly reducing the user's communication burden.

[0062] Thirdly, there is the proactive behavior based on context (situation and physical condition). The inference unit 120 senses the user's needs from biometric data (heart rate, fatigue level, etc.) and surrounding circumstances, and acts proactively even without explicit instructions from the user. For example, if it determines that the user is fatigued, it will suggest and make reservations for restaurants with easily digestible food rather than heavy meals. Also, if it determines that the user is in a conversation or fatigued, it will automatically block non-urgent notifications and report them all together later, supporting the user's information filtering and concentration.

[0063] Fourthly, the "proxy viewing" and "extraction" functions of visual information are realized. This is the function in which the inference unit 120 analyzes the video and real-world visual information acquired by the imaging unit 111 on behalf of the user, and extracts and presents only the information the user needs. Specifically, this can be done by extracting the main points from long explanatory videos, searching for objects on the roadside (such as flowers) in the real world, and creating reports of the latest information about the person in question before a meeting, allowing the user to obtain the necessary information through audio or other means without having to stare at the screen.

[0064] Fifth, adaptive output mode control is realized. The action determination unit 130 infers the user's activity state (walking, talking, standing still, etc.) and the type of information required by the user (presence or absence of visual information) based on the identified context, and autonomously determines the mode of feedback by the output unit 140 based on the inference result. Specifically, by switching between an image-less mode that does not use the image display unit and an image-enabled mode that uses the image display unit, information can always be provided to the user with the optimal modality (sound, vibration, light, or image), minimizing the burden of information presentation on the user.

[0065] Incidentally, devices that do not display anything on a screen (see Non-Patent Documents 1 to 3) are always limited to feedback means such as sound, vibration, or light modulation, which imposes constraints on the mode and amount of information transmitted, as well as on user approval.

[0066] In contrast, the information processing device 100 of this embodiment has an output unit 140 which includes an image display unit 141 (for example, a touch display 141d) for displaying images. The action determination unit 130 then infers the user's activity state (walking, talking, standing still, etc.) and the type of information required by the user (presence or absence of visual information) based on the identified context. Based on this inference result, it is possible to autonomously switch between two modes, namely, "no image mode" and "image mode".

[0067] The advantages of being able to select "image mode" are numerous. First, from the perspective of completeness of information transmission, in contexts where visual information presentation is inherently superior or essential, such as map information, detailed graphs and tables, or complex action proposals involving multiple options, it becomes possible to complete tasks without compromising the functionality and usability of the information processing device. In devices that do not display any screen information, this information would have to be conveyed only by voice, increasing the user's comprehension burden and raising the risk of misinterpretation.

[0068] Next, there is the aspect of the certainty of the user's approval action. The image display unit 141 of the output unit 140 also functions as a touch display 141d that constitutes the input unit 150. In response to the action decision unit 130's proposal for the execution of complex external services (e.g., complete travel arrangements or shopping assistance at an online supermarket), the user can visually confirm the content presented on the screen and then perform a "one-action approval" using touch operation. This improves the certainty of recognition of the proposed content and more clearly reflects the user's intentions compared to approval by voice alone.

[0069] In other words, the information processing device 100 of this embodiment is superior to a device that does not display a screen at all, in that it has the flexibility and expandability to always provide information to the user in the optimal modality, by eliminating screen displays when unnecessary, while making maximum use of image display capabilities when accuracy of information transmission, certainty of task completion, and clarity of user approval are required.

[0070] According to this embodiment, it is possible to realize a thoughtful information processing device 100 that normally hides the screen's presence (screen-less mode) and only provides reliable visual information when needed (screen-on mode).

[0071] The process described in the above embodiment can be realized by executing a pre-prepared program on a computer. This program is, for example, stored on a computer-readable storage medium and executed by being read from the storage medium. Alternatively, this program may be provided in the form of a non-volatile (non-transient) storage medium such as flash memory, or it may be provided via a network such as the Internet.

[0072] (Other embodiments) The information processing device of this disclosure is not limited to the embodiments described above. For example, the information processing device 100 of the above embodiment has an image display unit 141, but the information processing device of this disclosure also includes devices that do not have an image display unit (means for displaying images). Even in a configuration that does not have an image display unit at all, the information processing device 100 of the above embodiment employs a hybrid architecture that links on-device AI (NPU) and a private cloud (TPU / GPU), enabling it to complete processes requiring immediacy (simple notification decisions, biometric information analysis, basic speech recognition, etc.) within the device itself (on-device) without the need for external devices. Therefore, even in situations where there is no screen and feedback is limited to sound and vibration, the information processing device of this disclosure can provide a stress-free experience by returning real-time responses (such as vibration) to user operations at "thought speed".

[0073] Furthermore, according to the information processing device 100 of the above embodiment, the inference unit 120 has the function of determining whether the model necessary to execute the inference task is a large-scale language model for thinking or dialogue, or an action model for operating external services, and selecting from among the multiple cloud computing units 200 that has software or hardware resources capable of executing the determined action model as the delegate.Therefore, even in a configuration that does not perform any image display, the information processing device of this disclosure achieves effects such as improved processing speed, optimized operating costs, and efficient inference processing that makes the most of the characteristics of each cloud computing unit by allocating the optimal computing resources (hardware and software) according to the nature of the task (thinking or execution).

[0074] Furthermore, in the information processing device 100 of the above embodiment, the touch display 141d may be configured to visually blend in with the surface of the housing 101 when in "no image mode". Such a configuration can be achieved, for example, by making the housing 101 mirror-finished and covering the surface of the touch display 141d with a half-mirror that can visually blend in with the surface of the housing 101. Furthermore, this can be achieved by configuring the image display unit 141 as a non-volatile display device (e.g., electronic paper) and displaying a screen (image) on the image display unit 141 (non-volatile display device) that visually blends in with the surface of the housing 101 when in "no image mode". The screen that visually blends in with the surface of the housing 101 is, for example, a completely black screen if the surface of the housing 101 is black, or a screen with a pattern similar to the pattern on the entire surface if the surface of the housing 101 has a pattern. In short, it is a screen that resembles the color and pattern of the surface of the housing 101. By using a non-volatile display device, power consumption by the image display unit 141 in "no image mode" can be suppressed. By configuring it in this way, the information processing device 100 can be more effectively realized as a thoughtful device that normally hides the screen's presence (screen-less mode) and only provides reliable visual information when needed (screen-on mode).

[0075] Furthermore, although the above embodiment exemplified an image display unit 141 (display) as the output unit 140, this disclosure is not limited thereto. The output unit 140 may also include a "projection unit" that projects image information onto an object outside the housing 101 (for example, the user's palm, the surface of a desk, or a wall). With this configuration, the information processing device 100 can maintain a shape (texture) like a "stone" without a physical display, while generating a temporary screen in the environment only when visual confirmation is essential, such as for maps or ticket information. This makes it possible to realize an interface that does not impair convenience while eliminating reliance on a screen.

[0076] Furthermore, in the above embodiment, the information processing device 100 is configured to complete all input and output on its own, but it may also operate in cooperation with other external terminals owned by the user (e.g., smart glasses, smartphones, or wireless earphones) via the communication unit 170. For example, if the action decision unit 130 determines that it is necessary to present visual information, it may control the display of the connected smart glasses to display the information on the display of the connected smart glasses, rather than on the output unit 140 of its own device. Also, if long text input is required, the keyboard of the connected smartphone may be used as an auxiliary input device to accept input. In this way, the information processing device 100 may function as the "core of the AI," and the configuration may dynamically select and use the most suitable external device as I / O depending on the situation.

[0077] Furthermore, the housing 101 may have a pen shape that the user can grip and write with. In this case, the sensor unit 113 includes an inertial measurement unit (IMU) or optical sensor that detects the movement and acceleration of the tip of the housing 101. The inference unit 120 (or action determination unit 130) recognizes the handwriting actions performed by the user on a flat surface such as a desk or palm as character information or commands from the trajectory of the housing 101 detected by the sensor unit 113. This enables intuitive text input and instructions through physical writing actions without looking at a display, even in situations where voice (whisper voice) cannot be produced.

[0078] Furthermore, in controlling the vibration generating unit 142 and the light emitting unit 143, the control unit 180 may use a biomimetic pattern that mimics biological behavior. Specifically, when the AI ​​is processing something or showing empathy for the user, it generates fluctuating vibrations and flashing lights (Breathing Light) synchronized with the heartbeat and breathing rhythms of living organisms, rather than a constant mechanical blinking or vibration. This avoids giving the user a mechanical coldness, instead providing a sense of affinity and security as if interacting with a living creature, and promotes the formation of attachment to the device.

[0079] Furthermore, the surface material of the casing 101 is not limited to resin or metal, but may be made of ceramic, glass, wood, or stone (or a material that mimics the texture of these materials). In particular, in configurations without a display, the entire surface of the casing 101 becomes the user contact interface, making tactile quality important. Instead of visual information, the warmth and weight of the material make it easier for the information processing device 100 to blend into the environment as a work of art or a natural object.

[0080] Furthermore, while the above embodiment illustrates a configuration with a dedicated housing 101 having a unique shape and interface, the information processing device disclosed herein is not limited thereto. The information processing device disclosed herein includes an information processing device that has functions of an existing smartphone (for example, an Android® terminal or an iPhone® multifunctional mobile phone) such as a telephone function, camera function, music playback function, calendar function, web browsing function, and SNS (Social Networking Service) function. In this case, the inference unit 120 and the action decision unit 130 function as a higher layer of these existing functions (applications) or as part of the OS (Operating System). That is, the action decision unit 130 autonomously controls the telephone function, music playback function, or web browsing function, etc., in the background based on the identified context, without waiting for the user to manually launch or switch individual applications, and provides only the results required by the user via the output unit 140 (smartphone display or speaker). This makes it possible to realize an agent-centered user experience that moves away from an application-centered operating system while utilizing existing hardware resources.

[0081] (summary) This specification discloses at least the following configuration: (1) An information processing device comprising an imaging unit for acquiring visual information, a sound collection unit for acquiring auditory information, and a sensor unit for acquiring the user's biometric information, An inference unit (hybrid inference unit) integrates and analyzes the information acquired by the imaging unit, the sound receiving unit, and the sensor unit, and performs an inference task to identify the context, including the user's current situation, emotions, and intentions. If the computing power to perform the inference task is insufficient, the inference task is delegated to an external private cloud computing unit, and the context is identified based on the inference results returned from the private cloud computing unit. An action determination unit autonomously determines the action to be taken (use of an external service or presentation of information to the user) based on the identified context, An output unit that provides feedback to the user regarding the results of the action that was determined, An information processing device characterized by comprising the following:

[0082] The information processing device described in (1) solves the social problems associated with conventional app-centric operating systems in smartphones, such as forcing users to stare at the screen and memorize complex procedures, which in turn leads to reduced safety while walking and hinders face-to-face communication. It provides an information processing device that does not necessarily require users to manually launch or operate specific apps. This is achieved by integrating and analyzing multimodal information acquired by the imaging unit, sound collection unit, and sensor unit, and based on the context identified by the hybrid inference unit, the action decision unit autonomously determines the action to be performed.

[0083] (2) The inference unit determines whether the model necessary to perform the inference task is a large-scale language model for thinking or dialogue, or an action model for operating external services, and selects from among the multiple cloud computing units that has software or hardware resources capable of executing the determined action model as the delegate, as described in (1).

[0084] According to the information processing device in (2), the inference unit identifies the attributes of the inference task to be performed, such as whether it requires advanced language processing (e.g., thinking using a large-scale language model), operation of an external service (e.g., execution using an action model), or whether cost should be a consideration. It then compares these attributes with the expertise, usage costs, or performance characteristics of the installed hardware (TPU, GPU, etc.) of multiple connectable cloud computing units, and dynamically selects the optimal delegate. This makes it possible to allocate the optimal computing resources (hardware and software) according to the nature of the task, compared to performing all processing in a uniform cloud environment, thereby improving processing speed, optimizing operational costs, and achieving efficient inference processing that makes the most of the characteristics of each cloud computing unit.

[0085] (3) The information processing apparatus according to (1), characterized in that the sound receiving unit separates ambient sounds from the user's voice and includes a speech recognition processing unit that detects whispers or muscle movements during speech that are below a predetermined volume threshold and converts them into language information.

[0086] (3) The information processing device includes a sound pickup unit that separates ambient noise from the user's voice and a speech recognition processing unit that detects whispers or muscle movements during speech that are below a predetermined volume threshold and converts them into linguistic information. This not only allows the device to perceive "emotions" and "physical condition" contained in the tone of voice, but also enables silent speech (non-vocal input) which allows the user to operate the device without being noticed by those around them, even on trains or in quiet places, thereby reducing the user's communication burden and improving the convenience of operation.

[0087] (4) The information processing apparatus according to claim 1, characterized in that at least a portion of the processing in the inference unit and the action determination unit can be executed solely within the device without the need for an external device.

[0088] (4) The information processing device can perform at least a portion of the processing in the inference unit and the action decision unit solely within the device itself without the need for external devices. This ensures high-speed and real-time responsiveness in simple inference, speech recognition, and notification decision-making processes that require immediate response, thereby improving the user experience and reducing the communication load and reliance on the cloud.

[0089] (5) The information processing device according to (1), characterized in that the action determination unit automatically generates and executes access and operation procedures for multiple different external services (such as dispatch, payment, reservation, and communication) based on the user's natural language utterance or gaze indication, without requiring the user to launch and operate specific application software.

[0090] (5) The information processing device has an action decision unit that automatically generates and executes procedures for accessing and operating multiple different external services based on the user's natural language utterances or gaze indications, without requiring the user to launch and operate specific application software. This makes it possible to complete a series of tasks that previously required opening and running multiple applications with just a single instruction, or a single action of suggestion from the action decision unit and user approval, thereby significantly reducing the burden on the user.

[0091] (6) Having a housing that the user can grasp, The output unit is, The information processing apparatus according to (1), characterized in that the state inside the enclosure, which provides feedback to the user regarding the execution result of the determined action, can be represented by an image display visible from outside the enclosure, vibration of the enclosure, or a change in transmitted light from inside the enclosure.

[0092] The information processing device (6) has a housing that can be grasped by the user, and the output unit can express the result of the execution of a determined action by displaying an image visible from outside the housing, by vibration of the housing, or by a change in transmitted light from inside the housing. This makes it possible to provide a variety of feedback modes, such as visual, tactile, and luminous, depending on the user's situation and the type of information required, thereby increasing the flexibility and reliability of information transmission.

[0093] (7) The output unit includes an image display unit, The action determination unit, based on the identified context, infers the user's activity state (walking, talking, standing still, etc.) and the type of information required by the user (presence or absence of visual information), Based on the results of that reasoning, The output unit provides feedback using only sound, vibration, or light modulation, without using the image display unit, in an image-less mode. The information processing apparatus according to (1), characterized in that it autonomously determines one of the modes, at least an image mode, in which the feedback from the output unit is performed using the image display unit.

[0094] The information processing device (7) has an output unit that includes an image display unit, and an action determination unit that infers the user's activity state and the type of information required based on the specified context, and autonomously decides whether to use a no-image mode or an image-enabled mode. This allows the device to eliminate screen displays when unnecessary, while making maximum use of image display capabilities in contexts where visual information presentation is essential, such as map information or complex action suggestions. This ensures accuracy in information transmission, certainty in task completion, and clarity in user approval, while always providing information to the user in the optimal modality and minimizing the burden of information presentation.

[0095] (8) An information processing program for a computer equipped with an imaging unit for acquiring visual information, a sound collection unit for acquiring auditory information, and a sensor unit for acquiring the user's biometric information, to implement the following functions. A function that integrates and analyzes information acquired by the imaging unit, the sound receiving unit, and the sensor unit, and performs an inference task to identify the context, including the user's current situation, emotions, and intentions, and if the computing power required to perform the inference task is insufficient, delegates the inference task to an external private cloud computing unit, and identifies the context based on the inference results returned from the private cloud computing unit. Based on the identified context, an action decision function autonomously determines the action to be taken (use of an external service or presentation of information to the user), An output function that provides feedback to the user regarding the results of the action that was determined.

[0096] The information processing program in (8) enables a computer equipped with an imaging unit for acquiring visual information to implement the same reasoning, action decision, and output functions as in (1). This makes the program a foundational technology for transitioning the conventional application economy to an agent economy, freeing users from the need to master specific operating systems and applications, and providing a social infrastructure that allows all people, including the elderly and people with disabilities, to enjoy the benefits of advanced digital services using only natural language and intuitive signals.

[0097] (9) The information processing program described in (8) causes the computer to determine, as the inference function, whether the model necessary to perform the inference task is a large-scale language model for thinking and dialogue, or an action model for operating external services, and to select from among the multiple cloud computing units that has software or hardware resources capable of executing the determined model as the delegate.

[0098] (10) The information processing program described in (8) for the computer to cause the sound-collecting unit to perform speech recognition processing that separates ambient sounds from the user's voice, detects whispers or muscle movements during speech that are below a predetermined volume threshold, and converts them into language information.

[0099] The information processing program (10) causes the computer to perform speech recognition processing, in which the sound-collecting unit detects whispers or muscle movements during speech and converts them into linguistic information. This enables the computer to receive active instructions without vocal input, even in quiet environments or while walking, where consideration for the surroundings is necessary, and significantly reduces the limitations on user operation.

[0100] (11) The information processing program described in (8) for causing the computer to perform at least a portion of the processing in the inference function and the action decision function solely within the computer itself, without the use of external devices.

[0101] The information processing program (11) causes the computer to execute at least a portion of the processing in its inference and action decision functions solely within its own device, without the need for external devices. This enables the computer to execute core processing at high speed, regardless of the network connection status, and provides a foundation for providing highly immediate responses to the user.

[0102] (12) An information processing program as described in (8) for the computer to automatically generate and execute access and operation procedures for multiple different external services (such as dispatch, payment, reservation, and communication) based on the user's natural language utterances or gaze indications, without requiring the user to launch and operate specific application software, as an action determination function.

[0103] The information processing program (12) causes the computer to automatically generate and execute procedures for accessing and operating multiple different external services based on the user's natural language or eye-gaze instructions, without the need for specific application software, as an action decision function. This allows the computer to function as an action model, achieving application-less convenience, where the user can complete complex tasks with a single instruction without having to operate multiple apps.

[0104] (13) The information processing program described in (8) for the computer to provide feedback to the user regarding the results of the determined action, by expressing the state inside the enclosure that the user can grasp through an image display visible from outside the enclosure, vibration of the enclosure, or a change in transmitted light from inside the enclosure.

[0105] According to the information processing program in (13), the computer can express the result of a determined action through image display, vibration of the casing, or change in transmitted light. This allows the computer to provide multimodal feedback—visual, tactile, and luminous—through the casing held and carried by the user, thereby diversifying the modes of information transmission.

[0106] (14) An information processing program according to (8) for causing the computer to infer, as an action determination function, the user's activity state (walking, talking, standing still, etc.) and the type of information required by the user (presence or absence of visual information) based on the identified context, and to autonomously determine one of the following modes based on the inference result: an image-less mode in which feedback by the output function is provided only by sound, vibration, or light modulation without using the image display unit, or an image-enabled mode in which feedback by the output function is provided at least with images using the image display unit.

[0107] The information processing program (14) causes the computer to infer the user's activity state and the type of information required based on the identified context, as an action decision function, and to autonomously decide whether to use a mode without images or a mode with images. This allows the computer to always achieve optimal modality control, minimizing the burden of presenting information to the user while making maximum use of image display capabilities when accuracy of information transmission and certainty of user approval for complex actions are required. [Industrial applicability]

[0108] The technology disclosed herein can be used as a foundational technology to transition from the traditional application economy to an agent economy. Users will be freed from the need to master specific operating systems and applications, and it will function as a universal social infrastructure where all people, including the elderly and people with disabilities, can enjoy the benefits of advanced digital services using only natural language and intuitive cues.

[0109] Furthermore, the technology disclosed herein is also applicable to the medical and healthcare industry. The information processing device disclosed herein has the function of continuously monitoring changes in emotion and physical condition contained in the user's heart rate, respiration, and tone of voice, and inferring fatigue levels and stress levels. By utilizing this function, it is expected to be applied as a preventive medical device in collaboration with doctors and specialized institutions in the fields of detecting pre-illness conditions in daily life, monitoring the elderly, or mental health care.

[0110] Furthermore, the technology disclosed herein is also applicable to business and enterprise solutions. The ability of the action decision unit to autonomously handle tasks such as processing large volumes of emails, scheduling, and gathering information before meetings functions as a personal business assistant that dramatically improves corporate productivity. In particular, the whisper voice and silent speech input capabilities enable work to be performed during confidential business negotiations or in public spaces where silence is required.

[0111] Furthermore, the technology disclosed herein is applicable to advanced integration with IoT (Internet of Things) and smart city infrastructure. The action decision unit of the information processing device disclosed herein has the ability to trigger not only online services but also physical actions such as ride-hailing services (utilizing LAM: Large Action Model). In the future, it is expected to be used as an interface for a context-aware urban OS that directly interacts with physical devices such as autonomous vehicles, smart home devices, and drone delivery systems via 5G / 6G networks, and operates autonomously so that the entire urban infrastructure supports the user according to the user's context ("I'm in a hurry," "I'm tired," etc.).

[0112] Furthermore, the technology disclosed herein can be used as a hardware platform for realizing agentic AI, which has been rapidly developing in recent years. While conventional generative AI is limited to the "generation" of information, the action decision unit of this device is responsible for autonomous "action" and "completion." As a result, it is expected to be positioned as a standard device for next-generation AI services in the software industry that have the capability to solve real-world problems, going beyond mere chatbots.

[0113] Furthermore, the technology disclosed herein is also applicable in the fields of education and reskilling. The device can infer the user's level of concentration and comprehension in real time from their gaze, voice tone, and heart rate. Based on this context, it functions as an AI tutor that automatically provides supplementary explanations at points where the user is struggling and suggests appropriate breaks when concentration wavers. In particular, because it does not require the user to stare at a screen, it can provide extremely high learning efficiency in hands-on instruction involving manual work and in language learning while on the go.

[0114] Furthermore, in the tourism and entertainment industries, it can be used as a device that combines digital detox and augmented reality (AR). This device allows users to free themselves from smartphone screens while traveling or attending events, enabling them to immerse themselves in the scenery and experiences before them. Meanwhile, the inference unit senses the user's interests (what they are looking at) from location information and camera footage, and realizes audio AR by providing necessary guide information and historical background solely through auditory feedback such as voice (response to whispers into the sound-gathering unit) and bone conduction. This provides a new tourism solution that enhances the value of direct experiences through the five senses, rather than experiences viewed through a screen. [Explanation of symbols]

[0115] 100 Information Processing Devices 101 cabinets 111 Imaging Unit 112 Sound collection section 113 Sensor section 120 Reasoning Department 130 Action Decision Section 140 Output section 141 Image display section 150 Input section 160 Speech Recognition Processing Unit 170 Communications Department 180 Control Unit 190 Memory section 200 Private Cloud Computing Unit (Cloud Computing Unit)

Claims

1. An information processing device comprising an imaging unit for acquiring visual information, a sound collection unit for acquiring auditory information, and a sensor unit for acquiring the user's biometric information, An inference unit that integrates and analyzes the information acquired by the imaging unit, the sound receiving unit, and the sensor unit, and performs an inference task to identify the context, including the user's current situation, emotions, and intentions. If the computing power required to perform the inference task is insufficient, the inference unit delegates the task to an external cloud computing unit and identifies the context based on the inference results returned from the cloud computing unit. An action determination unit that autonomously determines the action to be taken based on the identified context, An output unit that provides feedback to the user regarding the results of the action that was determined, An information processing device equipped with the following.

2. The information processing apparatus according to claim 1, wherein the inference unit determines whether the model necessary to perform the inference task is a large-scale language model for thinking or dialogue, or an action model for operating external services, and selects from among the multiple cloud computing units that has software or hardware resources capable of executing the determined action model as the delegate.

3. The information processing apparatus according to claim 1, wherein the sound receiving unit separates ambient sounds from the user's voice and includes a speech recognition processing unit that detects whispers or muscle movements during speech that are below a predetermined volume threshold and converts them into linguistic information.

4. The information processing apparatus according to claim 1, wherein at least a portion of the processing in the inference unit and the action determination unit can be executed solely within the device without the need for an external device.

5. The information processing apparatus according to claim 1, wherein the action determination unit automatically generates and executes procedures for accessing and operating multiple different external services based on the user's natural language utterances or gaze indications, without requiring the user to initiate or operate specific application software.

6. Having a housing that the user can grasp, The output unit is, The information processing apparatus according to claim 1, wherein the state inside the enclosure, which provides feedback to the user regarding the execution result of the determined action, can be represented by an image display visible from outside the enclosure, vibration of the enclosure, or a change in transmitted light from inside the enclosure.

7. The output unit includes an image display unit, The action determination unit infers the user's activity state and the type of information required by the user based on the identified context. Based on the results of that reasoning, The output unit provides feedback using only sound, vibration, or light modulation, without using the image display unit, in an image-less mode. The information processing apparatus according to claim 1, wherein the feedback from the output unit is performed using the image display unit to autonomously determine one of the image-enabled modes, at least with an image.

Images