Information processing device and method, and program

Abstract

The present disclosure pertains to an information processing device and method capable of providing a service more suitable to the situation, and a program. On the basis of sensing information indicating a result of sensing a target space and general knowledge information for deriving language information indicating the situation, verbalization situation information, which is language information indicating the situation in the target space indicated by the sensing information, is generated. The present disclosure is applicable to, for example, information processing devices, electronic apparatuses, information processing methods, programs, and the like.

Description

Information processing device and method, and program

[0001] The present disclosure relates to an information processing device and method, and a program, and more particularly to an information processing device and method, and a program that are capable of providing services that are more suitable for a given situation.

[0002] Conventionally, there have been context-aware services (interaction services) that are provided according to the context of a target space (see, for example, Patent Document 1). This context may include objects present in the target space, events occurring in the target space, and the like. For example, it may include the behavior of a user and the state of the user's surroundings.

[0003] JP 2012-226741 A

[0004] In recent years, with the advancement of information processing, there has been a demand for providing such situation-adaptive services that are more suitable for the situation.

[0005] The present disclosure has been made in light of such circumstances, and aims to make it possible to provide services that are more suitable for the circumstances.

[0006] An information processing device according to one aspect of the present technology is an information processing device that includes a verbalization situation information generation unit that generates verbalization situation information, which is linguistic information that represents the situation within the target space indicated by the sensing information, based on sensing information that indicates the sensing results of the target space and general knowledge information for deriving linguistic information that represents the situation.

[0007] An information processing method according to one aspect of the present technology is an information processing method that generates verbalized situation information, which is linguistic information that represents the situation within a target space indicated by sensing information that indicates the sensing results of the target space and general knowledge information for deriving linguistic information that represents the situation, based on the sensing information.

[0008] A program according to one aspect of the present technology is a program that causes a computer to function as a verbalization situation information generation unit that generates verbalization situation information, which is linguistic information that represents the situation within the target space indicated by the sensing information, based on sensing information that indicates the sensing results of the target space and general knowledge information for deriving linguistic information that represents the situation.

[0009] In an information processing device, method, and program according to one aspect of the present technology, verbalized situation information, which is linguistic information representing the situation within the target space indicated by the sensing information, is generated based on sensing information indicating the sensing results of the target space and general knowledge information for deriving linguistic information representing the situation.

[0010] 1 is a diagram for explaining a context-adaptive service. A diagram for explaining a context-adaptive service. A diagram for explaining a context-adaptive service. A diagram for an example of virtual space sensing. A block diagram showing an example of the main configuration of a context-adaptive service providing device. A block diagram showing an example of the main configuration of a verbalization situation information generating unit. A diagram showing an example of situation recognition prior information. A diagram showing an example of general knowledge information. A diagram showing examples of verbalization situation information and reliability. A diagram showing an example of an importance score. A flowchart for explaining an example of the flow of a context-adaptive service providing process. A flowchart for explaining an example of the flow of a verbalization situation information generating process. A block diagram showing an example of the main configuration of a context-adaptive service providing device. A diagram showing an example of a personal characteristic information database. A block diagram showing an example of the main configuration of a verbalization situation information generating unit. A diagram showing an example of personal characteristic information. A diagram showing an example of dictionary detailing. A diagram showing an example of personal suppression condition information. A diagram showing an example of reaction detection. A diagram showing an example of personal behavioral characteristic information. A diagram showing an example of personalization control information. A flowchart for explaining an example of the flow of a context-adaptive service providing process. A flowchart for explaining an example of the flow of a verbalization situation information generating process. A flowchart for explaining an example of the flow of a service setting process. A block diagram showing an example of the main configuration of a computer.

[0011] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature, etc. supporting technical content and technical terminology 2. Context-aware service 3. Context-aware service using verbalized context information 4. Adaptation to individuals 5. Supplementary notes

[0012] <1. Literature, etc. supporting technical content and technical terminology> The scope of what is disclosed in the present technology includes not only the content described in the embodiments, but also the content described in the following patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following patent documents.

[0013] Patent Document 1: (mentioned above)

[0014] In other words, the contents of the above-mentioned patent documents and the contents of other documents referenced in the above-mentioned patent documents are also used as the basis for determining the support requirements.

[0015] <2. Context-Aware Services> <Overview of Context-Aware Services> Advances in computing, the widespread interconnection of devices via wireless networks, and advances in AI (Artificial Intelligence) such as DNN (Deep Neural Networks) are driving increasingly sophisticated services delivered by devices and systems. Until now, users analyzed their objectives and selected single-function devices, then achieved their goals through trial and error, adapting the controls to the device's needs. However, as devices began to integrate and offer more advanced functions, there was a risk that the traditional primitive controls, such as switches and menus, would no longer be fully operational. For example, even if viewing with xR (cross reality), a technology that combines the real and virtual worlds to create new experiences, allowed users to view content from their preferred viewpoint, constantly adjusting the viewpoint could become a cumbersome task for users. Thus, there is a need for interfaces that move away from primitive instructions in machine language and utilize advanced intelligent processing capabilities to more closely resemble the semantic units perceived by humans.

[0016] Therefore, a situation-adaptive service (interaction service) has been proposed that is provided according to the situation in a target space, such as automatically manipulating the viewpoint depending on the content being viewed, the user's preferences, and the user's judgment tendency. This situation may include objects present in the target space and events occurring in the target space. For example, it may include the user's behavior and the state of the user's surroundings. For example, as described in Patent Document 1, it has been proposed to generate a user profile and provide services according to the user's preferences based on the profile.

[0017] In such context-aware services, sensing of the real space in which the user exists is performed, as shown in Fig. 1, for example. For example, sensing of the user's surroundings, interactions with objects, the user's posture and movements (motion), the user's external state, etc. is performed. Note that external state refers to things other than the user's internal state, such as the user's facial expression, body temperature, heart rate, etc. Furthermore, interaction with objects refers to, for example, the relationship between a person and a tool (how a person uses a tool), etc.

[0018] Then, a context-aware service according to the sensing results (sensing information) is provided to the user in real space. This context-aware service may be any kind of service. For example, it may be a service according to what the user is doing. It may also be a service according to the location or situation. It may also be a service according to the user's mood. It may also be a service according to (a prediction of) the next event. It may also be a service that reduces the workload (effort) of the work the user is doing. It may also be a service related to recording the user's actions.

[0019] The context-aware service may be provided in a virtual space, as shown in FIG. 2, for example. In this case, sensing of the virtual space may be performed, and the context-aware service may be provided based on the sensing results of the virtual space. The sensing of the virtual space may be performed by a virtual sensor. Furthermore, as shown in FIG. 2, the context-aware service may be provided based on both the sensing results of the virtual space and the sensing results of the real space. Furthermore, the context-aware service may be provided to a user in the real space, or to a virtual user (e.g., an avatar) in the virtual space.

[0020] For example, as a context-aware service, control of sensing in a virtual space may be performed. For example, in Fig. 3, in the virtual space 1, a sensing object 11a has a camera function and captures an image of a whole body 13a of a tracking target object 12a from the front.

[0021] The tracking target object 12a is an object that is the target of tracking by the sensing object 11a. In the example of Fig. 3, the tracking target object 12a is an avatar, but it may be a person other than an avatar (NPC: Non-player character), an animal, a plant, a product, a background, etc. in the virtual space 1.

[0022] The sensing object 11a has a sensing function of acquiring representations in the three-dimensional virtual space 1 from the virtual space 1. The representations are, for example, images or sounds of objects. The sensing object 11a is set in correspondence with the tracked object 12a. Note that the sensing object 11a only needs to be functionally realized by software, and may or may not have an actual entity in the virtual space 1 (display an actual entity in the virtual space 1).

[0023] The sensing object 11a in FIG. 3 has a camera function as one of its sensing functions. When the sensing object 11a is set to sense the entire body 13a of the tracked object 12a from directly in front of the tracked object 12a, it moves to an appropriate spatial position according to the spatial position of the tracked object 12a, as indicated by the dotted arrow, maintains its spatial positional relationship with the tracked object 12a, and senses (photographs) the entire body 13a from directly in front of the tracked object 12a. The actual sensing process corresponds to rendering. In this case, generating (rendering) a two-dimensional image of the tracked object 12a corresponds to sensing. Hereinafter, the three-dimensional spatial position in the virtual space 1 will be simply referred to as "position," and the spatial positional relationship will be simply referred to as "positional relationship."

[0024] 3, since the sensing object 11a has a camera function, the entire body of the tracked object 12a is the sensing range, and processing is performed to acquire an image as a sensing result of the sensing range as virtual space sensing. Note that the sensing range is not limited to the entire body of the tracked object 12a, and it is also possible to set a part of the tracked object, the field of view range as seen from the tracked object, etc.

[0025] Then, services using images obtained as a result of sensing using virtual space sensing, analysis results of the images, or information based on the images or analysis results are provided (feedback) to the virtual space user terminal used by users of virtual space 1.

[0026] <Improving the Quality of Context-Aware Services> In recent years, with the advancement of information processing, there has been a demand for improving the quality (convenience) of such context-aware services. In other words, there is a demand for providing services that are more suitable for the situation. By providing a service that is more in line with the user's wishes, it is possible to reduce the need for instructions to switch between provided services, thereby improving the convenience of the service. Here, a service that is more in line with the user's wishes refers to a service that increases the user's satisfaction. For example, it may be a service that is closest to what the user previously desired, or a service that the user becomes more comfortable with after it is provided, even if they are not aware of it in advance.

[0027] <3. Context-Aware Service Using Verbalized Situation Information> <Verbalization of Situation> Therefore, verbalized situation information that expresses the situation of a target space as linguistic information is generated, and the verbalized situation information can be used to provide a context-aware service. In this way, it is possible to set services to be provided based on more comprehensive and advanced judgment. For example, it is possible to provide not only services that simply correspond to objects or events in the target space, but also services that are more suitable for the situation of the target space estimated based on those objects or events. As a result, it is possible to improve the convenience of the provided services and increase user satisfaction with the provided services. In other words, it is possible to improve the quality (convenience) of context-aware services.

[0028] <Context-aware service providing device> Fig. 4 is a block diagram showing an example of the configuration of a context-aware service providing device, which is one aspect of an information processing device to which the present technology is applied. The context-aware service providing device 100 shown in Fig. 4 is a device that provides a context-aware service. That is, the context-aware service providing device 100 senses a target space and provides a service according to the situation of the target space based on the sensing results. In doing so, the context-aware service providing device 100 generates verbalized situation information that represents the situation in the target space based on the sensing information, and provides a service corresponding to the verbalized situation information.

[0029] For example, the context-aware service providing apparatus 100 may provide a context-aware service that controls a television in a living room depending on the situation. For example, when a user sits on a sofa in the living room and faces the television, the context-aware service providing apparatus 100 detects the user's appearance, recognizes objects such as the user and the sofa, recognizes actions such as sitting, recognizes the surrounding situation, and, based on these recognition results, verbalizes the situation as "the user is sitting on the sofa and facing the TV." Then, the context-aware service providing apparatus 100 turns on the television as a service corresponding to the verbalized information. In this way, the context-aware service providing apparatus 100 can provide services depending on the sensed situation.

[0030] 4, the context-aware service providing apparatus 100 includes a detection unit 111, a verbalized context information generation unit 112, a service providing unit 113, and a general knowledge information database 114. Note that the detection unit 111 and the general knowledge information database 114 do not have to be provided within the context-aware service providing apparatus 100, and information output therefrom may be provided to the context-aware service providing apparatus 100 from an external device, an external service, or the like.

[0031] The detection unit 111 has a sensor (sensing device) and performs processing related to sensing. For example, the detection unit 111 may acquire sensing information (sensing results of the target space) using the sensor. The detection unit 111 may also supply the sensing information to the verbalization situation information generation unit 112.

[0032] The sensor may be any device that detects information about the target space. For example, the detection unit 111 may include a sensor that detects the physical and optical conditions of the target space. The physical conditions may include the appearance (shape, color, material, etc.) of objects present in the target space. The optical conditions may include the appearance of light in the target space (light source, brightness, etc.). The sensor may include, for example, an image sensor (visible light sensor), an infrared sensor, a Time of Flight (TOF) sensor, a millimeter-wave radar, etc. The detection unit 111 may also include a sensor (also referred to as other sensor) that detects things other than the physical and optical conditions of the target space. The "other sensor" may detect any information. For example, the "other sensor" may include, for example, a sensor that detects position, a sensor that detects temperature, a sensor that detects humidity, a sensor that detects audio information such as a microphone, a sensor that detects vital information such as pulse and heart rate, etc. The detection unit 111 may include any number of sensors. When the detection unit 111 includes multiple sensors, the detection unit 111 may include multiple types of sensors. In this case, the number of each type of sensor is arbitrary, and they may or may not be the same as each other.

[0033] The sensing information is information (sensing results) obtained by the applied sensor. That is, the sensing information may include information corresponding to the above-described description of the sensor types. For example, the sensing information may include the output of an image sensor (i.e., image data (also referred to as image sensing information)). The sensing information may also include the output of an infrared sensor. The sensing information may also include the output of a TOF sensor. The sensing information may also include the output of a millimeter-wave radar. The sensing information may also include sensing results such as position, temperature, humidity, audio information, and vital sign information. The sensing information may be information at a certain time, such as a still image, or information having a time axis, such as a moving image (information that can change over time). In the following description, this sensing information will be described as including image sensing information and other sensing results (also referred to as other sensing information). The other sensing information may include sensing results such as position, temperature, humidity, audio information, and vital sign information.

[0034] The target space may have any size and shape. The number of target spaces is arbitrary, and may be single or multiple. The target space may be a real space, a virtual space, or both. When the target space is a virtual space, the detection unit 111 may have a virtual sensor that senses the virtual space. In other words, in the case of a virtual space, the sensing information may include the output (sensing result) of the virtual sensor.

[0035] The verbalization situation information generation unit 112 executes processing related to verbalization of a situation. For example, the verbalization situation information generation unit 112 may acquire sensing information from the detection unit 111. Alternatively, the verbalization situation information generation unit 112 may acquire general knowledge information from a general knowledge information database 114.

[0036] The verbalization situation information generation unit 112 may generate verbalization situation information, which is linguistic information representing the situation in the target space indicated by the sensing information, based on the sensing information (sensing information indicating the sensing result of the target space) and general knowledge information for deriving linguistic information representing the situation. The verbalization situation information generation unit 112 may supply the generated verbalization situation information to the service provision unit 113.

[0037] The verbalization situation information generation unit 112 may also generate a reliability of the generated verbalization situation information. The verbalization situation information generation unit 112 may supply the generated reliability to the service providing unit 113. In this case, the verbalization situation information generation unit 112 may supply the reliability to the service providing unit 113 by associating the reliability with the verbalization situation information corresponding to the reliability.

[0038] The service providing unit 113 executes processing related to the provision of a context-aware service. For example, the service providing unit 113 may acquire the verbalization situation information from the verbalization situation information generating unit 112. The service providing unit 113 may provide a service corresponding to the verbalization situation information.

[0039] Furthermore, the service providing unit 113 may acquire a reliability corresponding to the verbalization situation information from the verbalization situation information generating unit 112. The service providing unit 113 may set a service to be provided based on the reliability. For example, the service providing unit 113 may provide a service corresponding to verbalization situation information with a higher reliability. Furthermore, the service providing unit 113 may prohibit (not provide) the provision of a service corresponding to verbalization situation information with a reliability lower than a predetermined standard.

[0040] For example, the service providing unit 113 may provide the context-aware service that is set according to the context of the target space to a user present in the target space, or to a user other than the user. For example, the service providing unit 113 may provide the context-aware service to (a user in) the real space, to (a virtual user in) the virtual space, to both, or to someone other than both.

[0041] The general knowledge information database 114 is a database for managing general knowledge information. The general knowledge information database 114 stores general knowledge information for deriving linguistic information representing a situation, and supplies the general knowledge information to the verbalization situation information generation unit 112 as needed.

[0042] <Verbalization Situation Information Generation Unit> Fig. 5 is a block diagram showing an example of the main configuration of the verbalization situation information generation unit 112. As shown in Fig. 5, the verbalization situation information generation unit 112 has an area division unit 151, an object recognition unit 152, a posture estimation unit 153, a behavior estimation unit 154, an object relation recognition unit 155, and a situation recognition unit 156.

[0043] The region division unit 151 performs processing related to region division of the target space. For example, the region division unit 151 may acquire image sensing information supplied from the detection unit 111. Based on the image sensing information, the region division unit 151 may detect regions in the target space where objects exist and divide the regions where each object exists from other regions. The region division unit 151 may generate object connection relationship information indicating the regions where each object exists and the connection relationships (positional relationships) between each region. This object connection relationship information may include, for example, information indicating the range of each region and identification information (label data) assigned to each region. The region division unit 151 may supply the object connection relationship information to the object recognition unit 152. The region division unit 151 may supply the object connection relationship information to the object relationship recognition unit 155. The region division unit 151 may also supply the image sensing information supplied from the detection unit 111 to the object recognition unit 152.

[0044] The object recognition unit 152 executes processing related to object recognition. For example, the object recognition unit 152 may acquire object connection relationship information from the region division unit 151. The object recognition unit 152 may also acquire image sensing information from the region division unit 151. The object recognition unit 152 may estimate (recognize) what an object is in an area where each object indicated by the object connection relationship information exists. Here, "estimation (recognition)" refers to assigning label data indicating what the object is. During this estimation (recognition), the object recognition unit 152 may use the image sensing information. The object recognition unit 152 may supply object label information including the label data of the object estimated (recognized) in this way to the object relation recognition unit 155.

[0045] The posture estimation unit 153 executes processing related to posture estimation. For example, the posture estimation unit 153 may acquire image sensing information supplied from the detection unit 111. When a human (user) is included in the target space based on the image sensing information, the posture estimation unit 153 may determine the position of the human skeleton (also referred to as bone coordinate information) by, for example, image analysis or the like. The posture estimation unit 153 may supply the bone coordinate information to the behavior estimation unit 154. The posture estimation unit 153 may supply the bone coordinate information to the object relation recognition unit 155.

[0046] The behavior estimation unit 154 executes processing related to behavior estimation. For example, the behavior estimation unit 154 may acquire bone coordinate information from the posture estimation unit 153. The behavior estimation unit 154 may estimate (recognize) what kind of behavior the human's movement (change in skeletal position over time) represents based on the bone coordinate information (change in skeletal position over time). Here, "estimation (recognition)" refers to assigning label data indicating what kind of behavior the human's movement represents. The behavior estimation unit 154 may supply behavior label information including label data of the behavior estimated (recognized) in this way to the object relation recognition unit 155.

[0047] The object relation recognition unit 155 executes processing related to the recognition of relationships between objects. For example, the object relation recognition unit 155 may acquire object connection relation information from the region division unit 151. The object relation recognition unit 155 may acquire object label information from the object recognition unit 152. The object relation recognition unit 155 may acquire bone coordinate information from the posture estimation unit 153. The object relation recognition unit 155 may acquire behavior label information from the behavior estimation unit 154. The object relation recognition unit 155 may generate situation recognition advance information based on the information.

[0048] Prior situation awareness information is a combination of elements of events occurring or existing in the target space. In prior situation awareness information, the elements of events are combined as components of sentences (also called sentence elements), such as subjects, predicates, objects, and complements. In other words, multiple elements of events are combined to form a single sentence.

[0049] FIG. 6 shows an example of situation awareness prior information. In the table of FIG. 6, the white rows indicate situation awareness prior information (sentences formed by combining elements of events), and the gray rows indicate item names. Therefore, when specifying a row in this table, the gray rows should be ignored. For example, "top row situation awareness prior information" refers to the top row of the white rows in the table of FIG. 6 (person, swing down, knife, fish, kitchen, Any, Any). Similarly, in tables of other figures, the gray rows indicate item names, so when specifying a row, the gray rows should be ignored. Note that the example of FIG. 6 is merely an example, and the sentence elements (items) constituting the situation awareness prior information may be any, and sentence elements (items) other than those in the examples described above may be included.

[0050] The object relation recognition unit 155 may use the sensing information to generate situation awareness prior information, which is a combination of elements of events occurring or existing in the target space. For example, the object relation recognition unit 155 may generate the situation awareness prior information by combining elements of events occurring or existing in the target space as sentence elements, as shown in each row of the table in FIG. 6 . In the example of FIG. 6 , object label information (i.e., recognized objects) is set for the subject and object (tool, object, etc.) of the situation awareness prior information. Furthermore, behavior label information (i.e., recognized behavior) is set for the predicate of the situation awareness prior information. Furthermore, complements (location, time, vital signs, etc.) of the situation awareness prior information are set based on other sensing information (i.e., other sensing results). Of course, these are merely examples, and each sentence element may be set based on any information. The object relation recognition unit 155 may supply the generated situation awareness prior information to the situation recognition unit 156.

[0051] The situation recognition unit 156 executes processing related to situation recognition. For example, the situation recognition unit 156 may acquire situation recognition prior information from the object relation recognition unit 155. The situation recognition unit 156 may acquire general knowledge information from the general knowledge information database 114.

[0052] General knowledge information is information that indicates a general correspondence between a combination of elements of an event (i.e., prior situation recognition information) and linguistic information that expresses a situation. FIG. 7 shows an example of general knowledge information. In the table of FIG. 7, the rows shown in white indicate general knowledge information, and the rows shown in gray indicate item names. That is, the general knowledge information has items including the same sentence elements as the prior situation recognition information and superordinate concepts (classes and instances). The superordinate concepts (classes) are linguistic information that express the situation indicated by the prior situation recognition information (combinations of sentence elements). In the example of FIG. 7, the superordinate concept (class) has the items "behavior" and "surrounding circumstances." That is, the superordinate concept (class) in this case expresses the situation indicated by the prior situation recognition information (combinations of sentence elements) using linguistic information that expresses "behavior" and linguistic information that expresses "surrounding circumstances." For example, in the example at the top, it is indicated that the situation indicated by the prior situation recognition information is "cooking in the kitchen."

[0053] The superordinate concept (instance) indicates an example of the situation indicated by the superordinate concept (class). For example, in the top example, the target (object) in the situation recognition prior information is "fish," so "fish cooking" is shown as the superordinate concept (instance) corresponding to the superordinate concept (class) ("cooking in the kitchen") corresponding to that situation recognition prior information. Note that the example in FIG. 7 is just one example, and any items may be included in the general knowledge information. For example, the superordinate concept may include items other than those mentioned above.

[0054] The situation recognition unit 156 may use general knowledge information to generate verbalized situation information corresponding to the generated prior situation recognition information. The verbalized situation information is linguistic information that represents the situation in the target space indicated by the sensing information. In other words, the situation recognition unit 156 may use a superordinate concept (class) and a superordinate concept (instance) of the items that make up the general knowledge information as the verbalized situation information. For example, the situation recognition unit 156 may select a row in the general knowledge information shown in FIG. 7 that has an item that matches the acquired prior situation recognition information ( FIG. 6 ), and use the superordinate concept (class) and the superordinate concept (instance) of that row as the verbalized situation information. For example, if the top row of the table in FIG. 6 is acquired as the situation recognition prior information, the situation recognition unit 156 may select a row in the general knowledge information (table in FIG. 7 ) that includes the situation recognition prior information (subject: “person,” predicate: “swing down,” object (tool): “knife,” object (object): “fish,” complement (place): “kitchen,” complement (time): “Any,” and complement (vital): “Any”), i.e., the top row, and use the superordinate concept (class) (“cooking in the kitchen”) and superordinate concept (instance) (“fish cooking”) of the top row as the verbalized situation information. In this specification, generating verbalized situation information corresponding to the situation recognition prior information in this manner is also referred to as “superimposition.” That is, the situation recognition unit 156 may use the general knowledge information to superimpose the situation recognition prior information.

[0055] The situation recognition unit 156 may supply the thus generated verbalized situation information to the service provision unit 113 (FIG. 4).

[0056] The situation recognition unit 156 may also generate a reliability of the generated verbalization situation information. This reliability is a parameter indicating the degree of correspondence between the generated verbalization situation information and the prior situation recognition information used to generate the verbalization situation information. In generating the verbalization situation information, general knowledge information is selected that includes a combination of sentence elements that is closest to the prior situation recognition information. For example, the situation recognition unit 156 may calculate a match rate (degree of match) between the combination of sentence elements of the selected general knowledge information and the prior situation recognition information, and use this as the reliability of the verbalization situation information.

[0057] 8 is a diagram showing examples of verbalization situation information and reliability. As shown in the table of FIG. 8, the reliability is associated with the generated verbalization situation information. The situation recognition unit 156 may supply the reliability generated in this manner to the service providing unit 113. In this case, the situation recognition unit 156 may supply the reliability to the service providing unit 113 by associating the reliability with the verbalization situation information corresponding to the reliability.

[0058] The general knowledge information may include an importance score for each element, and the situation recognition unit 156 may generate the reliability using the importance score. For example, in the general knowledge information, an importance score may be assigned to each element as shown in the table in FIG. 9. In the example of FIG. 9, an importance score of "H (High)," "M (Middle)," or "L (Low)" is assigned to each of the following items: predicate, object (tool), object (target), complement (place), complement (time), and complement (vital). "H" has the highest value, and "L" has the lowest value. This importance score is a parameter indicating the degree of contribution to the estimation (recognition) of the situation. In other words, the higher the assigned importance score, the more important the element is in the estimation (recognition) of the situation. In other words, the higher the importance score assigned to an element matches, the higher the matching rate (degree of matching) between the combination of sentence elements in the general knowledge information and the situation recognition prior information. Therefore, the situation recognition unit 156 may set the reliability based on the importance scores of elements that match between the general knowledge information and the situation recognition prior information. For example, the situation recognition unit 156 may set the reliability to a larger value as the importance score of the matching element is larger. Alternatively, the situation recognition unit 156 may set the reliability to a smaller value as the importance score of the matching element is smaller.

[0059] 9 is an example, and the method of assigning importance scores is not limited to this example. The importance scores may be two levels, four or more levels, or may be numerical values.

[0060] As described above, by generating verbalized situation information representing the situation in the target space indicated by the sensing information based on the sensing information and the general knowledge information, the context-aware service providing apparatus 100 can set a context-aware service using the verbalized situation information. This allows the context-aware service providing apparatus 100 to set the service to be provided based on more comprehensive and advanced judgment. For example, the context-aware service providing apparatus 100 can provide not only services that simply correspond to objects and events in the target space, but also services that are more appropriate for the situation in the target space estimated based on the objects and events (e.g., user behavior, surrounding circumstances, and instances thereof). In other words, it is possible to provide services that are aligned with the purpose of the user's actions. Furthermore, verbalization allows the situation to be recognized using a conceptual structure similar to how humans perceive it, allowing interactions such as confirming the recognition results and setting and adjusting the recognition level to be performed in human language. This minimizes interactions for the service. Furthermore, because actions are verbalized as meanings aligned with the purpose without being distracted by trivial actions, the accuracy of service provision is improved. As a result, the convenience of the provided service can be improved, and the user's satisfaction with the provided service can be increased, that is, the quality (convenience) of the context-aware service can be improved.

[0061] Furthermore, by generating situation recognition advance information using sensing information and generating verbalized situation information corresponding to that situation recognition advance information using general knowledge information, the situation-adaptive service providing device 100 can generate verbalized situation information more easily than when generating superordinate concepts (classes) or superordinate concepts (instances) directly from sensing information.

[0062] In addition, by further generating a reliability indicating the degree of correspondence with the verbalized situation information, the situation-adaptive service providing device 100 can set a situation-adaptive service based on that reliability, and can provide a service that is more suitable for the situation in the target space.

[0063] Furthermore, by generating a reliability using the importance score assigned to each element of the general knowledge information, the context-aware service providing apparatus 100 can generate a more useful reliability. Then, by applying the reliability thus generated, the context-aware service providing apparatus 100 can provide a service more suitable for the situation in the target space.

[0064] Furthermore, by providing a service corresponding to the generated verbalized situation information as described above, the situation-aware service providing device 100 can provide a service that is more suitable for the situation of the target space.

[0065] Furthermore, by providing a service corresponding to verbalized situation information with a higher degree of reliability, the context-aware service providing device 100 can provide a service that is more suitable for the situation in the target space.

[0066] Furthermore, by prohibiting the provision of services corresponding to verbalized situation information whose reliability is lower than a predetermined standard, the context-aware service providing device 100 can avoid providing services that are not suitable for the situation of the target space, thereby enabling the context-aware service providing device 100 to provide services that are more suitable for the situation of the target space.

[0067] Furthermore, the context-aware service providing apparatus 100 can provide services that are more suitable for the situation of the target space, not only in the real space but also in the virtual space.

[0068] <Flow of Context-Aware Service Providing Process> An example of the flow of context-aware service providing process executed by the context-aware service providing apparatus 100 will be described with reference to the flowchart of FIG.

[0069] When the context-aware service providing process is started, the detection unit 111 senses the target space in step S101.

[0070] In step S102, the verbalization situation information generating unit 112 executes a verbalization situation information generating process to generate verbalization situation information and reliability based on the sensing result and general knowledge information.

[0071] In step S103, the service providing unit 113 sets the service to be provided according to the generated verbalization situation information and reliability.

[0072] In step S104, the service providing unit 113 provides the set service as a context-aware service.

[0073] When the process of step S104 is completed, the context-aware service providing process is completed.

[0074] <Flow of Verbalization Situation Information Generation Process> An example of the flow of the verbalization situation information generation process executed in step S102 of FIG. 10 will be described with reference to the flowchart of FIG.

[0075] When the verbalization situation information generation process starts, in step S121, the area division unit 151 detects areas in the target space where objects exist, divides the areas where each object exists from other areas, and generates object connection relationship information.

[0076] In step S122, the object recognition unit 152 estimates (recognizes) what the object is in the area that is recognized as including the object, and generates object label information.

[0077] In step S123, the posture estimation unit 153 estimates (recognizes) the position of the human skeleton in the target space and generates bone coordinate information.

[0078] In step S124, the behavior estimation unit 154 estimates (recognizes) what kind of behavior the human's movement (change in skeletal position over time) represents based on the bone coordinate information (change in skeletal position over time), and generates behavior label information.

[0079] In step S125, the object relationship recognition unit 155 recognizes the relationships between objects based on the object connection relationship information, object label information, bone coordinate information, and behavior label information, and generates situation recognition advance information.

[0080] In step S126, the situation recognition unit 156 uses the general knowledge information to elevate the situation recognition prior information, i.e., generates verbalized situation information corresponding to the situation recognition prior information.

[0081] In step S127, the situation recognition unit 156 generates the reliability of the verbalized situation information based on the general knowledge information and the situation recognition prior information.

[0082] When the process of step S127 is completed, the verbalization situation information generation process ends.

[0083] By executing each process as described above, the context-aware service providing apparatus 100 can provide a service that is more suitable for the situation of the target space.

[0084] <4. Adaptation to Individuals> <Use of Personal Characteristic Information> In providing a context-aware service to which the present technology is applied as described above, the service may be provided further according to the characteristics of an individual. For example, verbalization situation information may be generated using personal characteristic information. This personal characteristic information is information related to the verbalization of a situation set for each user. Therefore, this personal characteristic information can also be said to be information expressing the characteristics of the user (preferences, judgment tendencies, behavioral tendencies, habits, etc.). By applying this personal characteristic information in generating the verbalization situation information of the present technology described above, it is possible to generate verbalization situation information that is more suitable for the user's characteristics. Therefore, it is possible to provide a service that is more suitable for the user's characteristics.

[0085] Furthermore, personal characteristic information may be applied when setting a service corresponding to the verbalization situation information, thereby enabling a service to be set that is more suitable for the characteristics of the user as a service to be provided.

[0086] In addition, the verbalization situation information may be filtered using personal characteristic information (or personal suppression condition information included therein). The personal suppression condition information is information set for each user that indicates a situation in which the provision of a service should be suppressed. For example, the reliability corresponding to the verbalization situation information may be updated based on this personal suppression condition information. For example, if the situation indicated by the verbalization situation information matches the suppression condition indicated by the personal suppression condition information, the reliability corresponding to the verbalization situation information may be reduced. In this way, it is possible to suppress the provision of services for situations that match the suppression conditions set in advance for each user. This makes it possible to provide services that are more suitable for the user's characteristics.

[0087] <Use of Personalization Control Information> Furthermore, personalization control information may be applied when setting a service corresponding to verbalization situation information. This personalization control information is information indicating the likelihood of providing a service to a user. This personalization control information is set for each service to be provided. In this way, a service that is more suitable for the characteristics of the user can be set as the service to be provided.

[0088] Furthermore, the user's reaction to the provided service may be detected, the user's satisfaction level may be determined, and the satisfaction level may be reflected in the personalized control information. This allows for more accurate personalized control information (more suitable for the user's characteristics) to be generated. Then, by applying such personalized control information as described above, it is possible to provide a service that is more suitable for the user's characteristics.

[0089] <Context-Aware Service Providing Device> Fig. 12 is a block diagram showing an example of the configuration of a context-aware service providing device, which is one aspect of an information processing device to which the present technology is applied. The context-aware service providing device 300 shown in Fig. 12 is a device similar to the context-aware service providing device 100 shown in Fig. 4. That is, the context-aware service providing device 300 senses a target space and provides a service according to the situation of the target space based on the sensing results. In this case, the context-aware service providing device 300 generates verbalized situation information representing the situation in the target space based on the sensing information, and provides a service corresponding to the verbalized situation information. In addition, in this case, the context-aware service providing device 300 applies personal characteristic information and personalized control information.

[0090] For example, the context-aware service providing apparatus 300 may provide, as a context-aware service, a service that controls a TV (television signal receiving display device) in a living room depending on the context. For example, when a user sits on a sofa in the living room and faces the TV, the context-aware service providing apparatus 300 detects this state, recognizes objects such as the user and the sofa, recognizes actions such as sitting, recognizes the surrounding context, and, based on these recognition results, verbalizes the situation as "the user is sitting on the sofa and facing the TV." In this case, the context-aware service providing apparatus 300 verbalizes this depending on the user's personal characteristic information.

[0091] Furthermore, if the user tends not to watch TV even when sitting on the sofa, the context-aware service providing apparatus 300 acquires such personal characteristic information from the database and filters out information to the TV control service.

[0092] Furthermore, the context-aware service providing apparatus 300 sets the content and volume to be initially presented when the TV is turned on based on personal characteristic information relating to the user's personal preferences. Furthermore, the context-aware service providing apparatus 300 determines the content and volume based on personalization control information set for each service. This personalization control information indicates the likelihood of providing a service to the user and is set for each service.

[0093] The context-aware service providing apparatus 300 may reflect the user's reaction (satisfaction level) to a service previously provided in the likelihood of the personalized control information. For example, if a user responds positively after watching TV content, such as by nodding, the context-aware service providing apparatus 300 increments the likelihood parameter of the personalized control information for that service. Conversely, if a user responds negatively after watching TV, such as by shaking their head, the context-aware service providing apparatus 300 decrements the likelihood parameter of the personalized control information for that service. In this way, the likelihood parameter of the personalized control information is a cumulative frequency of the user's satisfied / dissatisfied reactions for each activated service.

[0094] In this way, the context-aware service providing apparatus 300 can provide services further in accordance with the characteristics (preferences, judgment tendencies, behavioral tendencies, habits, etc.) of the user (individual).

[0095] 12 , the context-aware service providing apparatus 300 includes a detection unit 311, a verbalized context information generation unit 312, a filtering unit 313, a service providing unit 314, a response detection unit 315, a personalization processing unit 316, a general knowledge information database 317, a personal characteristic information database 318, and a service-specific personalization control information database 319. Note that the detection unit 311, the general knowledge information database 317, the personal characteristic information database 318, and the service-specific personalization control information database 319 do not have to be provided within the context-aware service providing apparatus 300, and information output therefrom may be provided to the context-aware service providing apparatus 300 from an external device, an external service, or the like.

[0096] The detection unit 311 is a processing unit similar to the detection unit 111, has a sensor (sensing device), and executes processing related to sensing. For example, the detection unit 311 may acquire sensing information (sensing results of the target space) using the sensor. The detection unit 311 may also supply the sensing information to the verbalization situation information generation unit 312.

[0097] The verbalization situation information generation unit 312 is a processing unit similar to the verbalization situation information generation unit 112 and performs basically the same processing as the verbalization situation information generation unit 112. For example, the verbalization situation information generation unit 312 may acquire sensing information from the detection unit 311. Alternatively, the verbalization situation information generation unit 312 may acquire general knowledge information from the general knowledge information database 117. Then, the verbalization situation information generation unit 312 may generate the verbalization situation information based on the sensing information and the general knowledge information. The verbalization situation information generation unit 312 may also generate a reliability of the generated verbalization situation information. However, the verbalization situation information generation unit 312 may further acquire personal authentication information and personal characteristic information from the personal characteristic information database 318. The verbalization situation information generation unit 312 may generate the verbalization situation information and the reliability using the personal authentication information and the personal characteristic information. For example, the verbalization situation information generation unit 312 may generate the reliability using personal characteristic information indicating the frequency of occurrence of a situation for each user. In this case, the verbalization situation information generating unit 312 may identify a user present in the target space using personal authentication information, and generate a reliability using personal characteristic information of the identified user.

[0098] The verbalization situation information generation unit 312 may supply the generated verbalization situation information and reliability to the filtering unit 313. The verbalization situation information generation unit 312 may supply the generated verbalization situation information to the reaction detection unit 315.

[0099] The filtering unit 313 performs processing related to filtering of the verbalization situation information. For example, the filtering unit 313 may acquire the verbalization situation information and the reliability from the verbalization situation information generation unit 312. The filtering unit 313 may also acquire personal suppression condition information from the personal characteristic information database 318. This personal suppression condition information is information indicating a predetermined situation as a condition for suppressing service provision (also referred to as a suppression condition). This personal suppression condition information is set for each user. In the case of a situation indicated as a suppression condition by this personal suppression condition information, the filtering unit 313 may filter the verbalization situation information so as to suppress the provision of a service corresponding to that situation. For example, the filtering unit 313 may reduce the reliability corresponding to the verbalization situation information that satisfies the suppression condition indicated by the personal suppression condition information. The filtering unit 313 may supply the filtered verbalization situation information and the reliability to the service providing unit 314.

[0100] The service providing unit 314 is a processing unit basically similar to the service providing unit 113, and executes processing related to the provision of context-aware services. For example, the service providing unit 314 may acquire verbalization situation information and reliability from the filtering unit 313. The service providing unit 314 may provide a service corresponding to the verbalization situation information. The service providing unit 314 may also set a service to be provided based on the reliability. The service providing unit 314 may inquire of the personalization processing unit 316 about a service that corresponds to the characteristics of the user to whom the service is to be provided. Then, the service set by the personalization processing unit 316 in response to the inquiry, i.e., a service that is set to correspond to the user's characteristics (preferences, judgment tendencies, behavioral tendencies, habits, etc.), may be provided.

[0101] The reaction detection unit 315 executes processing related to detection of the user's reaction to the provided service. For example, the reaction detection unit 315 may acquire verbalization situation information from the verbalization situation information generation unit 312. The reaction detection unit 315 may detect the user's reaction (satisfaction or dissatisfaction) to the provided service based on the verbalization situation information. The reaction detection unit 315 may generate satisfaction level information indicating the detection result and supply it to the personalization processing unit 316.

[0102] The personalization processing unit 316 executes processing related to the setting of the service to be provided. For example, the personalization processing unit 316 may acquire an inquiry from the service providing unit 314. The personalization processing unit 316 may acquire personal behavioral characteristic information corresponding to the inquiry from the personal characteristic information database 318. The personalization processing unit 316 may set the service to be provided based on the personal behavioral characteristic information. The personalization processing unit 316 may also acquire personalization control information corresponding to the inquiry from the service-specific personalization control information database 319. The personalization processing unit 316 may use the personalization control information to set the service to be provided.

[0103] Furthermore, the personalization processing unit 316 may acquire satisfaction level information from the response detection unit 315. The personalization processing unit 316 may update the personalization control information in the service-specific personalization control information database 319 using the satisfaction level information.

[0104] The general knowledge information database 317 is a processing unit similar to and executes similar processes as the general knowledge information database 114. For example, the general knowledge information database 317 may store general knowledge information and supply the general knowledge information to the verbalization situation information generation unit 312 as needed.

[0105] The personal characteristic information database 318 is a database that manages personal characteristic information. For example, as shown in A of FIG. 13 , the personal characteristic information database 318 may store personal authentication information 331 and personal characteristic information 332, and may supply this information to the verbalization situation information generation unit 312 as needed. The personal authentication information 331 is information related to the authentication of a user (individual). Similar to general knowledge information, the personal characteristic information 332 is information indicating a general correspondence between a combination of event elements (i.e., situation recognition prior information) and linguistic information that expresses a situation. However, the personal characteristic information 332 is composed only of information that matches the characteristics of the user (individual). The personal characteristic information 332 also includes information indicating the frequency of occurrence of the situation. In other words, the personal characteristic information 332 is information indicating the characteristics (preferences, judgment tendencies, behavioral tendencies, habits, etc.) of the user (individual).

[0106] The individual characteristic information 332 may include individual suppression condition information 332A. The individual characteristic information database 318 may supply the individual suppression condition information 332A to the filtering unit 313 as necessary.

[0107] Furthermore, the individual characteristic information database 318 may supply the individual characteristic information 332 to the personalization processing unit 316 as individual behavioral characteristic information as necessary.

[0108] Note that this personal characteristic information 332 depends on the user (individual) and is not dependent on the service. Therefore, the same personal characteristic information 332 may be used for multiple services. Therefore, there is a one-to-one correspondence between the personal characteristic information database 318 (the personal characteristic information 332 stored therein) and the user. Furthermore, there may be a one-to-many correspondence between the personal characteristic information database 318 (the personal characteristic information 332 stored therein) and the detection unit 111. Furthermore, there may be a one-to-many correspondence between the personal characteristic information database 318 (the personal characteristic information 332 stored therein) and the service application that provides the service, as shown in B of FIG. 13, for example.

[0109] The service-specific personalized control information database 319 is a database that manages personalized control information. The service-specific personalized control information database 319 stores personalized control information that is set independently for each service, and may supply the personalized control information to the personalization processing unit 316 as needed.

[0110] 14 is a block diagram showing an example of the main configuration of the verbalization situation information generation unit 312. As shown in Fig. 14, the verbalization situation information generation unit 312 has an area division unit 351, an object recognition unit 352, a posture estimation unit 353, a behavior estimation unit 354, an object relation recognition unit 355, and a situation recognition unit 356.

[0111] The region division unit 351 is a processing unit similar to the region division unit 151, has a configuration similar to the region division unit 151, and performs processing similar to that of the region division unit 151. For example, the region division unit 351 may acquire image sensing information supplied from the detection unit 311. Based on the image sensing information, the region division unit 351 may detect regions in the target space where objects exist and divide the regions where each object exists from other regions. The region division unit 351 may generate object connection relationship information indicating the regions where each object exists and the connection relationships (positional relationships) between each region. The region division unit 351 may supply the generated object connection relationship information to the object recognition unit 352. The region division unit 351 may supply the object connection relationship information to the object relationship recognition unit 355. The region division unit 351 may also supply the image sensing information supplied from the detection unit 311 to the object recognition unit 352.

[0112] The object recognition unit 352 is a processing unit basically similar to the object recognition unit 152, has basically the same configuration as the object recognition unit 152, and performs basically the same processing as the object recognition unit 152. That is, the object recognition unit 352 may acquire object connection relationship information from the region division unit 351 and estimate (recognize) what each object indicated by the object connection relationship information is in the region where the object exists. In this case, the object recognition unit 352 may acquire image sensing information from the region division unit 351 and use the image sensing information for the estimation (recognition). However, the object recognition unit 352 may also acquire personal authentication information from the personal characteristic information database 318. Then, if the object is a person, the object recognition unit 352 may use the personal authentication information to recognize who the person is (which user). That is, the object recognition unit 352 may identify a user present in the target space using the personal authentication information. In other words, the object label information may include label data indicating who the object is. The object recognition unit 352 may supply the generated object label information to the object relation recognition unit 355 .

[0113] The posture estimation unit 353 is a processing unit similar to the posture estimation unit 153, has a configuration similar to the posture estimation unit 153, and executes processing similar to that of the posture estimation unit 153. For example, the posture estimation unit 353 may acquire image sensing information supplied from the detection unit 311. The posture estimation unit 353 may obtain bone coordinate information of a human (user) included in the target space based on the image sensing information. The posture estimation unit 353 may supply the generated bone coordinate information to the behavior estimation unit 354. The posture estimation unit 353 may supply the bone coordinate information to the object relation recognition unit 355.

[0114] The behavior estimation unit 354 is a processing unit similar to the behavior estimation unit 154, has a configuration similar to the behavior estimation unit 154, and executes processing similar to that of the behavior estimation unit 154. For example, the behavior estimation unit 354 may acquire bone coordinate information from the posture estimation unit 353, and generate behavior label information based on the bone coordinate information (changes in the time direction). The behavior estimation unit 354 may supply the generated behavior label information to the object relation recognition unit 355.

[0115] The object relation recognition unit 355 is a processing unit similar to the object relation recognition unit 355, has a configuration similar to the object relation recognition unit 355, and executes processing similar to that of the object relation recognition unit 355. For example, the object relation recognition unit 355 may acquire object connection relation information from the region division unit 351, acquire object label information from the object recognition unit 352, acquire bone coordinate information from the posture estimation unit 353, and acquire behavior label information from the behavior estimation unit 354, and generate situation recognition advance information based on this information. The object relation recognition unit 355 may supply the generated situation recognition advance information to the situation recognition unit 356.

[0116] The situation recognition unit 356 is a processing unit basically similar to the situation recognition unit 156, has basically the same configuration as the situation recognition unit 156, and executes basically the same processing as the situation recognition unit 156. For example, the situation recognition unit 356 may acquire prior situation recognition information from the object relation recognition unit 355, acquire general knowledge information from the general knowledge information database 317, and use the general knowledge information to generate verbalized situation information corresponding to the prior situation recognition information. In other words, the situation recognition unit 356 may use the general knowledge information to elevate the prior situation recognition information. Furthermore, the situation recognition unit 356 may generate a reliability of the generated verbalized situation information. However, the situation recognition unit 356 may also acquire personal characteristic information from the personal characteristic information database 318. The situation recognition unit 356 may generate the verbalized situation information and the reliability based on the prior situation recognition information, general knowledge information, and personal characteristic information. For example, the situation recognition unit 356 may generate verbalized situation information and reliability based on prior situation recognition information and general knowledge information, and further update the reliability based on personal characteristic information.

[0117] FIG. 15 shows an example of personal characteristic information. In the table of FIG. 15, the rows shown in white indicate personal characteristic information, and the rows shown in gray indicate the item names. That is, the personal characteristic information has the same items (sentence elements and superordinate concepts) as the general knowledge information. Furthermore, the personal characteristic information has a frequency. This "frequency" indicates the occurrence frequency (for each user) of the situation indicated by the personal characteristic information (each row of FIG. 15).

[0118] The situation recognition unit 356 may generate the reliability using this frequency of the personal characteristic information. For example, the situation recognition unit 356 may add this frequency or a value corresponding to this frequency to the reliability. Alternatively, the situation recognition unit 356 may subtract this value from the reliability. Alternatively, the situation recognition unit 356 may generate the reliability using the personal characteristic information of the user recognized (identified) by the object recognition unit 352.

[0119] It is also possible to switch the dictionary data applied to the personal characteristic information depending on the frequency of occurrence of a situation. For example, if it is determined that Person A cooks frequently, the dictionary may be switched to one that can recognize cooking at a finer level of granularity. This can improve the accuracy of cooking-related services. An example of fine-grained personal characteristic information for cooking instances switched for a person who cooks frequently is shown in FIG. 16. This switching can be achieved, for example, by loading a new dictionary into the personal characteristic information database 318 the next time the application is started.

[0120] The situation recognition unit 356 may supply the thus generated verbalized situation information to the filtering unit 313 (FIG. 12).

[0121] <Filtering Unit> As described with reference to A in FIG. 13 , in the personal characteristic information database 318, the personal characteristic information 332 may include personal suppression condition information 332A. For example, as shown in FIG. 17 , the personal suppression condition information 332A lists situations that serve as suppression conditions. In the example of FIG. 17 , the situations of "cooking in the kitchen," "cooking," and "cooking fish" are listed as suppression conditions. For example, if the user cooks outside the kitchen, the situation of "cooking in the kitchen" is listed as the suppression condition. Furthermore, if the user does not cook anywhere, the situation of "cooking" is listed as the suppression condition. Furthermore, if the user does not particularly like cooking fish, the situation of "cooking fish" is listed as the suppression condition.

[0122] If the situation of the target space matches such a suppression condition, the filtering unit 313 may filter the verbalization situation information (the reliability corresponding to the verbalization situation information may be reduced).

[0123] For example, the filtering unit 313 may lower the reliability for an operation that ignores a specific behavior, and may raise the reliability for an operation that increases the sensitivity. Note that this personal suppression condition information may be read in advance from the personal characteristic information database 318 when the system is initialized, or may be read and updated during operation by a user operation. The user may explicitly specify the sensitivity for a specific behavior, but if the satisfaction level is low as a result of executing a service, the execution of the same service may be suppressed by lowering the sensitivity for that behavior.

[0124] <Response Detection Unit> The response detection unit 315 may acquire the verbalization status information from the verbalization status information generation unit 312 and detect the user's response (satisfaction or dissatisfaction) to the provided service based on the verbalization status information. For example, assume that the response detection unit 315 acquires the verbalization status information shown in A of FIG. 18 . The response detection unit 315 has predefined behaviors corresponding to satisfaction and dissatisfaction, as shown in the table shown in B of FIG. 18 . Note that the table shown in B of FIG. 18 is an example, and any behaviors corresponding to satisfaction and dissatisfaction may be used (not limited to this example). The response detection unit 315 may detect the user's response to the provided service (whether the user expressed satisfaction or dissatisfaction) in accordance with this definition. The response detection unit 315 may then provide the detection result (satisfaction or dissatisfaction) to the personalization processing unit 316 as satisfaction information. In other words, the response detection unit 315 may detect a predetermined user behavior and generate satisfaction information based on the detected behavior.

[0125] Feedback on satisfaction / dissatisfaction with the services provided is extremely important for the system, as it keeps a loop of gradual improvement in the accuracy of service provision. It is also desirable for users to customize the service to themselves as they use it, and the accuracy of the service provided improves. Because of this win-win relationship, it is thought that an agreement can be reached as long as it does not place a burden on users.

[0126] <Personalization Processing Unit> The personalization processing unit 316 may set the services to be provided using personal behavioral characteristic information that indicates the frequency of occurrence of a situation in the target space for each user. For example, when the personalization processing unit 316 receives an inquiry from the service providing unit 314, the personalization processing unit 316 may determine a superordinate concept (instance) and its frequency corresponding to the queried verbalization situation information (superordinate concept (class)) based on the personal behavioral characteristic information acquired from the personal characteristic information database 318.

[0127] For example, the personalization processing unit 316 may acquire individual behavioral characteristic information as shown in A of FIG. 19 . Using the individual behavioral characteristic information, the personalization processing unit 316 may determine the superordinate concept (instance) that best corresponds to the queried verbalization situation information (superordinate concept (class)) and its frequency, as shown in B of FIG. 19 . The personalization processing unit 316 may set the service to be provided based on such frequency (or the trend in frequency of each situation). In the example of B of FIG. 19 , the personalization processing unit 316 may instruct the service providing unit 314 to prioritize providing dish information for "meat dishes," which has the highest frequency.

[0128] Furthermore, the personalization processing unit 316 may set the service to be provided using the personalization control information in the service-specific personalization control information database 319. This personalization control information is information indicating the likelihood of providing the service to the user. For example, as shown in A of Fig. 20, the likelihood may be set for each genre of content to be provided in the personalization control information. Furthermore, as shown in B of Fig. 20, if the content to be provided is a cooking program, the likelihood may be set for each type of cooking.

[0129] The personalized control information may also be updated based on satisfaction information indicating the user's level of satisfaction with the provided service.

[0130] With the above-described configuration, the context-aware service providing device 300 can apply this personal characteristic information to the generation of the verbalization situation information of the present technology described above, and can generate verbalization situation information that is more suitable for the user's characteristics (preferences, judgment tendencies, behavioral tendencies, habits, etc.). Therefore, it is possible to provide a service that is more suitable for the user's characteristics.

[0131] <Flow of Context-Aware Service Providing Process> An example of the flow of context-aware service providing process executed by the context-aware service providing apparatus 300 will be described with reference to the flowchart of FIG.

[0132] When the context-aware service providing process is started, the detection unit 311 senses the target space in step S301.

[0133] In step S302, the verbalization situation information generating unit 312 executes a verbalization situation information generating process to generate verbalization situation information and reliability based on the sensing result, general knowledge information, and personal characteristic information.

[0134] In step S303, the filtering unit 313 filters the verbalization situation information based on the personal suppression condition information.

[0135] In step S304, the service providing unit 314 and the personalization processing unit 316 execute a service setting process to set a service to be provided in accordance with the verbalization situation information, the reliability, the personal behavioral characteristic information, and the personalization inhibition information.

[0136] In step S305, the service providing unit 314 provides the set service as a context-aware service.

[0137] In step S306, the reaction detection unit 315 detects the user's reaction to the provided service and determines whether satisfaction or dissatisfaction with the service has been detected. If it is determined that satisfaction or dissatisfaction has been detected, the process proceeds to step S307.

[0138] In step S307, the personalization processing unit 316 updates the personalization control information based on the satisfaction level information.

[0139] When the process of step S307 is completed, the context-aware service provision process is completed. Also, if it is determined in step S306 that satisfaction or dissatisfaction with the provided service is not detected, the process of step S307 is skipped and the context-aware service provision process is completed.

[0140] <Flow of Verbalization Situation Information Generation Process> An example of the flow of the verbalization situation information generation process executed in step S302 of FIG. 21 will be described with reference to the flowchart of FIG.

[0141] When the verbalization situation information generation process starts, in step S321, the area division unit 351 detects areas in the target space where objects exist, divides the areas where each object exists from other areas, and generates object connection relationship information.

[0142] In step S322, the object recognition unit 352 estimates (recognizes) what the object is for the area recognized as including the object, and generates object label information. If the object is a person, the object recognition unit 352 identifies who the sparse object is (recognizes the user) based on personal authentication information.

[0143] In step S323, the posture estimation unit 353 estimates (recognizes) the position of the human skeleton in the target space and generates bone coordinate information.

[0144] In step S324, the behavior estimation unit 354 estimates (recognizes) the behavior of the human movement (change in skeletal position over time) based on the bone coordinate information (change in the time direction), and generates behavior label information.

[0145] In step S325, the object relationship recognition unit 355 recognizes the relationships between objects based on the object connection relationship information, object label information, bone coordinate information, and behavior label information, and generates situation recognition advance information.

[0146] In step S326, the situation recognition unit 356 uses the general knowledge information to elevate the situation recognition prior information, i.e., generates verbalized situation information corresponding to the situation recognition prior information.

[0147] In step S327, the situation recognition unit 356 generates the reliability of the verbalized situation information based on the general knowledge information and the situation recognition prior information.

[0148] In step S328, the situation recognition unit 356 updates the reliability based on the personal characteristic information.

[0149] When the process of step S328 is completed, the verbalization situation information generation process ends.

[0150] <Flow of Service Setting Process> Next, an example of the flow of the service setting process executed in step S304 of FIG. 21 will be described with reference to the flowchart of FIG.

[0151] When the service setting process is started, the service providing unit 314 determines whether the reliability is sufficiently high in step S351. If it is determined that the reliability is sufficiently high, the process proceeds to step S352.

[0152] In step S352, the personalization processing unit 316 obtains the frequency corresponding to the instance of the verbalization situation information based on the individual behavioral characteristic information.

[0153] In step S353, the personalization processing unit 316 sets a service corresponding to the more frequent instance.

[0154] In step S354, the personalization processing unit 316 further configures the service according to the likelihood of the personalization control information.

[0155] When the process of step S354 is completed, the service setting process is completed. Also, if it is determined in step S351 that the reliability is not sufficiently high, the process proceeds to step S355.

[0156] In step S355, the service providing unit 314 sets the service not to be provided. When the process of step S355 ends, the service setting process ends.

[0157] By executing each process as described above, the context-aware service providing device 300 can provide a service that is more suitable for the context of the target space.

[0158] 5. Supplementary Notes Computer The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term computer includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0159] FIG. 24 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0160] In a computer 900 shown in FIG. 24, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0161] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0162] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives a removable recording medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0163] In a computer configured as described above, the CPU 901 loads a program stored in the storage unit 913, for example, into the RAM 903 via the input / output interface 910 and the bus 904, and executes the program. This performs the series of processes described above. The RAM 903 may store data and the like necessary for the CPU 901 to execute various processes, as appropriate.

[0164] The program executed by the computer may be applied by being recorded on a removable recording medium 921 such as a package medium, for example. In this case, the program may be read from the removable recording medium 921 attached to the drive 915 and installed in the storage unit 913 via the input / output interface 910.

[0165] This program may also be provided via any wired or wireless transmission medium, such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program may be received by the communication unit 914 and installed in the storage unit 913 via the input / output interface 910.

[0166] Alternatively, this program may be installed in advance in the ROM 902 or the storage unit 913, or both.

[0167] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0168] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0169] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0170] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0171] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0172] <Others> In this specification, the term "associate" means, for example, making it possible to use (link) one piece of data when processing the other piece of data. In other words, data that are associated with each other may be combined into one piece of data, or may be individual pieces of data. For example, information associated with encoded data (image) may be transmitted over a transmission path separate from that of the encoded data (image). Furthermore, for example, information associated with encoded data (image) may be recorded on a recording medium separate from that of the encoded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer to only a portion of the data, rather than the entire data. For example, a moving image and information corresponding to the moving image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0173] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0174] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0175] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0176] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0177] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0178] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0179] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0180] The present technology can also be configured as follows. (1) An information processing device comprising: a verbalization situation information generation unit that generates, based on sensing information indicating a sensing result of a target space and general knowledge information for deriving linguistic information indicating a situation, verbalization situation information, which is linguistic information indicating a situation in the target space indicated by the sensing information. (2) The general knowledge information indicates a general correspondence relationship between a combination of elements of events and linguistic information indicating a situation, and the verbalization situation information generation unit is configured to use the sensing information to generate prior situation recognition information, which is a combination of elements of events that have occurred or existed in the target space, and to generate the verbalization situation information corresponding to the generated prior situation recognition information, using the general knowledge information. (3) The information processing device according to (2), wherein the verbalization situation information generation unit is configured to further generate a reliability indicating a degree of correspondence between the prior situation recognition information and the verbalization situation information. (4) The information processing device according to (3), wherein the general knowledge information includes an importance score for each of the elements, and wherein the verbalization situation information generation unit is configured to generate the reliability using the importance score. (5) The information processing device according to (3) or (4), wherein the verbalization situation information generation unit is further configured to generate the reliability using personal characteristic information indicating an occurrence frequency of the situation for each user. (6) The information processing device according to (5), wherein the verbalization situation information generation unit is configured to identify a user present in the target space using personal authentication information, and generate the reliability using the personal characteristic information of the identified user. (7) The information processing device according to any of (3) to (6), further comprising a filtering unit that filters the generated verbalization situation information using personal suppression condition information for each user that indicates a predetermined situation as an suppression condition. (8) The information processing device according to (7), wherein the filtering unit is configured to reduce the reliability corresponding to the verbalization situation information that satisfies the suppression condition indicated by the personal suppression condition information.(9) The information processing device according to any one of (3) to (8), further comprising a service providing unit that provides a service corresponding to the generated verbalization situation information. (10) The information processing device according to (9), wherein the service providing unit is configured to provide a service corresponding to the verbalization situation information having a higher reliability. (11) The information processing device according to (9) or (10), wherein the service providing unit is configured to prohibit the provision of a service corresponding to the verbalization situation information having a reliability lower than a predetermined standard. (12) The information processing device according to any one of (9) to (11), further comprising a personalization processing unit that sets the service to be provided using personal behavioral characteristic information indicating the frequency of occurrence of the situation for each user. (13) The information processing device according to (12), wherein the personalization processing unit is further configured to set the service to be provided using personalization control information indicating the likelihood of providing the service to the user. (14) The information processing device according to (13), further comprising a reaction detection unit that detects a reaction of the user to the service based on the generated verbalization situation information and generates satisfaction information indicating a level of satisfaction of the user based on the detected reaction, wherein the personalization processing unit is configured to update the personalization control information based on the satisfaction information. (15) The information processing device according to (14), wherein the reaction detection unit detects a predetermined behavior of the user and generates the satisfaction information based on the detected behavior. (16) The information processing device according to any of (1) to (15), wherein the sensing information includes output from an image sensor. (17) The information processing device according to (16), wherein the sensing information further includes sensing results of position, temperature, humidity, voice information, or vital information. (18) The information processing device according to any of (1) to (17), wherein the target space is a real space, a virtual space, or both.

[0181] (19) An information processing method for generating verbalized situation information, which is linguistic information representing a situation in a target space indicated by sensing information indicating a sensing result of the target space and general knowledge information for deriving linguistic information representing the situation, based on the sensing information.

[0182] (20) A program that causes a computer to function as a verbalization situation information generation unit that generates verbalization situation information, which is linguistic information that represents the situation in the target space indicated by the sensing information, based on sensing information that indicates the sensing results of the target space and general knowledge information for deriving linguistic information that represents the situation.

[0183] 100 Context-aware service providing device, 111 Detection unit, 112 Verbalization situation information generation unit, 113 Service providing unit, 114 General knowledge information database, 151 Area division unit, 152 Object recognition unit, 153 Posture estimation unit, 154 Behavior estimation unit, 155 Object relation recognition unit, 156 Situation recognition unit, 300 Context-aware service providing device, 311 Detection unit, 312 Verbalization situation information generation unit, 313 Filtering unit, 314 Service providing unit, 315 Reaction detection unit, 316 Personalization processing unit, 317 General knowledge information database, 318 Personal characteristic information database, 319 Service-specific personalization control information database, 331 Personal recognition information, 332 Personal characteristic information, 332A Personal suppression condition information, 351 Area division unit, 352 Object recognition unit, 353 posture estimation unit, 354 behavior estimation unit, 355 object relationship recognition unit, 356 situation recognition unit, 900 computer

Claims

1. An information processing device having a verbalization situation information generation unit that generates verbalization situation information, which is linguistic information that represents the situation within the target space indicated by the sensing information, based on sensing information that indicates the sensing results of the target space and general knowledge information for deriving linguistic information that represents the situation.

2. The information processing device described in claim 1, wherein the general knowledge information indicates a general correspondence between a combination of elements of an event and linguistic information representing a situation, and the verbalization situation information generation unit is configured to use the sensing information to generate situation recognition advance information, which is a combination of elements of an event that occurs or exists in the target space, and to use the general knowledge information to generate the verbalization situation information corresponding to the generated situation recognition advance information.

3. The information processing device according to claim 2, wherein the verbalization situation information generating unit is further configured to generate a reliability indicating the degree of correspondence between the situation recognition prior information and the verbalization situation information.

4. The information processing device according to claim 3, wherein the general knowledge information includes an importance score for each of the elements, and the verbalization situation information generating unit is configured to generate the reliability using the importance score.

5. The information processing device according to claim 3, wherein the verbalization situation information generating unit is further configured to generate the reliability using personal characteristic information indicating the frequency of occurrence of the situation for each user.

6. The information processing device according to claim 5, wherein the verbalization situation information generation unit is configured to identify a user present in the target space using personal authentication information, and to generate the reliability using the personal characteristic information of the identified user.

7. The information processing device according to claim 3, further comprising a filtering unit that filters the generated verbalization situation information using personal suppression condition information for each user that indicates a predetermined situation as a suppression condition.

8. The information processing device according to claim 7, wherein the filtering unit is configured to reduce the reliability corresponding to the verbalization situation information that satisfies the suppression condition indicated by the personal suppression condition information.

9. The information processing device according to claim 3, further comprising a service providing unit that provides a service corresponding to the generated verbalization situation information.

10. The information processing device according to claim 9, wherein the service providing unit is configured to provide a service corresponding to the verbalization situation information with a higher reliability.

11. The information processing device according to claim 9, wherein the service providing unit is configured to prohibit the provision of a service corresponding to the verbalization situation information whose reliability is lower than a predetermined standard.

12. The information processing device according to claim 9, further comprising a personalization processing unit that sets the service to be provided using personal behavioral characteristic information indicating the frequency of occurrence of the situation for each user.

13. The information processing device according to claim 12, wherein the personalization processing unit is further configured to set the service to be provided using personalization control information indicating the likelihood of providing the service to the user.

14. An information processing device as described in claim 13, further comprising a reaction detection unit that detects the user's reaction to the service based on the generated verbalization situation information and generates satisfaction information indicating the user's satisfaction level based on the detected reaction, and the personalization processing unit is configured to update the personalization control information based on the satisfaction information.

15. The information processing device according to claim 14, wherein the reaction detection unit detects a predetermined behavior of the user and generates the satisfaction level information based on the detected behavior.

16. The information processing device according to claim 1, wherein the sensing information includes an output from an image sensor.

17. The information processing device according to claim 16, wherein the sensing information further includes sensing results of position, temperature, humidity, audio information, or vital information.

18. The information processing device according to claim 1, wherein the target space is a virtual space.

19. An information processing method for generating verbalized situation information, which is linguistic information representing the situation in the target space indicated by the sensing information, based on sensing information indicating the sensing results of the target space and general knowledge information for deriving linguistic information representing the situation.

20. A program that causes a computer to function as a verbalization situation information generation unit that generates verbalization situation information, which is linguistic information that represents the situation in the target space indicated by the sensing information, based on sensing information that indicates the sensing results of the target space and general knowledge information for deriving linguistic information that represents the situation.

Images