Information processing device, information processing method, and program

Abstract

The present disclosure pertains to an information processing device that makes it possible to achieve a highly accurate recognizer adapted to changes in a usage environment, an information processing method, and a program. On the basis of a recognition stability map obtained by mapping, for each spatial coordinate position, a recognition stability with regard to a predetermined recognition process, the present invention reads a recognition stability of a spatial coordinate position of a hand of a person who is a recognition target, selects, according to the recognition stability, a recognition process set in which a pre-process, a recognition model, and a post-process are combined, and recognizes the recognition target using the selected recognition process set. The present invention can be applied to an HMD.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that realize a highly accurate recognizer that adapts to changes in the usage environment.

[0002] 2. Description of the Related Art Technologies that recognize objects, sounds, and other targets based on input data such as images and sounds are becoming increasingly common.

[0003] This technology is realized by a recognizer that uses a recognition model formed by learning using training data consisting of pairs of input data and correct answer data for the input data.

[0004] However, due to changes in the environment in which the input data is acquired, unexpected noise may be added to the input data, which may cause the recognition accuracy to become unstable.

[0005] This is thought to be due to the fact that the robustness of the recognition model is reduced when input is outside the data distribution of the training data used for training, but it is difficult to create a dataset that is close to the actual distribution using training data.

[0006] Therefore, it is conceivable to prepare a plurality of different recognition models and switch between them to stabilize the recognition accuracy.

[0007] As a technique for preparing a plurality of different recognition models and switching between them, for example, there is a technique described in Patent Document 1 (see Patent Document 1).

[0008] Japanese Patent Application Laid-Open No. 2021-135793

[0009] However, the technology of Patent Document 1 switches the recognition process in order to reduce the processing load, and is not capable of stabilizing the recognition accuracy.

[0010] The present disclosure has been made in view of such circumstances, and in particular, aims to realize a highly accurate recognizer that can adapt to changes in the usage environment.

[0011] An information processing device and a program according to one aspect of the present disclosure include an information processing device including: a recognition processing unit that recognizes a recognition target using a recognition processing set that combines a plurality of recognition processing elements; and a recognition processing set selection unit that selects a combination of the plurality of recognition processing elements based on a recognition stability for a predetermined recognition process that is specified at the spatial coordinate position of the recognition target.

[0012] An information processing method according to one aspect of the present disclosure is an information processing method including a recognition process that recognizes a recognition target using a recognition process set that combines a plurality of recognition processing elements, and a recognition process set selection process that selects a combination of the plurality of recognition processing elements based on the recognition stability for a predetermined recognition process that is identified at the spatial coordinate position of the recognition target.

[0013] In one aspect of the present disclosure, a recognition target is recognized by a recognition processing set that combines multiple recognition processing elements, and the combination of the multiple recognition processing elements is selected based on the recognition stability for a specified recognition process that is identified at the spatial coordinate position of the recognition target.

[0014] FIG. 1 is a diagram illustrating the configuration of an HMD. FIG. 1 is a diagram illustrating an overview of an HMD according to the present disclosure. FIG. 2 is a diagram illustrating an example external configuration of an HMD according to the present disclosure. FIG. 3 is a block diagram illustrating the hardware configuration of the HMD of FIG. 3. FIG. 4 is a functional block diagram illustrating functions implemented by the HMD of FIG. 4. FIG. 5 is a diagram illustrating recognition stability. FIG. 6 is a diagram illustrating generation of a recognition stability map. FIG. 7 is a diagram illustrating a recognition processing set group. FIG. 8 is a diagram illustrating learning and selection by a recognition processing set selector. FIG. 9 is a flowchart illustrating recognition stability map generation processing. FIG. 10 is a flowchart illustrating recognition processing set selector learning processing. FIG. 11 is a flowchart illustrating hand posture recognition processing. FIG. 12 is a diagram illustrating the effect of hand posture recognition processing and presentation of a recognition stability map. FIG. 13 is a diagram illustrating a first modified example. FIG. 14 is a flowchart illustrating hand posture recognition processing in the first modified example. FIG. 15 is a diagram illustrating a second modified example. FIG. 16 is a flowchart illustrating hand posture recognition processing in the second modified example. FIG. 17 is a diagram illustrating a third modified example. FIG. 18 is a functional block diagram illustrating functions implemented by an HMD according to the third modified example. FIG. 19 is a flowchart illustrating hand posture recognition processing in the third modified example.

[0015] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0016] Hereinafter, embodiments of the present technology will be described in the following order.

[0017] 1. Overview of the present disclosure 2. Preferred embodiment 3. First modified example 4. Second modified example 5. Third modified example 6. Example of execution by software

[0018] <<1. Overview of the Present Disclosure>> The present disclosure is directed to realizing a highly accurate recognizer that can adapt to changes in the usage environment. Therefore, first, an overview of the present disclosure will be described.

[0019] As shown in FIG. 1 , an overview of the present disclosure will be described using an HMD (Head Mounted Display) 1 worn by a user H as an example.

[0020] The HMD 1 in FIG. 1 is worn on the head of a user H and is used for playing games and viewing content.

[0021] The HMD 1 includes a see-through display 2 and a camera 3. The camera 3 captures an image with a viewing angle substantially equivalent to the field of view that the user H can see through the see-through display 2.

[0022] It is assumed that user H wearing HMD 1 can see the outside world as field of view V1 through the see-through display 2. At this time, camera 3 captures an image with a viewing angle substantially equivalent to the field of view V1 visible to user H through the see-through display 2. Therefore, the field of view V1 visible to user H is substantially the same as the image captured by camera 3, and therefore the image captured by camera 3 can also be said to be field of view V1. For this reason, hereinafter, the image captured by camera 3 will also be referred to as image V1 (= field of view V1).

[0023] The HMD 1 displays a controller-shaped UI (User Interface) image C in AR (Augmented Reality) on the display 2. The UI image C displayed in AR on the display 2 is visually recognized by the user H as if it were floating above the field of view V1 of the outside world.

[0024] Furthermore, when the user H extends his / her hand 4 and makes a hand movement as if to operate the controller-like UI image C displayed in AR, the HMD 1 recognizes the movement and change in position of the hand 4' in the image as a hand posture (hand gesture) based on the image captured by the camera 3, and accepts the corresponding operation input by regarding it as an operation input to the UI image C functioning as a controller that corresponds to the recognized hand posture (hand gesture).

[0025] That is, in the case of Figure 1, when a hand posture (hand gesture) is detected in which the play button, represented by a right-convex triangular mark on the left side of UI image C displayed as a controller in field of view V1 displayed in the upper right corner, is being pressed by hand 4', HMD 1 assumes that the play button on UI image C displayed as a controller has been operated by hand 4', and accepts a play instruction as an operation input.

[0026] Here, consider a case where the HMD 1 operates as described above, and the UI image C is projected onto the display 2 in an AR display state on the image captured by the camera 3, so that the user H can see, for example, the field of view V2 in the lower right corner of Figure 1.

[0027] In the visual field V2, the area Zb is sufficiently bright due to the influence of the surrounding ambient light, while the area Zd is dark due to the influence of the surrounding ambient light for some reason.

[0028] In this way, when a field of view V2 with some dark areas is captured as image V2 by camera 3, if hand 4' is present within area Zb, the image recognition accuracy for recognizing hand posture by HMD 1 can be expected to be relatively stable.

[0029] However, as shown in image V2 in the lower right of Figure 1, if the hand 4' is present in the dark area Zd, the image recognition accuracy for recognizing the hand posture using the HMD 1 is likely to be lower than usual.

[0030] As a result, if hand 4' is present in the dark area Zd, as shown in image V2 in the lower right of Figure 1, the hand posture cannot be properly recognized, and there is a possibility that the operation input made by user H to UI image C by moving hand 4 cannot be recognized.

[0031] In general, image recognition processing for dark images tends to have lower recognition accuracy than image recognition processing for bright images, so image recognition processing that compensates for this decrease in recognition accuracy is necessary. For this reason, image recognition processing for bright images and image recognition processing for dark images generally need to be different.

[0032] Therefore, in the present disclosure, as shown in the HMD 11 of Figure 2, an output probability value P related to the recognition result when a reference image recognition process is performed is measured for each 3D coordinate (X, Y, Z) in the space in the usage environment of the HMD 11, and the measurement result is associated with the 3D coordinate and three-dimensionally mapped as recognition stability (X, Y, Z, P), thereby generating a three-dimensional recognition stability map M.

[0033] The HMD 11 in FIG. 2 has a configuration corresponding to that of the HMD 1, and includes a see-through display 12 and a camera 13 corresponding to the display 2 and the camera 3, respectively.

[0034] Furthermore, the distribution in three-dimensional space expressed by the shades of the recognition stability map M in Figure 2 represents the level of the output probability value P as recognition stability, with the darker the color, the lower the output probability value P, and the lighter the color, the higher the output probability value P.

[0035] The HMD 11 has various types of pre-processing, recognition models, and post-processing required for hand posture recognition, as a group of recognition processing sets 11'. Based on the recognition stability set for each coordinate of this recognition stability map M, the HMD 11 selects a recognition processing set, which is a combination of pre-processing, recognition model, and post-processing that is optimal for hand posture recognition, from the group of recognition processing sets 11', and executes hand posture recognition processing using the selected recognition processing set.

[0036] The optimum combination of pre-processing, recognition model, and post-processing is selected based on the recognition stability through prior learning.

[0037] With this configuration, when the user H actually wears the HMD 11 and operates the UI image C displayed in AR, the recognition stability is identified from the coordinates in the recognition stability map M that correspond to the position of the hand 14' in the image captured by the camera 13. Then, based on the identified recognition stability, a recognition processing set consisting of a combination of pre-processing, a recognition model, and post-processing that is optimal for hand posture recognition is selected from the recognition processing set group 11', and hand posture recognition processing is executed using the selected recognition processing set.

[0038] More specifically, in the field of view V2 of Figure 2, if an image corresponding to user H's hand 14 exists at coordinates (X, Y, Z) in a dark range like hand 14', the recognition stability (X, Y, Z, P) of the coordinates (X, Y, Z) is obtained from the recognition stability map M.

[0039] Next, a recognition process set R(X, Y, Z, P) consisting of optimal pre-processing, recognition model, and post-processing is selected for the recognition stability (X, Y, Z, P). Then, hand posture recognition is performed based on the recognition process set R(X, Y, Z, P) consisting of the selected optimal combination of pre-processing, recognition model, and post-processing.

[0040] In addition, Figure 2 shows an example in which a combination of "Auto Exposure," "Model M," and "Smoothing," each surrounded by dotted lines, is selected as a recognition processing set R (X, Y, Z, P) consisting of a combination of predetermined pre-processing, recognition model, and post-processing based on the recognition stability (X, Y, Z, P).

[0041] That is, in the present disclosure, first, a recognition stability map M is generated in advance in the usage space of the HMD 11. Next, based on the generated recognition stability map M, learning is performed so that a recognition processing set consisting of a combination of pre-processing, recognition model, and post-processing that is optimal for each coordinate according to the recognition stability is selected.

[0042] Then, when the HMD 11 that has been trained in this manner is worn by the user and the coordinates (X, Y, Z) of the hand 14' to be recognized are identified within the image V2 (= field of view V2) captured by the camera 13, the corresponding recognition stability (X, Y, Z, P) is identified based on the recognition stability map M, and a recognition processing set R (X, Y, Z, P) consisting of an optimal combination of pre-processing, recognition model, and post-processing is selected according to the identified recognition stability (X, Y, Z, P), and hand posture recognition processing is performed.

[0043] As a result, in the present disclosure, optimal hand posture recognition processing is performed based on the recognition stability measured in advance for each coordinate in the usage space of the HMD 11, making it possible to realize a highly accurate recognizer that can respond to changes in the environment.

[0044] <<2. Preferred Embodiment>> Next, the external configuration of the HMD of the present disclosure will be described with reference to FIG.

[0045] The HMD 31 in FIG. 3 has a configuration corresponding to the HMD 11 in FIG. 2, and is worn on the head of the user H and used for playing games and viewing content.

[0046] The HMD 31 includes a see-through display 52 and a camera 53. The camera 53 captures an image with a viewing angle substantially equivalent to the field of view that the user can see through the display 52.

[0047] 3, the user H wearing the HMD 31 can see the outside world through the see-through display 52 worn by the user H, as shown in the right part of Fig. 3. Note that this field of view V11 can also be considered as an image V11 captured by the camera 3.

[0048] The HMD 31 controls the display 52 to display a controller-shaped UI (User Interface) image C in AR (Augmented Reality) on the field of view V11, which is the view of the outside world viewed by the user H, as if it were floating in real space.

[0049] When user H extends hand 61 and makes a hand movement as if to operate UI image C displayed in AR, HMD 31 recognizes the movement and change in position of hand 61' in the image captured by camera 51 as a hand posture (hand gesture) based on the image, and accepts the corresponding operation input by assuming that an operation input has been made to UI image C corresponding to the recognized hand gesture.

[0050] <Example of Hardware Configuration of HMD in FIG. 3> Next, an example of the hardware configuration of the HMD 31 in FIG. 3 will be described with reference to FIG.

[0051] The HMD 31 is composed of a control unit 101, an input unit 102, an output unit 103, a memory unit 104, a communication unit 105, a drive 106, a removable storage medium 107, a camera 51, a sensor 111, and a microphone 112, which are connected to each other via a bus 108 and can send and receive data and programs.

[0052] The control unit 101 is composed of a processor and a memory, and controls the overall operation of the HMD 31. The control unit 101 also includes a recognition stability detection unit 131, a recognition stability map generation unit 132, a content display control unit 133, a recognition processing set selection unit learning unit 134, an importance calculation unit 135, a recognition processing set selection unit 136, a recognition processing unit 137, and a recognition stability map display processing unit 138.

[0053] The recognition stability detection unit 131, the recognition stability map generation unit 132, the content display control unit 133, the recognition processing set selection unit learning unit 134, the importance calculation unit 135, the recognition processing set selection unit 136, the recognition processing unit 137, and the recognition stability map display processing unit 138 will be explained together when explaining the functions realized by the HMD 31 of Figure 4, which will be explained with reference to Figure 5.

[0054] The input unit 102 is made up of input devices such as a keyboard, a mouse, and a touch panel for inputting various types of information, and supplies the control unit 101 with various signals corresponding to the input information.

[0055] The output unit 103 is controlled by the control unit 101 and includes a display 52 and an audio output unit 151. As described in Fig. 3, the display 52 is a see-through display that allows the user H wearing the HMD 31 to see through the display and also displays UI images and the like in AR. Note that the display 52 may be any other display device as long as it allows the user H wearing the HMD 31 to see what is ahead and can also display UI images. For example, the display is not limited to a see-through type and may be a display that covers the entire area in front of the user H's eyes.

[0056] The audio output unit 151 is made up of an audio output device such as a speaker, and outputs various types of voice, music, sound effects, and the like as audio.

[0057] The storage unit 104 is composed of a hard disk drive (HDD), a solid state drive (SSD), or a semiconductor memory, and is controlled by the control unit 101 to write or read various data and programs. The storage unit 104 also stores a recognition stability map 161 and a recognition processing set group 162, which are written or read as needed under the control of the control unit 101.

[0058] The communication unit 105 is controlled by the control unit 101 and realizes wired or wireless communication such as that represented by LAN (Local Area Network) or Bluetooth (registered trademark), and transmits and receives various data and programs to and from other users' HMDs 31 or other information processing devices via the network as necessary.

[0059] The drive 106 reads and writes data from and to a removable storage medium 107 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0060] As described above, the camera 51 captures an image of a recognition target within the field of view of the user H wearing the HMD 31 and within the viewing angle visible through the display 52, and supplies the image to the control unit 101. In this example, since the hand posture (hand gesture) is to be recognized, the recognition target is the hand of the user H wearing the HMD 31.

[0061] The sensor 111 detects image features corresponding to various information detected in association with the recognition stability for each coordinate position set in the space where the HMD 31 is used, and is a group of sensors that detect, for example, the brightness and contrast ratio of an image. Therefore, although shown as a single component in Fig. 4, the sensor 111 is configured in a number corresponding to the types set as image features.

[0062] The microphone 112 picks up the speech to be recognized and supplies it to the control unit 101 .

[0063] <Functions Realized by the HMD of FIG. 4> Next, functions realized by the HMD 31 of FIG. 4 will be described with reference to the functional block diagram of FIG.

[0064] The recognition stability detection unit 131 recognizes the hand posture in the space where the HMD 31 is used by performing hand posture recognition processing that uses a predetermined standard for each coordinate position (X, Y, Z) in the space based on an image captured by the camera 51, and further calculates the output probability value P of the recognition result as recognition stability (X, Y, Z, P) and supplies it to the recognition stability map generation unit 132.

[0065] At this time, when calculating the recognition stability, the recognition stability detection unit 131 also combines the sensing results (X, Y, Z, s1, s2, ..., sn) for each coordinate position (X, Y, Z) in space supplied from the sensor 111 with the recognition stability (X, Y, Z, P) and supplies this to the recognition stability map generation unit 132 as recognition stability (X, Y, Z, P, s1, s2, ..., sn).

[0066] Here, s1, s2, ..., sn represent detection results according to the type of sensor 111, such as brightness, contrast ratio, etc. In addition, other features that can be image features, such as output blur (output acceleration) and types of posture sets that are difficult to recognize, may be included.

[0067] FIG. 6 shows an example of the configuration of recognition stability (X, Y, Z, P, s1, s2, ..., sn). In FIG. 6, an example is shown in which the recognition stability is composed of three-dimensional values ​​of X, Y, and Z representing a coordinate position, a one-dimensional output probability value P, and n-dimensional feature quantities s1, s2, ..., sn. In FIG. 6, an example is shown in which the output probability value P is 0.912, the output acceleration as feature quantity s1 is 25.2, ..., the contrast ratio as feature quantity sn is 66. In other words, the recognition stability can be said to be a group of multiple feature quantities set for each coordinate position, each consisting of (n+1)-dimensional feature quantities, with the output probability value P being an essential component.

[0068] The recognition stability map generation unit 132 generates a recognition stability map 161 by mapping the recognition stability (X, Y, Z, P, s1, s2, ..., sn) supplied from the recognition stability detection unit 131, and stores it in the memory unit 104.

[0069] 7 , the recognition stability map generation unit 132 recognizes the hand posture for each coordinate position in the space used by the user H wearing the HMD 31 by performing a reference hand posture recognition process, calculates an output probability value P of the recognition result, and generates recognition stability (X, Y, Z, P, s1, s2, ..., sn) by summarizing the sensing results of the sensor 111 at that time and other feature amounts for each coordinate, and generates a recognition stability map 161 by mapping the recognition stability (X, Y, Z, P, s1, s2, ..., sn) of each coordinate. That is, in the recognition stability map 161, each coordinate position is stored in association with information on a group of (n+1)-dimensional feature amounts.

[0070] The distribution of shading in the recognition stability map 161 in Figure 7 represents the level of either the output probability value P or the n-dimensional feature quantity that constitutes the recognition stability, with a darker, darker distribution representing a lower level and a lighter, brighter distribution representing a higher level.

[0071] 7 , the recognition processing unit 137 executes hand posture recognition processing of the hand 61′ to be recognized based on an image V11 (which is the same as the field of view V11 visually recognized by the user H) captured by the camera 51, and based on the recognition result, recognizes an operation input to the UI image C. Then, based on the recognized operation input, the recognition processing unit 137 controls the content display control unit 133 to display content on the display 52 by display control according to the operation input.

[0072] More specifically, the recognition processing unit 137 identifies the coordinate position of the hand 61′ to be recognized based on the image V11 captured by the camera 51, and supplies information on the identified coordinate position to the recognition processing set selection unit 136.

[0073] When the recognition processing set selection unit 136 is supplied with the coordinate position of the hand to be recognized, it accesses the recognition stability map 161, reads out the recognition stability of the supplied coordinate position, and based on the read recognition stability, selects an appropriate recognition processing set from the recognition processing set group 162 and supplies it to the recognition processing unit 137.

[0074] The recognition processing set selection unit 136 has been trained in advance by the recognition processing set selection unit learning unit 134 using a decision tree algorithm to select the optimal recognition processing set from the recognition processing set group 162 for the recognition stability of each coordinate position in the recognition stability map 161.

[0075] Therefore, when the recognition processing set selection unit 136 is supplied with the coordinate position of the hand to be recognized, it accesses the recognition stability map 161, reads out the recognition stability of the supplied coordinate position, selects the recognition processing set that is optimal for the read recognition stability from the recognition processing set group 162, and supplies it to the recognition processing unit 137.

[0076] The hand posture recognition process uses a set of three types of recognition processing elements: pre-processing, recognition models, and post-processing. The recognition processing set group 162 includes, for example, a pre-processing group 162a consisting of multiple pre-processing steps, a recognition model group 162b consisting of multiple recognition models, and a post-processing group 162c consisting of multiple post-processing steps, as shown in Fig. 8. Note that there may be three or more types of recognition processing elements for one recognition process; for example, multiple pre-processing steps, multiple recognition models, and multiple post-processing steps may be included, or more than three types of recognition processing elements may be included.

[0077] 8, the pre-processing group 162a includes brightness normalization, auto exposure, and pre-processing L. The recognition model group 162b includes model 1, model 2, and model M. The post-processing group 162c includes smoothing, averaging, and post-processing N.

[0078] The recognition processing set selection unit learning unit 134 trains the recognition processing set selection unit 136 to select the optimal recognition processing set consisting of a combination of pre-processing, recognition model, and post-processing that results in the highest recognition stability from each of the pre-processing group 162a, recognition model group 162b, and post-processing group 162c of the recognition processing set group 162, according to the learned decision tree algorithm, for the recognition stability at each coordinate position of the recognition stability map 161.

[0079] For this reason, when the recognition processing set selection unit 136 is supplied with the coordinate position of the hand 61' to be recognized from the recognition processing unit 137, for example as shown in Figure 9, it reads out the corresponding recognition stability (X, Y, Z, P, s1, s2, ..., sn), and selects the combination of pre-processing, recognition model, and post-processing of the recognition processing set that is optimal for the read recognition stability from the pre-processing group 162a, recognition model group 162b, and post-processing group 162c of the recognition processing set group 162, respectively, and supplies it to the recognition processing unit 137.

[0080] Although the above description has been given of an example in which the recognition processing set selection unit 136 is trained using a decision tree algorithm, it is sufficient if the recognition processing set selection unit 136 is trained to select an optimal recognition processing set consisting of an optimal combination of pre-processing, recognition model, and post-processing for the recognition stability specified by the coordinate position on the recognition stability map. Therefore, the recognition processing set selection unit 136 may be formed by a method other than learning using a decision tree algorithm, and may be composed of, for example, a DNN (Deep Neural Network).

[0081] When selecting the optimal recognition processing set for the recognition stability, the importance calculation unit 135 calculates the contribution of each feature that constitutes the recognition stability (X, Y, Z, P, s1, s2, ..., sn) to determining the recognition processing set as importance, and outputs information on the importance of each feature to the recognition stability map display processing unit 138.

[0082] When the recognition processing set selection unit learning unit 134 causes the recognition processing set selection unit 136 to learn using a decision tree algorithm, the threshold for each feature is determined so as to divide the branches so as to reduce the Gini impurity of the data at each node, and at this time, the feature at the node where the amount of reduction in Gini impurity is large becomes the feature with high importance. Therefore, in the case of learning based on a decision tree algorithm, the importance of each of the feature that makes up the recognition stability (X, Y, Z, P, s1, s2, ..., sn) is found under the learning assumption.

[0083] Therefore, when learning is performed using a decision tree algorithm, the importance calculation unit 135 only processes to obtain the importance information for each feature that is obtained in the process in which the recognition processing set selection unit learning unit 134 trains the recognition processing set selection unit 136 using a decision tree algorithm.

[0084] On the other hand, when learning other than a decision tree algorithm, for example, learning that forms a DNN, is performed, the importance calculation unit 135 needs to calculate the importance separately. In this case, the importance calculation unit 135 calculates the importance by a calculation (Permutation Importance) using information that indicates how much an error would increase if a predetermined feature were to be completely randomized for inference, and outputs the calculated importance to the recognition stability map display processing unit 138.

[0085] That is, the importance calculation unit 135 calculates the importance so that the greater the error in a feature when inferred completely randomly, the greater the importance is, and outputs the calculated importance to the recognition stability map display processing unit 138.

[0086] When the input unit 102 is operated to instruct the display of the recognition stability map, the recognition stability map display processing unit 138 reads out from the recognition stability map 161 the feature quantities with the highest importance supplied by the importance calculation unit 135, and constructs, for example, a spatial distribution and displays it on the display 52.

[0087] Among the recognition stabilities (X, Y, Z, P, s1, s2, ..., sn) that make up the recognition stability map, the features that are important in determining the recognition processing set in the recognition processing unit 137 can be considered to be features that have a high contribution to the recognition stability, in other words, features that significantly change the recognition stability.

[0088] Therefore, it can be considered that the recognition stability map, which is the spatial distribution of important features, indicates that it may be possible to effectively improve recognition stability by taking measures to raise the recognition stability above a predetermined threshold value in areas that are smaller than the threshold value.

[0089] More specifically, for example, when the spatial distribution of a feature indicating brightness is presented as the spatial distribution of a feature with higher importance, it is possible to improve the recognition stability by taking measures such as increasing the brightness in areas where the brightness is lower than a predetermined threshold.

[0090] As a result, it becomes possible to suggest measures to improve recognition stability to the user even when the usage environment changes, and by implementing measures based on these suggestions, it becomes possible to realize a highly accurate recognizer that can adapt to changes in the usage environment.

[0091] In addition, when there are a plurality of feature amounts with high importance, for example, the importance of each feature amount may be displayed so that it can be selectively displayed.

[0092] <Recognition Stability Map Generation Process> Next, the recognition stability map generation process will be described with reference to the flowchart of FIG.

[0093] In step S31, the recognition stability detection unit 131 determines, based on the detection results of the sensor 111 or the image captured by the camera 51, whether the environment has changed or whether the input unit 102 has been operated by the user to instruct an update of the recognition stability map, and repeats the same processing until the environment has changed or an update is instructed.

[0094] If it is determined in step S31 that the environment has changed or an update has been instructed, the process proceeds to step S32.

[0095] In step S32, the recognition stability detection unit 131 sets an unprocessed coordinate position in space as the processing target coordinates (X, Y, Z).

[0096] In step S33, the recognition stability detection unit 131 performs a predetermined recognition process based on the image captured by the camera 51, calculates an output probability value P from the recognition result, and acquires recognition stability (P, s1, s2, ..., sn) that combines various detection results detected by the sensor 111 and other feature quantities, and supplies it to the recognition stability map generation unit 132.

[0097] In step S34, the recognition stability map generating unit 132 maps the acquired recognition stability (P, s1, s2, . . . , sn) onto the processing target coordinates (X, Y, Z).

[0098] In step S35, the recognition stability detection unit 131 determines whether or not there is an unprocessed coordinate position, and if there is an unprocessed coordinate position, the process returns to step S32.

[0099] That is, the processes of steps S32 to S35 are repeated until the recognition stability (P, s1, s2, . . . , sn) is mapped for all coordinate positions.

[0100] Then, in step S35, if it is determined that there are no unprocessed coordinate positions and that the recognition stability (P, s1, s2, . . . , sn) has been mapped for all coordinate positions, the process proceeds to step S36.

[0101] In step S36, the recognition stability map generation unit 132 generates a recognition stability map 161 from the recognition stability (X, Y, Z, P, s1, s2, ..., sn) of all mapped coordinate positions and registers it in the memory unit 104.

[0102] Through the above series of processes, the recognition stability map 161 is generated or updated when the usage environment of the HMD 31 changes or when there is a request for updating from the user.

[0103] <Recognition Processing Set Selector Learning Process> Next, the recognition processing set selector learning process will be described with reference to the flowchart of FIG.

[0104] In step S51, the recognition processing set selection unit learning unit 134 determines whether or not a recognition processing set selection unit learning process has been instructed, and repeats the same process until an instruction is received.

[0105] If it is determined in step S51 that a recognition processing set selection unit learning process has been instructed, the process proceeds to step S52.

[0106] In step S52, the recognition processing set selection unit learning unit 134 accesses the recognition stability map 161 and sets the coordinate position of the processing to the processing target coordinates.

[0107] In step S53, the recognition processing set selection unit learning unit 134 acquires the recognition stability at the processing target coordinates.

[0108] In step S54, the recognition process set selection unit learning unit 134 extracts from the recognition process set group 162 the recognition process set with the highest recognition stability at the processing target coordinates.

[0109] In step S55, the recognition processing set selection unit learning unit 134 registers a pair of the recognition stability at the processing target coordinates and the recognition processing set with the highest recognition stability as training data for the processing target coordinates.

[0110] In step S56, the recognition processing set selection unit learning unit 134 determines whether or not there are any unprocessed coordinate positions in the recognition stability map 161, and if there are any unprocessed coordinate positions, the process returns to step S52.

[0111] That is, the processes of steps S52 to S56 are repeated until teacher data is registered at all coordinate positions on the recognition stability map 161.

[0112] Then, in step S56, if it is determined that there are no unprocessed coordinate positions in the recognition stability map 161 and that teacher data has been registered for all coordinate positions, the process proceeds to step S57.

[0113] In step S57 , the recognition processing set selection unit learning unit 134 causes the recognition processing set selection unit 136 to learn using the registered teacher data at all coordinate positions on the recognition stability map 161 .

[0114] In step S58, the importance calculation unit 135 calculates the importance of the feature quantities that constitute the recognition stability. More specifically, when learning is performed using a decision tree algorithm, the importance calculation unit 135 obtains the importance generated in association with the learning by the decision tree algorithm and supplies it to the recognition stability map display processing unit 138.

[0115] Furthermore, when learning is performed using a method other than the decision tree algorithm, the importance calculation unit 135 calculates the importance by a calculation (Permutation Importance) using information indicating how much the error would increase if learning were performed with a predetermined feature amount being completely randomized, and outputs the calculated importance to the recognition stability map display processing unit 138.

[0116] The above processing enables the recognition processing set selection unit 136 to select an optimal recognition processing set based on the recognition stability and supply it to the recognition processing unit 137. At this time, it also becomes possible to supply to the recognition stability map display processing unit 138 feature quantities that are highly important in selecting an optimal recognition processing set from the feature quantities that make up the recognition stability.

[0117] <Hand Posture Recognition Processing> Next, the hand posture recognition processing will be described with reference to the flowchart of FIG.

[0118] In step S71, the recognition processing unit 137 determines whether or not the input unit 102 has been operated to instruct hand posture recognition processing.

[0119] In step S71, if a hand posture recognition process is instructed, the process proceeds to step S72.

[0120] In step S72, the content display control unit 133 displays the UI image C at a predetermined position on the display 52.

[0121] In step S73, the recognition processing unit 137 recognizes the position of the hand by object recognition processing based on the image captured by the camera 51, specifies the coordinate position of the recognized position, and supplies it to the recognition processing set selection unit 136.

[0122] In step S74, the recognition processing set selection unit 136 accesses the recognition stability map 161 and reads out the recognition stability corresponding to the supplied coordinate position.

[0123] In step S 75 , the recognition processing set selection unit 136 reads out the optimum recognition processing set from the group of recognition processing sets 162 based on the read recognition stability, and supplies it to the recognition processing unit 137 .

[0124] In step S76, the recognition processing unit 137 executes hand posture recognition processing using the recognition processing set supplied by the recognition processing set selection unit 136, recognizes the hand posture, and specifies the operation input based on the recognition result.

[0125] In step S77, the recognition processing unit 137 controls the content display control unit 133 to execute processing according to the operation input.

[0126] In step S78, the recognition stability map display processing unit 138 determines whether or not an instruction to display the recognition stability map has been issued by operating the input unit 102, for example.

[0127] If an instruction to display the recognition stability map is given in step S78, the process proceeds to step S79.

[0128] In step S79, the recognition stability map display processing unit 138 generates a recognition stability map based on the most important feature value among the feature values ​​constituting the recognition stability.

[0129] In step S80, the recognition stability map display processing unit 138 displays the generated recognition stability map on the display 52.

[0130] In step S81, it is determined whether or not an instruction to end has been given. If an instruction to end has not been given, the process returns to step S71, and the subsequent steps are repeated.

[0131] Then, in step S81, when an instruction to end the process is given, the process ends.

[0132] If the hand posture recognition process is not instructed in step S71, steps S72 to S77 are skipped. If the display of the recognition stability map is not instructed in step S78, steps S79 to S80 are skipped.

[0133] With the above processing, when an image Pi such as that shown in the upper left of Figure 13 is captured by camera 51, and an instruction is given to present a recognition stability map, a recognition stability map M such as that shown in the lower left is generated and presented.

[0134] That is, since the area Zd in the image Pi is somewhat dark, the dark portion of the recognition stability map M corresponding to the area Zd has a black distribution indicating low recognition stability.

[0135] When the UI image Pui1 is displayed near this area Zd and the user's hand H1 is detected in the vicinity thereof, the recognition processing set selection unit 136 selects a recognition processing set that is strong in dark places and supplies it to the recognition processing unit 137.

[0136] As a result, hand posture recognition processing is performed using a recognition processing set that is strong in dark places, making it possible to realize a highly accurate recognizer that can adapt to changes in the usage environment.

[0137] Furthermore, the recognition stability map M can be presented at any timing, and the spatial distribution of the feature amounts with the highest importance can be displayed on the display 52 as a recognition stability map.

[0138] In the recognition stability map M, the darker the color, the lower the recognition stability. Also, an importance display field Pt is provided in the upper left corner of the recognition stability map M, and the types of feature amounts are displayed in descending order of importance from the top to the bottom of the figure.

[0139] In the importance display field Pt of FIG. 13, the three most important feature quantities are displayed in order from top to bottom as "brightness," "contrast ratio," and "acceleration." The feature quantity "brightness," surrounded by a solid line, is the most important and is represented in the current recognition stability map M.

[0140] In addition, the type of feature represented in the recognition stability map M can be switched by selecting the type of feature in the importance display field Pt. For example, when "contrast ratio" is selected, the notation "contrast ratio" is displayed surrounded by a solid frame, and the notation "brightness" is displayed surrounded by a dotted frame, and the recognition stability map M is presented as a spatial distribution of the feature consisting of "contrast ratio".

[0141] This allows the spatial distribution of features that contribute significantly to recognition stability to be presented as a recognition stability map M, making it possible to show that by making improvements to areas where the recognition stability for the presented type of feature is reduced, it is possible to effectively improve recognition stability.

[0142] More specifically, for example, when the spatial distribution of a feature type indicating "brightness" as shown in FIG. 13 is presented as the spatial distribution of a feature with a higher degree of importance, it is possible to effectively improve the recognition stability by taking measures to improve brightness in areas where "brightness" is lower than a predetermined threshold.

[0143] This makes it possible to present specific measures to the user to improve recognition stability even when the usage environment changes, and by taking measures based on this presentation, it becomes possible to effectively improve recognition stability.

[0144] As a result, it is possible to realize a highly accurate recognizer that can adapt to changes in the usage environment.

[0145] Although the above description has been given of an example of hand posture recognition processing as recognition processing, other recognition processing may be used, such as processing for recognizing gestures using the whole body.

[0146] Furthermore, although the above has described hand posture recognition processing based on images captured by the camera 51, image recognition processing of an object other than a hand may also be used, or voice recognition processing of sound picked up by the microphone 112 may also be used.

[0147] <<3. First Modification>> In the above, an example has been described in which a highly accurate recognizer is realized in response to the usage environment by switching the recognition processing set and executing the recognition processing in accordance with the recognition stability at the coordinate position of the object to be recognized (hand H1 in FIG. 13 ) based on the recognition stability map.

[0148] However, based on the recognition stability map, the display position of the UI image may be adjusted to avoid areas with relatively low recognition stability, so that the object to be recognized is detected in an area with high recognition stability.

[0149] That is, for example, as shown in FIG. 14, when a UI image Pui1 is displayed in the vicinity of a slightly dark area Zd in an image Pi, and a user's hand H1 is detected in the vicinity, the user's hand H1 to be recognized will be detected in an area with low recognition stability.

[0150] Therefore, in such a case, for example, as shown in UI image Pui2, the UI image Pui2 may be displayed in an area that is relatively close to the display position of UI image Pui1 and where the recognition stability is higher than a predetermined value.

[0151] With this display, the user's hand H2 to be recognized is guided and detected in an area where the UI image Pui2 is displayed and where recognition stability is high, so that a recognition processing set corresponding to normal brightness can be used.

[0152] <Hand Posture Recognition Processing in First Modification> Next, the hand posture recognition processing in the first modification will be described with reference to the flowchart in FIG.

[0153] In the flowchart of FIG. 15, the processes of steps S91 and S94 to S102 are the same as the processes of steps S71 to S81 in the flowchart of FIG. 12, and therefore a description thereof will be omitted.

[0154] That is, when the hand posture recognition process is started by the process of step S91, the process proceeds to step S92.

[0155] In step S92, the recognition processing set selection unit 136 reads the recognition stability map 161, selects display positions of the UI image where the recognition stability is higher than a predetermined value, and supplies the selected display positions to the recognition processing unit 137. For example, the display positions of the UI image may be determined in advance to be near each of the four corners of the display 52, and a display position where the recognition stability is higher than a predetermined value may be selected from the four display positions.

[0156] In step S93, the recognition processing unit 137 controls the content display control unit 133 to display the UI image on the display 52 at the display position selected by the recognition processing set selection unit 136.

[0157] Through the above processing, the UI image is displayed at a position with high recognition stability where the user's hand to be recognized is relatively easy to recognize, and the user's hand is guided to an area with high recognition stability, so the recognition processing set can be made to correspond to normal brightness.

[0158] <<4. Second Modification>> In the above, we have described an example in which the display position of a UI image is set based on a recognition stability map so as to avoid areas with relatively low recognition stability, thereby enabling an object to be recognized to be detected in an area with high recognition stability.

[0159] However, when the hand to be recognized is in an area with low recognition stability, the image captured by camera 51 and a recognition stability map may be presented to prompt the user to shift their gaze to an area with high recognition stability.

[0160] For example, as shown in FIG. 16, when a UI image Pui1 is displayed near a slightly dark area Zd in an image Pi, and a user's hand H1 is detected in the vicinity thereof, the user's hand H1 to be recognized will be detected in an area with low recognition stability.

[0161] However, as shown in the left part of FIG. 16, in both the image Pi and the recognition stability map M, it can be recognized that the left side indicated by the dotted frame has a relatively high recognition stability.

[0162] Therefore, if the user's hand H1 to be recognized is detected in an area with low recognition stability, a wipe display Pi' of the image Pi and a wipe display M' of the recognition stability map M may be displayed, as shown in the right part of Figure 16, to allow the user to recognize an area with high recognition stability and encourage them to shift their gaze to an area with high recognition stability.

[0163] In the right part of Figure 16, when the user's viewpoint is shifted to the left in the image Pi shown in the dotted frame, the area Zd disappears from the image Pr captured by the camera 51, and the image changes overall to a brighter image with high recognition stability.

[0164] This allows the user's hand H1, which is the recognition target, to be detected in an area with high recognition stability, and as a result, the recognition processing set can be made to correspond to normal brightness.

[0165] <Hand Posture Recognition Processing in Second Modification> Next, the hand posture recognition processing in the second modification will be described with reference to the flowchart in FIG.

[0166] In the flowchart of FIG. 17, the processes of steps S111 to S114 and S117 to S123 are the same as the processes of steps S71 to S81 in the flowchart of FIG. 12, and therefore a description thereof will be omitted.

[0167] That is, the hand posture recognition process is started by the processes of steps S111 to S114, the UI image is displayed, and once the position of the hand to be recognized is detected, the process proceeds to step S115.

[0168] In step S115, the recognition process set selection unit 136 determines whether the recognition stability of the hand position to be recognized is lower than a predetermined value.

[0169] If it is determined in step S115 that the recognition stability of the hand position to be recognized is lower than the predetermined value, the process proceeds to step S116.

[0170] In step S116, the image captured by the camera 51 and the recognition stability map are wipe-displayed at a predetermined position on the display 52, and the process returns to step S112, where the subsequent processes are repeated.

[0171] More specifically, the recognition processing set selection unit 136 notifies the recognition processing unit 137 that the recognition stability of the hand position to be recognized is lower than a predetermined value. Based on this, the recognition processing unit 137 controls the content display control unit 133 to display a wipe image Pi' of the image Pi captured by the camera 51 on the display 52.

[0172] Furthermore, the recognition stability map display processing unit 138 reads out the recognition stability map 161 and displays a wipe image M′ of the recognition stability map M at a predetermined position on the display 52 .

[0173] By the above processing, the image Pi' and the recognition stability map M' are displayed in a wipe manner as shown in the right part of Figure 16, making it possible for the user to recognize areas with high recognition stability and to be prompted to shift their viewpoint to areas with high recognition stability.

[0174] This allows the user's hand H1, which is the recognition target, to be detected in an area with high recognition stability, and as a result, the recognition processing set can be made to correspond to normal brightness.

[0175] <<5. Third Variant>> In the above, we have described an example in which, when the hand to be recognized is in an area with low recognition stability, the image captured by camera 51 and the recognition stability map are wiped to prompt the user to shift their gaze to an area with high recognition stability.

[0176] However, when the area where the UI image is displayed, where the hand to be recognized is likely to be detected, is in an area with low recognition stability, the recognition stability itself may be increased by operating lighting or external devices to brighten the area.

[0177] For example, as shown in FIG. 18, when a UI image Pui1 is displayed near a slightly dark area Zd in an image Pi, and a user's hand H1 is detected in the vicinity thereof, the user's hand H1 to be recognized will be detected in an area with low recognition stability.

[0178] Therefore, in an area where the recognition stability of the display position of the UI image is low, the environment itself may be brightened by illuminating an external device with a lighting function, as shown in the right part of Fig. 18. That is, in the upper left part of Fig. 18, a display that can also function as a light and is present in image Pi is in an off state (Loff), but in the right part of Fig. 18, as shown in image Pr4, the display is turned on (Lon) and emits light, brightening the entire environment, and the dark area Zd in the upper left part of Fig. 18 changes to a bright area Zd' in the right part of Fig. 18.

[0179] This places the display position of the UI image in an area with high recognition stability, and the user's hand H1 to be recognized can be detected in a position with high recognition stability, making it possible to perform recognition processing using a recognition processing set that can be used in normal brightness.

[0180] <Configuration example of HMD in third modified example> Next, a configuration example of an HMD in the third modified example will be described with reference to Fig. 19. Note that in the configuration of the HMD 31' in Fig. 19, components having the same functions as those in the configuration of the HMD 31 in Fig. 5 are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0181] 19 differs from the HMD 31 in that a recognition processing set selection unit 136′ and a recognition processing unit 137′ are provided instead of the recognition processing set selection unit 136 and the recognition processing unit 137, and an external device control unit 201 is also provided. Furthermore, in the third modified example, an external device 211 that functions as lighting is essential in addition to the HMD 31′.

[0182] In other words, the recognition processing set selection unit 136' has the same basic functions as the recognition processing set selection unit 136, but in addition, if the recognition stability of the display position of the UI image from the recognition stability map 161 is lower than a predetermined value, it notifies the recognition processing unit 137' of this fact.

[0183] The basic functions of the recognition processing unit 137' are the same as those of the recognition processing unit 137, but in addition, when the recognition processing set selection unit 136' notifies the recognition processing unit 137' that the recognition stability of the display position of the UI image is lower than a predetermined value, the recognition processing unit 137' controls the external device control unit 201 to turn on the lighting of the external device 211 that has a lighting function.

[0184] In addition, when it is not notified that the recognition stability of the detected hand position is lower than a predetermined value, the recognition processing unit 137' controls the external device control unit 201 to turn off the lighting of the external device 211 that has a lighting function.

[0185] With this configuration, when the recognition stability of the display position of the UI image is low, it is possible to change to an area with high recognition stability that is brighter due to the illumination of the external device 211. This allows the detection position of the user's hand H1, which is the recognition target, to be in a position with high recognition stability, and as a result, it becomes possible to perform recognition processing using a recognition processing set that can be used in normal brightness.

[0186] <Hand Posture Recognition Processing in Third Modification> Next, the hand posture recognition processing in the third modification will be described with reference to the flowchart in FIG.

[0187] In the flowchart of FIG. 20, the processes of steps S131, S132 and S136 to S144 are the same as the processes of steps S71 to S81 in the flowchart of FIG. 12, and therefore a description thereof will be omitted.

[0188] That is, the hand posture recognition process is started by the processes of steps S131 and S132, and once the UI image is displayed, the process proceeds to step S133.

[0189] In step S133, the recognition processing set selection unit 136' determines whether the recognition stability of the display position of the UI image is lower than a predetermined value.

[0190] If it is determined in step S133 that the recognition stability of the display position of the UI image is lower than the predetermined value, the process proceeds to step S134.

[0191] In step S134, the recognition processing set selection unit 136′ notifies the recognition processing unit 137′ that the recognition stability of the display position of the UI image is lower than a predetermined value. In response to this, the recognition processing unit 137′ controls the external device control unit 201 to turn on the lighting of the external device 211 that has a lighting function.

[0192] On the other hand, if it is determined in step S133 that the recognition stability of the display position of the UI image is not lower than the predetermined value, the process proceeds to step S135.

[0193] In step S135, the recognition processing set selection unit 136′ notifies the recognition processing unit 137′ that the recognition stability of the display position of the UI image is not lower than a predetermined value. In response to this, the recognition processing unit 137′ controls the external device control unit 201 to turn off the lighting of the external device 211 that has a lighting function.

[0194] In the process of step S134, if the illumination of the external device 211 has already been turned on in the immediately preceding process, it is sufficient to maintain that state. Also, in the process of step S135, if the illumination has already been turned off in the immediately preceding process, it is sufficient to maintain that state.

[0195] By the above processing, when the recognition stability of the display position of the UI image is low, it is possible to change it to a brighter area with high recognition stability by the illumination of the external device 211, so that the detection position of the user's hand H1 to be recognized can be made to a position with high recognition stability.

[0196] As a result, it is possible to perform recognition processing using a recognition processing set that can be used under normal brightness.

[0197] <<6. Example of Execution by Software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes is executed by software, the program constituting the software is installed from a recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0198] 21 shows an example of the configuration of a general-purpose computer. This computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0199] The input / output interface 1005 is connected to an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected is a drive 1010 that reads and writes data from / to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0200] The CPU 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0201] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0202] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0203] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable storage medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0204] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0205] 40. The CPU 1001 in FIG. 40 realizes the functions of the control unit 101 in FIG.

[0206] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0207] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0208] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0209] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0210] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0211] The present disclosure may also be configured as follows: <1> An information processing device comprising: <1> a recognition processing unit that recognizes a recognition object using a recognition processing set that combines a plurality of recognition processing elements; and a recognition processing set selection unit that selects a combination of the plurality of recognition processing elements based on recognition stability for a predetermined recognition process specified at the spatial coordinate position of the recognition object. <2> The recognition processing elements include pre-processing, a recognition model, and a post-processing, and the recognition processing set is a combination of the pre-processing, the recognition model, and the post-processing that are the recognition processing elements, and the recognition processing set selection unit selects the pre-processing from a pre-processing group consisting of a plurality of types of the pre-processing, selects the recognition model from a recognition model group consisting of a plurality of types of the recognition model, and selects the post-processing from a post-processing group consisting of a plurality of types of the post-processing, according to the recognition stability, and selects the combination of the selected pre-processing, the recognition model, and the post-processing as the recognition processing set, and the recognition processing unit recognizes the recognition object using the recognition processing set selected by the recognition processing set selection unit. <3> The information processing device according to <2>, further comprising: a recognition stability detection unit that detects the recognition stability for the predetermined recognition process for each of the spatial coordinate positions; and a recognition stability map generation unit that generates a recognition stability map by mapping the recognition stability for the predetermined recognition process for each of the spatial coordinate positions detected by the recognition stability detection unit, wherein the recognition processing set selection unit selects the recognition processing set according to the recognition stability for the predetermined recognition process specified at the spatial coordinate position of the recognition target based on the recognition stability map. <4> The information processing device according to <3>, wherein the recognition processing set selection unit is formed by learning based on teacher data that pairs the recognition stability for the predetermined recognition process for each of the spatial coordinate positions with the recognition processing set consisting of the combination of the pre-processing, the recognition model, and the post-processing that maximizes the recognition stability. <5> The information processing device according to <4>, wherein the recognition stability for the predetermined recognition process includes an output probability value for the recognition result of the predetermined recognition process.<6> The information processing device according to <5>, wherein the recognition stability for the predetermined recognition process includes, in addition to an output probability value of the recognition result, a feature quantity formed from a sensing result of a predetermined sensor detected when the predetermined recognition process is performed on the recognition target. <7> The information processing device according to <6>, wherein the feature quantity formed from the sensing result of the predetermined sensor includes brightness, contrast ratio, and acceleration. <8> The information processing device according to <6>, further including an importance calculation unit that calculates an importance for each of the feature quantities in a process in which the recognition process set selection unit is formed by learning. <9> The information processing device according to <8>, further including a recognition stability map display control unit that controls display of a spatial distribution of feature quantities of a predetermined order of importance among the recognition stability for each spatial coordinate position in the recognition stability map. <10> The information processing device according to <9>, wherein the recognition stability map display control unit controls display of a spatial distribution of the feature quantity of the highest order of importance among the recognition stability for each spatial coordinate position in the recognition stability map. <11> The information processing device according to <10>, wherein the recognition processing is hand posture recognition processing that recognizes the posture of a user's hand relative to a UI (User Interface) image based on an image captured by a camera, and the recognition stability map display control unit controls display of a wipe image of the image and a wipe image of the recognition stability map when the image includes a spatial coordinate position where the recognition stability is smaller than a predetermined threshold. <12> The information processing device according to <8>, wherein the recognition processing set selection unit is formed by decision tree algorithm learning based on training data that pairs, for each spatial coordinate position, the recognition stability for the predetermined recognition processing with a recognition processing set consisting of a combination of the pre-processing, the recognition model, and the post-processing that maximizes the recognition stability. <13> The information processing device according to <12>, wherein the importance calculation unit acquires the importance of each of a plurality of feature amounts that constitute the recognition stability, which is generated in a process in which the recognition processing set selection unit is formed by decision tree algorithm learning.<14> The information processing device according to <8>, wherein the recognition processing set selection unit is a DNN (Deep Neural Network) formed by learning based on training data that pairs, for each of the spatial coordinate positions, a recognition stability for the predetermined recognition processing with a recognition processing set consisting of a combination of the pre-processing, the recognition model, and the post-processing that maximizes the recognition stability. <15> The information processing device according to <14>, wherein the importance calculation unit calculates the importance by a calculation (Permutation Importance) that uses an error when learning by completely randomizing a predetermined feature amount. <16> The information processing device according to <1>, wherein the recognition processing is a hand posture recognition process that recognizes the posture of a user's hand with respect to a UI image based on an image captured by a camera, and further includes a display control unit that displays the UI image in an area of ​​the spatial coordinate positions where the recognition stability is higher than a predetermined threshold. <17> The information processing device according to <1>, wherein the recognition processing is hand posture recognition processing that recognizes the posture of a user's hand relative to a UI image based on an image captured by a camera, and further includes an external device control unit that controls external devices in a range near where the image is captured to emit light when the image includes an area consisting of a group of spatial coordinate positions with the recognition stability smaller than a predetermined threshold. <18> The information processing device according to <17>, further including a see-through display that displays the UI image in AR (Augmented Reality). <19> An information processing device including: recognition processing that recognizes a recognition target using a recognition processing set that combines a plurality of recognition processing elements; and recognition processing set selection processing that selects a combination of the plurality of recognition processing elements based on the recognition stability for a predetermined recognition processing specified at the spatial coordinate position of the recognition target. <20> A program that causes a computer to function as: a recognition processing unit that recognizes a recognition target using a recognition processing set that combines a plurality of recognition processing elements; and a recognition processing set selection unit that selects the combination of the plurality of recognition processing elements based on the recognition stability for a predetermined recognition processing specified at the spatial coordinate position of the recognition target.

[0212] 31, 31' HMD, 51 camera, 52 display, 111 sensor, 131 recognition stability detection unit, 132 recognition stability map generation unit, 133 content display control unit, 134 recognition processing set selection unit learning unit, 135 importance calculation unit, 136 recognition processing set selection unit, 137 recognition processing unit, 138 recognition stability map display processing unit, 161 recognition stability map, 162 recognition processing set group

Claims

1. An information processing device comprising: a recognition processing unit that recognizes a recognition target using a recognition processing set that combines multiple recognition processing elements; and a recognition processing set selection unit that selects a combination of the multiple recognition processing elements based on the recognition stability for a specified recognition process that is identified at the spatial coordinate position of the recognition target.

2. The information processing device of claim 1, wherein the recognition processing elements include pre-processing, a recognition model, and post-processing; the recognition processing set is a combination of the pre-processing, the recognition model, and the post-processing, which are the recognition processing elements; the recognition processing set selection unit selects the pre-processing from a pre-processing group consisting of multiple types of pre-processing, selects the recognition model from a recognition model group consisting of multiple types of recognition models, and selects the post-processing from a post-processing group consisting of multiple types of post-processing, according to the recognition stability, and selects the selected combination of the pre-processing, the recognition model, and the post-processing as the recognition processing set; and the recognition processing unit recognizes the recognition object using the recognition processing set selected by the recognition processing set selection unit.

3. The information processing device of claim 2, further comprising: a recognition stability detection unit that detects the recognition stability for the specified recognition process for each spatial coordinate position; and a recognition stability map generation unit that generates a recognition stability map by mapping the recognition stability for the specified recognition process for each spatial coordinate position detected by the recognition stability detection unit, wherein the recognition processing set selection unit selects the recognition processing set based on the recognition stability map in accordance with the recognition stability for the specified recognition process identified at the spatial coordinate position of the recognition target.

4. The information processing device described in claim 3, wherein the recognition processing set selection unit is formed by learning based on training data that pairs the recognition stability for the specified recognition processing for each spatial coordinate position with the recognition processing set consisting of the combination of the pre-processing, the recognition model, and the post-processing that maximizes the recognition stability.

5. The information processing device according to claim 4, wherein the recognition stability for the predetermined recognition process includes an output probability value for the recognition result of the predetermined recognition process.

6. An information processing device as described in claim 5, wherein the recognition stability for the specified recognition process includes, in addition to the output probability value of the recognition result, a feature consisting of the sensing result of a specified sensor detected when the specified recognition process is performed on the recognition target.

7. The information processing device according to claim 6, wherein the feature values ​​obtained as a result of sensing by the predetermined sensor include brightness, contrast ratio, and acceleration.

8. The information processing device according to claim 6, further comprising an importance calculation unit that calculates the importance of each of the feature amounts in the process in which the recognition processing set selection unit is formed by learning.

9. The information processing device according to claim 8, further comprising a recognition stability map display control unit that controls the display of a spatial distribution of feature quantities of a predetermined order of importance among the recognition stabilities for each spatial coordinate position on the recognition stability map.

10. The information processing device according to claim 9, wherein the recognition stability map display control unit controls the display of the spatial distribution of the feature amount of the highest importance among the recognition stabilities for each spatial coordinate position on the recognition stability map.

11. The information processing device described in claim 10, wherein the recognition process is a hand posture recognition process that recognizes the posture of a user's hand relative to a UI (User Interface) image based on an image captured by a camera, and the recognition stability map display control unit controls the display of a wipe image of the image and a wipe image of the recognition stability map when the image contains a spatial coordinate position of the recognition stability that is smaller than a predetermined threshold.

12. The information processing device described in claim 8, wherein the recognition processing set selection unit is formed by decision tree algorithm learning based on training data that pairs the recognition stability for the specified recognition processing for each spatial coordinate position with a recognition processing set consisting of the combination of the pre-processing, the recognition model, and the post-processing that maximizes the recognition stability.

13. The information processing device according to claim 12, wherein the importance calculation unit acquires the importance of each of a plurality of feature quantities constituting the recognition stability, which are generated in the process in which the recognition processing set selection unit is formed by decision tree algorithm learning.

14. The information processing device described in claim 8, wherein the recognition processing set selection unit is a DNN (Deep Neural Network) formed by learning based on training data that pairs the recognition stability for the specified recognition processing for each spatial coordinate position with a recognition processing set consisting of a combination of the pre-processing, the recognition model, and the post-processing that maximizes the recognition stability.

15. The information processing device according to claim 14, wherein the importance calculation unit calculates the importance by a calculation (permutation importance) using an error when learning a predetermined feature amount completely randomly.

16. The information processing device according to claim 1, wherein the recognition process is a hand posture recognition process that recognizes the posture of the user's hand relative to a UI image based on an image captured by a camera, and further comprises a display control unit that displays the UI image in an area of ​​the spatial coordinate position where the recognition stability is higher than a predetermined threshold.

17. The information processing device according to claim 1, wherein the recognition process is a hand posture recognition process that recognizes the posture of the user's hand relative to a UI image based on an image captured by a camera, and further comprises an external device control unit that controls external devices in a range near where the image is captured to emit light when the image contains an area consisting of a group of spatial coordinate positions where the recognition stability is smaller than a predetermined threshold.

18. The information processing device according to claim 17, further comprising a see-through display that displays the UI image in AR (Augmented Reality).

19. An information processing method including: a recognition process for recognizing a recognition target using a recognition process set that combines a plurality of recognition process elements; and a recognition process set selection process for selecting a combination of the plurality of recognition process elements based on the recognition stability for a predetermined recognition process that is specified at the spatial coordinate position of the recognition target.

20. A program that causes a computer to function as a recognition processing unit that recognizes a recognition target using a recognition processing set that combines multiple recognition processing elements, and a recognition processing set selection unit that selects a combination of the multiple recognition processing elements based on the recognition stability for a specified recognition process, which is specified in the spatial coordinate position of the recognition target.

Images