Object tracking device, content processing system, and object tracking method

Abstract

A content processing device of the present invention acquires frame data of a captured image from a head-mounted display (S20), and cuts out the region of a hand which is an object (S22). A feature point of the hand is extracted from the cut-out image, and state information of the feature point is acquired and reflected on a display image. Meanwhile, an evaluation region of an image of the hand is set on the basis of the feature point (S26), and if the luminance is not within a proper range, the head-mounted display is requested to adjust a target luminance of AEAGC (Y in S28, S30). If it is not necessary to end content processing, the same processing is repeated for a subsequent frame (N in S32).

Description

Object tracking device, content processing system, and object tracking method

[0001] The present invention relates to an object tracking device, a content processing system, and an object tracking method for tracking an object based on a captured image.

[0002] Technologies that use head-mounted displays and other devices to create a sense of immersion in virtual spaces are becoming commonplace across a wide range of fields. For example, the sense of presence in virtual spaces can be enhanced by moving virtual objects on the screen in response to the user's movements or by providing tactile feedback. In content such as electronic games, using the user's movements as a control means allows for more intuitive operation than using input devices such as controllers.

[0003] In the above technology, the accuracy of tracking the movement of an object, such as a user, has a significant impact on the sense of realism in the virtual space and the operability of the content. On the other hand, when tracking movement by analyzing images of the object, the image characteristics can easily change depending on the relative positions and angles of the object's surface, camera, and light source, as well as the ambient illuminance, which can make the analysis results unstable. This problem is particularly evident when tracking parts of the body, such as a user's hand, whose position, posture, and shape can change arbitrarily and which move delicately.

[0004] The present invention has been made in view of the above-mentioned problems, and its purpose is to provide a technology that can stably track an object having any movement by photographing it.

[0005] One aspect of the present invention relates to an object tracking device, which includes a status information acquisition unit that analyzes an image of an object in an image captured by an imaging device and acquires status information of the object, and a target brightness control unit that determines whether the brightness of an evaluation area set according to a predetermined rule for the image of the object is within a predetermined appropriate range, and if not, causes the imaging device to adjust a target brightness in AEAGC (Auto Exposure Auto Gain Control) processing.

[0006] Another aspect of the present invention relates to a content processing system including a head-mounted display with an imaging device, and a content processing device that performs information processing based on status information of an object shown in an image captured by the imaging device, generates a display image, and displays the generated image on the head-mounted display, wherein the content processing device includes a status information acquisition unit that analyzes an image of the object in the image and acquires the status information, and a target brightness control unit that determines whether the brightness of an evaluation area set according to a predetermined rule for the image of the object is within a predetermined appropriate range, and causes the imaging device to adjust a target brightness in AEAGC (Auto Exposure Auto Gain Control) processing if the brightness is not within the appropriate range.

[0007] Yet another aspect of the present invention relates to an object tracking method, which includes the steps of: analyzing an image of an object in an image captured by an imaging device as a moving image, and acquiring status information of the object; and determining whether or not the luminance of an evaluation area set according to a predetermined rule for the image of the object is within a predetermined appropriate range, and, if not within the appropriate range, causing the imaging device to adjust a target luminance in AEAGC (Auto Exposure Auto Gain Control) processing.

[0008] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, or recording medium on which a computer program is recorded, are also valid aspects of the present invention.

[0009] According to the present invention, an object having any movement can be stably tracked by capturing images.

[0010] 1 is a diagram showing an example of the appearance of a head-mounted display to which the present embodiment can be applied; FIG. 2 is a diagram showing an example configuration of a content processing system to which the present embodiment can be applied; FIG. 3 is a diagram showing a schematic diagram of a basic processing procedure for acquiring state information of a user's hand in the present embodiment; FIG. 4 is a diagram showing an example of a change in a captured image by adjusting the target luminance in the present embodiment; FIG. 5 is a diagram showing another example of a change in a captured image by adjusting the target luminance in the present embodiment; FIG. 6 is a diagram showing an internal circuit configuration of a head-mounted display of the present embodiment; FIG. 7 is a diagram showing an internal circuit configuration of a content processing device of the present embodiment; FIG. 8 is a block diagram showing functional blocks of a head-mounted display and a content processing device of the present embodiment; FIG. 9 is a diagram showing an example of a relationship between feature points of a hand and an evaluation area in the present embodiment; FIG. 10 is a diagram showing an example of a luminance histogram evaluated by a target luminance control unit in the present embodiment; FIG. 11 is a diagram showing an example of a change over time in luminance of an evaluation area and target luminance in the present embodiment; FIG. 12 is a diagram showing a simplified example of a change in exposure time and a gain value by adjusting the target luminance in the present embodiment;

[0011] This embodiment relates to a technology for tracking the movement of an object by capturing an image. While the implementation of the imaging device and the type of object are not particularly limited, the more arbitrary movement is allowed for the object and imaging device, the more likely the image is to change in relation to the light source, making tracking accuracy more unstable. Therefore, as a more effective example, the following description will focus on tracking the movement of a user's hand based on an image captured by a camera mounted on a head-mounted display.

[0012] 1 shows an example of the appearance of a head-mounted display 100 to which this embodiment can be applied. In this example, the head-mounted display 100 is made up of an output mechanism unit 102 and a wearing mechanism unit 104. The wearing mechanism unit 104 includes a wearing band 106 that, when worn by a user, wraps around the head and secures the device in place. The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user is wearing the head-mounted display 100, and is equipped with a display panel inside so that it faces the eyes when worn.

[0013] The housing 108 also includes an eyepiece lens that is positioned between the display panel and the user's eyes when the head mounted display 100 is worn, and that magnifies the image. The head mounted display 100 may also include speakers or earphones at positions that correspond to the user's ears when worn. The head mounted display 100 may also include a built-in motion sensor such as an acceleration sensor, a gyro sensor, or a geomagnetic sensor, and may detect the translational and rotational movement of the head of the user wearing the head mounted display 100, as well as the position and posture at each time.

[0014] The head-mounted display 100 further includes cameras 110a, 110b, 110c, and 110d on the front surface of the housing 108, which capture video of the real space around the user. The number and arrangement of the cameras 110a, 110b, 110c, and 110d are not particularly limited, but in the example shown, the cameras 110a, 110b, 110c, and 110d are provided at the four corners of the front surface of the housing 108. Hereinafter, the cameras 110a, 110b, 110c, and 110d may be collectively referred to as cameras 110. By sequentially analyzing each frame of the video captured by the camera 110, the hand movements of the user within the field of view of the camera 110 can be tracked in three-dimensional space.

[0015] By representing a virtual three-dimensional object that interacts with the hand on an image according to the tracking results, it is possible to realize virtual reality or augmented reality in which the user can pick up and move the virtual object. In this embodiment, the image of the hand in the captured image is evaluated in real time, and the capturing conditions of the camera 110 are feedback-controlled, thereby improving the robustness of tracking accuracy against hand movement, movement of the head-mounted display 100, changes in the position and illuminance of the light source, and the like. Note that the tracking target may be not only the user's hand, but also another part of the user's body, a real object that the user holds or wears, and the like. Hereinafter, information in three-dimensional space, such as the position, posture, and shape of the target object, obtained from each frame of the captured image, will be collectively referred to as "state information."

[0016] The images captured by the camera 110 can also be used to acquire the position and orientation of the head mounted display 100, and ultimately the position and orientation of the user's head, by V-SLAM (Visual Simultaneous Localization and Mapping). V-SLAM is a technology that acquires the position and orientation of the camera while creating an environmental map by repeating a process of estimating the three-dimensional position of a real object from the positional relationship of images of the same real object shown in images captured from multiple viewpoints, and a process of estimating the position and orientation of the camera based on the position on the captured images of the images of the real object whose position has been estimated.

[0017] If the field of view of the image displayed on the head-mounted display 100 can be changed to correspond to the position and posture of the user's head acquired by V-SLAM, the user can feel immersed in the displayed world. Also, by instantly displaying an image captured by part of the camera 110 on the head-mounted display 100, it is possible to provide a see-through mode that shows the real world in the direction the user is facing.

[0018] 2 shows an example of the configuration of a content processing system to which this embodiment can be applied. The head-mounted display 100 is connected to the content processing device 200 via an interface for connecting peripheral devices, such as wireless communication or USB Type-C. The content processing device 200 may further be connected to a server via a network. In this case, the server may provide the content processing device 200 with an online application, such as a game in which multiple users can participate via the network.

[0019] The content processing device 200 basically processes the content program, generates display image and audio data, and transmits it to the head mounted display 100. The head mounted display 100 receives the display image and audio data and outputs them as content images and audio. Here, the content processing device 200 sequentially acquires frame data of moving images captured by the camera 110 of the head mounted display 100, and immediately acquires state information of the user's hands based on the frame data.

[0020] The processing that the content processing device 200 should perform based on the acquired state information is not particularly limited. For example, as described above, the content processing device 200 may generate a display image in which a virtual object is lifted or moved in accordance with a change in the state information of the hand, i.e., the movement of the hand. Alternatively, the content processing device 200 may recognize a gesture made by the user's hand as a command input and perform corresponding information processing.

[0021] The content processing device 200 may also sequentially acquire information on the position and orientation of the user's head using technology such as the above-mentioned V-SLAM, and generate a display image in the corresponding field of view. At this time, the content processing device 200 may acquire measurements from a motion sensor built into the head-mounted display 100 to acquire the position and orientation of the user's head with greater accuracy. It will be understood by those skilled in the art that various processes and display images can be conceived that are performed by the content processing device 200 and that are generated using state information on objects such as hands.

[0022] 3 is a diagram showing a basic processing procedure for acquiring state information of a user's hand in this embodiment. First, the four cameras 110 provided in the head-mounted display 100 capture video of the real space, thereby obtaining four images 50a, 50b, 50c, and 50d at a certain time (S10). While the figure shows equidistant projection images captured using a fisheye lens as an example, the type of lens is not intended to be limited.

[0023] The images 50a, 50b, 50c, and 50d show the user's hands in positions that depend on the respective fields of view. However, in this embodiment, it is not necessary for the hands to be shown in all images. Both hands may also be shown. The head-mounted display 100 performs appropriate signal processing on the data of the images 50a, 50b, 50c, and 50d and transmits them to the content processing device 200. The content processing device 200 uses well-known techniques such as pattern matching to extract the region of the hand, which is the target object, from the acquired images (S12). The figure shows, as an example, the extraction of the hand region 52 from the image 50a.

[0024] The content processing device 200 acquires hand state information in three-dimensional space at that time based on the image of the extracted hand region 52, and in this embodiment, a DNN (Deep Neural Network) is used as an example of the means for doing so (S14). That is, by performing deep learning in advance using a large number of hand images as training data, DNN data is prepared that inputs hand images and outputs state information. In this embodiment, a DNN is assumed that extracts feature points from hand images and derives state information in three-dimensional space from their positional relationships, etc.

[0025] Those skilled in the art will understand that there are various types of neural networks and learning algorithms that can be constructed using deep learning. However, the means by which the content processing device 200 acquires state information is not limited to deep learning. State information may also be acquired by fitting the positional relationship of feature points of the hand on the image to a three-dimensional model of the hand. In any case, the content processing device 200 acquires state information of the hand using the image of the hand region 52 extracted in S12 (S16). The content processing device 200 repeats the illustrated process at a predetermined rate to track changes in the state information as movement and reflects the changes in the display image as appropriate.

[0026] On the other hand, in the camera 110 of the head-mounted display 100, a target brightness is basically set so as to maintain an appropriate brightness for the entire image. The brightness of the entire image is expressed, for example, as the average value of the brightness of all pixels. The camera 110 adjusts at least one of the exposure time and the gain value so that the brightness of the entire captured image approaches the target brightness. For example, if the captured image does not have sufficient brightness due to a dark environment, the exposure time can be lengthened (the shutter speed can be slowed down) to increase the amount of accumulated charge after photoelectric conversion and thereby increase the brightness.

[0027] Increasing the exposure time increases the detection value of the original light, allowing for a relative reduction in noise, but at the expense of a change in the frame rate. Increasing the gain value also amplifies noise, but by appropriately setting filter parameters, such as a smoothing filter, that take the gain value into account, it is possible to suppress noise generation to some extent. Thus, the function of adjusting the exposure time to obtain a photographed image with optimal brightness regardless of the ambient brightness is known as AE (Auto Exposure), and the function of adjusting the gain value is known as AGC (Auto Gain Control). Hereinafter, these functions will be collectively referred to as "AEAGC."

[0028] However, with typical AEAGC processing, adjustments are made by focusing only on the average brightness of the entire images 50a, 50b, 50c, and 50d. Therefore, when focusing on the hand image, which requires high detail as in this embodiment, not only is there no improvement, but it may even result in an inappropriate state. In this embodiment, while utilizing the functions of a conventional AEAGC, control is performed so that hand images are stably acquired. Specifically, by adjusting the target brightness based on the brightness of the hand image and then operating the AEAGC, a captured image that is advantageous for acquiring status information is obtained.

[0029] 4 shows an example of how a captured image changes when the target brightness is adjusted. First, image 60a is the captured image before adjustment, and image 60b is the image from which the hand region has been extracted. The lower part shows a histogram of the brightness (pixel values) of image 60a, along with target brightness 62. In a typical AEAGC, a brightness histogram of the entire image 60a is obtained, and the exposure time and gain value are determined so that the average brightness thereof matches the target brightness 62. In other words, when viewed as a whole, image 60a has an average brightness that matches the target brightness 62, making it an image that is ideal for viewing.

[0030] On the other hand, in this example, as shown in image 60b, the entire hand image is concentrated near the maximum brightness, resulting in a so-called "blown-out" state. The brightness of the hand image at this time corresponds to region 64 in the brightness histogram. When the hand image is blown-out, the level of detail in the hand structure is lost, reducing the accuracy of acquiring feature points such as joints, and ultimately the accuracy of hand state information. In this embodiment, attention is focused on the brightness of the image of the hand, which is the target object, and the target brightness is adjusted so that the level of detail in the hand structure is not lost in the image.

[0031] As shown in the histogram at the bottom right of the figure, the target brightness 68 is lowered from the original target brightness 62. Operating the AEAGC with the target brightness lowered in this way reduces at least one of the exposure time and the gain value, lowering the average brightness of the entire image 66a. This makes it possible to capture a detailed image of the hand with a certain range of brightness and shading, as shown in image 66b of the hand region and the corresponding region 70 on the histogram. As a result, hand condition information can be captured with high accuracy regardless of the shooting environment.

[0032] 5 shows another example of how a captured image changes when the target brightness is adjusted. Image 72a is the captured image before adjustment, and image 72b is the image with the hand region extracted. As shown in the brightness histogram in the lower part, when the average brightness of the entire image 72a is adjusted to the target brightness 74 set by a typical AEAGC, the entire image of the hand is concentrated near the lowest brightness, resulting in a so-called "blackout" state, as shown in image 72b. The brightness of the hand image in this case corresponds to region 76 in the brightness histogram.

[0033] Such blackouts in the image can also result in a loss of detail in the hand structure and make it difficult to distinguish the outline from the background, ultimately reducing the accuracy of the status information. Therefore, as shown in the histogram at the bottom right, the target brightness 80 is adjusted to be higher than the original target brightness 74. Operating the AEAGC with the target brightness increased increases at least one of the exposure time and gain value, thereby increasing the average brightness of the entire image 78a. This allows for the acquisition of a detailed image of the hand with a certain range of brightness and shading, as shown in image 78b of the hand region and the corresponding region 82 on the histogram. As a result, hand status information can be acquired with high accuracy regardless of the shooting environment.

[0034] In this embodiment, the content processing device 200 determines the positions of feature points of the hand on the image with high accuracy in the process of acquiring state information of the hand in three-dimensional space. Therefore, even for an object with a complex shape such as a hand, the content processing device 200 can determine the occurrence of blown-out highlights or crushed shadows by evaluating a precise image region that excludes as much of the background and other objects as possible. When the content processing device 200 determines that the image of the hand is in a state that can be considered to be blown-out highlights or crushed shadows, it adjusts the target luminance and indirectly controls the exposure time and gain value to improve the state of the image.

[0035] 6 shows the internal circuit configuration of the head mounted display 100. The head mounted display 100 includes a CPU 120, a main memory 122, a display unit 124, and an audio output unit 126. These units are connected to one another via a bus 128. An input / output interface 130 is further connected to the bus 128. A communication unit 132 including a communication interface, a motion sensor 134, and the camera 110 are connected to the input / output interface 130.

[0036] The CPU 120 processes information acquired from each unit of the head mounted display 100 via the bus 128, and supplies display image and audio data acquired from the content processing device 200 to the display unit 124 and audio output unit 126. The main memory 122 stores programs and data necessary for processing by the CPU 120.

[0037] The display unit 124 includes a display panel such as a liquid crystal panel or an organic EL panel, and displays images in front of the eyes of the user wearing the head-mounted display 100. The display unit 124 may achieve stereoscopic vision by displaying a pair of stereo images in areas corresponding to the left and right eyes. The display unit 124 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the head-mounted display 100 is worn, and that expand the user's field of view. The audio output unit 126 is composed of speakers or earphones that are provided at positions corresponding to the user's ears when the head-mounted display 100 is worn, and allows the user to hear audio.

[0038] The communication unit 132 is an interface for sending and receiving data to and from the content processing device 200, and realizes communication using known communication technologies such as Bluetooth (registered trademark) and USB Type-C. The motion sensor 134 includes at least one of an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc., and acquires the angular velocity and acceleration of the head mounted display 100. As shown in FIG. 1 , the camera 110 is one or more video cameras that capture images of the real space, such as in front of the user. Measurement values ​​by the motion sensor 134 and data of images captured by the camera 110 are transmitted to the content processing device 200 via the communication unit 132.

[0039] 7 shows the internal circuit configuration of the content processing device 200. The content processing device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and a main memory 226. These components are connected to one another via a bus 230. An input / output interface 228 is further connected to the bus 230. A communication unit 232, a storage unit 234, an output unit 236, an input unit 238, and a recording medium drive unit 240 are connected to the input / output interface 228.

[0040] The communication unit 232 includes a peripheral device interface such as USB or IEEE 1394, and a network interface such as a wired LAN or wireless LAN. The storage unit 234 includes a hard disk drive, a nonvolatile memory, etc. The output unit 236 outputs data to the head mounted display 100. The input unit 238 accepts data input from the head mounted display 100. The recording medium drive unit 240 drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory.

[0041] The CPU 222 executes an operating system stored in the storage unit 234 to provide overall control over the content processing device 200. The CPU 222 also executes various programs that are read from the storage unit 234 or a removable recording medium and loaded into the main memory 226, or that are downloaded via the communication unit 232. The GPU 224 has the functions of a geometry engine and a rendering processor, performs drawing processing in accordance with drawing commands from the CPU 222, and outputs the drawing results to the output unit 236. The main memory 226 is composed of RAM (Random Access Memory), and stores programs and data required for processing.

[0042] Fig. 8 is a block diagram showing functional blocks of the head-mounted display 100 and the content processing device 200. Each device may perform general information processing such as running an application or communicating with a server, but Fig. 8 particularly shows functional blocks related to obtaining state information of an object using a captured image. Note that at least some of the functions of the content processing device 200 shown in Fig. 8 may be implemented in a server connected to the content processing device 200 via a network, or may be implemented in the head-mounted display 100.

[0043] 8 can be realized in hardware by the various circuits shown in Figures 6 and 7, and in software by a computer program that implements the functions of the multiple functional blocks. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof, and are not limited to any one of them.

[0044] The head mounted display 100 includes an imaging unit 140 that captures an image of real space, an image signal processing unit 142 that performs necessary processing on the captured image signal, an image data transmission unit 144 that transmits data of the image, an AEAGC processing unit 146 that performs AEAGC, and a display unit 148 that displays the image. The imaging unit 140 corresponds to the camera 110 in Fig. 6 and outputs, at a predetermined rate, a two-dimensional distribution (RAW image) of electric charges obtained by photoelectrically converting incident light using an imaging element array such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

[0045] The image signal processing unit 142 performs general processing such as black level correction and shading correction on the image signal output from the imaging unit 140 to generate frame data of the captured image. At this time, the image signal processing unit 142 acquires a luminance histogram of the entire image and supplies it to the AEAGC processing unit 146. The image data transmission unit 144 sequentially transmits the frame data of the captured image that has been subjected to various processes to the content processing device 200.

[0046] The AEAGC processing unit 146 adjusts at least one of the exposure time and the gain value based on the luminance histogram supplied from the image signal processing unit 142 so that the average luminance value becomes the target luminance. The specific adjustment method may be the same as that of conventional AEAGC processing, but in this embodiment, as described above, the target luminance can be adjusted by the content processing device 200. The display unit 148 corresponds to the display unit 124 in Fig. 6 and sequentially displays frames of the display image transmitted from the content processing device 200.

[0047] The content processing device 200 includes an area cutting unit 202 that cuts out the area of ​​the hand from the captured image, a feature point extraction unit 204 that extracts feature points from the image of the cut-out area, a status information acquisition unit 206 that acquires status information of the hand based on the positional relationship of the feature points, etc., an evaluation area setting unit 208 that sets an evaluation area for evaluating the brightness of the image of the hand, and a target brightness control unit 210 that controls the target brightness.

[0048] The content processing device 200 further includes an information processing unit 212 that performs information processing corresponding to the hand state information, a display image generation unit 214 that generates a display image as a result of the information processing, and an image data transmission unit 216 that transmits frame data of the generated display image to the head-mounted display 100. The area cutout unit 202, feature point extraction unit 204, state information acquisition unit 206, evaluation area setting unit 208, and target brightness control unit 210 are functional blocks that accurately track the state information of the hand, which is the object, and ultimately the movement, and these may be implemented together as an object tracking device.

[0049] 3, the area cropping unit 202 extracts the area of ​​the hand image from the captured image transmitted from the head mounted display 100 using an existing method such as pattern matching. When a plurality of images captured at the same time by a plurality of cameras 110 are transmitted from the head mounted display 100, the area cropping unit 202 may extract the area of ​​the hand image from all of the images in which the hand appears. However, images that do not contribute to obtaining state information, such as images in which only a part of the hand is visible, may be excluded from the cropping target.

[0050] The feature point extraction unit 204 extracts feature points of the hand from the cropped image. The feature point extraction unit 204 extracts feature points, for example, as lines representing the skeleton extending from the center of the palm to the fingertips, or points representing the positions of the joints. The state information acquisition unit 206 acquires state information of the hand in three-dimensional space based on the feature points in the image. As described above, the feature point extraction unit 204 and the state information acquisition unit 206 may actually be neural networks such as DNNs. In this case, the feature points become data output from the intermediate layers of the network.

[0051] The evaluation area setting unit 208 sequentially acquires the positions of feature points on the image from the feature point extraction unit 204, and sets a predetermined area including at least a part of the positions as the brightness evaluation area. For example, the evaluation area setting unit 208 sets the inside of a predetermined rectangle including the center of the palm and the second joints of the fifth finger as the evaluation area. However, the shape of the evaluation area is not particularly limited, and may be a circle, ellipse, triangle, polygon with five or more vertices, or the like. Alternatively, the evaluation area setting unit 208 may set the inside of the outline of the hand estimated from the feature points as the evaluation area.

[0052] In any case, by minimizing the possibility of images of objects other than hands being included and by setting the evaluation area to be as wide as possible, the brightness of the hand image can be evaluated with high accuracy. When hand images are detected from multiple images captured simultaneously, the evaluation area setting unit 208 sets evaluation areas for all of those images. The target brightness control unit 210 obtains a brightness histogram of the captured images limited to the evaluation area, determines whether it is appropriate, and adjusts the target brightness if it is not appropriate.

[0053] An upper threshold value Th_u and a lower threshold value Th_l of the appropriate range for the brightness of the evaluation area are prepared in advance. For example, when the average brightness of the evaluation area exceeds the upper threshold value Th_u, the target brightness control unit 210 determines that the image of the hand is overexposed and reduces the target brightness. Furthermore, when the average brightness of the evaluation area is less than the lower threshold value Th_l, the target brightness control unit 210 determines that the image of the hand is underexposed and increases the target brightness. Note that the determination of overexposed or underexposed areas is not limited to being based on the average brightness of the evaluation area, and any statistical method may be used, such as the maximum brightness value in the evaluation area or whether the proportion of pixels outside the appropriate range exceeds a threshold.

[0054] When hand images are detected from multiple images captured simultaneously, the target brightness control unit 210 may add up the brightness in those evaluation areas to obtain a single histogram. In this case, the target brightness control unit 210 determines whether the brightness of the evaluation area is appropriate and adjusts the target brightness in common for all cameras 110.

[0055] Alternatively, the target brightness control unit 210 may acquire a brightness histogram of the evaluation area for each image captured by the multiple cameras 110, evaluate whether the brightness is appropriate, and adjust the target brightness for each camera 110. This enables adjustments that take into account differences in brightness of captured images due to individual differences between the cameras 110 and differences in field of view. In this case, however, changing the exposure time in response to the adjustment of the target brightness will result in differences in frame rate between the cameras 110, and therefore the target brightness control unit 210 preferably requests the AEAGC processing unit 146 to respond by changing only the gain value.

[0056] Alternatively, the target brightness control unit 210 may select images for brightness evaluation by prioritizing, among multiple images in which a hand image is detected, images with a large image area or images in which the hand is captured at a high ratio to the entire image, etc. On the other hand, if a hand image is not detected in any of the captured images, i.e., if an evaluation area is not set, the target brightness control unit 210 may set the target brightness to a default value, i.e., a value set by a general AEAGC. Alternatively, in this case, the target brightness control unit 210 may gradually change the target brightness until a hand image is detected.

[0057] Furthermore, when both left and right hands are captured in a captured image, the target brightness control unit 210 may add up the brightnesses of the evaluation regions set for each hand by the evaluation region setting unit 208 to obtain a single histogram. In this case, the target brightness control unit 210 may determine whether the brightness is appropriate and adjust the target brightness in common based on the combined result for both the left and right hands. However, if the exposure time and gain value can be adjusted in pixel units or pixel block units for the captured image, the target brightness control unit 210 may evaluate the brightness for each of the left and right hands and adjust the target brightness individually. This enables adjustments that take into account differences in how the hands appear depending on their position and posture.

[0058] The target luminance control unit 210 determines whether to change the target luminance, and if so, the changed value, for each frame of the captured image or for each predetermined number of frames, and transmits the result to the head mounted display 100. When a change in target luminance is requested, the AEAGC processing unit 146 of the head mounted display 100 adjusts at least one of the exposure time and the gain value, and controls so that the average luminance of the most recent captured image approaches the target luminance.

[0059] The information processing unit 212 performs information processing based on the hand status information acquired by the status information acquisition unit 206. The display image generation unit 214 generates display images showing the results of the information processing at a predetermined rate. As described above, in this embodiment, the specific content of the information processing using the hand status information and the display images is not particularly limited. The image data transmission unit 216 sequentially transmits frame data of the generated display images to the head-mounted display 100 and displays them on the display unit 148.

[0060] FIG. 9 illustrates the relationship between hand feature points and evaluation areas. The figure shows an image of a hand region cut out from a captured image. As shown in (a), the feature point extraction unit 204 extracts hand feature points 250 from the captured image. In this example, lines representing the skeleton of the hand and points representing the joints within the lines are extracted as feature points 250. The evaluation area setting unit 208 sets an evaluation area 252 on the image based on the distribution of feature points 250. The target brightness control unit 210 acquires a brightness histogram of the evaluation area 252 and determines whether the brightness is within an appropriate range, and if it is outside the appropriate range, whether it is overexposed or underexposed.

[0061] In the example shown in the figure, the image of the hand itself is close to being blocked up, so it is desirable for the target brightness control unit 210 to determine this and adjust the target brightness upward. On the other hand, as shown in the figure, if the evaluation area 252 includes a background that is an image of high-luminance lighting, the brightness histogram may show blown-out areas that exceed the upper threshold of the appropriate range. In other words, a state in which blocked-up shadows and blown-out highlights coexist in the evaluation area may occur.

[0062] To prepare for such a case, the target brightness control unit 210 may first check whether there are any blown-out highlights in the evaluation area and then determine whether to eliminate them based on the average brightness value of the evaluation area. In the example shown in the figure, since the average brightness value of the evaluation area is below the lower threshold, the target brightness control unit 210 determines that the image of the hand itself is blocked up and increases the target brightness. (b) shows a captured image in which the target brightness is increased and the gain is increased based on such a determination, resulting in a clearer outline of the hand.

[0063] 10 illustrates two cases of brightness histograms evaluated by the target brightness control unit 210. Both (a) and (b) assume that there is a blown-out highlight portion in the evaluation area. For example, the target brightness control unit 210 determines that there is a blown-out highlight portion in the evaluation area when the number of pixels whose brightness exceeds the upper threshold value Th_u is equal to or greater than a predetermined value. In this case, as described above, the target brightness control unit 210 determines whether or not to eliminate the blown-out highlight portion based on the average brightness value.

[0064] For example, in the histogram (a), in addition to a peak 260 indicating overexposed areas, there is a peak 262 below the lower brightness threshold Th_l. The latter corresponds to the image of the hand with crushed black areas shown in FIG. 9. The target brightness control unit 210 calculates the average brightness L of the evaluation area. ａｖｅ is less than the lower limit threshold Th_l, the image of the hand itself is determined to be blacked out, and the target luminance is adjusted to be higher. In this case, areas that are already blown out remain white, but the image of the hand is given priority and the luminance is optimized.

[0065] In the histogram of (b), in addition to a peak 264 indicating overexposure, there is a relatively gentle peak 266 that is within the appropriate range. In this case, the target brightness control unit 210 adjusts the average brightness L ａｖｅ Based on the fact that ' is equal to or greater than the lower threshold value Th_l, the brightness of the hand image is determined to be within the appropriate range, and the target brightness is adjusted to be lower. This eliminates blown-out highlights in the evaluation area and reduces problems such as the outline becoming unclear due to strong light from lighting or the like being reflected on the hand.

[0066] As shown in the figure, when there are two brightness peaks, high brightness values ​​due to lighting or the like may raise the average brightness value of the entire evaluation area. Therefore, in this case, the lower threshold value Th_l assigned to the average brightness value may be higher than when there is only one peak. Alternatively, when there are two brightness peaks, the overexposed areas may be excluded from the calculation of the average brightness value. This also allows the brightness of the hand image itself to be correctly evaluated.

[0067] 11 illustrates an example of temporal changes in the brightness of the evaluation area and the target brightness. The upper row shows the average brightness of the evaluation area in the captured image, and the lower row shows the target brightness controlled by the target brightness control unit 210, relative to the horizontal time axis. In this example, the average brightness of the evaluation area is assumed to be the basis for target brightness control, and an upper threshold value Th_u and a lower threshold value Th_l of the appropriate range are set for this average value. First, if the average brightness value is within the appropriate range, the target brightness control unit 210 does not adjust the target brightness.

[0068] If the average brightness value exceeds the upper threshold value Th_u at a certain time, the target brightness control unit 210 determines this and requests the AEAGC processing unit 146 to lower the target brightness (arrow A). In response to the request, the AEAGC processing unit 146 reduces at least one of the exposure time and the gain value, thereby lowering the average brightness of the entire captured image and, ultimately, the evaluation area so that it falls within the appropriate range (arrow B). If, after a while, the average brightness value again exceeds the upper threshold value Th_u due to a change in the state of the hand, for example, the target brightness control unit 210 determines this and again requests the AEAGC processing unit 146 to lower the target brightness (arrow C).

[0069] In response to the request, the AEAGC processing unit 146 again reduces at least one of the exposure time and the gain value, thereby decreasing the average brightness of the evaluation area and bringing it within the appropriate range (arrow D). If the average brightness falls below the lower threshold value Th_l due to a change in the state of the hand or other reason, the target brightness control unit 210 determines this and requests the AEAGC processing unit 146 to increase the target brightness (arrow E). In response to the request, the AEAGC processing unit 146 increases at least one of the exposure time and the gain value, thereby increasing the average brightness of the evaluation area and bringing it within the appropriate range (arrow F).

[0070] If the average brightness again exceeds the upper threshold value Th_u, the target brightness control unit 210 determines this and requests the AEAGC processing unit 146 to lower the target brightness (arrow G). The AEAGC processing unit 146 responds to the request by reducing at least one of the exposure time and the gain value, thereby lowering the average brightness of the evaluation area and bringing it into the appropriate range (arrow H). By repeating this process, the brightness of the hand image can be kept within the appropriate range as much as possible, and status information can be continuously acquired with high accuracy.

[0071] To facilitate understanding, the changes in brightness are somewhat exaggerated in the diagram. However, the brightness of the image of the hand can change due to various factors, such as changes in the ambient brightness, the field of view, and the positional relationship between the light source and the surface of the hand. As shown in the diagram, there is a certain time lag between when the brightness of the evaluation area falls outside the appropriate range and when the target brightness is adjusted to return it to the appropriate range. Therefore, if the target brightness is frequently changed based solely on threshold judgment, the exposure time and gain value may oscillate. Furthermore, frequent changes in the brightness of the captured image due to changes in the exposure time and gain value may actually have a negative impact on the accuracy of status information acquisition.

[0072] For these reasons, it is desirable to impose certain restrictions on switching the target luminance. For example, even if the luminance of the evaluation area falls outside the appropriate range, the target luminance control unit 210 waits without adjusting the target luminance until a predetermined time has elapsed since the previous adjustment of the target luminance. If the luminance of the evaluation area remains outside the appropriate range even after a predetermined time has elapsed since the previous adjustment, the target luminance control unit 210 adjusts the target luminance. The target luminance control unit 210 may prevent oscillation of the determination result by applying a low-pass filter to a signal indicating the average luminance of the evaluation area and then determining whether the average luminance is within the appropriate range. Alternatively, the target luminance control unit 210 may perform hysteresis control by varying the threshold value assigned to the luminance of the evaluation area, which is the basis for adjusting the target luminance, depending on the situation.

[0073] For example, once the average brightness exceeds the upper threshold value Tu_u and the target brightness is lowered, the target brightness control unit 210 lowers the lower threshold value Tu_l by a predetermined amount for a predetermined period of time, thereby reducing the possibility that the target brightness will have to be immediately increased. Conversely, once the average brightness falls below the lower threshold value Tu_l and the target brightness is increased, the target brightness control unit 210 raises the upper threshold value Tu_u by a predetermined amount for a predetermined period of time, thereby reducing the possibility that the target brightness will have to be immediately decreased. This also reduces the possibility that unnecessary adjustments will be made due to temporary brightness changes, etc. The target brightness control unit 210 may employ either temporal adjustment using a low-pass filter or hysteresis control, or a combination of both.

[0074] FIG. 12 shows a simplified example of how exposure time and gain value change as a result of adjusting the target brightness. In the captured image plane 270, each of the 8×8 rectangular regions represents a pixel, and 256 levels of brightness (pixel value) are shown within each rectangle. In this example, there are two blocks of four pixels with the highest brightness value of "255," totaling eight pixels, 44 surrounding pixels with a low brightness value of "10," and 12 pixels outside the valid range with a brightness value of "0." The image signal processor 142 generates a histogram 272 as part of the image signal processing.

[0075] The average brightness of this captured image is 38. This situation corresponds to the AEAGC processing unit 146 setting the target brightness to 38. As an example, the exposure time in this state is 4 msec, and the gain value is 8x. On the other hand, if the area with a brightness value of "255" is the area of ​​the image of the hand, which is the subject, the target brightness control unit 210 determines that there is blown-out highlights and decides to lower the target brightness. For example, the target brightness control unit 210 decides to halve the target brightness from the original average brightness to 19. In this case, the AEAGC processing unit 146 leaves the exposure time unchanged, as shown in the figure, and halves the gain value from 8x to 4x.

[0076] As a result, the average brightness value of subsequent captured images will be halved. However, the rules for adjusting the exposure time and gain value are not limited, and the exposure time may be halved, or both the exposure time and the gain value may be adjusted and the multiplied result may be halved. On the other hand, if the area with a brightness value of "10" is the area of ​​the hand image, the target brightness control unit 210 determines that there is blocked-up shadows and decides to increase the target brightness.

[0077] For example, the target brightness control unit 210 determines to set the target brightness to 1.5 times the original brightness average value, i.e., 57. In this case, the AEAGC processing unit 146, for example, leaves the gain value unchanged and changes the exposure time from 4 msec to 6 msec. Alternatively, the target brightness control unit 210 determines to set the target brightness to twice the original brightness average value, i.e., 76. In this case, the AEAGC processing unit 146, for example, leaves the gain value unchanged and changes the exposure time from 4 msec to 8 msec.

[0078] In this case, the adjustment rule for the exposure time and the gain value is not limited, and the gain value may be adjusted, or both the exposure time and the gain value may be adjusted. The rate at which the target luminance is lowered when the object is overexposed and the rate at which the target luminance is increased when the object is underexposed may be fixed values, or may be adjusted in stages depending on the shape of the original histogram, the size of the overexposed or underexposed areas, etc.

[0079] Next, an operation that can be realized by the configuration described above will be described. Fig. 13 is a flowchart showing the processing procedure in which the content processing device 200 controls the target brightness. The illustrated flowchart starts when the user wears the head-mounted display 100 and performs an operation such as selecting target content. By selecting content, the object from which status information is to be acquired is determined. Alternatively, it may be determined that a body part such as a hand will be used for operation when the user does not hold an input device such as a controller (not shown).

[0080] When the content processing device 200 acquires frame data of a captured image from the head-mounted display 100 (S20), the area extraction unit 202 extracts an area in which an image of a hand appears from the acquired image (S22). Next, the feature point extraction unit 204 extracts feature points of the hand from the image of the extracted area (S24). As described above, this process may actually acquire intermediate data obtained by inputting an image of the hand into a DNN. Although not shown in the figure, the content processing device 200 acquires state information of the object from the acquired feature points and performs appropriate information processing to generate a display image in parallel with the processing procedure shown in the figure.

[0081] Meanwhile, the evaluation area setting unit 208 sets an evaluation area for evaluating the brightness of the hand image based on the distribution of the extracted feature points (S26). When hands are simultaneously captured in images captured by multiple cameras 110, the evaluation area setting unit 208 sets an evaluation area for each of them. Furthermore, when multiple objects, such as both hands, are captured in a single captured image, the evaluation area setting unit 208 sets an evaluation area for each of them. The target brightness control unit 210 checks whether the brightness of the evaluation area is within an appropriate range (S28). For example, if the average brightness of the evaluation area exceeds the upper threshold of the appropriate range, the target brightness control unit 210 determines that the hand image is overexposed, and if it is below the lower threshold of the appropriate range, the target brightness control unit 210 determines that the hand image is underexposed.

[0082] As described above, the target brightness control unit 210 may generate a brightness histogram that aggregates all the brightness values ​​of the evaluation areas set in multiple captured images and determine whether highlights are blown out or shadows are blocked up in common based on the brightness histogram, or may perform a determination individually for multiple images captured simultaneously, or for each camera 110. The target brightness control unit 210 may also determine whether highlights are blown out or shadows are blocked up by distinguishing between the left and right hands. The target brightness control unit 210 may also change the criteria for determining whether highlights are blown out or shadows depending on the shape of the brightness histogram of the evaluation area and the previous adjustment state of the target brightness.

[0083] If it is determined that the brightness of the evaluation area is within the appropriate range, the target brightness control unit 210 maintains the previous target brightness (N in S28). If it is determined that the brightness of the evaluation area is not within the appropriate range (Y in S28), the target brightness control unit 210 determines to adjust the target brightness (S30). That is, if it is determined that the image of the hand is overexposed, the target brightness control unit 210 lowers the target brightness, and if it is determined that the image of the hand is underexposed, the target brightness control unit 210 raises the target brightness. The optimum value for the degree of adjustment of the target brightness is determined in advance by experiment or the like.

[0084] Alternatively, the target brightness control unit 210 may change the target brightness in stages. When it is decided to adjust the target brightness, the target brightness control unit 210 requests the head-mounted display 100 to do so. As a result, the head-mounted display 100 changes at least one of the exposure time and the gain value to correspond to the requested target brightness, and then continues capturing images. If there is no need to end content processing due to a user operation or the like, the content processing device 200 obtains data of the next frame from the head-mounted display 100 and repeats the illustrated processing (N in S32, S20 to S30).

[0085] When the target luminance is adjusted, the content processing device 200 can eventually acquire frame data in which at least one of the exposure time and the gain value has been appropriately changed. If it becomes necessary to end content processing due to a user operation or the like, the content processing device 200 ends all processing (Y in S32).

[0086] 14 is a diagram illustrating how the evaluation area setting unit 208 sets the evaluation area. As described above, the evaluation area setting unit 208 sets the evaluation area 302 based on the distribution of hand feature points (e.g., feature points 300a and 300b) extracted by the feature point extraction unit 204. In this case, if the evaluation area is determined so that images of objects other than the hand are minimized, it becomes easier to accurately evaluate a brightness histogram specialized for the image of the hand. For example, if the smallest rectangle 304 that encompasses all feature points is set as the evaluation area, background objects reflected between the fingers may affect the brightness histogram, making it impossible to set the target brightness appropriately.

[0087] Therefore, if the evaluation area 302 is set so that part of the feature point, and therefore part of the hand image, protrudes, the influence of the background can be reduced and the target brightness can be made more stable. Therefore, the evaluation area setting unit 208 may, for example, reduce the rectangle 304 by a predetermined reduction rate and set the rectangle centered on the feature point 300b indicating the base of the middle finger as the evaluation area 302. Here, the reduction rate may be set for each of the vertical and horizontal directions based on the characteristics of the hand shape, etc.

[0088] The evaluation area setting unit 208 may sequentially acquire hand state information from the state information acquisition unit 206 and change the evaluation area setting rules, such as the reduction rate and center position, accordingly. For example, when the hand is in a clenched fist, the evaluation area setting unit 208 increases the reduction rate of the evaluation area compared to when the hand is open, and moves the center by a predetermined percentage from the base of the middle finger toward the wrist. The evaluation area setting unit 208 may change the evaluation area setting rules according to various hand states (poses) other than whether the hand is open or closed.

[0089] In any case, by having the evaluation area setting unit 208 set an evaluation area smaller than the actual hand image, the influence of images other than the hand can be suppressed, and as a result, the target luminance can be appropriately controlled based on the actual brightness of the skin. For example, even in the same environment, the darker the skin, the higher the target luminance can be, and thus the accuracy of hand tracking can be maintained.

[0090] In the explanation so far, the AEAGC processing unit 146 adjusts at least one of the exposure time and the gain value in response to a request to change the target brightness from the content processing device 200. However, adjusting the exposure time uniformly regardless of the situation may actually reduce the accuracy of hand tracking. For example, if the exposure time is increased when the hand is moving relatively quickly, the image will be blurred, reducing the accuracy of acquiring feature points. Therefore, if the exposure time is not increased, the exposure time will be excessively reduced even when the hand is stationary and there is no need to worry about blurring. Instead, the increased gain value will increase the possibility of increased noise and crushed shadows.

[0091] Therefore, the target brightness control unit 210 may control the AEAGC processing unit 146 so as to change the allowable range of exposure time depending on the relative speed of the hand with respect to the camera 110. Fig. 15 is a diagram for explaining an aspect in which the target brightness control unit 210 controls the allowable range of exposure time depending on the speed of the hand with respect to the camera 110. The horizontal direction of the diagram indicates changes in the relative speed of the hand with respect to the camera 110, and the vertical direction indicates, with filled rectangles, the allowable range of exposure time set for the range of relative speed.

[0092] In this embodiment, camera 110 is mounted on head-mounted display 100, so even if the user's hand is stationary, moving their head creates the same situation as if the hand is moving in the captured image. Therefore, target brightness control unit 210 controls the allowable range of exposure time according to the "relative speed" of the hand with respect to camera 110.

[0093] For example, the target brightness control unit 210 acquires the relative speed based on the displacement of feature points on the captured image plane. Alternatively, the target brightness control unit 210 may acquire the relative speed using the hand status information acquired by the status information acquisition unit 206 and the movement of the camera 110 obtained from sensor data of the head-mounted display 100, etc. The target brightness control unit 210 may acquire the relative speed for the entire hand, or may acquire the relative speed for a part of the hand, such as a specific finger. The target brightness control unit 210 may change the granularity of the part from which the relative speed is acquired, depending on the information processing realized using the hand tracking results.

[0094] In the illustrated example, an upper threshold Vl is set for the relative velocity, at which the hand is considered to be stationary relative to the camera 110, and a lower threshold Vu is set for the relative velocity, at which the hand is considered to be moving at high speed. As an example, the thresholds Vl and Vu are set to 0.1 m / sec and 0.7 m / sec, respectively. A state where the relative velocity v is Vl≦v<Vu, that is, a state where the hand is considered to be moving relative to the camera 110 but not at high speed, is considered to be the standard state, and the lower limit of the exposure time is set to El1 and the upper limit to Eu1. In other words, the AEAGC processing unit 146 adjusts the exposure time within this range to achieve the desired target brightness in accordance with a request from the target brightness control unit 210, and adjusts the gain value if the exposure time deviates from this range.

[0095] When the relative velocity v is 0≦v<Vl, which allows the hand to be considered stationary (i.e., when the relative velocity v satisfies this condition), the AEAGC processor 146 adjusts the exposure time in accordance with a request from the target brightness control unit 210 by setting the lower limit El0 and upper limit Eu0 of the exposure time higher than the lower limit El1 and upper limit Eu1 of the standard state, respectively. During the period when the hand is considered stationary relative to the camera 110, even if the exposure time is increased slightly, image blurring is unlikely to occur. Therefore, by increasing the margin for increasing the exposure time in this way, the possibility of the image becoming crushed even if the area around the hand temporarily becomes dark can be reduced. Furthermore, it is possible to reduce situations where the gain value must be increased to achieve the target brightness, thereby increasing the margin for noise reduction.

[0096] When the relative velocity v is Vu≦v, which indicates that the hand is moving at high speed (i.e., when the relative velocity v satisfies this condition), the AEAGC processing unit 146 adjusts the exposure time by setting the lower limit El2 and upper limit Eu2 of the exposure time lower than the lower limit El1 and upper limit Eu1 of the standard state, respectively, in accordance with a request from the target brightness control unit 210. By controlling the exposure time to suppress an increase in the exposure time during the period when the hand is considered to be moving at high speed relative to the camera 110, it is possible to reduce the adverse effects of image blurring on the accuracy of acquiring feature points.

[0097] The specific setting values ​​of the thresholds Vl and Vu to be assigned to the relative velocity v and the upper and lower limits of the exposure time are determined in advance by experiment, etc. However, the settings shown in the figure are merely an example, and the number of thresholds for the relative velocity v may be one or three or more, and the upper and lower limits of the exposure time may be gradually decreased as the relative velocity increases. Furthermore, only one of the upper and lower limits of the allowable range of the exposure time may be changed.

[0098] The target brightness control unit 210 is not limited to changing the allowable range of exposure time in accordance with the relative speed of the hand, but may also change which of the exposure time and the gain value is given priority in adjustment, or the proportion of the adjustment amounts between the exposure time and the gain value. For example, when increasing the target brightness, the target brightness control unit 210 may reduce the increase in exposure time as the relative speed of the hand increases, and compensate for that by increasing the gain value. Furthermore, when decreasing the target brightness, the target brightness control unit 210 may prioritize decreasing the exposure time as the relative speed of the hand increases. In this case, the same effect as the method illustrated in the figure can be obtained.

[0099] According to the present embodiment described above, an object with arbitrary movement, such as a hand, is tracked based on a captured image, and the brightness of the area of ​​the image of the object is checked to see if it is within an appropriate range, and if it is not, the target brightness in the AEAGC processing is adjusted. This makes it possible to easily achieve brightness optimization specific to the image of the object using a conventional AEAGC mechanism. The area used to check whether the brightness is within the appropriate range is determined by the distribution of feature points acquired during tracking of the object. This allows the brightness of the image to be accurately evaluated, and the accuracy of target brightness adjustment can be improved.

[0100] Furthermore, by adaptively changing the luminance judgment criteria depending on whether there is overexposure due to background lighting, the timing of the previous target luminance, the adjustment direction, etc., it is possible to further improve the accuracy of target luminance adjustment and avoid worsening of tracking accuracy due to unnecessary adjustments. With this configuration, even for objects with arbitrary movement, it is possible to continue capturing images suitable for analysis regardless of the situation or environment, and to perform stable tracking using captured images.

[0101] Furthermore, the exposure time tolerance range in AEAGC processing is changed according to the relative speed of the object relative to the camera. This allows the exposure time to be adjusted appropriately according to the situation, such as reducing the exposure time to prevent image blur when the object is moving at high speed, and increasing the exposure time to improve resistance to dark places when the object is stationary, thereby making tracking accuracy more stable.

[0102] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the respective components and treatment processes, and that such modifications are also within the scope of the present invention.

[0103] The present disclosure may include the following aspects. [Item 1] An object tracking device including a circuit configured to: analyze an image of an object in an image obtained by video capture by an imaging device, acquire status information of the object, determine whether the luminance of an evaluation area set for the image of the object according to a predetermined rule is within a predetermined appropriate range, and, if not within the appropriate range, cause the imaging device to adjust a target luminance in an auto exposure auto gain control (AEAGC) process. [Item 2] The object tracking device according to item 1, wherein the circuit acquires status information including the shape of the object by extracting feature points in the image of the object, and sets the evaluation area based on a distribution of the feature points in the image. [Item 3] The object tracking device according to item 1, wherein the circuit adjusts the target luminance to lower the target luminance when the average value of the luminance of the evaluation area exceeds an upper threshold of the appropriate range, and to raise the target luminance when the average value is below a lower threshold of the appropriate range. [Item 4] The object tracking device according to item 1, wherein the circuit, when there is a portion in the evaluation area having a brightness exceeding a predetermined value, increases the target brightness if the average value of the brightness of the evaluation area is less than a lower threshold of the appropriate range, and decreases the target brightness if the average value is equal to or greater than the lower threshold. [Item 5] The object tracking device according to item 1, wherein the circuit changes a condition for determining adjustment of the target brightness depending on the timing of the previous adjustment. [Item 6] The object tracking device according to item 5, wherein the circuit changes a threshold that defines the appropriate range for a predetermined time after adjusting the target brightness. [Item 7] The object tracking device according to item 5, wherein the circuit determines the next adjustment of target brightness on the condition that a predetermined time has elapsed since adjusting the target brightness.[Item 8] The object tracking device according to item 1, wherein the circuit aggregates brightness histograms of the evaluation areas set for multiple objects that appear simultaneously in one image, and commonly determines whether or not they are within the appropriate range. [Item 9] The object tracking device according to item 1, wherein the circuit individually determines whether or not the brightness of the evaluation areas set for multiple objects that appear simultaneously in one image is within the appropriate range, and determines whether or not to adjust the target brightness for each image of the object based on the result. [Item 10] The object tracking device according to item 1, wherein the circuit aggregates brightness histograms of the evaluation areas set for multiple images that are video images taken by multiple imaging devices, and commonly determines whether or not they are within the appropriate range. [Item 11] The object tracking device according to item 1, wherein the target brightness control unit individually determines whether the brightness of the evaluation area set for each of a plurality of images obtained by video capture by a plurality of imaging devices is within the appropriate range, and determines whether or not to adjust the target brightness for each of the imaging devices based on the result. [Item 12] The object tracking device according to item 1, wherein the circuit, when an image of the object is not detected in the image, changes the target brightness until it is detected. [Item 13] The object tracking device according to item 2, wherein the circuit sets the evaluation area to an area smaller than the image of the object. [Item 14] The object tracking device according to item 13, wherein the circuit changes the rule for setting the evaluation area based on the status information. [Item 15] The object tracking device according to item 1, wherein the circuit changes the allowable range of exposure time in the AEAGC processing based on the range of relative speed of the object with respect to the imaging device.16. The object tracking device according to claim 15, wherein the circuit increases at least one of an upper limit and a lower limit of an allowable range of the exposure time when the relative velocity satisfies a condition under which the object can be considered stationary, and decreases at least one of the upper limit and the lower limit of the allowable range of the exposure time when the object can be considered moving at high speed. [Item 17] A content processing system including: a head-mounted display equipped with an imaging device; and a content processing device that performs information processing based on status information of an object shown in an image captured by the imaging device, generates a display image, and displays the image on the head-mounted display, wherein the content processing device has a circuit configured to: analyze an image of the object in the image, acquire the status information, determine whether the luminance of an evaluation area set according to a predetermined rule for the image of the object is within a predetermined appropriate range, and, if not, cause the imaging device to adjust a target luminance in AEAGC (Auto Exposure Auto Gain Control) processing. [Item 18] An object tracking method, comprising: analyzing an image of an object in an image obtained by video capture with an imaging device; acquiring status information of the object; determining whether the luminance of an evaluation area set according to a predetermined rule for the image of the object is within a predetermined appropriate range; and, if it is not within the appropriate range, having the imaging device adjust a target luminance in AEAGC (Auto Exposure Auto Gain Control) processing. [Item 19] A recording medium having recorded thereon a program for causing a computer to implement the following functions: analyzing an image of an object in an image obtained by video capture with an imaging device; and acquiring status information of the object; determining whether the luminance of an evaluation area set according to a predetermined rule for the image of the object is within a predetermined appropriate range; and, if it is not within the appropriate range, having the imaging device adjust a target luminance in AEAGC (Auto Exposure Auto Gain Control) processing.

[0104] As described above, the present invention can be used in information processing devices such as object tracking devices, content processing devices, game devices, and personal computers, as well as information processing systems including any of these devices.

[0105] REFERENCE SIGNS LIST 100 head-mounted display, 110 camera, 140 imaging unit, 142 image signal processing unit, 144 image data transmission unit, 146 AEAGC processing unit, 148 display unit, 200 content processing device, 202 area extraction unit, 204 feature point extraction unit, 206 status information acquisition unit, 208 evaluation area setting unit, 210 target brightness control unit.

Claims

1. An object tracking device comprising: a status information acquisition unit that analyzes an image of an object in an image captured by an imaging device and acquires status information of the object; and a target brightness control unit that determines whether the brightness of an evaluation area set according to a predetermined rule for the image of the object is within a predetermined appropriate range, and if it is not within the appropriate range, causes the imaging device to adjust the target brightness in AEAGC (Auto Exposure Auto Gain Control) processing.

2. The object tracking device described in claim 1, characterized in that the state information acquisition unit acquires state information including the shape of the object by extracting feature points in the image of the object, and further comprises an evaluation area setting unit that sets the evaluation area based on the distribution of the feature points in the image.

3. The object tracking device of claim 1 or 2, characterized in that the target brightness control unit adjusts the target brightness by lowering it when the average brightness value of the evaluation area exceeds the upper threshold of the appropriate range, and by raising it when the average brightness value is below the lower threshold of the appropriate range.

4. The object tracking device of claim 1 or 2, characterized in that when there is a portion in the evaluation area having a brightness exceeding a predetermined value, the target brightness control unit increases the target brightness if the average value of the brightness of the evaluation area is less than the lower threshold value of the appropriate range, and decreases the target brightness if the average value is equal to or greater than the lower threshold value.

5. The object tracking device according to claim 1 or 2, characterized in that the target brightness control unit changes the conditions for determining the adjustment of the target brightness depending on the timing of the previous adjustment.

6. The object tracking device according to claim 5, wherein the target brightness control unit changes the threshold value that defines the appropriate range for a predetermined time after adjusting the target brightness.

7. The object tracking device according to claim 5, wherein the target brightness control unit determines the next adjustment of the target brightness on the condition that a predetermined time has elapsed since the target brightness was adjusted.

8. The object tracking device described in claim 1 or 2, characterized in that the target brightness control unit aggregates the brightness histograms of the evaluation areas set for multiple objects that appear simultaneously in one image, and commonly determines whether or not they are within the appropriate range.

9. The object tracking device described in claim 1 or 2, characterized in that the target brightness control unit individually determines whether the brightness of the evaluation area set for each of multiple objects that appear simultaneously in one image is within the appropriate range, and depending on the result, decides whether or not to adjust the target brightness for each image of the object.

10. The object tracking device described in claim 1 or 2, characterized in that the target brightness control unit aggregates the brightness histograms of the evaluation areas set for multiple images captured by multiple imaging devices, and commonly determines whether or not they are within the appropriate range.

11. The object tracking device described in claim 1 or 2, characterized in that the target brightness control unit individually determines whether the brightness of the evaluation area set for each of multiple images captured by multiple imaging devices is within the appropriate range, and based on the result, decides whether or not to adjust the target brightness for each imaging device.

12. An object tracking device according to claim 1 or 2, characterized in that, when an image of an object is not detected in the image, the target brightness control unit changes the target brightness until the image is detected.

13. The object tracking device according to claim 2, wherein the evaluation area setting unit sets the evaluation area to an area smaller than the image of the object.

14. The object tracking device according to claim 13, wherein the evaluation area setting unit changes the setting rules for the evaluation area according to the state information of the object acquired by the state information acquisition unit.

15. An object tracking device according to claim 1 or 2, characterized in that the target brightness control unit changes the allowable range of exposure time in the AEAGC processing according to the range of the relative speed of the object with respect to the imaging device.

16. The object tracking device of claim 15, wherein the target brightness control unit increases at least one of the upper and lower limits of the permissible range of the exposure time when the relative speed satisfies the conditions under which the object can be considered to be stationary, and decreases at least one of the upper and lower limits of the permissible range of the exposure time when the relative speed satisfies the conditions under which the object can be considered to be moving at high speed.

17. A content processing system comprising: a head-mounted display equipped with an imaging device; and a content processing device that performs information processing based on status information of an object shown in an image captured by the imaging device, generates a display image, and displays it on the head-mounted display, wherein the content processing device comprises: a status information acquisition unit that analyzes the image of the object in the image and acquires the status information; and a target brightness control unit that determines whether the brightness of an evaluation area set according to a predetermined rule for the image of the object is within a predetermined appropriate range, and if it is not within the appropriate range, causes the imaging device to adjust the target brightness in AEAGC (Auto Exposure Auto Gain Control) processing.

18. A method for tracking an object comprising the steps of: analyzing an image of the object in an image captured by an imaging device and acquiring status information of the object; and determining whether the brightness of an evaluation area set according to a predetermined rule for the image of the object is within a predetermined appropriate range, and if it is not within the appropriate range, causing the imaging device to adjust the target brightness in AEAGC (Auto Exposure Auto Gain Control) processing.

19. A computer program that enables a computer to perform the following functions: analyze an image of an object in an image captured by an imaging device and obtain status information of the object; determine whether the brightness of an evaluation area set according to a predetermined rule for the image of the object is within a predetermined appropriate range, and, if it is not within the appropriate range, have the imaging device adjust the target brightness in AEAGC (Auto Exposure Auto Gain Control) processing.

Images