Image processing apparatus, information processing system, and image acquisition method

Abstract

To acquire real-world state information from a captured image with high accuracy even in an environment where an imaging apparatus moves.SOLUTION: A parameter control section 30 in an image processing apparatus 10 acquires motion information of a movable body in which an imaging apparatus 14 is mounted from a motion measurement apparatus 12, and controls a filter parameters for an exposure time, a gain value, and noise reduction in accordance therewith. A gain processing section 22 performs a gain process on an image captured at a controlled exposure time. An image processing section 26 performs a noise reduction process. An output unit 28 outputs image data to an image analysis apparatus 16.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for acquiring a photographed image for acquiring state information of the real world, an information processing system for generating the state information, and a method for acquiring a photographed image.

Background Art

[0002] There is known a game in which an imaging device photographs a user's body or a marker, and the area of the image is replaced with another image and displayed on a display (see, for example, Patent Document 1). Further, there is also known a technique in which an imaging device is attached to a moving object such as a vehicle or a robot, and the self-position is estimated or surrounding objects are recognized by analyzing the photographed image. As described above, techniques for acquiring and using various situation information of the real world using photographed images are used in a wide range of fields from daily life to special surveys.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In an environment where the imaging device itself moves, such as when mounted on a moving object, the relative movement with respect to the subject becomes larger compared to a fixed imaging device, and motion blur in which the image blurs easily occurs. Motion blur makes it difficult to extract feature points and feature amounts important in image analysis and deteriorates the analysis accuracy. The higher the degree of freedom of movement of the imaging device, and the more an environment where sudden movement is likely to occur, the more the problem of motion blur becomes apparent, and the analysis result may become indeterminate.

[0005] The present invention has been made in view of such problems, and an object thereof is to provide a technique capable of accurately acquiring state information of the real world from a captured image even in an environment where an imaging device moves.

Means for Solving the Problems

[0006] One aspect of the present invention relates to an image processing apparatus. This image processing apparatus includes a parameter control unit that acquires motion information of an imaging device that captures a moving image at a predetermined rate and controls an exposure time based on the motion information, and an output unit that outputs data obtained by performing predetermined processing on an image captured by the imaging device at the exposure time, and is characterized by including these.

[0007] Another aspect of the present invention relates to an information processing system. This information processing system is characterized by including the above-described image processing apparatus, an imaging device, and a motion measurement device that measures motion information at a predetermined rate.

[0008] Still another aspect of the present invention relates to an image acquisition method. This image acquisition method includes a step of acquiring motion information of an imaging device that captures a moving image at a predetermined rate, a step of controlling an exposure time based on the motion information, and a step of outputting data obtained by performing predetermined processing on an image captured by the imaging device at the exposure time, and is characterized by including these.

[0009] Note that any combination of the above-described components, and those obtained by converting the expression of the present invention among a method, an apparatus, a system, a computer program, a recording medium recording the computer program, etc. are also effective as aspects of the present invention.

Effects of the Invention

[0010] According to the present invention, even in an environment where an imaging device moves, state information of the real world can be accurately acquired from a captured image.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0012] The present embodiment relates to a technique for analyzing an image photographed in an environment that allows movement of an imaging device and acquiring state information of the real world. As long as this is the case, the mounting form of the imaging device is not particularly limited, and it may be mounted on a device having another use such as a head-mounted display, a robot, a mobile terminal, a vehicle, a drone, etc., or may be a single imaging device that can be moved by a person. Hereinafter, an imaging device mounted on a head-mounted display will be mainly described as an example.

[0013] FIG. 1 shows an external appearance example of a head-mounted display 100 to which the present embodiment can be applied. In this example, the head-mounted display 100 is composed of an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that wraps around the head when worn by the user to achieve fixation of the device. The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head-mounted display 100, and a display panel is provided inside so as to face the eyes when worn.

[0014] Inside the housing 108, there is further an eyepiece lens that is positioned between the display panel and the user's eyes when the head-mounted display 100 is worn and that expands the user's viewing angle. The head-mounted display 100 also incorporates an IMU (Inertial Measurement Unit) sensor, which detects the translational and rotational movements of the head of the user wearing the head-mounted display 100, and thus the position and orientation at each moment. The head-mounted display 100 may further include speakers or earphones at positions corresponding to the user's ears when worn. Furthermore, the head-mounted display 100 includes an imaging device 110 for shooting a moving image of the real space on the front surface of the housing 108.

[0015] For example, as shown in the drawing, if the imaging device 110 is a stereo camera and the images captured from the left and right viewpoints are displayed on the display panel, even a closed-type head-mounted display 100 can realize a video see-through where the real space can be surveyed. On the other hand, by analyzing the frames of the captured moving image, the position and orientation of the user's head can be acquired at the frame period. By using this information to draw a virtual world in the field of view corresponding to the user's line of sight, an immersive virtual reality can be realized. Also, by synthesizing computer graphics so as to match the field of view of the captured image, augmented reality and mixed reality can also be realized.

[0016] As a method for analyzing the captured image and estimating the self-position and orientation while creating a surrounding map, various algorithms such as V-SLAM (Visual Simultaneous Localization and Mapping) have been proposed. By integrating the measurement values of the IMU sensor with the analysis results from the captured image, the detail and accuracy of the position and orientation information can be enhanced. However, the content of the image analysis is not particularly limited, and objects in the field of view of the imaging device 110 may be recognized or measured. Also, the imaging device 110 mounted on the head-mounted display 100 is not limited to a stereo camera, and may be a multi-eye camera with three or more lenses or a monocular camera.

[0017] In any case, when an imaging device is mounted on a movable body such as a head-mounted display 100, motion blur caused by the movement of the imaging device side occurs. As a result, large errors occur in parameters used for analysis, such as the contour and feature points of the image, and the accuracy of image analysis is likely to decrease. Similar problems occur in various movable bodies such as automobiles, robots, and drones equipped with an imaging device for analysis. In addition, it also occurs when the imaging device itself is in a movable state.

[0018] However, when the speed and acceleration change continuously, it is possible to maintain a certain degree of analysis accuracy by predicting or correcting parameters based on the previous movement and the like. On the other hand, when the movement direction and speed can freely change due to human intention or the like, as in the case of the head-mounted display 100, there is a high possibility that the change exceeds the allowable range, and the analysis accuracy deteriorates significantly or the analysis result becomes indeterminate.

[0019] For example, when a user wears a head-mounted display 100 and plays a combat game, it is conceivable that the head moves complexly and suddenly in order to avoid the approach of an enemy or the flying of a bullet. In addition, in the case of using a game controller, the controller may hit the head, and an acceleration caused by the impact may occur unintentionally. Such unpredictable movements can occur in various scenes, not limited to head-mounted displays, such as when an autonomous mobile robot crosses a step on the ground.

[0020] In order to stably obtain the analysis result, it is desirable to be able to continuously capture an image with less motion blur regardless of the movement of the imaging device. On the other hand, in a general imaging device, automatic adjustment of shooting conditions and image processing is performed so that a person feels that the captured image is beautiful. Elements that a person feels are beautiful include, for example, high contrast, high sharpness, and high resolution. In order to preferably realize these elements, various processes and adjustments such as adjustment of exposure time and gain according to the ambient brightness, noise reduction processing, gamma correction, contrast enhancement, demosaicing, and super resolution are performed in a general imaging device.

[0021] Figure 2 illustrates the dependency relationships of parameters related to the processing of general captured images. Among them, (a) shows the chain of main parameters. First, a target luminance 50 indicating a suitable brightness for the entire image is set. The brightness of the entire image can be represented, for example, by the centroid in the luminance histogram. The imaging device controls the exposure time 54 according to the brightness of the shooting environment 52 so that the brightness of the entire captured image approaches the target luminance 50. For example, if sufficient luminance cannot be obtained in a dark environment, by increasing the exposure time (shutter speed), the incident light per pixel increases and the luminance rises.

[0022] By first setting the exposure time 54 with respect to the target luminance 50, the original light detection value can be made as high as possible, and noise can be relatively reduced. The gain value 56 is set to amplify the detection value thus obtained and bring it closer to the target luminance 50. Then, in order to reduce the noise amplified by the gain adjustment, filter parameters 58 such as a smoothing filter based on the gain value 56 are set. Actually, as described above, processing such as gamma correction, contrast enhancement, and super-resolution is also performed. For a color image, further, demosaicing processing and color tone adjustment are also performed.

[0023] In this way, the function of adjusting the exposure with respect to the ambient brightness is known as AE (Auto Exposure), and the function of adjusting the gain is known as AGC (Auto Gain Control). (b) illustrates the relationship between the shooting environment and motion blur when shooting with a general imaging device. According to the parameter chain shown in (a), the exposure time is determined by the brightness or darkness of the shooting environment. Qualitatively, as shown in the figure, if the shooting environment is bright, the exposure time is shortened, and if it is dark, the exposure time is lengthened. Note that the expressions here are relative to the brightness and darkness.

[0024] Here, if there is no movement in the imaging device, and by extension, the head-mounted display 100 on which it is mounted, motion blur will not occur regardless of the exposure time. On the other hand, if there is movement in the head-mounted display 100, the longer the exposure time, the greater the degree of occurrence of motion blur (the size of the blur). As a result, even if the movement of the head-mounted display 100 is the same, the darker the shooting environment, the greater the motion blur.

[0025] To suppress motion blur, it is conceivable not to perform automatic control of the exposure time. In this case, however, it is necessary to always set the exposure time shorter. As a result, including when the head-mounted display 100 is stationary, the detected value of light constantly becomes smaller, so it is necessary to increase the gain value, and as a result, the noise component increases. Such an increase in noise becomes prominent in a dark environment, and ultimately the accuracy of image analysis decreases. That is, in any case when a general imaging device is adopted, the analysis accuracy of the captured image, and by extension, the quality of the user experience using the head-mounted display 100, will depend on the surrounding environment.

[0026] Therefore, in the present embodiment, the movement of the imaging device is used as the basis for determining the exposure time. FIG. 3 shows the dependency of the parameters related to the processing of the captured image in the present embodiment. (a) shows the parameter chain in the present embodiment in the same format as FIG. 2. Compared with FIG. 2, in the present embodiment, the exposure time 64 is determined from the motion information 60 instead of the shooting environment, and the target luminance 62 is used in the determination of the gain value 66, which is different.

[0027] Specifically, the motion information 60 is acquired from the IMU sensor provided in the head-mounted display 100, and the exposure time 64 is determined based on it. Qualitatively, if the head-mounted display 100 is stationary or has a low-speed motion where motion blur is not a problem, the exposure time is lengthened, and if the motion is medium-speed or high-speed where motion blur becomes a problem, the exposure time is shortened. Hereinafter, the former state may be referred to as the "no movement" state, and the latter state may be referred to as the "movement" state.

[0028] Then, by determining the gain value 66 through comparison between the detected value of the light obtained by such exposure time control and the target luminance 62, the luminance of the entire image is suitably maintained. However, as will be described later, when switching the exposure time, the gain value 66 is determined so as to correspond to the change. Further, filter parameters 68 for noise reduction processing are determined so as to correspond to the gain value 66. Thus, in the present embodiment, the parameters of the processing of the captured image are controlled based on the motion information 60 of the head-mounted display 100 and, by extension, the imaging device.

[0029] Here, the motion information 60 may be at least either the angular velocity or the acceleration measured by the IMU sensor, or may be a simple detection result such as whether there is motion or not. Naturally, the more detailed the motion information is, the higher the accuracy and the degree of detail of the determined exposure time can be. For example, the occurrence of motion blur is more prominent in rotational motion than in translational motion. Therefore, when rotational motion occurs, fine control such as shortening the exposure time becomes possible.

[0030] (b) illustrates the relationship between the motion of the head-mounted display 100 and motion blur in the present embodiment. By shortening the exposure time in a state where there is motion as described above, motion blur can be suppressed to be smaller compared to the example in FIG. 2. Also, by increasing the gain value by the amount corresponding to the shortened exposure time, the luminance of the entire image can be maintained. For example, when the exposure time is switched to 1 / N, the gain value is set to N times the previous value. As a result, the noise component (SN ratio) increases but only temporarily, and since the influence of motion blur is suppressed, the accuracy of image analysis can be improved overall.

[0031] In this way, by ensuring an appropriate exposure time in the presence of motion, in the absence of motion, the exposure time can be made as long as possible, and a captured image with a small gain and low noise can be obtained. Naturally, in the absence of motion, no motion blur occurs, so the accuracy of image analysis can be improved due to the synergistic effect with the low noise. The figure also shows the changes in the shooting environment. In this embodiment, since the exposure time is determined based on the presence or absence of motion, the influence of the ambient brightness can be minimized.

[0032] Note that since the light detection value itself naturally changes depending on brightness, the absolute values of the gain value and filter parameters change. Qualitatively, in a dark place, the detection value is smaller than in a bright place, so a larger gain is applied and the noise component increases. However, as long as the head-mounted display 100 is not moving, the exposure time can be kept long, so the noise can be minimized.

[0033] When the captured image is only used for image analysis, there is no need to add elements that people feel beautiful when viewed later as described above. Rather, processing applied to the light detection result by the image sensor, such as contrast enhancement and super-resolution, changes the original image information or adds extra information, and can be an inhibitory factor from the perspective of the accuracy of image analysis. Therefore, in this embodiment, it is desirable to perform only noise reduction as the image processing after gain adjustment. Also, considering that the spatial resolution decreases due to spectroscopy when a color image is used, preferably a grayscale image with only the luminance magnitude as the pixel value is captured.

[0034] FIG. 4 shows the configuration of an information processing system to which the present embodiment can be applied. The information processing system 8 includes an imaging device 14, an image processing device 10, a motion measurement device 12, and an image analysis device 16, and generates information on the real world based at least on a captured image. Here, the information processing system 8 is incorporated in a movable body such as a head-mounted display 100. However, at least a part of the information processing system 8, for example, at least one of the image processing device 10 and the image analysis device 16, or a part of the image processing device 10 may be provided in another device that is communicatively connected to the movable body. Also, at least a part of the image processing device 10 may be included in the imaging device 14.

[0035] When applied to the head-mounted display 100 of FIG. 1, the imaging device 14 and the motion measurement device 12 correspond to the imaging device 110 and the built-in IMU sensor. Hereinafter, the information processing system 8 will be described as being mounted on the head-mounted display 100. The imaging device 14 includes an imaging unit 20 that outputs a two-dimensional distribution of charges (RAW image) obtained by photoelectrically converting incident light with an imaging element array such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) at a predetermined rate, and an exposure adjustment unit 18 that adjusts the exposure time.

[0036] The motion measurement device 12 is an IMU sensor including, for example, a gyro sensor and an acceleration sensor, and detects the posture, translational motion, and rotational motion of the head-mounted display 100 at a predetermined rate. The image analysis device 16 generates information on the real world, such as the position and posture of the head-mounted display 100 and information on surrounding objects, at a predetermined rate by using well-known techniques such as V-SLAM, object detection, and object recognition on the captured image. Preferably, the image analysis device 16 generates information on the real world with high accuracy by complementarily using the measured values of the motion measurement device 12 and the results of image analysis.

[0037] The image processing device 10 processes the RAW image before gain processing captured by the imaging device 14 and generates data of the captured image used for analysis by the image analysis device 16 at a predetermined rate. The image processing device 10 can be realized hardware-wise by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), various arithmetic units, and memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and software-wise by a program that exhibits various functions such as an information processing function, an image drawing function, a data input / output function, and a communication function, which is loaded from a recording medium into the memory. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof, and are not limited to any one of them.

[0038] The image processing device 10 includes a gain processing unit 22 that performs gain processing for amplifying the pixel values of the RAW image, an image processing unit 26 that performs necessary image processing such as noise reduction processing, and an output unit 28 that outputs the data after image processing to the image analysis device 16. The image processing device 10 also includes a luminance information acquisition unit 24 that acquires information related to the luminance of the entire image after gain processing, and a parameter control unit 30 that controls parameters used for shooting and image processing, such as the exposure time, gain value, and filter parameters.

[0039] Specifically, the gain processing unit 22 may be composed of an analog gain unit that applies gain to the analog value output by the imaging unit 20, an AD conversion unit that converts the data into a digital value, and a digital gain unit that applies gain to the digital value. The image processing unit 26 sequentially acquires the data of the image after gain adjustment and performs necessary image processing. As described above, in this embodiment, the image processing unit 26 preferably performs noise reduction processing and does not perform other processing.

[0040] The output unit 28 temporarily stores the data output from the image processing unit 26 in an internal buffer memory and then supplies it to the image analysis device 16. The image analysis device 16 may read the data from the buffer memory at a suitable timing. The luminance information acquisition unit 24 acquires the data of the image whose gain has been adjusted by the gain processing unit 22, generates information representing the luminance of the entire image such as the centroid of the luminance histogram, and supplies it to the parameter control unit 30. The parameter control unit 30 determines the exposure time, gain value, and filter parameters at a predetermined rate by the parameter chain shown in Fig. 3(a).

[0041] For example, the parameter control unit 30 acquires the measurement value by the motion measurement device 12 at a predetermined rate, and derives the change amount of the head-mounted display 100 in units of frame periods. Here, the change amount is, for example, the rotation angle around three axes, and is obtained by integrating the angular velocity over the frame period. Then, the parameter control unit 30 determines an appropriate exposure time based on the derived change amount per unit time, that is, the speed, and transmits a control signal for the exposure time to the exposure adjustment unit 18. For example, if the speed is below the threshold value at which the head-mounted display 100 can be regarded as stationary, the parameter control unit 30 sets the exposure time to a preset specified exposure time.

[0042] The exposure time is set to an upper limit value determined by the frame rate, or the longest time within a non-saturating range, etc. At this time, the parameter control unit 30 may adjust the exposure time according to the ambient brightness. Then, when the speed of the head-mounted display 100 exceeds the threshold value, the parameter control unit 30 shortens the exposure time, and when the speed drops below the threshold value again, the parameter control unit 30 returns the exposure time to its original value. Note that the threshold value of the speed serving as the trigger for switching the exposure time may be the same or different in the increasing direction and decreasing direction of the speed. By making the threshold values different, it is possible to avoid the exposure time being switched in a short time.

[0043] Alternatively, a plurality of thresholds may be provided for the speed, and the parameter control unit 30 may stepwise change the exposure time such that the exposure time becomes shorter as the speed increases. The exposure adjustment unit 18 adjusts the exposure time of the imaging unit 20 according to the control signal from the parameter control unit 30. General AE technology can be used for the adjustment process itself.

[0044] The parameter control unit 30 also determines a gain value for realizing the target luminance by comparing the luminance information of the entire image supplied from the luminance information acquisition unit 24 with the target luminance held internally. On the other hand, when the parameter control unit 30 changes the exposure time due to the presence or absence of motion or a change in speed, it immediately changes the gain value so as to correspond to the change. This suppresses the adverse effects such that frames with insufficient luminance continue due to shortening of the exposure time, or frames with large noise due to excessive gain continue even though the exposure time has been restored.

[0045] The parameter control unit 30 transmits a control signal to the gain processing unit 22 so that gain adjustment is performed with the determined value. The gain processing unit 22 performs gain processing on the RAW image supplied from the imaging unit 20 with a value corresponding to the control signal. The parameter control unit 30 may set gain values for the above-described analog gain unit and digital gain unit, respectively.

[0046] Furthermore, the parameter control unit 30 estimates the noise that may occur according to the magnitude of the gain, and determines parameters of a spatial filter or a temporal filter corresponding to the amount. The parameter control unit 30 transmits a control signal to the image processing unit 26 so that noise reduction processing is performed with the determined parameters. The image processing unit 26 performs noise reduction processing on the image after gain adjustment by filter processing corresponding to the control signal, and outputs it to the output unit 28.

[0047] Next, from the perspective of responsiveness to exposure time control, a general imaging device is compared with the present embodiment. FIG. 5 illustrates the relationship between exposure control and image changes in a general imaging device. The horizontal axis of the figure represents the passage of time, and each frame is captured in synchronization with the vertical synchronization signal at the uppermost stage. Here, the vertical rectangular pairs shown in "shooting environment" and "output image" represent the shooting environment and the state of the output image of the same frame among frame numbers #1 to #8.

[0048] Also, the "exposure time / gain" in the third row is the exposure time and gain value set when each frame is captured, and the "reference destination luminance centroid" in the fifth row is the centroid of the luminance histogram that is the basis for setting the exposure time and gain of each frame. In this example, first, assume that the frame #1 is captured in a bright environment, and the frames after #2 are captured in a dark environment.

[0049] In this case, in the frame #1 (and the frames before it), the exposure time and gain are suppressed so that the captured image does not become too bright, and for example, they are set to the minimum value (Min). As a result, a bright and suitable image is output such that the centroid of the luminance histogram becomes the target luminance (Target). Here, even if the surroundings become dark during the shooting period of the subsequent frame #2, the exposure time and gain are not immediately adjusted. This is because at the shooting timing of the frame #2, the luminance centroid (S10) of the immediately preceding frame #1 is used as the basis for control.

[0050] As a result, the shooting is performed with the exposure time and gain remaining at the minimum value, and the output image of the frame #2 becomes quite dark. The imaging device detects the deviation from the target luminance, for example, by referring to the luminance centroid (S12) of the frame #2 at the shooting time of the frame #3, and increases the exposure time and gain at the shooting timing of the frame #4 (S14). As a result, in the frame #4, the darkness of the image is somewhat improved (S16).

[0051] Thereafter, the exposure time and gain of the frame #6 are adjusted with reference to the shooting time of the frame #5 for the luminance center of gravity of the frame #4, and the exposure time and gain of the frame #8 are adjusted with reference to the shooting time of the frame #7 for the luminance center of gravity of the frame #6, and so on (S18). Finally, it converges to an exposure time and gain suitable for a dark environment, and an image with a luminance center of gravity corresponding to the target luminance is output.

[0052] According to such general automatic exposure control, the luminance center of gravity does not reach the target luminance at once, and the exposure time and gain are adjusted over a period of several frames by convolution processing or the like. This prevents the occurrence of blinks in which the output image becomes dark or bright due to the influence of noise or lighting, and makes it a gentle change that does not feel uncomfortable even when a person looks through the viewfinder.

[0053] FIG. 6 illustrates the relationship between the control of the exposure time and the change of the image in the image processing apparatus 10 of the present embodiment. The manner of showing the figure is the same as that of FIG. 5. However, in the present embodiment, basically, since the exposure time is controlled based on the presence or absence of the movement of the head-mounted display 100, the second stage shows the presence or absence of "movement" instead of "shooting environment". Also, the "output image" in the fifth stage indicates the degree of motion blur. The "exposure time" in the third stage and the "gain" in the fourth stage are the exposure time and gain values set when each frame is shot, respectively.

[0054] In this example, it is assumed that the head-mounted display 100 is stationary at the shooting time of the frame #1, and the head-mounted display 100 is moving after the frame #2. In this case, the parameter control unit 30 of the image processing apparatus 10 sets the exposure time to the maximum value (Max) in the frame #1 and suppresses the gain, thereby outputting a bright image with little noise suitable for image analysis. Even in this case, since the head-mounted display 100 is stationary, no motion blur occurs in the output image.

[0055] When the parameter control unit 30 of the image processing apparatus 10 detects the occurrence of motion, such as the speed of the head-mounted display 100 exceeding the threshold value at the shooting time of the frame of #2, it shortens the exposure time at the shooting timing of the frame of #3 (S20). At the same time, the gain value is increased so as to compensate for the decrease in the amount of light due to the shortening of the exposure time (S22). At this time, convolution processing or the like is not performed as in the control of the exposure time and gain by a general imaging device shown in FIG. 5, and the target value is immediately reached.

[0056] As a result, although a large motion blur appears in the output image of the frame of #2 taken at the time of the occurrence of motion, it is immediately suppressed in the frame of #3, and an output image with little overall change in brightness is obtained. The parameter control unit 30 switches the exposure time and the gain value to the target values from the immediately subsequent frame when the motion of the head-mounted display 100 satisfies the condition of changing the exposure time due to a speed change or the like, regardless of the presence or absence of motion. For this reason, the parameter control unit 30 holds in the internal memory a table associating the presence or absence of motion and the speed range with the target values of the exposure time and the gain value.

[0057] Also, as described above, when the head-mounted display 100 becomes stationary again, the parameter control unit 30 immediately returns the exposure time and the gain value to their original values. By these controls, the situation where the accuracy of image analysis deteriorates can be suppressed to a minimum period. In a situation where the output image is not viewed by a person, the accuracy of image analysis can be maintained without any problem in this way. When it is necessary to display the captured image, the head-mounted display 100 may be equipped with an imaging device for display in addition to the imaging device for image analysis.

[0058] FIG. 7 compares the shooting by a conventional imaging device and the shooting by the information processing system 8 of the present embodiment in terms of actual output images. (a) is an image shot by a conventional imaging device. Since the imaging device itself moves, motion blur occurs and the outline of the ceiling light, which is the subject, is blurred. On the other hand, in the captured image according to the present embodiment shown in (b), even in the same situation, by suppressing the exposure time and increasing the gain, motion blur is suppressed and an image with unchanged overall brightness can be obtained.

[0059] According to the present embodiment described above, the exposure time of the imaging device is controlled based on the motion information of the imaging device or the moving body on which it is mounted. Specifically, when moving at a high speed such that motion blur becomes a problem, the degree of motion blur is reduced by shortening the exposure time. Thereby, even in a situation where the imaging device moves, an image with less motion blur can be captured, and a decrease in the accuracy of image analysis can be minimized.

[0060] Also, by ensuring such control of the exposure time, the exposure time during the period when the imaging device is stationary can be maximally lengthened, and an image with high brightness and low noise can be obtained. Since noise can be suppressed even in a dark place, the accuracy of image analysis at rest can be stably maintained regardless of the surrounding environment. Furthermore, by not performing image processing assuming human appreciation, the original information of the real world can be accurately acquired and the delay time until output can be reduced.

[0061] Also, by immediately changing the exposure time and the gain value to the target values according to the change in the presence or absence of motion, the period during which motion blur or a high signal-to-noise ratio occurs, and thus the period during which the analysis accuracy is likely to deteriorate, can be minimized. In the present embodiment, the motion information uses the measured values of a motion sensor provided in the moving body. In recent years, since electronic devices equipped with an imaging device acquire postures and movements for use in information processing, many cases also incorporate a motion sensor. By using this, the present embodiment can be easily introduced without being affected by cost and weight.

[0062] The above has described the present invention based on embodiments. It is understood by those skilled in the art that the above embodiments are examples, and various modifications are possible for each of these components and combinations of each processing process, and such modifications are also within the scope of the present invention.

Description of Reference Numerals

[0063] 8 Information processing system, 10 Image processing device, 12 Motion measurement device, 14 Imaging device, 16 Image analysis device, 18 Exposure adjustment unit, 20 Imaging unit, 22 Gain processing unit, 24 Luminance information acquisition unit, 26 Image processing unit, 28 Output unit, 30 Parameter control unit, 100 Head-mounted display, 110 Imaging device.

Claims

1. A parameter control unit that acquires motion information of an imaging device that captures a video at a predetermined rate, determines an exposure time based on the motion information, and determines a gain value by comparing a detection value of light obtained by the exposure time with a predetermined target luminance; A gain processing unit that performs gain processing on an image captured by the imaging device at the exposure time with the gain value; An output unit that outputs data of the image subjected to the gain processing; An image processing apparatus comprising the above.

2. The image processing apparatus according to claim 1, wherein the parameter control unit shortens the exposure time when the speed of the imaging device exceeds a threshold value.

3. The image processing apparatus according to claim 1 or 2, wherein the parameter control unit shortens the exposure time as the speed of the imaging device increases.

4. The image processing apparatus according to any one of claims 1 to 3, wherein the parameter control unit sets the exposure time to an upper limit value when a condition that the imaging device can be regarded as stationary is satisfied.

5. The image processing apparatus according to any one of claims 1 to 4, wherein the parameter control unit switches the exposure time and the gain value to target values from the immediately subsequent frame when the motion information satisfies a condition for changing the exposure time.

6. Further comprising an image processing unit that performs noise reduction processing on the image subjected to the gain processing, The image processing apparatus according to any one of claims 1 to 5, wherein the parameter control unit controls filter parameters used for noise reduction based on the gain value.

7. The image processing apparatus according to claim 6, wherein the image processing unit supplies data of an image obtained by performing only the noise reduction processing on the image subjected to the gain processing to the output unit, and the output unit outputs the supplied data to an image analysis apparatus that generates information related to the real world at a predetermined rate by image analysis.

8. An image processing apparatus according to any one of claims 1 to 7; The imaging device; A motion measurement device that measures the motion information at a predetermined rate; An information processing system comprising the above.

9. A step of acquiring motion information of an imaging device that captures a video at a predetermined rate; A step of determining an exposure time based on the motion information; A step of determining a gain value by comparing a detection value of light obtained by the exposure time with a predetermined target luminance; A step of outputting data of an image obtained by performing gain processing on the image captured by the imaging device at the exposure time with the gain value; An image acquisition method characterized by including the above.

10. A function of acquiring motion information of an imaging device that captures a moving image at a predetermined rate; A function of determining an exposure time based on the motion information; A function of determining a gain value by comparing a detection value of light obtained by the exposure time with a predetermined target luminance; A function of outputting data of an image obtained by performing gain processing on the image captured by the imaging device at the exposure time with the gain value; A computer program characterized by causing a computer to realize the above.

Images