Information processing apparatus, information processing system, information processing method, and program

Abstract

To enable an apparatus for requiring distance information to acquire highly accurate distance information at a higher speed.SOLUTION: An information processing apparatus has: acquisition means for acquiring a first image and a second image having parallax to each other; first operation means for calculating distance information of a first range of the first image by a first calculation based on the first image and the second image; first transmission means for transmitting a second range included in the first range of the first image and smaller than the first range and a third range of the second image to an external device; receiving means for receiving distance information of the second range calculated by a second calculation different from the first calculation based on the second range and the third range in the external device; and correction means for correcting the distance information of the first range calculated by the first calculation means based on the distance information of the second range received by the receiving means.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] Conventionally, MR (Mixed Reality) systems obtain distance information to the object based on images captured by a camera. This distance information is used to generate a 3D map of the captured space and to calculate the seamless positional relationship between the hand of the user using the MR system and the CG in the virtual space.

[0003] Generally, in distance calculation (depth calculation) to a photographed object, which is typified by stereo processing, the distance to a subject present in a stereo image photographed in real space by a pair of stereo cameras is calculated by triangulation. Triangulation is a method of detecting the amount of positional shift of the image of the subject photographed by the left and right cameras, and calculating the distance between the camera and the subject based on the amount of positional shift. This amount of positional shift is inversely proportional to the distance from the camera to the subject. Therefore, it is possible to calculate the distance to the subject by identifying the amount of positional shift, which is the difference between the positions of the same subject in two images.

[0004] In Patent Document 1, the depth information processing device acquires distance information by block matching of two images captured by two cameras. If there is a pixel block for which it is determined that acquisition of distance information has failed, the depth information processing device sets the block size to a smaller value and performs the same distance calculation (depth calculation) again. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-140344 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, as in Patent Document 1, when highly accurate depth calculation is realized by the hardware of a depth information processing device, there is a possibility that implementation or modification of the depth information processing device may be hindered due to a lack of memory or computing unit resources. On the other hand, when highly accurate depth calculation is realized using a server machine on a cloud that has abundant computing resources and is capable of high-speed calculation, there may be a restriction on the transmission bandwidth on the network. In this case, a large delay occurs before a device that actually uses the result of the depth calculation obtains the result of the depth calculation, and the performance of the entire system decreases.

[0007] Therefore, an object of the present invention is to enable a device requiring distance information to obtain highly accurate distance information more quickly. [Means for solving the problem]

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: An acquisition means for acquiring a first image and a second image having a parallax with respect to each other; a first calculation means for calculating distance information of a first range of the first image by a first calculation based on the first image and the second image; a first transmission means for transmitting a second range included in the first range of the first image and smaller than the first range, and a third range of the second image to an external device; The second range and the second range are obtained in the external device by a second calculation different from the first calculation. a receiving means for receiving distance information of the second range calculated based on the third range; a correction means for correcting the distance information of the first range calculated by the first calculation means, based on the distance information of the second range received by the receiving means; The information processing device is characterized by having:

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: An acquisition step of acquiring a first image and a second image having a parallax with respect to each other; a first calculation step of calculating distance information of a first range of the first image by a first calculation based on the first image and the second image; a first transmission step of transmitting a second range included in the first range of the first image and smaller than the first range, and a third range of the second image to an external device; a receiving step of receiving distance information of the second range calculated in the external device based on the second range and the third range by a second calculation different from the first calculation; This information processing method is characterized by having a correction step of correcting the distance information of the first range calculated in the first calculation step based on the distance information of the second range received in the receiving step. Effect of the Invention

[0010] According to the present invention, a device requiring distance information can obtain highly accurate distance information more quickly. [Brief description of the drawings]

[0011] [Figure 1A] 1 is a block diagram of an imaging and display system according to a first embodiment. [Figure 1B] 1 is an external view of an imaging and display system according to a first embodiment. [Diagram 2] 3 is an internal configuration diagram of a position detection unit according to the first embodiment. FIG. [Diagram 3] 5A to 5C are diagrams illustrating calculation of distance information by a SAD according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining re-search according to the first embodiment. [Diagram 5] 4 is a flowchart of processing performed by the image processing device according to the first embodiment. [Figure 6] FIG. 11 is a diagram showing the internal configuration of a position detection unit according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and all of the combinations of features described in the embodiments are not necessarily essential to the solution of the invention.

[0013] <Embodiment 1> An imaging and display system 11 (information processing system) according to embodiment 1 will be described below. Fig. 1A and Fig. 1B are diagrams showing the imaging and display system 11 according to embodiment 1. Fig. 1A is a block diagram of the imaging and display system 11. Fig. 1B is a schematic diagram showing the external appearance of the imaging and display system 11.

[0014] 1B, the imaging display system 11 includes a head-mounted imaging display device (hereinafter referred to as "HMD") 10, a cable 20, and an image processing device (PCWS: PC WorkStation) 30. The image processing device 30 generates an image (hereinafter referred to as "synthetic image") of a mixed reality (MR) space in which real space and virtual space are fused together, and provides the image to the HMD 10. Note that although the cable 20 is illustrated as a communication path for a wired connection, a communication path for a wireless connection may be used instead of the cable 20.

[0015] (HMD configuration) The configuration of the HMD 10 will be described. The HMD 10 includes a control unit 100, two display elements 1 The display device 110 has a display element 10 (110L, 110R), two background imaging elements 130 (130L, 130R), two position imaging elements 140 (140L, 140R), and two gaze detection units 150 (150L, ​​150R). Here, the display element 110L, the background imaging element 130L, the position imaging element 140L, and the gaze detection unit 150L are used to control an image for the left eye. The display element 110R, the background imaging element 130R, the position imaging element 140R, and the gaze detection unit 150R are used to control an image for the left eye.

[0016] The control unit 100 performs necessary image processing on the image. The control unit 100 also controls each device and cooperates with the image processing device 30.

[0017] The display element 110L and the display element 110R use an OLED (Organic Light-Emitting Diode) or an LCD (Liquid Crystal Display), etc. The display element 110L presents a composite image for the left eye (hereinafter referred to as a "left composite image") to the user through an optical system (not shown). The display element 110R presents a composite image for the right eye (hereinafter referred to as a "right composite image") to the user through an optical system (not shown). Through such processing, the imaging display system 11 allows the user to perceive a stereoscopic image (mixed reality space) based on the right composite image and the left composite image. Technologies related to imaging and displaying stereoscopic images are known and will not be described here.

[0018] A CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor is used as each of the two background imaging elements 130 and the two position imaging elements 140. Each of these imaging elements captures an image of the real space according to an exposure time, a sensor gain, an exposure start timing, and the like that are set based on a control signal from the control unit 100, thereby acquiring a captured image.

[0019] The background imaging element 130L captures a background image for the left eye (hereinafter referred to as a "left background image") by imaging the real space. The background imaging element 130R captures a background image for the right eye (hereinafter referred to as a "right background image") by imaging the real space. The position imaging element 140L and the position imaging element 140R each capture a position image, which is an image used to generate a CG (Computer Graphics) image, which will be described later, by imaging the real space. The image captured by the position imaging element 140L is called a "left position image," and the image captured by the position imaging element 140R is called a "right position image." The left position image and the right position image have a parallax from each other.

[0020] Note that the background image and the position image have different requirements (angle of view, resolution, image processing, etc.). For this reason, the background image and the position image may be captured by different imaging elements as shown in Fig. 1A. Alternatively, one of the background image and the position image may be a captured image captured by a single imaging element, and the other image may be cut out from that captured image.

[0021] The position image sensor 140L and the position image sensor 140R are arranged side by side in the horizontal direction when a user facing forward wears the HMD 10 on his / her head. The gaze detection unit 150L and the gaze detection unit 150R are pupil photographing cameras that capture an image of the user's pupil (pupil image) in order to detect the gaze position (the position where the user is looking) of the user wearing the HMD 10. A known method can be used to detect the gaze position based on the pupil image, so a detailed description will be omitted.

[0022] (Configuration of image processing device) The image processing device 30 includes a position detection unit 300, a CG generation unit 310, a synthesis unit 320, and a network It has a work portion 340.

[0023] The position detection unit 300 acquires a left position image and a right position image from the HMD 10. The position detection unit 300 generates a distance image indicating the distance between the subject and the HMD 10 (camera) (the distance in the captured real space) based on the left position image and the right position image.

[0024] Furthermore, the position detection unit 300 calculates information (such as the user's position and user's orientation) necessary for generating a CG image of the virtual space to be displayed to the user based on the distance image. Known techniques can be used to calculate the user's position and user's orientation, so detailed explanations will be omitted.

[0025] In general, a known pattern arranged in real space is photographed by a camera, and a three-dimensional space with the pattern as the origin coordinate is newly defined. In the newly defined three-dimensional space, a numerical value indicating the user's position and the user's orientation is calculated. Then, the known pattern is photographed again, and at the same time, an arbitrary feature point is acquired from the image acquired by the camera, and the coordinates of the feature point in the three-dimensional space are obtained. Based on the positional relationship of the feature points and the distance from the camera to the feature point, information on the position and orientation of the camera in the three-dimensional space is acquired. This information on the position and orientation of the camera is used as user position and direction information in the three-dimensional space. Even if the user moves, information on the current position and orientation of the camera (i.e., information on the user's position and direction in the updated three-dimensional space) can be acquired based on changes in the position and distance of the feature point on the image (the distance from the camera to the feature point).

[0026] The CG generation unit 310 generates a CG image for the left eye (hereinafter referred to as a "left CG image") and a CG image for the right eye (hereinafter referred to as a "right CG image") based on information such as the user's position and the user's orientation. Rendering is performed based on CAD data (not shown) stored in the 3D model (such as a 3D model) or on a computer-readable medium (such as a hard disk).

[0027] The synthesis unit 320 generates a left synthesis image by superimposing the left CG image on the left background image. The synthesis unit 320 generates a right synthesis image by superimposing the right CG image on the right background image. The synthesis unit 320 outputs the left synthesis image and the right synthesis image to the HMD 10.

[0028] The network unit 340 is a distributed system for reducing the processing load of the position detection unit 300. In general, the position detection performed by the position detection unit 300 imposes a high processing load. For this reason, the network unit 340 performs part of the calculations that should be performed by the position detection unit 300, and notifies the position detection unit 300 of the calculation results. This realizes a reduction in the processing load of the position detection unit 300. In the first embodiment, depending on the status of the imaging display system 11, the network unit 340 also executes part of the calculations that should be performed by the position detection unit 300.

[0029] In the first embodiment, the network unit 340 calculates distance information of the image transmitted from the position detection unit 300 in response to an instruction from the position detection unit 300. The network unit 340 transmits the calculated distance information to the position detection unit 300.

[0030] The network unit 340 and the position detection unit 300 are connected to each other via a network. Therefore, in the image processing device 30, it can also be understood that "an information processing device having the position detection unit 300, the CG generation unit 310, and the synthesis unit 320" is connected to "an external device having the network unit 340." The network may be realized either wirelessly or wired. In the first embodiment, the network is described as being realized wirelessly.

[0031] (Internal structure of the position detection unit) The internal configuration of the position detection unit 300 will be described with reference to Fig. 2. The position detection unit 300 includes a distance image generation unit 301 and a position determination unit 307.

[0032] The distance image generation unit 301 acquires the left position image and the right position image via the cable 20. The distance image generation unit 301 generates a distance image based on the left position image and the right position image, and outputs the distance image to the position determination unit 307.

[0033] The details of distance image generation by distance image generation unit 301 will be described later with reference to FIG. 3A and subsequent figures. A "distance image" is an image that expresses the distance to a subject by color shading or color saturation. Each pixel in the distance image according to the first embodiment has a value (pixel value) that is a discretized integer value of "1" to "255" that expresses shading. Here, the pixel value is smaller as the distance from the camera increases. Note that a pixel value of "0" is a special value that indicates that it was not possible to calculate the distance. The screen resolution of the distance image is the same as that of the left position image and the right position image.

[0034] The position determination unit 307 calculates the position (three-dimensional position) and direction of the user wearing the HMD 10 based on the distance image, and transmits the information on the user's position and direction to the CG generation unit 310. A description of the method for calculating the user's position and direction based on the distance information will be omitted.

[0035] (Internal configuration of distance image generation unit) Next, a detailed description will be given of the internal configuration of distance image generation unit 301. Distance image generation unit 301 has a memory unit 302A, a memory unit 302B, a first depth calculation unit 303A, a transmission unit 304A, a reception unit 304B, an invalidation determination unit 305, and an execution determination unit 306.

[0036] The memory unit 302A has a RAM (Random Access Memory) which is a readable and writable storage device. The memory unit 302A stores the left position image and the right position image received from the cable 20 as frame images. When the memory unit 302A completes receiving one frame of data, it transmits the frame image to the first depth calculation unit 303A in accordance with a read command from the first depth calculation unit 303A. The memory unit 302A can also transmit an image within a range designated by the execution determination unit 306 (described later) to the transmission unit 304A.

[0037] The memory unit 302B has a RAM like the memory unit 302A. The memory unit 302B is used as a storage location for a distance image corresponding to "distance information calculated by the first depth calculation unit 303A and the network unit 340." The memory unit 302B operates as a correction unit that acquires a distance image by correcting the distance information calculated by the first depth calculation unit 303A based on the distance information calculated by the network unit 340. The vertical and horizontal sizes of the distance image are the same as those of the left position image or the right position image.

[0038] The first depth calculation unit 303A acquires a left position image and a right position image having a parallax with each other. The first depth calculation unit 303A defines either the left position image or the right position image as a "base image" and the other as a "reference image". The first depth calculation unit 303A may select either the left position image or the right position image as a base image. In the first embodiment, it is assumed that the first depth calculation unit 303A selects the left position image as a "base image" and the right position image as a "reference image". The base image and the reference image are composed of a set of a plurality of pixels (pixel blocks). Therefore, hereinafter, a pixel block (a set of pixels) in the base image is called a "base block", and a pixel block in the reference image is called a "reference block".

[0039] Then, the first depth calculation unit 303A executes a block matching process (first calculation) on a standard block (a set of pixels) in which a specific object appears, thereby determining the position of a reference block in which the specific object appears. The block matching process of the first depth calculation unit 303A is a process of calculating an evaluation value indicating the dissimilarity between two pixel blocks using a known method, SAD (Sum of Absolute Differences). In SAD, the sum of absolute values ​​of the differences in luminance between corresponding pixels in the standard block and the reference block is calculated as an evaluation value (SAD value) of the reference block. The lower the SAD value, the higher the similarity between the two pixel blocks. For this reason, the first depth calculation unit 303A finds a reference block with the highest evaluation value (i.e., the smallest SAD value), and uses the difference in position between this reference block and the standard block as distance information of the standard block. The difference in position between the standard block and the reference block, which are in a corresponding relationship to each other, is inversely proportional to the distance from the imaging element (camera) to the object appearing in the standard block. Therefore, "calculating the difference in position between the standard block and the reference block" is equivalent to "calculating the distance from the imaging element to the subject."

[0040] In addition, there may be a case where there are multiple reference blocks that have the smallest SAD value in relation to the base block. In this case, the first depth calculation unit 303A determines that accurate distance information has not been acquired (distance information has not been calculated) and sets the distance information of the base block to "0."

[0041] The first depth calculation unit 303A may obtain the SAD values ​​of all reference blocks on the epipolar line, or may obtain the SAD values ​​limited to reference blocks in a predetermined range on the epipolar line. Hereinafter, "obtaining the coordinates of the reference block with the highest evaluation value in relation to the base block" is referred to as "searching". The size of the pixel block may be 1 pixel x 1 pixel (i.e., 1 pixel) or more. However, the larger the size of the pixel block, the lower the resolution of the distance image. In the first embodiment, the size of the pixel block is assumed to be 3 pixels x 3 pixels.

[0042] The first depth calculation unit 303A receives a base block for which distance information is to be obtained from the memory unit 302A, and also receives a plurality of reference blocks used in block matching of the base block. The first depth calculation unit 303A calculates distance information for each base block and stores it in the memory unit 302B. The first depth calculation unit 303A also stores the minimum value of the SAD values ​​used to calculate the distance information in the memory unit 302B.

[0043] The invalid determination unit 305 monitors the memory unit 302B. Then, the invalid determination unit 305 judges whether or not the distance information of each reference block is "0". The invalid determination unit 305 sets (determines) the reference block whose distance information is "0" as an "invalid reference block".

[0044] When the execution determination unit 306 acquires information (information of center coordinates) of the invalid reference block from the invalid determination unit 305, the execution determination unit 306 controls the network unit 340 to calculate distance information of the invalid reference block. Specifically, the execution determination unit 306 instructs the memory unit 302A to transmit the invalid reference block and a plurality of reference blocks used for block matching of the invalid reference block to the transmission unit 304A.

[0045] The transmitting unit 304A is an interface connected to the memory unit 302A. The transmitting unit 304A acquires a plurality of pixel blocks (an invalid base block and a plurality of reference blocks) transmitted by the memory unit 302A. The transmitting unit 304A transmits the acquired plurality of pixel blocks to the receiving unit 344C of the network unit 340 connected by wireless communication. The transmitting unit 304A performs communication using a communication method such as wireless LAN (Local Area Network) using an antenna (not shown) for wireless transmission or a modulation / demodulation circuit.

[0046] The receiving unit 304B is an interface that receives distance information of a partial range of the reference image from the network unit 340 connected by wireless communication. The receiving unit 304B transmits the received distance information to the memory unit 302B. The memory unit 302B merges the distance information of the partial range of the reference image with the distance information calculated by the first depth calculation unit 303A. That is, the memory unit 302B corrects the distance information calculated by the first depth calculation unit 303A based on the distance information of the partial range of the reference image. A distance image indicating the corrected distance information is transmitted from the memory unit 302B to the position determination unit 307.

[0047] The first depth calculation unit 303A, the invalidation determination unit 305, and the execution determination unit 306 are configured with specific hardware. The specific hardware is a field programmable gate array (FPGA) that is a rewritable logic circuit, or an application specific integrated circuit (ASIC), etc.

[0048] (Internal configuration of the network section) The internal configuration of the network section 340 will be described with reference to Fig. 2. The network section 340 has a second depth calculation section 343B, a receiving section 344C, and a transmitting section 344D.

[0049] The receiving unit 344C is an interface that receives pixel blocks from the transmitting unit 304A of the distance image generating unit 301. The receiving unit 344C transmits the received pixel blocks to the second depth calculation unit 343B.

[0050] The second depth calculation unit 343B acquires an invalid reference block and a plurality of reference blocks for which block matching with the invalid reference image is performed from the receiving unit 344C. The second depth calculation unit 343B calculates distance information of the invalid reference block by block matching (second calculation) using the acquired reference block and the plurality of reference blocks. The first depth calculation unit 303A employs a method of calculating corresponding points of the same subject by SAD values ​​(first depth calculation method) as a method of calculating distance information. The second depth calculation unit 343B employs a method of calculating distance information (second depth calculation method) different from that of the first depth calculation unit 303A.

[0051] In the first embodiment, the second depth calculation unit 343B performs block matching using SGM (Semi Global Matching) so as to realize calculation with higher accuracy. In SAD, search is possible only in the horizontal axis direction, but in SGM, the search range can be expanded to the vertical axis direction and diagonal directions. In addition, in SGM, the final distance information can be obtained by referring to the distance information of pixel blocks surrounding the pixel block for which distance information is to be calculated. Therefore, when calculating distance information of one pixel block (one range), the distance information is expected to be more accurate in SGM compared to SAD, but the calculation cost (number of processing steps) increases. For example, SGM requires a larger calculation hardware resource.

[0052] A delay in the calculation of distance information of the entire reference image by the position detection unit 300 leads to a deterioration in the quality of the user experience. For this reason, it is necessary to implement the position detection unit 300 in an edge device close to the user so as to prevent communication delays between devices. On the other hand, if distance information is calculated only for a part of the reference image, it is considered that transmission delays due to wireless communication are sufficiently small. For this reason, the network unit 340 can also be implemented using high-speed hardware resources (external device; equipment) on the cloud. In other words, the second depth calculation unit 343B arranged in an external device (equipment) on the cloud can calculate distance information of a part of the image and transmit the calculation result to the position detection unit 300 by wireless communication.

[0053] The transmission unit 344D is an interface. The transmission unit 344D acquires distance information of the invalid reference blocks from the second depth calculation unit 343B. The transmission unit 344D transmits the acquired distance information to the reception unit 304B of the distance image generation unit 301.

[0054] 3A to 3C are conceptual diagrams for explaining calculation of distance information by the first depth calculation unit 303A. Fig. 3A shows a standard image which is the left position image 21 and a reference image which is the right position image 31. In this example, the reference block having the highest evaluation value in the reference image (right position image 31) is calculated based on the SAD value in relation to a standard block 401 of the left position image 21 which is the standard image.

[0055] In the first embodiment, since the position imaging element 140L and the position imaging element 140R are arranged side by side in the horizontal direction, the epipolar line 402 extends in the horizontal direction. The coordinate values ​​of the right position image 31 are determined based on the position of the reference block in the left position image 21 from which distance information is calculated. Therefore, if the horizontal width of the left position image 21 and the right position image 31 is 1000 pixels and the center of the reference block 401 is the 400th pixel from the left end, a numerical value of -400 to +600 is assigned as the coordinate value of the right position image 31, which is the reference image.

[0056] The first depth calculation unit 303A calculates distance information by determining the position of the reference block with the minimum SAD value in the region 405 of the reference image on the epipolar line 402. For example, as shown in Fig. 3A, if the reference block with the minimum SAD value is reference block 403 centered on the 500th pixel from the left of the reference image, the distance information is calculated as "100" (=500-400).

[0057] 3B is a diagram showing an example of the SAD value of each reference block in the region 405 of the reference image in relation to the reference block 401. In the example shown in FIG. 3B, there is one peak with the smallest SAD value and no other peaks, so the distance information of the reference block 401 is calculated as "100". The first depth calculation unit 303A stores the calculated distance information in the memory unit 302B. At the same time, the first depth calculation unit 303A also stores the smallest SAD value in the memory unit 302B.

[0058] FIG. 3C is a diagram showing another example of the SAD value of each reference block in the region 405 of the reference image in relation to the reference block 401. In the example shown in FIG. 3C, there are two peaks with the smallest SAD value, and the values ​​are substantially the same, so it is not possible to determine whether the distance information of the reference block 401 should be set to "-180" or "+115". In such a case, in the first embodiment, the first depth calculation unit 303A sets "0" as the distance information of the reference block 401 and stores the distance information in the memory unit 302B. Note that the first depth calculation unit 303A may set "-180" or "115" as the distance information of the reference block 401. The first depth calculation unit 303A also stores the smallest SAD value in the memory unit 302B. In this case, the invalidation determination unit 305 transmits information on the center coordinates of the reference block 401, which is an invalid reference block, to the execution determination unit 306.

[0059] 4A to 4C are conceptual diagrams illustrating the operations of execution determining section 306 and second depth calculating section 343B.

[0060] The execution determination unit 306 receives information on the center coordinates of the invalid base block from the invalid determination unit 305 (FIG. 4A). Then, the execution determination unit 306 determines the invalid base block and a plurality of reference blocks (hereinafter referred to as a "search block group") for which block matching with the invalid base block is performed in the second depth calculation unit 343B. The memory unit 302A reads out the invalid base block and the search block group from the frame memory and outputs them to the second depth calculation unit 343B. Send to 343B.

[0061] The second depth calculation unit 343B calculates the distance information of the invalid reference block by SGM. The search block group can be a plurality of reference blocks located vertically, horizontally, or diagonally around a position corresponding to the position of the invalid reference block. In the first embodiment, as shown in FIG. 4B, a plurality of reference blocks located vertically or horizontally around a position corresponding to the position of the invalid reference block are the search block group.

[0062] As shown in Fig. 4C, the second depth calculation unit 343B performs block matching by SGM based on the invalid reference block and the search block group. As a result, the second depth calculation unit 343B calculates distance information of the invalid reference block. The memory unit 302B stores (overwrites and stores) the calculated distance information of the invalid reference block.

[0063] The processing of distance image generation unit 301 and network unit 340 in embodiment 1 will be described with reference to the flowchart in Fig. 5. In the following description, it is assumed that there are N pixel blocks in each of the standard image and the reference image. The N pixel blocks are arranged in a matrix of M vertically and M horizontally.

[0064] In step S1001, first depth calculation unit 303A calculates distance information for each of N reference blocks. Here, the reference block for which distance information is to be calculated is referred to as a "target block."

[0065] The first depth calculation unit 303A calculates the SAD value of each of M reference blocks arranged in the horizontal direction of the target block between the target block and the M reference blocks. The first depth calculation unit 303A detects the minimum value (hereinafter referred to as the "minimum SAD value") among the SAD values ​​of the M reference blocks in relation to the target block. The first depth calculation unit 303A calculates the coordinate value of the reference block corresponding to the minimum SAD value as distance information of the target block. Furthermore, if there are two or more reference blocks corresponding to the minimum SAD value, the first depth calculation unit 303A cannot calculate the distance information of the target block, and sets the distance information to "0". The first depth calculation unit 303A stores the minimum SAD value of the target block and the distance information together in the memory unit 302B.

[0066] In step S1001, through such processing, the first depth calculation unit 303A obtains distance information for each of the N reference blocks.

[0067] In step S1002, the invalid determination unit 305 determines a reference block whose distance information indicates "0" among the N reference blocks. Here, a reference block whose distance information indicates "0" is a reference block in which there are multiple distances corresponding to the minimum SAD value. If there is no reference block whose distance information indicates "0", the process of this flowchart ends. If there is a reference block whose distance information indicates "0" (an invalid reference block), the invalid determination unit 305 transmits information on the center coordinates of the invalid reference block to the execution determination unit 306. Then, the process proceeds to step S1003. In the following, it is assumed that the number of invalid reference blocks is "P".

[0068] 5, for the sake of simplicity, block matching using SAD is first performed once for N reference blocks of the reference image, and distance information and the minimum SAD value of the N reference blocks are stored in the memory unit 302 A. However, the invalidity determination process may be started before block matching using SAD for the N reference blocks of the reference image is completed.

[0069] The process of steps S1003 to S1005 is performed for each of the P invalid standard blocks. Therefore, one standard block that is the target of the process of steps S1003 to S1005 is called an "invalid target block."

[0070] In step S1003, the execution determination unit 306 calculates information on the invalid target block and a plurality of reference blocks (search block group) used for block matching in the second depth calculation unit 343B based on the information on the center coordinates of the invalid pixel block. The search block group is 2M-1 reference blocks arranged at a position corresponding to the invalid target block or at a position in the vertical or horizontal direction of the position. The memory unit 302A reads out information on the invalid target block and the search block group, and transmits it to the second depth calculation unit 343B.

[0071] In step S1004, the second depth calculation unit 343B performs a search using the SGM based on the invalid target block and the search block group. As a result, the second depth calculation unit 343B calculates distance information of the invalid target block. If there are two or more reference blocks corresponding to the smallest evaluation value (local value) even by the SGM, the second depth calculation unit 343B sets the distance information to "0".

[0072] In step S1005, the second depth calculation unit 343B transmits the distance information of the invalid target block to the memory unit 302B via the transmission unit 344D and the reception unit 304. In the memory unit 302, the distance information of the invalid target block is written, and thereby the information on the range of the invalid target block in the distance information of the reference image is updated (corrected).

[0073] According to the first embodiment, it is possible to effectively limit pixel blocks for which highly accurate distance calculation processing is performed in a cloud on a network, and therefore the image processing device can quickly obtain highly accurate distance information.

[0074] In addition, if the surrounding environment (especially brightness) changes significantly during calculation of distance information and the synchronization between the position image sensor 140L and the position image sensor 140R is incomplete, the luminance values ​​of the left and right position images will differ even if the same subject is imaged. For this reason, the position detection unit 300 and the network unit 340 cannot obtain correct distance information. Therefore, the first depth calculation unit 303A (and the second depth calculation unit 343B) calculates the overall average luminance of the left and right position images, respectively, and controls not to calculate distance information when the difference between the average luminance of the left and right position images is greater than a predetermined value. This is because it is considered that the brightness of the surrounding environment has changed significantly between the imaging timing of the position image sensor 140L and the imaging timing of the position image sensor 140R. According to such processing, it is possible to prevent erroneous distance information from being obtained.

[0075] <Embodiment 2> In the first embodiment, the image processing device 30 calculates distance information of an area according to the result of the first depth calculation section 303A with high accuracy by the second depth calculation section 343B, regardless of whether the area is the area that the user is paying attention to.

[0076] On the other hand, by performing a process of calculating distance information with higher accuracy for a pixel block (range) in which a subject that the user focuses on is captured, it is possible to provide the user with a higher quality MR experience. In the second embodiment, the image processing device 30 calculates high-accuracy distance information for a reference block in an area in which the user focuses on.

[0077] The reference block that the user focuses on can be determined from the user's line of sight direction, which is the calculation result of line of sight detection unit 150L and line of sight detection unit 150R. In other words, when the user's line of sight direction is acquired to be the front direction, the region that the user focuses on is the base block of the central range of the reference image. In this case, the image processing device 30 may set the network unit 340 to calculate distance information only for the reference block in the central range of the reference image.

[0078] Fig. 6 is a diagram showing the configuration of the position detection unit 300 according to embodiment 2. In Fig. 6, the position detection unit 300 newly includes a line-of-sight determination unit 308. On the other hand, the position detection unit 300 does not include the invalidation determination unit 305 and the execution determination unit 306.

[0079] As in the first embodiment, the gaze detection unit 150L and the gaze detection unit 150R are pupil photographing cameras for detecting the gaze position (the position where the user is looking) of the user wearing the HMD 10. The gaze detection unit 150L and the gaze detection unit 150R detect the gaze position from the acquired pupil image.

[0080] Based on the gaze position, the gaze determination unit 308 determines a reference block corresponding to the gaze position (a reference block arranged at the gaze position). Then, the gaze determination unit 308 instructs the memory unit 302A to transmit the reference block corresponding to the gaze position and a search block group corresponding to the reference block to the second depth calculation unit 343B.

[0081] The processing of other components is the same as that of the first embodiment, and therefore the description will be omitted. That is, the second depth calculation unit 343B calculates the distance information of the reference block corresponding to the gaze position by SGM based on the reference block corresponding to the gaze position and the search block group.

[0082] According to the second embodiment, the image processing device 30 can calculate highly accurate distance information in a focused manner for a pixel block that the user is paying attention to (a pixel block that the user is looking at), thereby making it possible to obtain highly accurate distance information for the entire reference image.

[0083] Since a user often looks at a subject in the front direction, important subjects are often captured in the central area of ​​an image. Therefore, as another example of the second embodiment, the image processing device 30 may execute highly accurate calculation of distance information by the network unit 340 for a reference block located in the central area of ​​the reference image.

[0084] <Embodiment 3> In the first embodiment, when there are multiple reference blocks (ranges in a reference image) with the smallest SAD value in relation to the reference block, the invalid determination unit 305 determines (determines) that the reference block is an invalid reference block. In the third embodiment, the invalid determination unit 305 determines an invalid reference block by another method.

[0085] For example, the invalid determination unit 305 may determine that a reference block is an invalid reference block when the peripheral luminance change degree of the reference block is lower than a threshold value. This is because the accuracy of block matching by SAD decreases when the difference in luminance between a reference block and its surrounding pixel blocks is small. The peripheral luminance change degree of a pixel block is, for example, the difference between the "average luminance of the pixel block" and the "average luminance of the eight surrounding (eight surrounding) pixel blocks". The invalid determination unit 305 may set a different threshold value for each reference block. For example, the closer the reference block is to the center position of the reference image, the larger the threshold value for determining whether the reference block is an invalid reference block. Also, the closer the reference block is to the gaze position (the position where the user is looking), the larger the threshold value for determining whether the reference block is an invalid reference block. This is because the closer a reference block is to the center position of the reference image, or the closer a reference block is to the gaze position, the more likely it is that the user is paying attention to that reference block, and therefore the more accurately distance information can be calculated. This is because it is preferable.

[0086] Furthermore, the position detection unit 300 (first depth calculation unit 303A) may change the size of the pixel block depending on the communication state (such as communication speed or network load status) between the position detection unit 300 and the network unit 340. Specifically, the position detection unit 300 may increase the size of the pixel block as the communication speed between the position detection unit 300 and the network unit 340 decreases or the network load increases. Furthermore, when the communication speed between the position detection unit 300 and the network unit 340 is higher than a predetermined speed, the position detection unit 300 may transmit the entire standard image and the entire reference image to the network unit 340. Then, the network unit 340 may calculate distance information of the range of the standard image specified by the position detection unit 300.

[0087] In the first to third embodiments, the distance image generating unit 301 generates a distance image showing distance information of the entire reference image. However, the distance image generating unit 301 may generate a distance image showing distance information of a part of the range of the reference image. For example, the first depth calculation unit 303A calculates distance information of a specific range close to the center of the reference image. Then, the second depth calculation unit 343B may calculate distance information of a range of the specific range of the reference image where the first depth calculation unit 303A was unable to accurately calculate distance information. According to such processing, the memory unit 302 generates a distance image showing distance information of the specific range of the reference image. According to this, the distance image can be efficiently generated in cases where distance information of positions away from the center of the reference image is not necessary, for example.

[0088] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0089] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."

[0090] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0091] (Other embodiments) The present invention provides a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and executes the program on a computer of the system or device. The present invention can be realized by a process in which one or more processors in a computer read and execute a program, or by a circuit (for example, an ASIC) that realizes one or more functions.

[0092] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An acquisition means for acquiring a first image and a second image having a parallax with respect to each other; a first calculation means for calculating distance information of a first range of the first image by a first calculation based on the first image and the second image; a first transmission means for transmitting a second range included in the first range of the first image and smaller than the first range, and a third range of the second image to an external device; a receiving means for receiving distance information of the second range calculated in the external device based on the second range and the third range by a second calculation different from the first calculation; a correction means for correcting the distance information of the first range calculated by the first calculation means, based on the distance information of the second range received by the receiving means; 13. An information processing device comprising: (Configuration 2) A calculation cost when the distance information of the second range is calculated by the second calculation is higher than a calculation cost when the distance information of the second range is calculated by the first calculation. 2. The information processing device according to configuration 1. (Configuration 3) The external device is a device in a cloud on a network. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The method further includes a determination means for determining, as the second range, a range in which distance information is not calculated by the first calculation, out of the first range. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The determining means determines a range in which two or more distances are calculated by the first calculation as the second range. 5. The information processing device according to configuration 4. (Configuration 6) The apparatus further includes a determination means for determining a range that a user is viewing from among the first range as the second range. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 7) The method further includes a determination unit for determining a central range of the first image as the second range from among the first range. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 8) the first range is made up of a plurality of pixel blocks, The method further includes a determination unit that determines, from the first range, a range of pixel blocks in which a difference in luminance between the pixel blocks and surrounding pixel blocks is smaller than a first threshold value as the second range. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 9) the determining means determines the first threshold for each pixel block included in the first range according to a distance between a center position of the first image and the pixel block, or a distance between a position in the first image where a user is looking and the pixel block. 9. The information processing device according to configuration 8. (Configuration 10) the first calculation means determines a size of the pixel block according to a communication state between the information processing device and the external device. 10. The information processing device according to configuration 8 or 9. (Configuration 11) the first calculation means does not calculate the distance information of the first range when a difference in brightness between the first image and the second image is greater than a second threshold value; 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) An information processing device according to any one of configurations 1 to 11, The external device; having the external device has a second calculation means for calculating distance information of the second range based on the second range and the third range by the second calculation, and a second transmission means for transmitting the distance information of the second range to the information processing device. An information processing system comprising: (method) An acquisition step of acquiring a first image and a second image having a parallax with respect to each other; a first calculation step of calculating distance information of a first range of the first image by a first calculation based on the first image and the second image; a first transmission step of transmitting a second range included in the first range of the first image and smaller than the first range, and a third range of the second image to an external device; a receiving step of receiving distance information of the second range calculated in the external device based on the second range and the third range by a second calculation different from the first calculation; a correction step of correcting the distance information of the first range calculated in the first calculation step based on the distance information of the second range received in the receiving step. (program) 12. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0093] 30: image processing device, 300: position detection unit, 340: network unit, 302B: memory unit, 303A: First depth calculation unit, 343B: Second depth calculation unit

Claims

1. An acquisition means for acquiring a first image and a second image having a parallax with respect to each other; a first calculation means for calculating distance information of a first range of the first image by a first calculation based on the first image and the second image; a first transmission means for transmitting a second range included in the first range of the first image and smaller than the first range, and a third range of the second image to an external device; a receiving means for receiving distance information of the second range calculated in the external device based on the second range and the third range by a second calculation different from the first calculation; a correction means for correcting the distance information of the first range calculated by the first calculation means, based on the distance information of the second range received by the receiving means; 13. An information processing device comprising:

2. a calculation cost when the distance information in the second range is calculated by the second calculation is higher than a calculation cost when the distance information in the second range is calculated by the first calculation; 2. The information processing apparatus according to claim 1,

3. The external device is a device in a cloud on a network.

2. The information processing apparatus according to claim 1,

4. The method further includes a determination unit that determines, as the second range, a range in which distance information has not been calculated by the first calculation, out of the first range.

2. The information processing apparatus according to claim 1,

5. The determining means determines a range in which two or more distances are calculated by the first calculation as the second range.

5. The information processing apparatus according to claim 4.

6. The display device further includes a determination unit for determining a range of the first range that is viewed by the user as the second range.

2. The information processing apparatus according to claim 1,

7. The method further includes a determination unit for determining a central range of the first image as the second range from among the first range.

2. The information processing apparatus according to claim 1,

8. the first range is made up of a plurality of pixel blocks, The method further includes a determination unit for determining, as the second range, a pixel block range in which a difference in luminance between the pixel block and a surrounding pixel block is smaller than a first threshold value, from the first range.

2. The information processing apparatus according to claim 1,

9. the determining means determines the first threshold for each pixel block included in the first range according to a distance between a center position of the first image and the pixel block, or a distance between a position in the first image at which a user is looking and the pixel block.

9. The information processing apparatus according to claim 8,

10. the first calculation means determines a size of the pixel block in accordance with a communication state between the information processing device and the external device.

9. The information processing apparatus according to claim 8,

11. the first calculation means does not calculate the distance information of the first range when a difference in brightness between the first image and the second image is greater than a second threshold value; 2. The information processing apparatus according to claim 1,

12. An information processing device according to any one of claims 1 to 11, The external device; having the external device has a second calculation means for calculating distance information of the second range based on the second range and the third range by the second calculation, and a second transmission means for transmitting the distance information of the second range to the information processing device. An information processing system comprising:

13. An acquisition step of acquiring a first image and a second image having a parallax with respect to each other; a first calculation step of calculating distance information of a first range of the first image by a first calculation based on the first image and the second image; a first transmission step of transmitting a second range included in the first range of the first image and smaller than the first range, and a third range of the second image to an external device; a receiving step of receiving distance information of the second range calculated in the external device based on the second range and the third range by a second calculation different from the first calculation; a correction step of correcting the distance information of the first range calculated in the first calculation step based on the distance information of the second range received in the receiving step.

14. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 11.

Images