Information processing device and warning display method

Abstract

To provide an information processor and a warning presentation method capable of presenting images free from restriction by an HMD (head mounted display) due to actual restriction conditions.SOLUTION: In an actual space including a user 120, the information processor sets a boundary of a play area 184 using boundaries 182a and 182b in a field of vision through an imaging apparatus 12 according to a predetermined rule. When the user gets out of the play area 184, a warning situation determination part determines that warning situation has occurred and superimpose a warning image over a display image. The information processor determines whether to set a margin area between the boundaries 182a and 182b in the field of vision and the boundary of the play area 184, or whether to change the width of the margin area depending on the direction and a situation in the actual space.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device that processes information based on a captured image, and a warning presentation method performed by the information processing device. [Background technology]

[0002] A head mounted display (hereinafter referred to as "HMD") connected to a game machine is worn on the head, and a game is played while looking at the displayed screen (see, for example, Patent Document 1). For example, if the position and posture of the user's head are acquired, and an image of the virtual world is displayed so that the field of view changes according to the direction of the face, it is possible to create a situation in which the user feels as if they have entered the virtual world. The position and posture of the user are generally acquired based on the analysis results of visible light or infrared images captured of the user, or measurements from a motion sensor built into the HMD. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5580855 specification Summary of the Invention [Problem to be solved by the invention]

[0004] Technologies that perform some kind of information processing based on captured images assume that the user or other object is within the camera's field of view. However, when wearing an HMD, the user cannot see the outside world, so they may lose their sense of direction or become so engrossed in the game that they move to an unexpected location in the real world. If the user moves out of the camera's field of view, the information processing may fail or its accuracy may deteriorate, and the user himself may not even notice the cause. The more the user is made aware of real-world circumstances such as the positional relationship with the camera, the higher the accuracy of the information processing can be maintained, but at the same time, a dilemma arises in that it is easier to disrupt the worldview of the virtual space constructed using the HMD.

[0005] The present invention has been made in consideration of these problems, and its purpose is to provide a technology that can optimize the balance between the entertainment value and information processing accuracy achieved by an HMD. [Means for solving the problem]

[0006] One aspect of the present invention relates to an information processing device, which includes an information processing unit that detects the position of an object in real space based on an image captured by an imaging device and performs information processing based on the object, an image generating unit that generates image data to be displayed as a result of the information processing, a warning status determining unit that sets a play area by changing a setting rule for the field of view of the imaging device according to a state of the real space acquired by analyzing the captured image, determines the necessity of a warning to a user based on the play area, and superimposes a warning image on the image generating unit, and an output unit that outputs data of the image generated by the image generating unit to a display device.

[0007] Yet another aspect of the present invention relates to a method for presenting a warning, the method for presenting a warning including a step of an information processing device detecting a position of an object in a real space based on an image captured by an imaging device, performing information processing based on the object, a step of generating data of an image to be displayed as a result of the information processing, a step of setting a play area by changing a setting rule for the field of view of the imaging device according to a situation of the real space acquired by analyzing the captured image, and a step of determining the necessity of a warning to a user based on the play area, a step of superimposing a warning image on the image to be displayed according to the determination, and a step of outputting data of the generated image to a display device.

[0008] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, or recording medium having a computer program recorded thereon, are also effective aspects of the present invention. Effect of the Invention

[0009] According to the present invention, a user wearing an HMD can enjoy the representational world without being conscious of restrictive conditions in the real world as much as possible. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing system to which the present embodiment can be applied. [Diagram 2] 2A to 2C are diagrams illustrating examples of the external shape of an HMD according to the present embodiment. [Diagram 3] 1 is a diagram showing an internal circuit configuration of an information processing device according to an embodiment of the present invention; [Figure 4] 2 is a diagram showing the internal circuit configuration of an HMD according to the present embodiment. [Diagram 5] 1 is a diagram showing a functional block configuration of an information processing device according to an embodiment of the present invention; [Figure 6] 1 is a diagram for explaining information that can be acquired from a captured image in the present embodiment. FIG. [Figure 7] 10 is a diagram showing a schematic example of a setting of a play area that is referred to when a warning status determination unit in the present embodiment determines the need for a warning. FIG. [Figure 8] FIG. 11 is a diagram showing a schematic diagram of another example of the setting of the play area in the present embodiment. [Figure 9] 11A and 11B are diagrams for explaining the effect of the presence or absence of a margin area on processing in the present embodiment. [Figure 10] 13 is a diagram for explaining a method for setting a play area on an image plane in the present embodiment. FIG. [Figure 11] 11 is a diagram illustrating an example of a display screen that is generated when it is determined that a warning is necessary in the present embodiment. FIG. [Figure 12] 10 is a flowchart showing a processing procedure in which the information processing device generates output data according to a user's movement in the present embodiment. [Figure 13] 10A to 10C are diagrams showing a schematic diagram of a state transition between display and non-display of a warning image in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] 1 shows an example of the configuration of an information processing system to which this embodiment can be applied. The information processing system 8 includes an imaging device 12 that captures an image of an object, an information processing device 10 that processes information based on the captured image, a flat panel display 16 and an HMD 18 that display an image obtained as a result of the information processing, and an input device 14 that is operated by a user.

[0012] The information processing device 10, the imaging device 12, the input device 14, the flat display 16, and the HMD 18 may be connected by wired cables or by known wireless communication technology such as Bluetooth (registered trademark). Depending on the information processing performed by the information processing device 10, the flat display 16 may not be required. The external shapes of these devices are not limited to those shown in the drawings. Furthermore, a device may be provided that is equipped with two or more of these devices in an integrated manner. For example, the information processing device 10, the input device 14, and the flat display 16 may be realized by a mobile terminal equipped with them.

[0013] The imaging device 12 has a camera that captures an object such as a user at a predetermined frame rate, and a mechanism that generates output data of the captured image by performing general processing such as demosaic processing on the output signal from the camera and sends the output data to the information processing device 10. The camera has a visible light sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor that is used in general digital cameras and digital video cameras. The imaging device 12 may have only one camera, or may be a so-called stereo camera in which two cameras are arranged on the left and right at a known distance as shown in the figure.

[0014] Alternatively, the imaging device 12 may be configured by combining a monocular camera with a device that irradiates an object with reference light such as infrared light and measures the reflected light. When a stereo camera or a mechanism for measuring reflected light is introduced, the position of the object can be found in three-dimensional real space, and information processing by the information processing device 10 and image display by the display device can be further diversified. Both the method of determining the distance of the object from the camera based on the principle of triangulation using stereo images captured from left and right viewpoints by a stereo camera, and the method of determining the distance of the object from the camera based on the TOF (Time of Flight) or pattern irradiation method by measuring reflected light are widely known.

[0015] Hereinafter, the imaging device 12 will be mainly described as capturing stereo images, but as described above, this is not intended to limit the present embodiment, and the imaging device 12 may include at least one camera. The information processing device 10 performs necessary information processing using data transmitted from the imaging device 12, and generates output data such as images and sounds. The content of the processing performed by the information processing device 10 is not particularly limited, and may be appropriately determined depending on the functions and application contents desired by the user.

[0016] For example, the information processing device 10 performs general face detection and tracking processing on the captured image to progress a game in which a character reflecting the user's movements as a target appears, or converts the user's movements into command inputs to perform information processing. At this time, markers provided on the input device 14 may be used to acquire the movements of the input device 14. In addition, by tracking multiple markers provided on the outer surface of the HMD 18, the position and posture of the head of the user wearing the HMD 18 may be identified, and the virtual world viewed from a viewpoint that moves in response to the position and posture may be displayed on the HMD 18. The output data generated by the information processing device 10 is transmitted to at least the HMD 18.

[0017] The HMD 18 is a display device that displays an image on a display panel such as an organic EL panel located in front of the user's eyes when worn on the user's head. For example, parallax images viewed from left and right viewpoints may be generated and displayed in the left and right areas obtained by dividing the display screen into two, thereby allowing the user to view the image stereoscopically. However, this is not intended to limit the present embodiment, and one image may be displayed on the entire display screen. The HMD 18 may further include built-in speakers or earphones that output sound to positions corresponding to the user's ears.

[0018] The flat display 16 may be a television having a display for outputting two-dimensional images and a speaker for outputting sound, such as a liquid crystal television, an organic electroluminescence television, a plasma television, a PC display, or the like. Alternatively, it may be the display and speaker of a tablet terminal or a mobile terminal. The input device 14 is operated by a user to receive requests such as starting and ending a process, selecting a function, and inputting various commands, and supplies the requests to the information processing device 10 as electrical signals.

[0019] The input device 14 may be realized by any of common input devices such as a game controller, a keyboard, a mouse, a joystick, a touch pad provided on the display screen of the flat display 16, or a combination of these. The input device 14 may further include a light-emitting marker consisting of an element that emits light in a predetermined color or a collection of such an element. In this case, the information processing device 10 tracks the movement of the marker using the captured image, so that the movement of the input device 14 itself can be treated as a user operation. The input device 14 may be configured only by a light-emitting marker and a mechanism for holding it.

[0020] 2 shows an example of the external shape of the HMD 18. In this example, the HMD 18 is composed of an output mechanism 102 and a wearing mechanism 104. The wearing mechanism 104 includes a wearing band 106 that goes around the head when worn by the user to secure the device. The wearing band 106 is made of a material or has a structure that allows the length to be adjusted to fit the head circumference of each user. For example, it may be made of an elastic body such as rubber, or a buckle, gears, etc. may be used.

[0021] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the HMD 18, and has a display panel inside so as to face the eyes when worn. Light-emitting markers 110a, 110b, 110c, 110d, etc. are provided on the outer surface of the housing 108. The number and arrangement of the light-emitting markers are not particularly limited, but in this embodiment, they are provided at the four corners of the front surface of the housing of the output mechanism unit 102.

[0022] Furthermore, light-emitting markers 110e and 110f are provided on both sides of the rear of the attachment band 106. By arranging the light-emitting markers in this manner, even if the user faces sideways or backwards with respect to the imaging device 12, the situation can be identified based on the number and positions of the images of the light-emitting markers in the captured image. Note that the light-emitting markers 110c and 110d are located below the output mechanism unit 102, and the light-emitting markers 110e and 110f are located outside the attachment band 106. Since they are not actually visible from the viewpoint in FIG. 2, their outer peripheries are indicated by dotted lines.

[0023] 3 shows the internal circuit configuration of the information processing device 10. The information processing device 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are connected to each other via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 is connected to a communication unit 32 including a peripheral device interface such as USB or IEEE1394, or a network interface for wired or wireless LAN, a storage unit 34 such as a hard disk drive or nonvolatile memory, an output unit 36 ​​that outputs data to the flat panel display 16 or HMD 18, an input unit 38 that inputs data from the imaging device 12, the input device 14, and the HMD 18, and a recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory.

[0024] The CPU 22 executes an operating system stored in the storage unit 34 to control the entire information processing device 10. The CPU 22 also executes various programs read from a removable recording medium and loaded into the main memory 26, or downloaded via the communication unit 32. The GPU 24 has a geometry engine function and a rendering processor function, performs drawing processing according to drawing commands from the CPU 22, and stores a display image in a frame buffer (not shown). The GPU 24 then converts the display image stored in the frame buffer into a video signal and outputs it to the output unit 36. The main memory 26 is composed of a RAM (Random Access Memory), and stores programs and data required for processing.

[0025] 4 shows the internal circuit configuration of the HMD 18. The HMD 18 includes a CPU 50, a main memory 52, a display unit 54, and an audio output unit 56. These units are connected to one another via a bus 58. An input / output interface 60 is further connected to the bus 58. A communication unit 62, which is formed of a wired or wireless LAN network interface, an acceleration sensor 64, and a light-emitting unit 66 are connected to the input / output interface 60.

[0026] The CPU 50 processes information acquired from each unit of the HMD 18 via the bus 58, and supplies output data to the display unit 54 and the audio output unit 56. The main memory 52 stores programs and data necessary for processing in the CPU 50. However, depending on the application to be executed and the design of the device, it may be sufficient for the information processing device 10 to perform almost all processing, and for the HMD 18 to only output data transmitted from the information processing device 10. In this case, the CPU 50 and the main memory 52 can be replaced with simpler devices.

[0027] The display unit 54 is composed of a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of the user wearing the HMD 18. As described above, stereoscopic vision may be realized by displaying a pair of parallax images in areas corresponding to the left and right eyes. The display unit 54 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the HMD 18 is worn, and that expand the user's viewing angle.

[0028] The audio output unit 56 is composed of speakers or earphones provided at positions corresponding to the user's ears when the HMD 18 is worn, and allows the user to hear audio. The number of channels of the output audio is not particularly limited, and may be any of monaural, stereo, and surround. The communication unit 62 is an interface for transmitting and receiving data between the information processing device 10 and the flat display 16, and can be realized by using a known wireless communication technology such as Bluetooth (registered trademark).

[0029] The acceleration sensor 64 detects the inclination of the HMD 18 by measuring the gravitational acceleration in a predetermined axis direction. The HMD 18 may further be provided with various other sensors such as a gyro sensor. Measurement values ​​by the sensor are transmitted to the information processing device 10 via the communication unit 62. The light emitting units 66 are elements or groups of elements that emit light in a predetermined color, and are provided at multiple locations on the outer surface of the HMD 18 as shown in FIG. 2. The position of the HMD 18 is obtained by tracking these as markers, and the attitude of the HMD 18 is obtained from the number and positional relationship of their images in the captured image.

[0030] The information processing device 10 can obtain the position and posture of the user's head with higher accuracy by integrating information obtained by a plurality of means such as the acceleration sensor 64 and the light emitting unit 66. Meanwhile, in this embodiment, the acceleration sensor 64 can be omitted in some cases.

[0031] Fig. 5 shows the configuration of functional blocks of the information processing device 10. Each functional block shown in Fig. 5 can be realized in terms of hardware by the configuration of the CPU, GPU, various memories, data bus, etc. shown in Fig. 3, and in terms of software by a program that performs various functions such as a data input function, a data holding function, an image processing function, and a communication function, which are loaded into memory from a recording medium, etc. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof, and are not limited to any one of them.

[0032] The information processing device 10 includes an input information acquisition unit 72 that acquires input information from the input device 14 and the HMD 18, a captured image acquisition unit 74 that acquires captured image data from the imaging device 12, an information processing unit 76 that performs information processing according to content such as a game, an output data generation unit 78 that generates data to be output, and a content data storage unit 84 that stores data necessary for information processing and image generation. The information processing device 10 further includes a position information acquisition unit 80 that acquires user position information based on the captured image, a warning situation determination unit 82 that determines a situation requiring a warning based on the user's position, an area setting information storage unit 85 that stores setting information related to an area in real space required for the determination, and an output data transmission unit 86 that transmits data to be output to the HMD 18.

[0033] The input information acquisition unit 72 acquires the contents of a user operation from the input device 14. Here, the user operation may be an operation performed on a general information processing device, such as selection of an application or content to be executed, start / end of processing, command input, etc. The input information acquisition unit 72 supplies the information acquired from the input device 14 to the captured image acquisition unit 74 or the information processing unit 76 depending on the information. The input information acquisition unit 72 further receives a measurement value from the acceleration sensor 64 of the HMD 18 and supplies it to the information processing unit 76.

[0034] The captured image acquisition unit 74 acquires, at a predetermined rate, data of captured images, such as stereo images, obtained by the imaging device 12 capturing a moving image. The captured image acquisition unit 74 may further control the start / end of imaging in the imaging device 12 according to a processing start / end request from a user acquired by the input information acquisition unit 72, or may control the type of data acquired from the imaging device 12 according to the result of processing in the information processing unit 76.

[0035] The position information acquisition unit 80 acquires the user's position information at a predetermined rate by detecting the image of a predetermined object from the captured image. For example, the position of the user's head or hand in real space is acquired based on the image of a light-emitting marker provided on the HMD 18 or the input device 14. Alternatively, an image analysis technique may be combined to track a part of the user's body using a contour line, or to recognize a face or an object having a specific pattern by pattern matching. Depending on the configuration of the imaging device 12, the distance to the user may be determined by measuring the reflection of infrared rays as described above.

[0036] The information processing unit 76 processes electronic content such as a game specified by the user. This processing includes the use of the user's position information acquired by the position information acquisition unit 80. The information processing unit 76 may identify the user's posture by integrating the measurement results of the acceleration sensor of the HMD 18 as necessary. As described above, the content of the subsequent information processing performed by the information processing unit 76 in response to the user's operation via the input device 14 or the user's movement is not particularly limited.

[0037] The output data generating unit 78 generates image and audio data to be output as a result of information processing in accordance with a request from the information processing unit 76. For example, the output data generating unit 78 generates a left and right parallax image of the virtual world as seen from a viewpoint corresponding to the position and posture of the user's head. If the parallax image is displayed in front of the left and right eyes on the HMD 18 and audio in the virtual world is output, the user can feel as if he or she has entered the virtual world. Programs and image and audio data required for information processing in the information processing unit 76 and data generation processing in the output data generating unit 78 are stored in the content data storage unit 84.

[0038] The warning situation determination unit 82 constantly monitors the occurrence of a situation requiring a warning based on the user's position information acquired by the position information acquisition unit 80, and determines the occurrence of such a situation. Specifically, the unit performs an inside / outside determination for a play area set directly or indirectly in a three-dimensional space in the real world, and determines that a warning is necessary when the user leaves the play area. In this embodiment, the unit acquires position information using a captured image, so information processing proceeds on the premise that the user is within the angle of view of the camera.

[0039] However, when wearing an HMD such as that shown in Figure 2, it is difficult for a user to recognize the surrounding situation, and even whether or not they are within the field of view. In order to continue normal information processing, it is necessary to provide such information related to the real world by some means, but this may damage the world view expressed by the content. One possible measure to achieve this is to warn the user only when they are about to leave the field of view. However, if this timing is too early, warnings will be given frequently for even small movements, which will effectively narrow the range of movement. On the other hand, if the timing is too late, it will be inevitable to go out of the field of view, and there may be a period when position information cannot be obtained.

[0040] Therefore, in this embodiment, as described above, the timing to issue a warning is determined by determining whether the area is inside or outside the area set as the play area in the real world, and the relationship between the play area and the angle of view is further adjusted to optimize the balance between the range of motion and processing accuracy according to the situation. The setting information for the play area is stored in advance in the area setting information storage unit 85. The specific setting contents will be described later. When the warning status determination unit 82 determines that a warning is necessary, it notifies the output data generation unit 78 of that effect.

[0041] At this time, the output data generation unit 78 additionally draws a component image representing the warning on the display image. The warning status determination unit 82 also determines that a warning is no longer necessary, for example because the user has returned to a normal position, and notifies the output data generation unit 78 of this fact. In response to this, the output data generation unit 78 hides the component image representing the warning. Note that the warning may be accompanied by sound as well as an image. The output data transmission unit 86 sequentially acquires the output data generated by the output data generation unit 78, formats it as necessary, and transmits it to the HMD 18.

[0042] FIG. 6 is a diagram for explaining information that can be acquired from a captured image in this embodiment. In the figure, a user 120 holds an input device 14 and wears an HMD 18. The input device 14 is provided with a light-emitting marker 122 at a position directly facing the imaging device 12 when the input device 14 is held in a manner suitable for operation. The light-emitting markers of the HMD 18 are as shown in FIG. 2. When the imaging device 12 is a stereo camera, a distance Z from the imaging plane of the imaging device 12 to each light-emitting marker is obtained based on the parallax of the images in the stereo images. In addition, the position of the image of the light-emitting marker in the image plane (XY plane) of either captured image represents the apparent position from the imaging device 12.

[0043] The position of each light-emitting marker in the three-dimensional space of the real world is obtained by integrating this information, specifically by inversely projecting the position on the XY plane using the distance Z from the imaging device 12. Furthermore, the orientation (vector va) of the HMD 18 in real space can be obtained from the number and positional relationship of the images of the light-emitting markers of the HMD 18, and the orientation (vector vb) of the input device 14 in real space can be obtained from the shape of the images of the light-emitting markers 122 of the input device 14.

[0044] Based on the positions and orientations of these devices in real space, the information processing unit 76 of the information processing device 10 can express a virtual world in which the field of view changes according to the direction of the user's 120's face, and how objects in an image move according to the movement of the input device 14. Note that the image capturing device 12 does not have to be a stereo camera when estimating the distance from the image capturing device 12 based on the apparent size of a marker, or when performing information processing that does not require movement in the depth direction. The same applies when a distance measurement technique using a reference light is introduced. Also, it is not necessary to track both the HMD 18 and the input device 14.

[0045] When using images of light-emitting markers in this way, strictly speaking, what is obtained directly is the position of the light-emitting markers of the HMD 18 and the input device 14. However, since the HMD 18 is worn on the user's head, it can also be considered as the position of the user's head. Even when specifying the position based on subjects other than the light-emitting markers, strictly speaking, the positions of those subjects are obtained, and the position of the user's center of gravity, etc. is estimated from that information as necessary.

[0046] In the following explanation, the "user's position" may be the user's own position or the position of the subject that is the basis for position acquisition. In addition, rules related to the judgment criteria may be set according to the nature and situation of the subject, such as not requiring a warning if the input device 14 goes outside the play area but the HMD 18 is not present.

[0047] FIG. 7 schematically shows an example of the setting of the play area, which is referred to when the warning situation determination unit 82 determines the necessity of a warning. The figure shows a state of overlooking the real space, and the user 120 is present facing the imaging device 12 directly. The horizontal viewing angle θ of the camera of the imaging device 12 h defines the boundary surfaces 182a and 182b of the field of view indicated by the dotted lines. In the case of a stereo camera, it is the intersection of the viewing angles of both. Based on such a vertical boundary surface of the field of view, the boundary surface of the play area 184 is set according to a predetermined rule. The warning situation determination unit 82 determines that a warning is necessary when the user goes outside the play area 184.

[0048] In the illustrated example, in the range where the distance Z from the imaging device 12 satisfies Z1 ≦ Z < Z3, the play area 184 is set inside the boundary surfaces 182a and 182b of the field of view. In the range of Z1 ≦ Z < Z2 close to the imaging device 12, the width is increased in proportion to the distance Z around the optical axis of the camera, and in the range of Z2 ≦ Z < Z3, it is set to have a constant width W. The vertical surfaces 186a and 186b that determine the boundary of the play area in the range of Z1 ≦ Z < Z2 correspond to positions where, regardless of the position in the depth direction, they are imaged inside the left and right ends by a predetermined amount on the captured image. Therefore, by setting at least the inside of the vertical surfaces 186a and 186b as the play area, a margin area of the predetermined amount on the image can be used as a margin area, and a warning can be given at a timing when the user is about to deviate from the viewing angle.

[0049] The closer to the imaging device 12, the narrower the field of view and thus the narrower the movable range. Therefore, in the range close to the imaging device 12, by maximizing the condition of being inside the vertical surfaces 186a and 186b, a wider range can be obtained without giving a warning. On the other hand, at a position far from the imaging device 12, since the field of view expands, if the inside of the vertical surfaces 186a and 186b is set as the play area based on whether it has deviated from the viewing angle, a wider range can be obtained without giving a warning. However, in the illustrated example, considering the appropriateness of information processing and the presence of surrounding objects, etc., the play area is limited to a narrower width W.

[0050] In this way, in addition to reducing the possibility of deviation from the angle of view, the play area 184 can also simultaneously serve to set an appropriate range of movement. Also, by setting the play area according to different rules depending on the distance range from the camera, it is possible to adapt to changes in priorities such as giving priority to the range of movement in an area with a narrow field of view and giving priority to other elements such as the circumstances of the content in an area with a wide field of view.

[0051] FIG. 8 schematically shows another example of the setting of the play area. This figure shows the state of the real space as seen from the left side of the user, and the user 120 exists facing the imaging device 12 directly. The vertical angle of view θ of the camera of the imaging device 12 v defines the boundary surfaces 192a and 192b of the field of view indicated by the dotted lines. Based on such a horizontal boundary surface of the field of view, the boundary surface of the play area is set in the range where Z1 ≦ Z < Z3 from the imaging device 12.

[0052] In the example of this figure, the boundary surfaces 192a and 192b of the field of view are made to coincide with the boundary surfaces of the play area 194 over the range where Z1 ≦ Z < Z3. That is, compared with the play area shown in FIG. 7, no margin area is set between the boundary surface of the field of view. That is, no margin such as the horizontal planes 196a and 196b that are imaged inside by a predetermined amount from the upper and lower ends on the captured image is provided, and the area corresponding to the upper end to the lower end of the captured image is set as the play area. By doing so, as a result, a warning is given at the same time as deviating from the angle of view, so there is a high possibility that position acquisition will become impossible even for a very short time. On the other hand, it becomes possible to avoid giving a warning over a wider range.

[0053] This setting is effective for cameras with a narrow angle of view. When the angle of view is narrow, setting a margin area as shown in FIG. 7 may cause frequent warnings with small movements. Therefore, comfort can be prioritized by securing a wide play area, even at the expense of some deterioration in the accuracy of information processing. For example, when using a camera with a wide horizontal angle of view and a narrow vertical angle of view, a good balance between information processing accuracy and a comfortable operating environment can be achieved by varying the positional relationship of the boundary surface of the play area to the boundary surface of the field of view according to the direction in real space, as shown in FIG. 7 and FIG. 8.

[0054] 9 is a diagram for explaining the effect on processing of the presence or absence of a margin area. The upper part of the figure (a) shows a time chart of the user's movements and each process when a margin area is provided for the field of view and the play area is set, and the lower part (b) shows a time chart of the user's movements and each process when the field of view and the play area are aligned. That is, as shown diagrammatically on the right, a play area shown in white is set for the imaging device 12, but in (a) a margin area shown in shaded is provided between the area outside the field of view shown in black. The white, shaded, and black rectangles in the "position" row in the time chart respectively represent the time the user is in the play area, margin area, and outside the field of view.

[0055] For comparison, in both settings, it is assumed that the user leaves the field of view at time t2, and the warning is displayed for the same amount of time, resulting in the user returning to the play area. For such user movement, the period during which the position information acquisition unit 80 can acquire position information is shown by a rectangle in the "Position Acquisition" column, and the period during which the warning is displayed is shown by a rectangle in the "Warning" column. In the case of (a), the user leaves the play area at time t1, before the time t2 when the user leaves the field of view, so the warning is displayed from that point until the user returns to the play area at time t3. On the other hand, in the case of (b), no margin area is set, so the warning is displayed from time t2 when the user leaves the field of view to time t4.

[0056] As shown in the figure, in (a), since location information can be obtained even after the warning is displayed, it is guaranteed that the user is outside the play area based on that information. In the case of (b), since location information cannot be obtained at the same time as the warning, the validity of the warning cannot be guaranteed during the period in which the warning is displayed. Therefore, for example, the fact that the user's image is not detected due to occlusion at the boundary of the field of view may be mistaken as the user having gone outside the field of view, causing the warning to be displayed.

[0057] In addition, even if the user returns to the play area at the same timing after the warning is displayed in (a) and (b), the period during which position information cannot be acquired will be longer in (b). In the illustrated example, a period during which the user goes out of the field of view also occurs in (a), but depending on the width of the margin area, there is a possibility that the user will return in response to the warning display before going out of the field of view, so the period during which position information cannot be acquired is less likely to occur compared to the case of (b). On the other hand, even if the user moves in the same way, in the case of (a), the warning is displayed at an earlier time t1, so the constraints on the user become stricter. In light of these characteristics, it is desirable to set the optimal play area according to the priority order according to the situation, such as whether it is desired to maximize the area in which a warning is not issued, or whether there is enough area without doing so.

[0058] In addition to deciding whether or not to provide a margin area, the width of the margin area and the shape of the play area may be optimized according to the expected situation and the direction in real space. For example, the shape of the play area may be changed even with the same angle of view depending on the user's posture expected depending on the content and environment of the game, specifically, whether the user is playing standing or sitting, sitting at a desk or sitting on a sofa.

[0059] In addition, when the settings of FIG. 7 and FIG. 8 are applied simultaneously, the play area to be set has a shape having a quadrilateral cross section with respect to the distance Z from the camera, but this embodiment is not limited to this shape, and any shape such as a cylinder may be used. In addition, when the optical axis is not horizontal, the distance Z from the camera is not horizontal with the floor surface. In this case, the shape of the play area may be defined by the distance in the optical axis direction, or may be defined by the horizontal distance in real space. If the angle between the optical axis of the camera and the horizontal plane of the real space is measured by an acceleration sensor or the like built into the imaging device 12, the coordinate conversion can be easily realized. It is understood by those skilled in the art that various specific data formats for setting an area in a three-dimensional space are possible.

[0060] On the other hand, the play area may be indirectly set by defining an area corresponding to the play area on the image plane. FIG. 10 is a diagram for explaining a method for setting the play area on the image plane. The figure shows the field of view of a camera facing the subject space, and thus the plane of the captured image, with the upper part (a) being a monocular camera and the lower part (b) being a stereo camera. In the case of a monocular camera, the left and right ends of the plane of the captured image 300 having a predetermined width x are defined as margin areas, and the area 302 excluding these is defined as the area corresponding to the play area.

[0061] The area in three-dimensional space defined by this area 302 is area 304 whose horizontal width increases in proportion to the distance Z in the depth direction. In other words, when focusing on the horizontal direction, setting area 302 sets the inside of vertical surfaces 186a, 186b shown in Fig. 8 as the play area. When play area corresponding area 302 is set on the image plane in this way, the cross section of the play area in real space will be a similar shape with a size according to distance Z, so there is less freedom in the setting shape compared to when it is set directly in real space.

[0062] On the other hand, since the necessity of a warning can be determined directly by determining whether the image on the captured image is inside or outside the play area corresponding region 302, there is less room for error than when using the user's position information. The illustrated example shows a case where a margin region is provided in the horizontal direction, but the same applies when a margin region is provided in the vertical direction. For the same reasons as above, whether or not to provide a margin region and the adjustment of its width x may be performed independently depending on the direction. These elements may be changed depending on the content and situation of the content.

[0063] In the case of the stereo camera shown in (b), the camera that leaves the field of view first differs depending on whether the user is at the left or right edge of the camera's field of view. For example, when the user is near the right edge of the camera's field of view, as shown in the figure, image 308a in image 306a of the left viewpoint camera is closer to the right edge than image 308b in image 306b of the right viewpoint camera. Therefore, if the user continues to move to the right, he or she will leave the field of view of the left viewpoint camera first. Conversely, if the user is near the left edge of the camera's field of view, he or she will leave the field of view of the right viewpoint camera first.

[0064] Therefore, when setting the play area within the field of view that can be captured by both stereo cameras, the area 310a excluding the margin area of ​​a given width x on the right edge in the image plane captured by the left viewpoint camera and the area 310b excluding the margin area of ​​a given width x on the left edge in the image plane captured by the right viewpoint camera are simultaneously set as the play area corresponding area. Then, when the image of the user leaves the play area corresponding area in either image, the need for a warning is determined.

[0065] 11 shows an example of a display screen that is generated when the warning status determination unit 82 determines that a warning is necessary. The display screen 200 has a configuration in which a warning image 202 that indicates a warning is added to an image of a content such as a game screen. In the illustrated example, the warning image 202 indicates that the user has gone outside the play area by displaying text information such as "Out of play area." The urgency is further expressed by including a mark indicating danger.

[0066] However, the form of the warning image 202 is not limited to this, and it may be expressed only by figures or marks by devising colors and shapes, or such figures or marks may be combined with text information. The warning image 202 may also be changed for each content, or according to the content displayed at that time. Since the warning image 202 is intended to alert the user, it is basically displayed superimposed on the content image. On the other hand, when the content image is a virtual world with a sense of depth, the user may be surprised or feel uncomfortable if a component image without a sense of depth suddenly appears in front of the user.

[0067] Therefore, as shown in the figure, the edges of the warning image 202 are blurred by blurring or alpha blending. The entire warning image 202 may be made semi-transparent. This increases the affinity with the virtual world depicted behind it, even though it is in the foreground. It is also desirable to display the warning image 202 at a position that avoids the viewpoint so as not to give the user the feeling that their line of sight has been suddenly blocked. For example, in a display format in which the field of view changes according to the user's line of sight, the viewpoint is inevitably fixed to the center of the screen, so the image is displayed at a position at least a predetermined distance away from the center of the screen.

[0068] Also, because it is easier for people to lower their gaze than to raise it, by displaying the warning image 202 in the lower half of the screen (the area below the vertical center line C) regardless of the display format, the gaze point can be moved naturally from the content image. This prevents the eyes from being unable to focus on a suddenly appearing image and taking time to recognize it, and prevents motion sickness caused by small eye movements.

[0069] Next, the operation of the information processing device 10 that can be realized by the above-mentioned configuration will be described. Fig. 12 is a flowchart showing the processing procedure in which the information processing device generates output data according to the user's movements in this embodiment. This flowchart is started when the user requests the information processing device 10 to start processing via the input device 14.

[0070] First, the captured image acquisition unit 74 of the information processing device 10 requests the imaging device 12 to start capturing images, and in response starts acquiring data of the captured images captured and output by the imaging device 12 (S10). Meanwhile, the warning status determination unit 82 reads out setting information related to the play area from the area setting information storage unit 85 (S12). At this time, the setting information associated with the content to be played, such as selected by the user, may be selected. Also, the captured image may be analyzed to determine whether the user is seated or not, the surrounding situation, and the like, and the setting information may be selected based on the information.

[0071] Next, the information processing unit 76 requests the output data generating unit 78 to output the real-time captured image acquired by the captured image acquiring unit 74 to the HMD 18. The captured image is displayed on the HMD 18 to prompt the user to check for any obstacles in the surroundings (S14, S16). At this time, the output data generating unit 78 superimposes text information such as "Is there anything dangerous?" and a GUI (Graphical User Interface) for the user to input that he or she has checked the captured image. This allows the user to recognize the camera's angle of view in advance, and therefore allows the user to move while keeping in mind an area equivalent to the play area.

[0072] By knowing the play area, the user's range of movement becomes clear, and it becomes possible to determine whether surrounding objects are potential obstacles. If the user moves objects away as necessary and inputs a confirmation that they have been removed before the next process can begin, the camera will monitor for deviations from the field of view and collisions with obstacles at the same time. The captured image will continue to be displayed until there is a confirmation input from the user (N in S16, S14).

[0073] When the user inputs a confirmation (Y in S16), the position information acquisition unit 80 starts acquiring the user's position information based on the captured image (S18). The information processing unit 76 then uses the acquired information to perform information processing, and the output data generation unit 78 draws the display image of the content as a result (S20). Meanwhile, the warning status determination unit 82 monitors whether the user has left the play area (S22). Strictly speaking, it is determined that the user has left the play area when the HMD 18 worn by the user, or a marker provided on the input device 14 held by the user, leaves the play area.

[0074] However, the criteria for this determination may vary depending on the means for acquiring the user's position information. For example, if the position information is obtained by creating a depth map from the image of the user itself in a stereo image, the criteria may be when the center of gravity of the user leaves the play area. The same applies when tracking a contour line. Also, if tracking is performed by pattern matching such as face detection processing, the criteria may be when the face is no longer recognized within the play area.

[0075] If the user has not left the play area (N in S22), the output data transmission unit 86 outputs the image of the content drawn in S20 to the HMD 18 as is (S26). If the user has left the play area (Y in S22), the warning status determination unit 82 notifies the output data generation unit 78 of this, causing the output data generation unit 78 to superimpose a warning image on the image of the content (S24). The output data transmission unit 86 then outputs the image with the warning image superimposed to the HMD 18 (S26). Note that in the process of S26, audio data may also be output at the same time.

[0076] When there is no need to end the process, such as when there is a request from the user to stop the process, the process of drawing an image of the content based on the position information, superimposing a warning image as necessary, and outputting the image to the HMD 18 is repeated at a predetermined rate (N of S28, S20 to S26). Then, when it becomes necessary to end the process, all the processes are ended (Y of S28).

[0077] In the flowchart in the figure, the period during which the user is outside the play area coincides with the period during which the warning image is superimposed. In other words, the warning image is hidden when the user returns to the play area. On the other hand, the conditions for hiding the warning image may be set independently of the conditions for displaying it. FIG. 13 shows a schematic diagram of the state transition of displaying / hiding the warning image. First, as described above, the criterion for determining when the warning image changes from a state in which it is not displayed to a state in which it is displayed is that the user has left the play area (rightward arrow).

[0078] On the other hand, the criterion for changing from a displayed warning image to a hidden state is not when the user enters the play area, but when a separately set condition is met (arrow pointing left). For example, a warning hidden area is set as an area narrower than the play area, and the warning image is hidden when the user enters that area. Alternatively, the warning image is hidden when a predetermined time has passed since entering the play area. In this way, once a warning is displayed, it is difficult for it to be hidden. As a result, when the user is near the boundary of the play area, a slight movement can switch the inside / outside determination result for the play area, preventing chattering in which the warning is repeatedly displayed and hidden.

[0079] According to the embodiment described above, in a system that acquires user position information based on an image in a captured image and processes information using the result, a play area is set in a three-dimensional space in the real world, and when the user leaves the play area, a warning to that effect is displayed to the user, etc. This prevents the user from being unable to see the outside world while wearing an HMD, or from being immersed in a displayed virtual world, and suddenly going out of the field of view of the camera, causing information processing to fail.

[0080] Here, the play area is set based on the camera's field of view. Elements such as whether to set the play area inside the field of view with a margin area in mind, or to make it the same as the field of view, and if a margin area is to be set, its width, are optimized. Adjusting the margin area in this way determines the range in which movement can be made without warning, and at the same time adjusts the frequency with which the user will go outside the field of view and location information cannot be acquired, and thus the accuracy of location information acquisition. This allows for a setting that matches the priority, such as whether to allow as wide a range of movement as possible even at the expense of some deterioration in the accuracy of location information acquisition, or to maintain the accuracy of location information acquisition even if the range is limited.

[0081] Since such priorities change depending on the horizontal / vertical angle of view of the camera, the content of information processing, the user and the surrounding situation, etc., it is possible to achieve both the accuracy of acquiring position information and the comfort of the user by optimizing the settings for each. Also, since there are no particular limitations on the shape of the play area, if the shape and size are set appropriately, it is possible to simultaneously fulfill roles such as complying with the appropriate value of the amount of movement expected from content such as games and avoiding collisions with surrounding obstacles, in addition to preventing deviation from the camera's field of view.

[0082] When displaying a warning, a component image showing the warning is superimposed on the original display image such as the game screen. At this time, the edges are blurred or made semi-transparent to enhance affinity with the original display image and reduce the sense of incongruity caused by the sudden appearance of an unrelated image. In addition, by setting the image at a specified distance from the viewpoint or displaying it in the lower half of the screen, discomfort and nausea caused by the line of sight being blocked are suppressed, and the user can easily focus their gaze and recognize the situation.

[0083] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0084] 8 Information processing system, 10 Information processing device, 12 Imaging device, 14 Input device, 16 Flat display, 18 HMD, 22 CPU, 24 GPU, 26 Main memory, 72 Input information acquisition unit, 74 Captured image acquisition unit, 76 Information processing unit, 78 Output data generation unit, 80 Position information acquisition unit, 82 Warning status determination unit, 85 Area setting information storage unit, 86 Output data transmission unit.

Claims

1. an information processing unit that detects a position of an object in a real space based on an image captured by the imaging device and processes information based on the object; an image generating unit that generates data of an image to be displayed as a result of the information processing; a warning status determination unit that sets the play area by changing a setting rule for the position of the boundary surface of the play area based on the boundary surface of the field of view of the imaging device according to the situation of the real space acquired by analyzing the captured image, and determines the need for a warning to the user based on the play area and superimposes a warning image on the image generation unit; an output unit that outputs data of the image generated by the image generation unit to a display device; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the warning state determination unit changes the setting rule depending on a posture state of the user.

3. 3. The information processing apparatus according to claim 1, wherein the warning state determination section further changes the setting rule in accordance with a width of a visual field of the imaging device.

4. 4. The information processing device according to claim 1, wherein the warning state determination unit sets the play area to have boundaries defined by two parallel vertical planes.

5. 5 . The information processing device according to claim 1 , wherein the warning state determination unit sets the play area to have a quadrilateral cross section relative to the distance from the imaging device.

6. 6. The information processing device according to claim 1, wherein the warning status determination unit determines that a warning to the user is necessary when the object moves outside the play area.

7. The information processing device according to claim 6, characterized in that the warning situation determination unit makes the superimposed warning image invisible under conditions other than the object entering the play area used when it was determined that the warning was necessary.

8. 8. The information processing apparatus according to claim 1, wherein the output unit outputs to the display device a confirmation screen that prompts the user to check whether there is an obstacle, and includes a real-time captured image on the confirmation screen.

9. 9. The information processing device according to claim 1, wherein the warning state determination unit acquires a distance of the object in an optical axis direction based on a stereo image captured by a stereo camera constituting the imaging device.

10. The information processing apparatus according to claim 9 , wherein the optical axis direction of the stereo cameras is not in a horizontal direction in real space.

11. The information processing device described in claim 9 or 10, characterized in that when the optical axis of the imaging device is not in a horizontal direction in real space, the warning situation determination unit obtains the positional relationship between the play area and the object by defining the shape of the play area with respect to the distance in the optical axis direction.

12. The information processing device described in claim 9 or 10, characterized in that the warning situation determination unit defines the shape of the play area in relation to the horizontal distance in real space, and when the optical axis of the imaging device is not in the horizontal direction, obtains the positional relationship between the play area and the object by coordinate transformation based on the measured value of the angle between the optical axis and the horizontal direction.

13. The information processing device according to claim 1, characterized in that the warning situation determination unit defines an area corresponding to the play area on the image plane captured by the imaging device, and determines the need for the warning based on the position of the image of the object on the captured image.

14. The information processing device according to claim 13, wherein the warning situation determination unit determines that the warning is necessary when an image of the object in either one of the images falls outside a set area by setting an area corresponding to a space captured by both stereo cameras constituting the imaging device in each of the stereo images captured by the stereo cameras.

15. The information processing device according to claim 1 , wherein the warning state determination unit sets a boundary surface of the play area based on a boundary surface of a product of angles of view of stereo cameras constituting the imaging device.

16. Detecting a position of an object in a real space based on an image captured by an imaging device, and performing information processing based on the object; generating data of an image to be displayed as a result of the information processing; a step of setting a play area by changing a setting rule for a boundary surface position of the play area based on a boundary surface of a field of view of the imaging device according to a state of a real space acquired by analyzing the captured image, and determining the need for a warning to a user based on the play area; superimposing a warning image on the image to be displayed in response to the determination; outputting data of the generated image to a display device; 2. A method for presenting a warning by an information processing device, comprising:

17. A function of detecting the position of an object in real space based on an image captured by an imaging device and performing information processing based on the object; A function of generating data of an image to be displayed as a result of the information processing; a function for setting a play area by varying a setting rule for the position of the boundary surface of the play area based on the boundary surface of the field of view of the imaging device according to the situation of the real space acquired by analyzing the captured image, determining the necessity of warning the user based on the play area, and notifying a function for generating data of the image of the necessity, thereby superimposing a warning image; A function of outputting the generated image data to a display device; A computer program characterized by causing a computer to execute the above.

Images