Information processing device and method, and non-volatile computer-readable storage medium

Abstract

An information processing device and method and a non-volatile computer-readable storage medium are provided. The information processing device includes a circuit configured to: generate a virtual change object obtained by changing the state of an object recognized in a real space; generate an object image of the object; and control display of the object image on the virtual change object.

Description

[0001] This application is a divisional application of an invention patent application with an application date of October 3, 2014, application number 201480067824.9 (international stage application number PCT / JP2014 / 076618), and invention name “INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD, AND PROGRAM”. Technical Field

[0002] The present disclosure relates to an information processing device, an information processing method, and a program. Background Art

[0003] In recent years, a technology called augmented reality (AR) has attracted attention, which superimposes additional information on a real space and presents the information to a user. In AR technology, the information presented to the user is visualized by using various forms of virtual objects (such as text, icons, or animations). The virtual object is positioned in the AR space according to the position of the real object associated with the virtual object, and in the AR space, for example, the virtual object can move, collide, and deform.

[0004] For example, Patent Document 1 cited below discloses a technique for deforming a virtual object and a technique for moving a virtual object while determining a collision with a terrain represented by the virtual object.

[0005] Reference List

[0006] Patent Literature

[0007] Patent Document 1: JP2012-141822A Summary of the invention

[0008] Technical issues

[0009] However, the technology disclosed in the patent document 1 cited above only represents changes in virtual objects (such as deformation of virtual objects and interactions between virtual objects). In order to give users a stronger impression that the real world is enhanced by using AR technology, it is desirable to provide a representation of changing objects in real space.

[0010] Solution to the problem

[0011] According to the present disclosure, there is provided an information processing device including: a recognition unit configured to recognize an object included in a real space to distinguish the object from a background based on three-dimensional data of the real space so as to generate a virtual object image obtained by changing a state of the object.

[0012] According to the present disclosure, there is provided an information processing method including: recognizing an object included in a real space by a processor to distinguish the object from a background based on three-dimensional data of the real space so as to generate a virtual object image obtained by changing a state of the object.

[0013] According to the present disclosure, a program is provided that causes a computer to function as: a recognition unit configured to recognize an object included in a real space to distinguish the object from a background based on three-dimensional data of the real space so as to generate a virtual object image obtained by changing the state of the object.

[0014] Beneficial effects of the present invention

[0015] As described above, according to the present disclosure, it is possible to give the user a stronger impression that the real world is enhanced by using AR technology. Please note that the above effects are not necessarily limited, and any effect that is desired to be introduced in this specification or other effects that can be expected from this specification may be displayed together with or instead of these effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A diagram for explaining an overview of an AR display process according to an embodiment of the present disclosure.

[0017] Figure 2 : is a block diagram showing a configuration example of a smartphone according to this embodiment.

[0018] Figure 3 A diagram for explaining a process for recognizing a target object.

[0019] Figure 4 An explanatory diagram for explaining the process for generating a background texture.

[0020] Figure 5 An explanatory diagram for explaining the process for generating a background texture.

[0021] Figure 6 An example of a UI for generating a background texture is shown.

[0022] Figure 7 A diagram to explain the dynamic rendering mask generation process caused by the state change of the target object.

[0023] Figure 8 An explanatory view for explaining a process for displaying a background texture.

[0024] Fig. 9 An explanatory view for explaining a process for displaying a background texture.

[0025] Fig.10 An explanatory view for explaining a process for displaying a background texture.

[0026] Fig.11 An example of a process for correcting the brightness of a varying texture is shown.

[0027] Fig.12 The flowchart is a flowchart showing an example of a process flow for generating a change object, which process flow example is executed in a smartphone according to this embodiment.

[0028] Fig.13 The following is a flowchart showing an example of a cache processing flow that is executed in a smartphone according to this embodiment.

[0029] Fig.14 The flowchart is a flowchart showing an example of a process flow for generating a background texture, which is executed in a smartphone according to this embodiment.

[0030] Fig.15 A flowchart showing an example of a process flow for generating a changing texture, which process flow example is executed in a smartphone according to this embodiment.

[0031] Fig.16 The following is a flowchart showing an example of an AR display processing flow that is executed in a smartphone according to this embodiment. DETAILED DESCRIPTION

[0032] The preferred embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. In this specification and the accompanying drawings, structural elements having substantially the same function and structure are denoted by the same reference numerals, and repeated explanation of these structural elements is omitted.

[0033] Note that the description will be given in the following order.

[0034] 1. Overview

[0035] 2. Configuration example of smartphone

[0036] 3. Operation processing

[0037] 3-1. Processing of generating change objects

[0038] 3-2. Cache processing

[0039] 3-3. Processing of generating background texture

[0040] 3-4. Processing of Generating Changing Textures

[0041] 3-5. AR display processing

[0042] 4. Conclusion

[0043] <1. Overview>

[0044] Will combine Figure 1 An overview of an AR display process according to an embodiment of the present disclosure is described.

[0045] Figure 1 FIG. 1 is a diagram for explaining an overview of an AR display process according to an embodiment of the present disclosure. Figure 1 As shown in FIG. 1 , the user looks at the display unit 7 while holding the smartphone 1, and the image pickup unit on the back side of the display unit 7 faces the table. On the table, a phone 10, a strap 11, a helmet 12, a beverage can 13, and a spray can 14 are placed. Figure 1 As indicated by reference numeral 100 in FIG. 1 , the smartphone 1 according to the present embodiment can perform a display obtained by changing the objects placed on the table in the real space.

[0046] like Figure 1 As indicated by the reference numeral 100 in the figure, the smartphone 1 displays a state in which a ball 120 (which is a virtual object) is thrown from the smartphone 1 and collides with an object on the table to produce an effect on the real space. Specifically, the smartphone 1 displays by superimposing AR images 111 and 112 showing the flying packing tape 11 and the damaged helmet 12 on a captured image obtained by capturing a real-space image in real time. Moreover, the smartphone 1 displays by superimposing AR images 121 and 122 showing the background on respective areas in which the packing tape 11 and the helmet 12 are initially placed, and these areas are those parts newly exposed due to movement or destruction. Although the phone 10, the beverage can 13, and the spray can 14 are real objects, the ball 120 does not collide with the phone 10, the beverage can 13, or the spray can 14. Therefore, as Figure 1 As indicated by the reference numeral 100 in the figure, the smartphone 1 displays through images of the phone 10, the beverage can 13, and the spray can 14 in their current conditions. Hereinafter, among real objects, an object that becomes a target subject to a state change, such as an object that flies out or is broken, will be referred to as a target object, and objects other than the target object, such as a table or a wall, will also be referred to as a background object. A captured image obtained by capturing a real space image in real time will also be referred to as a through image.

[0047] As a first prior preparation for realizing such representation, first, the smartphone 1 acquires three-dimensional data of the real space. The three-dimensional data is data containing position information about the vertices of objects in the real space, the line segments connecting these vertices, and the surfaces surrounded by these line segments, and is information representing the three-dimensional shape (surface) of the real space. By using this three-dimensional data, the smartphone 1 can display an AR image (object image) at an appropriate position, and can represent interactions such as collisions between virtual objects and real objects.

[0048] As a second prior preparation, the smartphone 1 collects images based on which AR image related to the background object is generated, and generates a texture (surface image) of the surface of the background object. Specifically, first, with respect to the portion corresponding to the background object in each surface in the three-dimensional data, the smartphone 1 collects captured images obtained by capturing an image of an area corresponding to the portion in the real space, and caches the captured images. For example, in Figure 1 In the example shown in , when there is only a captured image captured on the front side of the strapping tape 11, the background object on the back side of the strapping tape 11 is not captured because the background object is hidden. Therefore, the smartphone 1 caches not only the captured image captured on the front side of the strapping tape 11 but also the captured image captured on the back side of the strapping tape 11. Thereafter, the smartphone 1 combines these cached captured images (cached images), thereby generating an image of a virtual texture (background texture) in which the target object does not exist.

[0049] As a third prior preparation, the smartphone 1 generates a texture based on which AR image related to the target object is generated. Specifically, first, the smartphone 1 generates a virtual object (changed object) obtained by changing the state of the target object, and calculates the current position. For example, Figure 1 In the example shown in , the smartphone 1 generates three-dimensional data of the destroyed portion 112 of the helmet 12. Thereafter, the smartphone 1 generates a changing texture to be displayed on the surface of the changing object.

[0050] In the case where the above-mentioned AR display is performed, the smartphone 1 displays the AR image while dynamically shielding the through image according to the state change (such as destruction or movement) of the object. Specifically, first, the smartphone 1 calculates the state change of the change object generated as a previous preparation, and dynamically determines the current position of the change object in the through image. For example, Figure 1 In the example shown in , the current position of the flying strap 111 and the current position of the damaged portion 112 of the helmet 12 are calculated. Then, the smartphone 1 generates a drawing mask that, when displaying AR, allows the area used for displaying differently from the real space to be shown in the through image. Figure 1 The strapping tape 11 shown in FIG. 1 is used to generate a drawing mask that is transparent to the background newly exposed due to the strapping tape 11 flying out and the area for displaying the flying strapping tape 11. Afterwards, the smartphone 1 displays the texture corresponding to these areas that are transparent in the drawing mask.

[0051] Regarding the AR image of the background object, the smartphone 1 displays the background texture generated in advance on the surface of the background object by using the drawing mask. Figure 1In the example shown in , the corresponding area 112 of the background texture is displayed in the background area newly exposed due to the flying of the strapping tape 11. In this way, the smartphone 1 can display a natural background in the area newly exposed due to the change in the state of the target object.

[0052] Regarding the AR image of the target object, the smartphone 1 displays the change texture generated in advance on the surface of the change object. However, in order to more naturally express the state change of the target object, the smartphone 1 attaches the corresponding area in the through image to the surface portion of the change object exposed in the through image, and attaches the change texture to other areas. For example, Figure 1 In the example shown in , since the peripheral portion of the strapping tape 11 is exposed, the smartphone 1 displays the peripheral portion by attaching the image of the peripheral portion in the through image to the peripheral portion 111 of the strapping tape that flew out. At the same time, since, for example, the inside of the helmet 12 is hidden, the smartphone 1 displays the change texture generated in advance in the broken cross section of the broken portion 112. In this way, the smartphone 1 can more naturally express the target object whose state has changed.

[0053] Note that although Figure 1 An example is shown in which the information processing device according to the embodiment of the present disclosure is implemented as a smartphone 1, but the technology according to the present disclosure is not limited thereto. For example, the information processing device may be a head mounted display (HMD), a digital camera, a digital video camera, a tablet terminal, a mobile phone terminal, and the like. In the case where the information processing device is implemented as an HMD, the HMD| may display an AR image by superimposing it on a through image captured by a camera, or may display an AR image on a display unit formed in a transparent or translucent finish state.

[0054] The above has described the overview of the AR display processing according to the present embodiment. Figures 2 to 11 The configuration of the smartphone 1 according to the present embodiment is described.

[0055] <2. Configuration example of a smartphone>

[0056] Figure 2 1 is a block diagram showing a configuration example of the smartphone 1 according to the present embodiment. Figure 2 As shown in , the smartphone 1 includes an image pickup unit 2, a posture information acquisition unit 3, a three-dimensional data acquisition unit 4, a control unit 5, a display control unit 6, and a display unit 7.

[0057] (1) Image pickup unit 2

[0058] The image pickup unit 2 includes, for example, a lens system including an image pickup lens, an aperture, a zoom lens, and a focusing lens, a drive system for causing the lens system to perform focusing and zooming operations, and a solid-state image pickup element array for photoelectrically converting light obtained in the lens system and generating an image pickup signal. The solid-state image pickup element array can be implemented by, for example, a charge coupled device (CCD) sensor array or a complementary metal oxide semiconductor (CMOS) sensor array. The image pickup unit 2 can be a single-lens camera or a stereo camera. Using the image generation unit 53 described below, the captured image captured by the single-lens camera or the stereo camera can be used to generate textures, or, using the three-dimensional data acquisition unit 4, the captured image can be used to generate dynamic three-dimensional data.

[0059] The image pickup unit 2 according to the present embodiment has a function of capturing a real-space image to acquire an image for generating a texture using the below-described change object generation unit 52. The image pickup unit 2 also has a function of acquiring a through image obtained by capturing the real-space image in real time. The image pickup unit outputs the captured image to the control unit 5 and the display control unit 6.

[0060] (2) Posture information acquisition unit 3

[0061] The posture information acquisition unit 3 has a function of acquiring posture information indicating the position and angle (posture) of the smartphone 1. The posture information acquisition unit 3 particularly acquires posture information about the image pickup unit 2. In AR technology, in order to present truly useful information to the user, it is important that the computer accurately captures the situation in the real space. Therefore, a technology for capturing the situation in the real space has been developed as the basis of AR technology. One such technology is a technology called simultaneous localization and mapping (SLAM), which can, for example, simultaneously estimate the position and posture of a camera and the position of feature points present in the image of the camera. The basic principles of SLAM technology using a single-lens camera are described in "Andrew J. Davison," Real-Time Simultaneous Localization and Mapping with a Single Camera ", Proceedings of the 9th IEEE International Conference on Computer Vision Volume 2, 2003, pp. 1403-1410". Note that, specifically, SLAM technology for visually estimating positions by using camera images is also referred to as visual SLAM (VSLAM). In SLAM technology, the position and posture of the camera are estimated by using an environment map and a camera image. In a case where SLAM technology is used, for example, the posture information acquisition unit 3 can accurately locate polygon information, which is used to form the shape of the object relative to the real object by matching the environment map with the three-dimensional position of the feature points belonging to the object. Based on the result of the positioning, the posture information acquisition unit 3 collects posture information about the image pickup unit 2. In addition, the posture information acquisition unit 3 can use a posture estimation technology using a marker (a technology called dense real-time tracking and mapping (DTAM) or a technology called Kinect fusion). The posture information acquisition unit 3 can also collect posture information based on information detected by an acceleration sensor, an angular velocity (gyroscope) sensor, and a geomagnetic sensor. The posture information acquisition unit 3 outputs the collected posture information to the control unit 5 and the display control unit 6.

[0062] (3) Three-dimensional data acquisition unit 4

[0063] The three-dimensional data acquisition unit 4 has a function of acquiring three-dimensional data of a real space. The three-dimensional data can be prepared by, for example, a single-lens image pickup sensor, a multi-lens image pickup sensor, or a shape sensor using infrared rays. The three-dimensional data acquisition unit 4 can generate three-dimensional data by using the image pickup unit 2 or a shape sensor using infrared rays (not shown) and the posture information acquisition unit 3, or can acquire three-dimensional data generated in advance in another terminal from an external device. The three-dimensional data is implemented as, for example, computer-aided design (CAD) data. The three-dimensional data acquisition unit 4 outputs the acquired three-dimensional data to the control unit 5 and the display control unit 6.

[0064] (4) Control unit 5

[0065] The control unit 5 functions as an arithmetic processing unit and a control device, and controls the entire operation in the smartphone 1 according to various programs. The control unit 5 is implemented, for example, by an electronic circuit such as a central processing unit (CPU) or a microprocessor. Note that the control unit 5 may include a read-only memory (ROM) for storing programs to be used, calculation parameters, etc., and a random access memory (RAM) for temporarily storing parameters that change from time to time, etc. Figure 2 As shown in , the control unit 5 functions as a recognition unit 51 , a change object generation unit 52 , and an image generation unit 53 .

[0066] (4-1) Identification Unit 51

[0067] The recognition unit 51 has a function of recognizing a target object included in the real space based on the three-dimensional data of the real space collected by the three-dimensional data collection unit 4 to distinguish the target object from the background object. For example, assuming that the three-dimensional data indicates an artificial object (e.g., the internal structure of a room), the recognition unit 51 recognizes the floor surface (dominant plane) and the wall surface from the three-dimensional data. After that, the recognition unit 51 recognizes a three-dimensional shape other than the floor surface or the wall surface as the target object, which protrudes on the floor surface and has a volume equal to or greater than a threshold value. Note that, for example, a part of the identified target object that was initially in contact with the floor surface or the wall surface may be a missing missing part (hole) because the part is not registered as a surface in the three-dimensional data. Therefore, the recognition unit 51 interpolates the missing part by extending the vertices around the missing part, connecting the vertices around the missing part with lines, or compensating the surface. In addition, the recognition unit 51 can interpolate the missing part by using a previously existing algorithm. Using this method, it becomes possible to calculate the volume because the diversity of the target object is guaranteed. Therefore, the recognition unit 51 can appropriately recognize the target object. The target object identified by the recognition unit 51 is a target to be used for generating a change object using the change object generation unit 52 described below. Among the three-dimensional shapes indicated by the three-dimensional data, the recognition unit 51 recognizes the three-dimensional shapes other than the target object as background objects. The background objects identified by the recognition unit 51 as being distinguishable from the target object are excluded from the targets to be used for generating a change object using the change object generation unit 52 described below. The recognition unit 51 can interpolate the missing part of the background object in the same way as the method of interpolating the missing part of the target object. Figure 3 A process for recognizing a target object using the recognition unit 51 is described.

[0068] Figure 3 is a diagram for explaining the process of identifying a target object. In this paper, it is assumed that Figure 3The three-dimensional data indicated by reference numeral 200 in the figure is collected by the three-dimensional data collection unit 4. The recognition unit 51 extracts a plane from the three-dimensional data 200, and recognizes the plane 201 having the largest area as the floor surface. At this time, the recognition unit 51 can extract the floor surface 201 based on the gravity direction detected by the acceleration sensor of the terminal that has collected the three-dimensional data 200. In the case where the three-dimensional data 200 is generated by the three-dimensional data collection unit 4, the floor surface 201 is extracted based on the gravity direction detected by the acceleration sensor when the data is generated. The use of the gravity direction improves the accuracy of the floor surface extraction. Note that it is expected that in the case where the three-dimensional data indicates a natural shape, the floor surface (ground) is rough, and therefore the recognition unit 51 can extract the floor surface by using a moderate gradient threshold. Further, the recognition unit 51 recognizes a wall surface 202 installed on the floor surface from the object. For example, the recognition unit 51 defines a rectangle (axis-aligned bounding box) and recognizes a shape group that is perpendicular to the floor surface and provided in an end portion of the three-dimensional shape indicated by the three-dimensional data as the wall surface 202. Using the above-mentioned processing, the recognition unit 51 recognizes the target object 210 and the background objects 220 other than the target object to separate the target object 210 from the background objects 220.

[0069] (4-2) Change object generation unit 52

[0070] The change object generation unit 52 has a function of generating a change object obtained by changing the state of the target object recognized by the recognition unit 51. Specifically, the change object generation unit 52 generates data containing position information about the vertices of the change object, the line segments connecting the vertices, and the surface surrounded by the line segments. The change object may be, for example, a destroyed portion obtained by destroying the target object or a target object having an uneven surface. For example, the change object generation unit 52 generates a change object indicating the destroyed portion by applying an algorithm such as Voronoi Fracture or Voronoi Shatter to the target object. Figure 3 A process of generating a change object using the change object generating unit 52 is described.

[0071] like Figure 3 As shown in FIG. 2 , the change object generation unit 52 forms a destroyed cross section generated by the destruction on the target object 210 recognized by the recognition unit 51, and divides the target object into a plurality of parts, thereby generating a change object 230 indicating the destroyed parts. At this time, the change object generation unit 52 sets flag information indicating whether the surface of each destroyed part was a surface of the target object before the target object was destroyed. When the display control unit 6 described below draws these destroyed parts, the flag information is referred to.

[0072] (4-3) Image Generating Unit 53

[0073] The image generation unit 53 has a function of generating in advance a texture prepared in advance for AR display based on which AR image is generated. Specifically, the image generation unit 53 generates a virtual object image of the background object identified by the recognition unit 51, that is, a texture (background texture) displayed on the surface of the background object newly exposed due to the state change. The image generation unit 53 also generates a virtual target image obtained by changing the state of the target object identified by the recognition unit 51, that is, a texture (changing texture) displayed on the surface of the changing object generated by the changing object generation unit 52. The image generation unit 53 generates a texture based on a captured image obtained by, for example, capturing an image of a real space using the image pickup unit 2.

[0074] (4-3-1) Generate background texture

[0075] The image generation unit 53 combines the background parts exposed in one or more captured images obtained by capturing images of the real space, thereby generating a background texture (first surface image) to be displayed on the surface of the background object. Specifically, the image generation unit 53 collects images corresponding to the background object from the one or more captured images, and combines these images by using an area divided by a line segment linking vertices in the three-dimensional data (hereinafter also referred to as a polygon) as a minimum amount unit, thereby generating an image of the background texture. Figure 4 An explanatory diagram for explaining the process for generating a background texture. Figure 4 The correspondence between the captured image of the real space and the three-dimensional data is shown. Figure 4 In FIG. 5 , the surface of the target object is shown as a polygon surrounded by solid lines, and the surface of the background object is shown as a polygon surrounded by dotted lines. The image generation unit 53 collects the image corresponding to the image generated by the one or more captured images. Figure 4 The dashed lines in the figure indicate the polygonal images of the background objects and these images are combined to generate the background texture.

[0076] The image generation unit 53 caches a plurality of captured images captured by the image pickup unit 2, and forms a background texture by combining one or more cached images. When caching, the image generation unit 53 preferentially caches captured images with high independence. High independence means that the overlapping portion where the background is hidden by the target object is small. In this way, the image generation unit 53 can generate a background texture with more polygons and fewer images, and thus the number of seams generated when combined is reduced. Thus, a more natural AR display is achieved. The image generation unit 53 preferentially caches the captured images that have been recently captured. In this way, the image generation unit 53 can generate a background texture with a more recently captured captured image. This reduces the difference between the moment when the cached image is captured and the moment when the through image is captured based on which AR image is generated. Thus, a more natural AR display is achieved. Note that in this specification, it will be assumed that the image generation unit 53 uses multiple cached images to generate a background texture for description, but the technology according to the present disclosure is not limited thereto. For example, the image generation unit 53 can generate the following changing texture by using multiple cached images. Furthermore, the image generation unit 53 may generate an arbitrary texture related to a real object by using a plurality of cache images.

[0077] The image generation unit 53 determines the degree of independence for each vertex. Specifically, the image generation unit 53 performs cache to visualize as many vertices of the background object as possible. For example, the image generation unit 53 determines the invisible vertices and visible vertices of the background object, and caches them so that all vertices become visible vertices in any captured image. Because visibility is determined for each vertex, the number of determinations is reduced, and the amount of calculation is reduced, compared to a situation in which visibility is determined for each vertex. Note that, among the vertices of the three-dimensional data, the visible vertices are vertices whose corresponding positions in the captured image are exposed. At the same time, among the vertices of the three-dimensional data, the invisible vertices are vertices whose corresponding positions in the captured image are hidden. For example, in Figure 4 In the example shown, the vertex 310 in front of the helmet 12 is a visible vertex, and the hidden vertex 320 behind the helmet 12 is an invisible vertex. Of course, when the machine capability increases, the image generation unit 53 can determine visibility for each pixel.

[0078] The image generation unit 53 determines visibility of vertices in the captured image and determines whether to cache by calculating at what position and at what angle the image pickup unit 2 captures the image in the real space using the posture information acquired by the posture information acquisition unit 3 and three-dimensional data. Figure 5 is an explanatory view explaining the process for generating a background texture. Figure 5As shown in , the image generation unit 53 dynamically calculates the vertex positions of the target object by using the posture information and the three-dimensional data, and generates a mask 410 that penetrates the area of ​​the target object in the captured image 400. Thereafter, the image generation unit 53 determines the vertices included in the area penetrated by the mask 410 as invisible vertices, and determines the other vertices as visible vertices. Note that, unlike using the mask 410, the image generation unit 53 can determine the visibility of the vertices by using a mask 420 obtained by two-dimensionally increasing the penetrated area in the mask 410. In this case, vertices within a wider range are determined as invisible vertices, and thus the posture estimation error of the posture information acquisition unit 3 is corrected. This prevents erroneous determination, that is, prevents the invisible vertices from being erroneously determined as visible vertices.

[0079] As described above, the image generation unit 53 generates an image of the background texture by combining one or more buffered images. The image generation unit 53 preferentially combines captured images having a large visible area. More specifically, the image generation unit 53 combines the buffered images so that the area obtained by combining captured images having a large visible area occupies a larger area in the background texture. In this way, the background texture is generated with fewer captured images, and thus a more consistent and natural AR display is achieved. Note that the image generation unit 53 can determine that the visible area is large in a case where the number of visible vertices is large, and can determine that the visible area is small in a case where the number of visible vertices is small.

[0080] In the background object, a polygon that is not visible in all cached images, such as a floor surface in contact with the target object, may be a blank missing portion in the combined background texture. Therefore, the image generation unit 53 may interpolate the missing portion in the background texture by using an image interpolation algorithm such as Inpainting. Note that, in the background texture, an area obtained by combining cached images will be referred to as a background texture in a visible area, and an area interpolated by using an image interpolation algorithm such as Inpainting will be referred to as a background texture in an invisible area.

[0081] The smartphone 1 may provide a user interface (UI) for caching captured images with a high degree of independence. For example, the smartphone 1 displays a UI for guiding the image pickup unit 2 so that the entire area corresponding to a portion of the background in the surface of the solid body indicated by the three-dimensional data is exposed in at least one of the one or more cached captured images. Specifically, the smartphone 1 guides the user so that all vertices in the background in the three-dimensional data are visible vertices in any cached captured image. Figure 6 An example of a UI for generating a background texture is shown. Figure 6The UI 500 shown in shows, for example, a display 512 showing that the rear surface of the helmet 12 and the surface thereof in contact with the floor are not included in the cached captured image. Therefore, the UI 500 shows a display 520 for guiding the capture of an image from the rear surface side of the helmet 12. In a case where a captured image obtained by capturing the strapping tape 11 from the background side of the strapping tape 11 has been cached, the UI shows a display 511 showing that the surface of the strapping tape 11 in contact with the floor is not included in the cached captured image. Further, the smartphone 1 can perform caching while giving a feeling that the user is playing a game by displaying a display 530 showing a collection rate. Such display of the UI is performed by the display control unit 6 described below.

[0082] For example, environmental changes occur in real space over time, and in some cases multiple cached images have different brightness values. In view of this, when caching or generating textures, the image generation unit 53 records brightness values ​​near the positions corresponding to vertices in the three-dimensional data, which exist in the background exposed in the cached captured image. The display control unit 6 described below compares these brightness values ​​so that these images match the through images, thereby correcting the brightness value of the texture. Therefore, a more natural AR display is achieved. To improve the accuracy of the correction, the image generation unit 53 can record multiple brightness values ​​associated with the vertex as a brightness value distribution. Among the visible vertices, the image generation unit 53 only records the pixel values ​​of the visible vertices of the background object near the position where the background texture is displayed. This reduces the number of brightness value comparisons, which are performed by the display control unit 6 described below for matching, and improves the accuracy of the brightness value correction. Of course, when the machine capacity increases, the recording of pixel values ​​and the comparison of brightness values ​​can be performed for all visible vertices. Note that the image generation unit 53 can determine whether to record the brightness value based on the normal vector corresponding to the position of the vertex in the captured image. For example, when the image generation unit 53 records brightness values ​​in a state where the direction of the normal vector is toward the image pickup unit 2, the image generation unit 53 can prevent brightness values ​​at positions that may cause interference (such as lateral interference) from being recorded. The image generation unit 53 records a sufficient number of pixel values ​​of the vertices to exclude the influence caused by abnormal values.

[0083] (4-3-2) Generate changing texture

[0084] The image generation unit 53 generates a change texture to be displayed on the surface of the change object generated by the change object generation unit 52. As the texture about the change object to be displayed, for example, there are two textures, that is, the texture of the invisible part such as the cross section of the destroyed part and the texture of the visible part that was originally exposed. The image generation unit 53 generates the former texture as the change texture (second surface image). Regarding the latter texture, the corresponding part in the through image is displayed by the display control unit 6 described below.

[0085] The image generation unit 53 generates a changing texture by making an inference based on the portion of the exposed surface corresponding to the target object in the captured image. For example, when generating a changing texture, the image generation unit 53 may determine a single color based on, for example, the average value of pixel values ​​in a portion of the target object exposed in the captured image. The polygons in the invisible area of ​​the changing object are painted with the single color by the display control unit 6 described below. In addition, the image generation unit 53 may generate the changing texture by using an image interpolation algorithm such as Inpainting. Figure 4 In the example shown, the surface of the helmet 12 is visible, and the interior of the helmet 12 is not visible. Figure 1 In the case of the AR image 112 of the destroyed helmet 12 shown in , the image generation unit 53 averages the pixel values ​​of the surface of the helmet 12 to determine a single color as the color of the changing texture of the invisible destroyed cross section of the destroyed portion.

[0086] As described above, the background texture and the changing texture are generated based on the image captured by the image pickup unit 2, but the technology according to the present disclosure is not limited to this. For example, the image generation unit 53 can generate the background texture and the changing texture based on the captured image captured in advance by an external image pickup device. However, considering the possibility that the image pickup conditions are different, it is more preferable that the image pickup unit for capturing the captured image for generating the background texture and the changing texture and the image pickup unit for capturing the through image on which the AR image is superimposed are the same. Note that the background texture and the changing texture may not be generated using the captured image. For example, the image generation unit 53 may generate a texture coated with an arbitrary monochrome or may generate a texture representing perspective based on the result of sensing performed by a depth sensor (not shown).

[0087] (5) Display control unit 6

[0088] The display control unit 6 has a control display unit 7 so that the display unit 7 performs AR display by using the texture or through-image generated by the image generation unit 53. Specifically, first, the display control unit 6 dynamically determines the vertex positions of the background object and the changing object in the through-image by using the posture information collected by the posture information collection unit 3 and the three-dimensional data collected by the three-dimensional data collection unit 4. After that, the display control unit 6 displays the through-image on which the AR image is superimposed, and dynamically masks the through-image according to the state change of the changing object. The control performed by the display control unit 6 can be roughly divided into the following four categories: calculating the state of the changing object; generating a dynamic drawing mask; displaying the background texture; and texture display on the changing object.

[0089] (5-1) Calculate the state of the changed object

[0090] The display control unit 6 calculates the state of the changed object. Specifically, first, the display control unit 6 calculates the movement of the changed object. Thereafter, the display control unit 6 dynamically determines the vertex positions of the background object and the changed object in the through image by using the posture information and the three-dimensional data. For example, in the case where the state is changed to destruction, the display control unit 6 completely calculates the position and posture of the destroyed part, and determines the vertex position of the background object in the through image and the vertex position of each destroyed part.

[0091] (5-2) Generate dynamic drawing mask

[0092] The display control unit 6 dynamically generates a drawing mask that passes through a region in the through image for performing AR display different from the real space according to the state change of the target object. Figure 1 In the example shown in , the helmet 12 moves backward due to the impact of the collision with the ball 120, and thus a drawing mask is generated that transmits the background newly exposed due to the movement and the area in which the destroyed portion of the destroyed helmet 12 is located. Note that the display control unit 6 can increase such a transparent area two-dimensionally or apply Gaussian blur to the drawing mask. In this way, the seam between the original background and the texture drawn by using the mask is more naturally displayed. Figure 7 A diagram to explain the dynamic rendering mask generation process caused by the state change of the target object. Figure 7 Reference numerals 610 and 620 in FIG. 7 indicate images displayed by the display unit 7, and reference numerals 612 and 622 indicate drawing masks dynamically generated by the display control unit 6. Figure 7As indicated by reference numeral 610 in FIG. 1 , in the case where there is no state change of the target object, the display control unit 6 does not perform AR display, displays the through image as it is, and thus generates a drawing mask having no through area as indicated by reference numeral 612. Figure 7 As indicated by reference numeral 620 in FIG. 1 , in a case where the state of the target image changes, as indicated by reference numeral 622 , the display control unit 6 generates a drawing mask that penetrates the area for displaying the AR image.

[0093] (5-3) Display background texture

[0094] The display control unit 6 displays an AR image in which a background texture is attached to a polygon, which is a background object hidden by a target object and is newly exposed due to a change in the state of the target object. Specifically, the display control unit 6 displays an AR image in which a corresponding area in the background texture is attached to a newly exposed polygon in the background object and overlays it on the through image, while dynamically determining the vertex position of the background object in the through image using posture information and three-dimensional data. At this time, the display control unit 6 can correct the brightness value or pixel value of the background texture. For example, based on a comparison result between the brightness value distribution recorded during caching and the brightness value distribution in the corresponding position in the through image, the display control unit 6 corrects the brightness value of the background texture so that the two brightness values ​​become close to each other. In this way, the match between the background texture and the through image is improved, and a more natural AR display is achieved. Below, the display control unit 6 will be combined with Figures 8 to 10 The process of displaying background texture is described in detail.

[0095] Figures 8 to 10 An explanatory diagram showing background texture is used for explanation. Figures 8 to 10 A display example in which all target objects are deleted from the through image is shown as an example. Figure 8 An example is shown in which the background texture in the visible area is displayed in the background hidden by the target object. Figure 8 As shown in, for example, in the case where a captured image captured from the rear surface side of the helmet 12 is cached, a background texture in a visible area is displayed in an area of ​​the upper half of the helmet 12. In this way, display is performed as if the upper half of the helmet 12 is transparent. Fig. 9 An example is shown in which not only the background texture in the visible area but also the background texture in the invisible area is displayed in the background hidden by the target object. Fig. 9 As shown in FIG. 1 , in a case where a captured image obtained by capturing an image of a floor surface that the helmet 12 contacts is not cached, a background texture in an invisible area is displayed in an area corresponding to the floor surface of the lower half of the helmet 12. In this way, the display is performed as if the helmet 12 does not exist. Fig.10 Showing that from Fig. 9 A display example in which the display of three-dimensional data is deleted.

[0096] (5-4) Texture display on changing objects

[0097] The display control unit 6 displays the texture by attaching it to the surface of the changing object using the vertex positions of the changing object that have been dynamically determined. As described above, as the texture to be displayed regarding the changing object, there are two kinds of textures, namely, the texture in the invisible part and the texture in the visible part. Regarding the former texture, the display control unit 6 displays an AR image in which the changing texture generated by the image generation unit 53 is attached to the polygon newly exposed due to the change in the state of the changing object. For example, in Figure 1 In the example shown in , the display control unit 6 attaches a texture coated with a single color determined by the image generation unit 53 to a polygon corresponding to the destroyed cross section among the polygons corresponding to each surface of the destroyed portion. With regard to the latter texture, the display control unit 6 displays an AR image in which an image of the target object exposed in the through image is attached to a corresponding polygon in the changed object. For example, in Figure 1 In the example shown in , the display control unit 6 attaches the surface image of the helmet 12 exposed in the through image to the polygon corresponding to the surface of the helmet 12 among the polygons of each surface of the destroyed portion. Note that the display control unit 6 may determine whether to display the changing texture or to display a portion of the through image with reference to the flag information set by the changing object generating unit 52.

[0098] The display control unit 6 corrects the brightness according to the difference between the initial position of the target object and the dynamically determined position of the changing object. For example, the display control unit 6 estimates the position of the light source based on the brightness distribution of the target object in the through image, and calculates how to expose the changing object to the light from the estimated light source, thereby correcting the brightness distribution of the changing texture and displaying the changing texture. In this way, the display control unit 6 can express how to naturally expose the target object to light according to the state change (such as destruction or movement) of the target object. Fig.11 An example of processing for correcting the brightness of a varying texture is shown. Fig.11 As shown in , the position and angle of the destroyed portion 710 of the destroyed telephone 10 are changed due to the destruction. Therefore, the display control unit 6 represents the shadow by correcting the brightness value of the outside of the handset in the destroyed portion 710. By generating the shadow at the time of destruction, the display control unit 6 can more naturally represent the destruction of the target object. Note that the position of the light source may be a predetermined position.

[0099] (6) Display unit 7

[0100] Based on the control performed by the display control unit 6, the display unit 7 combines the through image captured by the image pickup unit 2 and the AR image generated by the image generation unit 53, and displays the combined image. The display unit 7 is implemented by, for example, a liquid crystal display (LCD) or an organic light emitting diode (OLED). In the case where the information processing device according to the present embodiment is implemented as an HMD, the display unit 7 can be formed into a transparent or translucent completed state, and the AR image displayed in the real space on the display unit 7 is displayed in this completed state. In addition, based on the control performed by the display control unit 6, the display unit 7 displays a UI for caching captured images with a high degree of independence, which has been combined Figure 6 Describe the UI.

[0101] In the above, the configuration example of the smartphone 1 according to the present embodiment has been described. Figures 12 to 16 The operation process of the smartphone 1 according to the present embodiment is described.

[0102] <3. Operation Processing>

[0103] Hereinafter, an operation process in an example in which the smartphone 1 displays an AR image in which a target object is destroyed will be described as an example.

[0104] [3-1. Processing of generating change objects]

[0105] Fig.12 The flowchart is a flowchart showing an example of a process flow for generating a change object, which process flow example is executed in the smartphone 1 according to the present embodiment.

[0106] like Fig.12 As shown in FIG. 1 , in step S102, the recognition unit 51 extracts the floor surface. Specifically, the recognition unit 51 extracts a plane from the three-dimensional data acquired by the three-dimensional data acquisition unit 4, and recognizes the plane with the largest area as the floor surface. At this time, the recognition unit 51 can recognize a shape group that is perpendicular to the floor surface and provided in the end portion of the three-dimensional shape indicated by the three-dimensional data as the wall surface 202.

[0107] In step S104, the recognition unit 51 separates the target object from the background object. Specifically, the recognition unit 51 recognizes a three-dimensional shape other than the floor surface or the wall surface as the target object, the three-dimensional shape protruding from the floor surface and having a volume equal to or greater than the threshold value. Moreover, among the three-dimensional shapes indicated by the three-dimensional data, the recognition unit 51 recognizes the three-dimensional shape other than the target object as the background object.

[0108] In step S106, the recognition unit 51 interpolates the missing portion of the target object. Specifically, with respect to the missing portion of a portion of the target object that is initially in contact with the floor surface or the wall surface, the recognition unit 51 interpolates the missing portion by extending vertices around the missing portion, connecting the vertices around the missing portion with lines, or compensating the surface. Similarly, the recognition unit 51 interpolates the missing portion of the background object.

[0109] In step S108, the changed object generation unit 52 generates a changed object obtained by changing the state of the target object. Specifically, the changed object generation unit 52 generates a changed object showing a destroyed portion by applying an algorithm such as Voronoi Fracture or Voronoi Shatter to the target object.

[0110] [3-2, Cache processing]

[0111] Fig.13 This is a flowchart showing an example of a cache processing flow that is executed in the smartphone 1 according to the present embodiment.

[0112] like Fig.13 As shown in , in step S202 , the image pickup unit 2 captures an image of the real space and outputs the captured image to the image generation unit 53 .

[0113] In step S204, the image generation unit 53 estimates the position and posture of the image pickup unit 2. Specifically, in the case where the image pickup unit 2 captures the image in step S202, the image generation unit 53 uses the posture information acquired by the posture information acquisition unit 3 and the three-dimensional data acquired by the three-dimensional data acquisition unit 4 to estimate at what position and at what angle in the real space the image pickup unit 2 captured the captured image as a cache candidate.

[0114] In step S206, the image generation unit 53 determines whether to cache the captured image captured by the image pickup unit 2 in step S202. Specifically, first, the image generation unit 53 determines the visibility of the vertices of the background object in the captured image based on the position and angle of the image pickup unit 2 in the real space, which position and angle have been estimated in step S204. Thereafter, the image generation unit 53 determines whether to cache so that captured images with high independence and recently captured are preferentially cached. With the aid of this determination about caching, a background texture is generated by using fewer cached images captured at a more recent time, and thus a more natural AR display is achieved. Note that in the case where a captured image with a higher degree of independence than the already cached cached images is newly captured, the image generation unit 53 can update the cached contents by replacing those cached images.

[0115] In the case where it is determined to perform caching (S206 / Yes), in step S208, the image generation unit 53 caches the captured image. Conversely, in the case where it is determined not to perform caching (S206 / No), the process returns to step S202.

[0116] In step S210, the image generation unit records a plurality of brightness values ​​near the positions corresponding to the vertices in the three-dimensional data as brightness value distributions associated with the vertices, which are in the background exposed in the cached captured image. As described above, the brightness value distribution recorded at this time is referred to by the display control unit 6, and the brightness value of the background texture is corrected during AR display. Using this method, a more natural AR display is achieved.

[0117] By repeating the above-described process, the smartphone preferentially caches a captured image that has a higher degree of independence and is captured more recently.

[0118] [3-3. Processing of generating background texture]

[0119] Fig.14 The flowchart is a flowchart showing an example of a process flow for generating a background texture, which is executed in the smartphone 1 according to the present embodiment.

[0120] like Fig.14 As shown in FIG. 1 , in step S302, the image generation unit 53 sorts the cached images in ascending order (i.e., in order from the smallest visible area) based on the visible area. The image generation unit 53 combines the cached images in the sorted order in step S306 described below, and thus the image generation unit 53 preferentially combines the cached images having a large visible area.

[0121] In step S304, the image generation unit 53 corrects the pixel values ​​for each cache image based on the cache image with the largest visible area. This prevents the generation of unnatural background textures in which the pixel values ​​differ greatly around the seam.

[0122] In step S306, the image generation unit 53 generates a background texture by combining the cached images in the order arranged in step S302 (i.e., the order starting from the smallest visible area). Specifically, the image generation unit 53 draws the cached images by overlapping these cached images in order starting from the smallest visible area, and finally drawing the cached image with the largest visible area. The overlapping parts of the cached images drawn earlier are covered by the cached images drawn later, and therefore, since the visible area is smaller, the drawing area is smaller. Accordingly, the image generation unit 53 combines so that the area obtained by combining the captured images with a large visible area occupies a larger area in the background texture. By combining the cached images using a Gaussian blur mask, the image generation unit 53 can represent the seams more naturally.

[0123] In step S308, the image generation unit 53 interpolates the missing portion of the generated background texture. Specifically, by using an image interpolation algorithm such as Inpainting, the image generation unit 53 interpolates an area that is not visible in all cache images (such as a floor surface in contact with the target object).

[0124] Using the above-described processing, the smartphone 1 generates an image of the background texture.

[0125] [3-4. Processing of generating changing texture]

[0126] Fig.15 The flowchart is a flowchart showing an example of a process flow for generating a variation texture, which is executed in the smartphone 1 according to the present embodiment.

[0127] like Fig.15 As shown in FIG. 4 , in step S402, the image generation unit 53 determines a single color to be used for the varying texture by the display control unit 6 based on the visible portion of the target object. In addition, the image generation unit 53 may generate the varying texture by using an image interpolation algorithm such as Inpainting.

[0128] Using the above-described processing, the smartphone 1 generates a changing texture. The processing described so far in [3-1, Processing for generating a changing object] to [3-4, Processing for generating a changing texture] is performed in advance as a preliminary preparation, or immediately before the processing described in [3-5, AR display processing] below. This reduces the difference between the time when the image whereby various textures are generated is captured and the time when the through image is captured. Therefore, a more natural AR display is achieved.

[0129] [3-5. AR display processing]

[0130] Fig.16 FIG. 2 is a flowchart showing an example of AR display processing executed in the smartphone 1 according to the present embodiment. Fig.16 In the example in which AR display is performed, an example will be described in detail. Figure 1 Destruction of the target object shown in .

[0131] like Fig.16 As shown in , in step S502 , the display control unit 6 acquires the through image captured in real time by the image pickup unit 2 and the posture information acquired in real time by the posture information acquisition unit 3 .

[0132] In step S504, the display control unit 6 calculates the position and posture of the destroyed part. Specifically, using the posture information and three-dimensional data collected in step S502, the display control unit 6 completely calculates the movement of the destroyed part and calculates the current position of each vertex of the background object and the destroyed part in the through image.

[0133] In step S506, the display control unit 6 dynamically generates a drawing mask according to the destruction situation. Specifically, the display control unit 6 generates a drawing mask that transmits the area for displaying the destroyed part and the area for displaying the background texture by using the vertex position of the destroyed part determined in step S504. Note that the display control unit 6 two-dimensionally increases the transparent area or applies Gaussian blur to the drawing mask, and thus more naturally displays the seam between the original background and the texture drawn by using the drawing mask.

[0134] In step S508, the display control unit 6 calculates parameters for correcting the brightness. Specifically, based on the comparison result between the brightness value distribution recorded in the buffer and the brightness value distribution of the corresponding position in the through image, the display control unit 6 corrects the brightness value of the background texture so that the two brightness values ​​become close to each other.

[0135] In step S510, the display control unit 6 draws the through image captured in real time by the image pickup unit 2. In the subsequent processing, the changing texture and the background texture are drawn to be superimposed, and thereby the AR image is superimposed on the through image.

[0136] In step S512 , the display control unit 6 fills the depth buffer with all objects such as the changing objects and the background objects.

[0137] In step S514, the display control unit 6 draws the background texture based on the drawing mask generated in step S506. Specifically, in conjunction with the depth buffer, the display control unit 6 draws the background texture by attaching the corresponding area in the background texture to the polygon included in the area of ​​the background object that is transmitted through the drawing mask.

[0138] In step S516, the display control unit 6 draws the destroyed portion. Specifically, using the depth buffer, the display control unit 6 paints the polygon of the cross section of the destroyed portion newly exposed, for example, due to the state change in the destroyed portion, with a single color determined by the image generation unit 53. Further, the display control unit 6 draws the surface image of the target object exposed in the through image in the corresponding polygon of the destroyed portion.

[0139] In step S518, the display control unit 6 performs various post-processing. For example, the display control unit 6 draws other virtual objects or estimates the position of the light source to draw shadows.

[0140] Using the above-described processing, the smartphone 1 performs display so as to superimpose an AR image representing a state in which a target object existing in the real space is destroyed on the through image.

[0141] <4. Conclusion>

[0142] So far, it has been combined Figures 1 to 16 Embodiments according to the technology disclosed herein are described. According to the above-described embodiments, a representation of an object existing in a real space is changed, and thus it is possible to give a user a stronger impression that the real world is enhanced by using AR technology. For example, the smartphone 1 according to the present embodiment can naturally represent the destruction of an object existing in the real world.

[0143] The preferred embodiments of the present disclosure have been described above in conjunction with the accompanying drawings, but the present disclosure is not limited to the above examples. Those skilled in the art may find various changes and modifications within the scope of the appended claims, and should understand that these changes and modifications can be naturally derived under the technical scope of the present disclosure.

[0144] For example, in the above-mentioned embodiment, an example has been described in which the image pickup unit 2, the posture information acquisition unit 3, the three-dimensional data acquisition unit 4, the control unit 5, the display control unit 6, and the display unit 7 are formed in the same device as the smartphone 1, but the present technology is not limited to this example. For example, the image pickup unit 2 may be included in an external device, and the smartphone 1 may perform the above-mentioned AR display based on the captured image acquired from the external device. In addition, for example, a server in the cloud may include a three-dimensional data acquisition unit 4, a control unit 5, and a display control unit 6, and a client device connected to the server via a network may include an image pickup unit 2, a posture information acquisition unit 3, and a display unit 7. In this case, the client device can send the captured image and posture information to the server, and display an AR image according to various calculations and controls performed by the server.

[0145] In the above-mentioned embodiment, an example in which the state of the target object is changed to destruction has been described, but the present disclosure is not limited to this example. For example, the smartphone 1 can indicate the size increase / decrease and movement of the target object. In the case in which the movement of the target object is indicated, the technology according to the present disclosure can be applied to, for example, a remodeling simulator to virtually move a table or a chair.

[0146] A series of control processes performed by each device described in this specification can be implemented by software, hardware, or a combination of software and hardware. A program including such software may be stored in advance on a storage medium (non-transitory medium) provided inside and outside each device. As an example, during execution, such a program is written to a RAM (random access memory) and executed by a processor such as a CPU.

[0147] In addition, the effects described in this specification are merely illustrative and exemplary, and are not restrictive. In other words, together with or different from the effects based on this specification, the technology according to the present disclosure can show other effects that are obvious to those skilled in the art.

[0148] Furthermore, the present technology may also be configured as follows.

[0149] (1) An information processing device comprising:

[0150] A recognition unit is configured to recognize an object included in a real space to distinguish the object from a background based on three-dimensional data of the real space so as to generate a virtual object image obtained by changing a state of the object.

[0151] (2) The information processing device according to (1), further comprising:

[0152] An image generating unit is configured to generate the object image of the object recognized by the recognition unit.

[0153] (3) The information processing device according to (2), further comprising:

[0154] a change object generating unit configured to generate a virtual change object obtained by changing the state of the object; and

[0155] A display control unit is configured to control a display unit so that the display unit displays the object image generated by the image generation unit on a surface of the changing object.

[0156] (4) The information processing device according to (3),

[0157] wherein the image generation unit generates a second surface image obtained by estimating a surface of the object hidden in the captured image, based on a portion corresponding to an exposed surface of the object in a captured image obtained by capturing an image of the real space, and

[0158] The display control unit displays an object image in which the second surface image is attached to a region of the changed object newly exposed due to the change.

[0159] (5) The information processing device according to (3) or (4),

[0160] The display control unit displays an object image obtained by attaching an image of a target object exposed in a through image obtained by capturing an image of the real space in real time to a corresponding region of the changing object.

[0161] (6) The information processing device according to any one of (3) to (5),

[0162] The display control unit displays the object image by estimating a position of a light source in the real space and correcting the brightness of the object image according to the estimated position of the light source.

[0163] (7) The information processing device according to any one of (3) to (6),

[0164] The display control unit displays the object image obtained by attaching the first surface image to a portion of the background that is hidden by the object and is newly exposed due to a change in the state of the object.

[0165] (8) The information processing device according to (7), wherein the image generation unit generates the first surface image by combining backgrounds exposed in one or more captured images obtained by capturing images of the real space.

[0166] (9) The information processing device according to (8),

[0167] Among the captured images obtained by capturing images of the real space, the image generation unit preferentially caches the captured images in which the overlapping portion of the background hidden by the object is small and which are recently captured, and uses the cached captured images to generate the first surface image.

[0168] (10) The information processing device according to (9),

[0169] Herein, based on posture information indicating a position and an angle of an image pickup unit that has captured a captured image and the three-dimensional data, the image generation unit determines whether to cache the captured image.

[0170] (11) The information processing device according to (9) or (10),

[0171] Wherein, in order to expose the entire area of ​​the part corresponding to the background in the surface of the solid body indicated by the three-dimensional data in at least one image of the one or more cached captured images, the display control unit displays to guide the image pickup posture of the image pickup unit, and the image pickup unit is configured to capture the captured image used by the image generation unit to generate an image.

[0172] (12) The information processing device according to any one of (9) to (11),

[0173] Wherein, the display control unit corrects the brightness value of the first surface image based on a comparison result between a brightness value near a position corresponding to a vertex in the three-dimensional data and a brightness value of a corresponding position in a through image obtained by capturing an image of the real space in real time, the position being in the background exposed in the cached captured image.

[0174] (13) The information processing device according to any one of (3) to (12), further comprising:

[0175] an image pickup unit configured to capture an image used by the image generation unit to generate an image,

[0176] The display control unit combines a captured image captured in real time by the image pickup unit and an image generated by the image generation unit, and displays the combined image.

[0177] (14) The information processing device according to any one of (3) to (13), wherein the background recognized by the recognition unit as to be distinguished is excluded from a target to be used for generating the changing object using the changing object generating unit.

[0178] (15) An information processing device according to any one of (1) to (14), wherein the recognition unit extracts a floor surface from the three-dimensional data, recognizes a portion protruding on the extracted floor surface as the object, and recognizes a portion other than the object as the background.

[0179] (16) The information processing device according to (15), wherein the recognition unit extracts the floor surface based on a gravity direction.

[0180] (17) The information processing apparatus according to any one of (1) to (16), wherein the change in state includes destruction of the object.

[0181] (18) An information processing method comprising:

[0182] By processor:

[0183] An object included in a real space is recognized to be distinguished from a background based on three-dimensional data of the real space so as to generate a virtual object image obtained by changing a state of the object.

[0184] (19) A program for causing a computer to:

[0185] A recognition unit is configured to recognize an object included in a real space to distinguish the object from a background based on three-dimensional data of the real space so as to generate a virtual object image obtained by changing a state of the object.

[0186] Reference numerals list

[0187] 1. Smartphone

[0188] 2 Image pickup unit

[0189] 3 Posture Information Collection Unit

[0190] 4 3D data acquisition unit

[0191] 5 Control unit

[0192] 51 Identification Unit

[0193] 52 Change object generation unit

[0194] 53 Image generation unit

[0195] 6 Display control unit

[0196] 7 Display unit

[0197] 10 Telephone

[0198] 11 Packing tape

[0199] 12. Helmet

[0200] 13 Beverage Cans

[0201] 14 Spray cans

Claims

1. An information processing device, comprising: The circuit is configured as: generating a virtual changed object obtained by changing a state of an object recognized in a real space, the change being caused by a collision between the virtual object and the recognized object; generating an object image of the identified object that has undergone the change; as well as controlling display of the object image on the virtual changing object, The changing includes: changing the position, angle or size of only a part of the recognized object instead of the entire recognized object, or deforming a part of the recognized object.

2. The information processing device according to claim 1, further comprising: An image pickup unit is configured to capture an image of the real world.

3. The information processing device according to claim 1 or 2, wherein: The circuit is further configured to superimpose the object image displayed on the virtual changing object on a capture image captured in real time.

4. The information processing device according to claim 1, wherein: The circuit is further configured to generate the virtual transformed object based on the three-dimensional data of the identified object.

5. An information processing method, comprising: generating a virtual changed object obtained by changing a state of an object recognized in a real space, the change being caused by a collision between the virtual object and the recognized object; generating an object image of the identified object that has undergone the change; as well as controlling display of the object image on the virtual changing object, The changing includes: changing the position, angle or size of only a part of the recognized object instead of the entire recognized object, or deforming a part of the recognized object.

6. A non-volatile computer-readable storage medium storing a program for causing a computer to implement an information processing method, the method comprising: generating a virtual changed object obtained by changing a state of an object recognized in a real space, the change being caused by a collision between the virtual object and the recognized object; generating an object image of the identified object that has undergone the change; as well as controlling display of the object image on the virtual changing object, The changing includes: changing the position, angle or size of only a part of the recognized object instead of the entire recognized object, or deforming a part of the recognized object.

Images