Information processing device, information processing program, and information processing method

Abstract

To provide an information processing device, an information processing program, and an information processing method that can reproduce a photographed subject to a high degree in a reconstructed image.SOLUTION: An information processing device 30 includes an acquisition unit that acquires multiple pieces of shooting information based on shooting from different viewpoints, which include color information and depth information generated by shooting a target including a moving object, a point cloud generation unit that generates multiple point cloud data from the multiple pieces of shooting information by generating point cloud data for each piece of shooting information from the color information and depth information, and a playback data generation unit that includes a mesh generation unit that generates a mesh model from the point cloud, and generates playback data including data of a mesh model corresponding to at least a portion of a composite point cloud, which is a point cloud formed by combining multiple point cloud data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing program, and an information processing method for processing shooting information including color information and depth information. [Background technology]

[0002] In recent years, imaging devices equipped with a depth sensor as well as an image sensor have been developed. By photographing an object, these imaging devices obtain depth information indicating the distance to the object in addition to color information indicating the color of the object (see, for example, Patent Document 1). By processing the photographed data including the color information and depth information with a reproducing device, it is possible to reproduce an image that reproduces the color and three-dimensional structure of the object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-539169 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to expand the applications of the system including the above-mentioned imaging device and playback device, there is a demand for a more advanced reproduction of the object when reproducing the image of the object. Specifically, it is desired that an image be constructed so that the three-dimensional structure of the object can be grasped from any viewpoint, and that the movement of a moving object be accurately reproduced. [Means for solving the problem]

[0005] Various aspects of an information processing device, an information processing program, and an information processing method for solving the above problems will be described below. [Aspect 1] An information processing device comprising: an acquisition unit that acquires multiple pieces of shooting information including color information and depth information generated by shooting a subject including a moving object, the shooting information being based on shooting from different viewpoints; a point cloud generation unit that generates multiple pieces of point cloud data from the multiple pieces of shooting information by generating point cloud data for each piece of shooting information from the color information and the depth information; and a playback data generation unit that includes a mesh generation unit that generates a mesh model from the point cloud, and generates playback data including data of a mesh model corresponding to at least a portion of a composite point cloud, which is a point cloud formed by combining the multiple point cloud data.

[0006] According to the above configuration, since the reproduction data is generated by synthesizing a plurality of pieces of data based on shooting from different viewpoints, the reproduced image reproduced using the reproduction data has a three-dimensional structure that can be confirmed from a plurality of viewpoints. Therefore, the user can confirm the reproduced image from a viewpoint selected from a wide range, and the photographed subject can be reproduced in a highly sophisticated manner in the reproduced image.

[0007] [Aspect 2] The point cloud data, or mesh data which is data of a mesh model generated from the point cloud data, is synthesis target data, and the process of generating the playback data by the playback data generation unit includes synthesizing the multiple synthesis target data by unifying reference positions of position information contained in each of the multiple synthesis target data corresponding to the multiple point cloud data.The information processing device described in [Aspect 1].

[0008] According to the above configuration, the reference positions of multiple point cloud data or mesh data based on images taken from different viewpoints are unified and these data are synthesized. Therefore, the positional deviation of the superimposed point cloud or mesh model is suppressed, and the structure of the photographed object is accurately reproduced in the reconstructed image.

[0009] [Aspect 3] The information processing device described in [Aspect 1] or [Aspect 2], wherein the shooting information includes timestamp information indicating timestamps attached to the color information and the depth information, the acquisition unit acquires a plurality of shooting data including a series of the shooting information generated by photographing the subject from one viewpoint during a specified shooting period, the shooting data being based on shooting from different viewpoints, and in the plurality of shooting data, the shooting information including the timestamp information indicating a common point in time during the shooting period is subject shooting information, and the composite point cloud is a point cloud obtained by combining the plurality of point cloud data generated from the plurality of shooting information including the subject shooting information of each shooting data.

[0010] According to the above configuration, the reproduction data is generated from the imaging information at a common point in time that is specified based on the time stamp information among the imaging information of the plurality of imaging information taken from different viewpoints and during a predetermined imaging period. Therefore, the structure of the imaging subject at each point in time is accurately reproduced in the reproduced image.

[0011] [Aspect 4] The information processing device described in [Aspect 3], wherein the playback data generation unit outputs the playback data generated by corresponding each of the series of shooting information to the target shooting information in a time series based on the timestamp information. According to the above configuration, since it is possible to reproduce the reproduced image in chronological order at the time of shooting, the movement of the photographed subject is accurately reproduced in the reproduced image.

[0012] [Aspect 5] An information processing device according to any one of [Aspect 1] to [Aspect 4], wherein the playback data generation unit generates the playback data including data of a mesh model corresponding to a portion of the composite point group.

[0013] According to the above configuration, by making only the characteristic parts of the reconstructed image into a mesh model, the user can accurately grasp the structure and color characteristics of the reconstructed image, while reducing the processing load of generating the mesh model compared to generating a mesh model corresponding to the entire composite point group.

[0014] [Aspect 6] An information processing program that causes one or more computers to execute the following steps: acquire multiple pieces of shooting information including color information and depth information generated by photographing a subject including a moving object, the multiple pieces of shooting information being based on shooting from different viewpoints; generate multiple pieces of point cloud data from the multiple pieces of shooting information by generating point cloud data for each piece of shooting information from the color information and the depth information; and generate playback data including data of a mesh model corresponding to at least a portion of a composite point cloud, which is a point cloud formed by combining the multiple point cloud data.

[0015] According to the above configuration, since the reproduction data is generated by synthesizing a plurality of pieces of data based on shooting from different viewpoints, the reproduced image reproduced using the reproduction data has a three-dimensional structure that can be confirmed from a plurality of viewpoints. Therefore, the user can confirm the reproduced image from a viewpoint selected from a wide range, and the photographed subject can be reproduced in a highly sophisticated manner in the reproduced image.

[0016] [Aspect 7] An information processing method in which one or more computers acquire multiple pieces of shooting information including color information and depth information generated by photographing a subject including a moving object, the multiple pieces of shooting information being based on shooting from different viewpoints, generate point cloud data for each piece of shooting information from the color information and the depth information, thereby generating multiple pieces of point cloud data from the multiple pieces of shooting information, and generate a mesh model from the point cloud, thereby generating playback data including data of a mesh model corresponding to at least a portion of a composite point cloud, which is a point cloud formed by combining the multiple point cloud data.

[0017] According to the above configuration, since the reproduction data is generated by synthesizing a plurality of pieces of data based on shooting from different viewpoints, the reproduced image reproduced using the reproduction data has a three-dimensional structure that can be confirmed from a plurality of viewpoints. Therefore, the user can confirm the reproduced image from a viewpoint selected from a wide range, and the photographed subject can be reproduced in a highly sophisticated manner in the reproduced image. Effect of the Invention

[0018] According to the present disclosure, the subject can be reproduced to a higher degree in the reconstructed image. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a three-dimensional reconstruction system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a configuration of an information processing device according to an embodiment. [Diagram 3] FIG. 3 is a flowchart showing the process of the information processing device according to the embodiment. [Figure 4] FIG. 4 is a diagram showing an example of an image represented by a point cloud. [Diagram 5] FIG. 5 is a diagram showing an example of an image represented by a mesh model. [Figure 6] FIG. 6 is a diagram showing an example of a usage form of the three-dimensional reconstruction system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] An embodiment of an information processing device, an information processing program, and an information processing method will be described with reference to the drawings. [Overall configuration of the 3D reconstruction system] The overall configuration of the three-dimensional reconstruction system will be described with reference to Fig. 1. As shown in Fig. 1, the three-dimensional reconstruction system 100 includes a plurality of imaging devices 10, an information management device 20, an information processing device 30, and a playback device 40. Of these, the information processing device 30 and the playback device 40 form a playback system 50.

[0021] The image capturing device 10 captures an image of an object. The object is the image capturing subject. The image capturing subject includes a moving object such as a person or an animal. The period from the start to the end of capturing the image of the object is the image capturing period.

[0022] The photographing device 10 includes a photographing unit that generates color information and depth information by photographing an object to be photographed, and an information processing unit that processes the information generated by the photographing unit. The color information is information that indicates the color of the object to be photographed. The color information is, for example, information that expresses color by additive mixing using an RGB color model, and includes information that indicates each of the three primary colors with a predetermined number of gradations. The depth information is information that indicates the distance from a reference position to the object to be photographed. In other words, the depth information is measurement information of the depth of the photographing range, and the reference position is, for example, the position of the photographing device 10.

[0023] The photographing unit includes an image sensor and a depth sensor, and color information is generated from the detection result by the image sensor, and depth information is generated from the detection result by the depth sensor. The image sensor may be an imaging element of a known type, for example, a CCD image sensor or a CMOS image sensor. The depth sensor may be a sensor using a known type, such as a ToF sensor that calculates distance based on the time required to observe reflected light such as infrared light, a structured light that determines shape and depth based on detection of distortion of pattern-like irradiated light, a stereo camera that uses parallax, or the like.

[0024] The information processing unit includes a calculation device and a memory. The information processing unit assigns a timestamp to the color information and the depth information to generate shooting information including timestamp information indicating the timestamp, the color information, and the depth information. The timestamp information is information indicating a specific point in time, and indicates, for example, the date and time when the information processing unit acquired the color information and the depth information. The color information and the depth information included in the shooting information are information based on the detection results of each sensor acquired at the same timing or with a small difference that can be considered to be the same timing.

[0025] The photographing device 10 is composed of, for example, a camera having an image sensor, a camera having a depth sensor, and a processing unit including an arithmetic device and a memory. The photographing device 10 may be composed of multiple separate mechanisms, and the photographing section and the information processing section may be connected by wire or wirelessly. Alternatively, all of the mechanisms constituting the photographing device 10 may be assembled together.

[0026] The multiple image capture devices 10 capture the same subject from different positions. That is, the multiple image capture devices 10 simultaneously capture the common subject from different viewpoints. A group of the image capture devices 10 that simultaneously capture the common subject is the image capture device group 11.

[0027] The shooting ranges of the multiple imaging devices 10 included in the imaging device group 11 may include overlapping portions. The number of imaging devices 10 included in the imaging device group 11 is not particularly limited, but it is preferable that the number and arrangement positions of the imaging devices 10 are set so that the target is photographed from all directions by the multiple imaging devices 10, in other words, so that no blind spots are generated for the target in the shooting ranges of the multiple imaging devices 10. Fig. 1 shows an example in which one imaging device group 11 includes four imaging devices 10, namely, imaging device 10A, imaging device 10B, imaging device 10C, and imaging device 10D.

[0028] In addition, the information processing unit may be a computer device separate from the imaging device 10, and one information processing unit may perform the generation process of the imaging information based on the color information and depth information obtained from each of the multiple imaging devices 10.

[0029] The photographing information for each photographing device 10 is transmitted to the information management device 20 via a network such as the Internet. The information management device 20 is a computer device such as a personal computer or a server device. The information management device 20 includes electronic circuits that are arithmetic units such as a CPU, an MPU, and a GPU, memories such as a ROM, a RAM, a registered memory, and an unbuffered memory, and storage devices such as an SSD and an HDD.

[0030] The information management device 20 stores the shooting data 60, which is a collection of shooting information generated during a shooting period. The shooting data 60 is composed of shooting information for each shooting device 10. In other words, the shooting data 60 includes a series of shooting information generated by shooting a subject from one viewpoint during the shooting period.

[0031] The information management device 20 stores imaging data 60 for the multiple imaging devices 10 included in the imaging device group 11, i.e., multiple imaging data 60 based on imaging from different viewpoints. In the example shown in Fig. 1, the information management device 20 stores imaging data 60A for the imaging device 10A, imaging data 60B for the imaging device 10B, imaging data 60C for the imaging device 10C, and imaging data 60D for the imaging device 10D. The multiple imaging data 60 are managed for each imaging device group 11 and each imaging period.

[0032] The information management device 20 and the information processing device 30 are capable of transmitting and receiving data to and from each other via a network such as the Internet, and the information processing device 30 obtains the photographing data 60 from the information management device 20 .

[0033] The information processing device 30 is a computer device such as a personal computer, a tablet terminal, a smartphone, a server device, etc. The information processing device 30 generates reproduction data, which is data for reproducing an image of a subject, from a plurality of shooting data 60 generated based on shooting by the group of shooting devices 11.

[0034] The information processing device 30 outputs the playback data to the playback device 40. The information processing device 30 and the playback device 40 may be connected by wire or wirelessly, or the information processing device 30 and the playback device 40 may be an integrated device.

[0035] The playback device 40 plays back an image of a subject by using the playback data. The playback device 40 is, for example, a display device equipped with a display panel such as a liquid crystal panel, a head-mounted display for VR (Virtual Reality) or AR (Augmented Reality), a projection device, or the like. The information processing device 30 may also have at least a part of the functions of the playback device 40.

[0036] [Configuration of information processing device] 2, a detailed configuration of the information processing device 30 will be described. The information processing device 30 includes a communication unit 31, a control unit 32, and a storage unit 33.

[0037] The communication unit 31 performs communication processing between the information processing device 30 and the information management device 20, such as connection to a network and transmission and reception of data. When the information processing device 30 transmits and receives data with the playback device 40 via short-range wireless communication, the communication unit 31 may have a processing unit that performs communication processing through the above-mentioned network and a processing unit that performs communication processing such as connection of devices via short-range wireless communication and transmission and reception of data.

[0038] The communication unit 31 functions as an acquisition unit that acquires a plurality of pieces of shooting data 60 based on shooting by the group of shooting devices 11 from the information management device 20 . The control unit 32 executes the information processing program stored in the storage unit 33, thereby functioning as a point cloud generation unit 34, a mesh generation unit 35, and a synthesis unit 36.

[0039] The point cloud generator 34 generates point cloud data from color information and depth information of the shooting information included in the shooting data 60. The point cloud data is data indicating a set of points having position information and color information. That is, in the point cloud data, the shape of the object is represented by a set of points, and the color of each part of the object is represented by the color of the point at the corresponding position. A known algorithm may be used to generate the point cloud data from the color information and depth information.

[0040] The mesh generating unit 35 generates mesh data from the point cloud data. The mesh data is data that represents an object represented as a set of points in the point cloud data as a three-dimensional mesh model that is a set of triangular or quadrangular faces, in other words, data made up of polygons. A known algorithm such as the Ball Pivoting Algorithm may be used to generate the mesh model from the point cloud.

[0041] The synthesis unit 36 ​​synthesizes a plurality of point cloud data generated from the shooting information of a plurality of shooting data 60. That is, the positions of the point clouds included in each point cloud data are converted to positions relative to a common reference position, and the points of each point cloud data are superimposed to form one set. As a result, synthetic point cloud data, which is data of the synthesized point clouds, is generated, and the synthetic point cloud data is processed by the mesh generation unit 35.

[0042] The synthesis unit 36 ​​specifies the point cloud data to be synthesized based on the time stamp information included in the shooting information. In detail, the point cloud data generated from the shooting information at the time considered to be the same time among the shooting information included in the multiple shooting data 60 is set as the synthesis target.

[0043] Note that the synthesis unit 36 ​​may synthesize a plurality of mesh data after mesh data is generated from each of the plurality of point cloud data. In this case, the position of the mesh model included in each mesh data is converted to a position relative to a common reference position, and a set of polygonal faces, which is the mesh model included in each mesh data, is superimposed to form a single set.

[0044] A single mesh data is generated from shooting information of a plurality of shooting data 60 by cooperation of the point cloud generation unit 34, the mesh generation unit 35, and the synthesis unit 36. This mesh data is the playback data 61, and is output to the playback device 40. Of the above, the mesh generation unit 35 and the synthesis unit 36 ​​constitute the playback data generation unit.

[0045] The storage unit 33 stores various programs and data necessary for the control unit 32 to execute processing. The storage unit 33 stores an information processing program as an example of such a program. The storage unit 33 also stores position management data 65 as an example of such data. The position management data 65 is data used by the synthesis unit 36 ​​to unify position information when synthesizing data, and includes, for example, information indicating the relative positional relationship of the multiple image capture devices 10 that constitute the image capture device group 11.

[0046] The physical configuration, i.e., the hardware configuration, of the information processing device 30 having the above functions will be described. The information processing device 30 includes electronic circuits that are arithmetic devices such as a CPU, MPU, GPU, etc., memories such as ROM, RAM, registered memory, unbuffered memory, etc., and storage such as an SSD or HDD. The arithmetic devices load an operating system and various programs from the storage into the memory, and execute commands retrieved from the memory. The information processing device 30 may include integrated circuits such as an ASIC and an FPGA.

[0047] The information processing device 30 also includes a communication interface for transmitting and receiving data to and from a connected device via a network. The communication interface is implemented as hardware, software, or a combination of these.

[0048] Each process executed by the information processing device 30 may be executed by software included in the information processing device 30, or may be realized by a combination of an integrated circuit included in the information processing device 30 and software.

[0049] [Processing of information processing equipment] The flow of processing by the information processing device 30 in the three-dimensional reconstruction system 100 will be described with reference to FIG.

[0050] First, in the process of step S10, the information processing device 30 acquires, from the information management device 20, a plurality of pieces of shooting data 60 generated based on shooting by the group of shooting devices 11. For example, in response to a request from the information processing device 30, the information management device 20 transmits, to the information processing device 30, a plurality of pieces of shooting data 60 corresponding to the specified group of shooting devices 11 and shooting period.

[0051] Here, the configuration of the shooting data 60 will be described in detail. The shooting data 60 for each shooting device 10 includes multiple pieces of shooting information generated at predetermined intervals during a shooting period. As described above, the shooting information includes color information, depth information, and time stamp information, and the order of generation of the multiple pieces of shooting information can be identified based on the time stamp information. In other words, the time stamp information indicates the time point when the shooting information was generated and indicates the chronological order of the multiple pieces of shooting information.

[0052] In order to reduce the load on communication, it is preferable that the color information and the depth information are compressed. In this case, it is preferable that the color information is encoded using a compression method with a higher compression rate than the depth information. For example, lossy compression is used for compressing the color information, and lossless compression is used for compressing the depth information. By increasing the compression rate of the color information, which is less likely to be unnatural due to missing or altered information compared to the depth information, it is possible to reduce the amount of data transmitted while suppressing deterioration in the quality of the reproduced image.

[0053] In this way, when color information and depth information are processed separately, timestamp information indicating the same point in time is assigned to each of the color information and the depth information, and the color information and the depth information are associated with each other based on the timestamp information.

[0054] When the information processing device 30 acquires a plurality of shooting data 60, in the process of step S11, the information processing device 30 generates point cloud data from color information and depth information for each piece of shooting information of each piece of shooting data 60. That is, the point cloud data is generated for each piece of shooting information. The process of step S11 is a process performed by the point cloud generating unit 34.

[0055] Next, in step S12, the information processing device 30 synthesizes the point cloud data generated from the shooting information of each of the photographing devices 10 included in the group of photographing devices 11 to generate synthesized point cloud data. At this time, the information processing device 30 synthesizes point cloud data generated from shooting information at a common time point during the shooting period, among point cloud data generated from a plurality of photographing data 60, based on time stamp information included in the photographing information that is the source of the point cloud data. One point cloud data for each photographing data 60 is to be synthesized. If the time stamp information included in the photographing information is configured to be added to the generated point cloud data, the information processing device 30 can identify the data to be synthesized based on the time stamp information added to the point cloud data.

[0056] 1, when the group of imaging devices 11 is made up of four imaging devices 10, four point cloud data generated from the imaging information of the imaging data 60 for each imaging device 10 are synthesized. That is, the point cloud data generated from the imaging information of the imaging data 60A, the point cloud data generated from the imaging information of the imaging data 60B, the point cloud data generated from the imaging information of the imaging data 60C, and the point cloud data generated from the imaging information of the imaging data 60D are synthesized.

[0057] The information processing device 30 unifies the reference positions of the point clouds included in each point cloud data and synthesizes the point clouds of each point cloud data. For example, when the position of the point cloud is defined in each point cloud data based on the position of the image capture device 10 that generated the original image capture information, the information processing device 30 converts the position of the point cloud of the point cloud data based on the image capture information of another image capture device 10 so that the position of one image capture device 10 in the image capture device group 11 is used as the reference, thereby unifying the reference positions of the point clouds included in each point cloud data.

[0058] Information such as the position of each image capture device 10 required for unifying the reference position is included in the position management data 65. The position management data 65 may be stored in advance in the information processing device 30 as data for each image capture device group 11, or may be managed by the information management device 20 and transmitted to the information processing device 30 together with the image capture data 60. The process of step S12 is a process performed by the synthesis unit 36.

[0059] Next, the information processing device 30 generates mesh data from the composite point cloud data as processing in step S13. The processing in step S13 is processing performed by the mesh generating unit .

[0060] Then, the information processing device 30 outputs the generated mesh data to the playback device 40 as playback data 61 in step S14. Reproduction data 61 is generated for each time point indicated by the time stamp information from the multiple shooting data 60. Then, the information processing device 30 outputs the reproduction data 61 generated from each of the series of shooting information via the point cloud data in correspondence with the time series of the shooting information based on the time stamp information.

[0061] For example, the mesh data is provided with time stamp information contained in the shooting information that is the source of the mesh data, or information indicating a specific time point in the time series that corresponds to the time stamp information. The provided information is time series information indicating a time series that corresponds to the time series of the shooting information in a plurality of mesh data generated for each time point indicated by the time stamp information from a plurality of shooting data 60. The information processing device 30 outputs the playback data 61, which is mesh data, in the order according to the time series indicated by the time series information. Alternatively, the information processing device 30 may output the playback data 61 to the playback device 40 with time series information added thereto.

[0062] With the above, the processing of the information processing device 30 ends. In the above processing, the point cloud data is synthesized, and then mesh data is generated from the synthesized point cloud data, but the reproduction data 61 may be generated by creating mesh data for each point cloud data from the point cloud data for each piece of shooting information, and synthesizing a plurality of mesh data. In this case, for the position information contained in each mesh data, the reference position is unified using the position management data 65 in the same manner as described above, and the mesh models of each mesh data are synthesized. The synthesized mesh data generated by synthesis is the reproduction data 61.

[0063] In short, the playback data 61 needs to be mesh model data corresponding to a composite point group, which is a point group formed by combining multiple point cloud data, and may be synthesized in the form of a point cloud or in the form of a mesh model.

[0064] The playback device 40 plays back the images in the order of the time series indicated by the time series information, using the playback data 61. As a result, the images are played back in the time series of the images taken, and the movement of the subject is accurately reproduced in the played back images.

[0065] In addition, since the redevelopment is represented by a mesh model, a user who sees the reconstructed image can easily grasp the three-dimensional structure and color of the reconstructed image. Fig. 4 shows an example of an image represented by a point cloud represented by point cloud data, and Fig. 5 shows an example of an image represented by a mesh model represented by mesh data generated from the point cloud data.

[0066] As shown in Figure 4, in an image represented by a point cloud, the depths of the image can be seen through the gaps between the points, and the area in which colors are displayed is limited to the positions of the points, so it can be difficult to grasp the detailed structure and colors of the image. Increasing the density of the points makes it easier to grasp the details, but increasing the density of the points also increases the data volume and the processing load.

[0067] On the other hand, as shown in Figure 5, in an image represented by a mesh model, the image is represented by connecting surfaces, so the depth of the image is suppressed and the area in which the color is displayed spreads over the entire surface. This makes it easier to grasp the detailed structure of the image and also reduces the processing load required to render the image.

[0068] In this embodiment, data based on images captured by the multiple image capture devices 10 from different viewpoints is synthesized to generate the reproduction data 61, so that the reproduced image has a three-dimensional structure that can be viewed from multiple viewpoints. Therefore, the user can view the reproduced image from a selected viewpoint within a wide range.

[0069] [How the 3D reconstruction system is used] The three-dimensional reconstruction system 100 is used for recording and playing back a subject, including a moving object. The three-dimensional reconstruction system 100 is preferably used for practical training, such as sports, playing a musical instrument, dancing, and traditional performing arts. In this case, the person demonstrating the practical skill is the subject of the recording, and the person being taught the practical skill is the user who views the reproduced image.

[0070] 6 shows an example in which the three-dimensional reconstruction system 100 is used for lessons on playing a musical instrument. The first space S1 is a space in which the subject Tg is present, that is, a space in which the subject Tg is photographed. The subject Tg includes a musical instrument and a performer, and the subject Tg is photographed by a plurality of image capturing devices 10 while being played. As a result, photographing data 60 from the viewpoint of each image capturing device 10 is generated and stored in the information management device 20.

[0071] The second space S2 is a space in which users P1 and P2 are present, viewing a reconstructed image Ig using reproduction data 61 based on the shooting data 60. In order to facilitate understanding, in Fig. 6, the reconstructed image Ig being reproduced by the reproduction device 40 attached to each of the users P1 and P2 is shown at a corresponding position in the second space S2 based on the viewpoints of the users P1 and P2.

[0072] According to the three-dimensional reconstruction system 100 of this embodiment, the reconstructed image Ig has a three-dimensional structure that can be confirmed from a plurality of viewpoints, so that the users P1 and P2 can view the reconstructed image Ig from a viewpoint corresponding to a portion of the reconstructed image Ig that they want to see, and can also view the reconstructed image Ig by moving the viewpoint. Therefore, the characteristics of the photographed object Tg can be accurately grasped from the reconstructed image Ig.

[0073] Furthermore, since the movement of the subject Tg is accurately reproduced in the reproduced image Ig in chronological order at the time of shooting, users P1 and P2 can accurately grasp the movements that constitute the practical performance of the subject Tg, such as the movements of a performer's fingers, from the reproduced image Ig.

[0074] This allows the practical training to be smoothly conducted remotely even when it is difficult for the instructor and the student to meet in person. The playback data 61 may include audio data, and the audio may be played back together with the playback image Ig.

[0075] The three-dimensional reconstruction system 100 may also be used to observe the movements of animals. For example, the three-dimensional reconstruction system 100 is used to create works such as animal sketches or to take lectures on animals. In this case, the animal is the subject Tg to be photographed, and the observer is the user who views the reconstructed image.

[0076] When reproducing the reproduced image Ig, it is also possible to stop the reproduction of the reproduced image Ig at any timing. Therefore, it is also possible to stop the reproduction at a timing corresponding to when the moving photographed subject Tg takes a specific pose, and observe the reproduced image Ig of the specific pose. When the photographed subject Tg is an animal, it is difficult to instruct the photographed subject Tg itself to stop in a specific pose, but by stopping the reproduced image Ig at a desired timing, it is possible to observe the specific pose from various angles over a long period of time.

[0077] The three-dimensional reconstruction system 100 may also be used to check a work site such as a factory or a construction site. For example, the three-dimensional reconstruction system 100 is used to monitor the work site from a remote location and to give instructions to workers. In this case, the workers and the objects to be worked on are included in the photographed object Tg, and the checker is a user who views the reconstructed image.

[0078] In this case, too, the reconstructed image Ig can be confirmed from various angles, and the movement of the photographed target Tg can be confirmed from the reconstructed image Ig, so the situation at the work site can be accurately grasped, and therefore accurate monitoring and instructions can be given.

[0079] It should be noted that the uses of the three-dimensional reconstruction system 100 are not limited to those mentioned above. The three-dimensional reconstruction system 100 may be used for watching or appreciating sports and the like, analyzing various movements, simulating actions, and the like.

[0080] [Variations] The reproduction data 61 may be configured such that a part of a composite point cloud, which is a point cloud obtained by combining a plurality of point cloud data, is made into a mesh model. The part other than the part represented by the mesh model may be represented by a point cloud, may be represented in a representation format different from the point cloud, or may be omitted.

[0081] Specifically, in the reproduced image, a portion having a movement or structure that is expected to attract the user's attention is made into a mesh model. For example, only the hands of a player of an instrument included in the subject of shooting may be made into a mesh model. With this configuration, it is possible to reduce the processing load of generating a mesh model in the information processing device 30 while making it easier for the user to grasp the movement and structure of the subject of shooting in the portion that is required.

[0082] For example, a part of the reconstructed image may be made into a mesh model by making a point cloud based on the shooting information of only a part of the shooting devices 10 among the group of shooting devices 11 into a mesh model. Alternatively, a region to be made into a mesh model may be specified in the composite point cloud.

[0083] Since the position information included in the point cloud data and the position information included in the mesh data generated from the point cloud data correspond to each other, even if point cloud data and mesh data are mixed in the data to be synthesized, it is possible to synthesize these data to generate data in which a part of the synthesized point cloud is made into a mesh model. Furthermore, the data to be synthesized may include data in which a part of a point cloud in one point cloud data is made into a mesh model. In this way, even if a part of the synthesized point cloud is made into a mesh model, either the process of synthesizing data corresponding to multiple pieces of shooting information or the process of generating a mesh model may be performed first.

[0084] The portion to be made into the mesh model may be designated by a viewer of the reproduced image or a user who is an administrator of the three-dimensional reconstruction system 100. Based on designation information indicating the portion of the composite point cloud designated by the user being input to the information processing device 30, reproduction data 61 is generated, which includes data of a mesh model corresponding to the portion of the composite point cloud designated by the designation information.

[0085] Also, it may be possible to switch between a state in which a part of the combined point cloud is a mesh model and a state in which the entire combined point cloud is a mesh model. In short, the reproduction data 61 only needs to include data of a mesh model corresponding to at least a part of the composite point group. Furthermore, the reproduction data 61 including the mesh model may be generated only for the shooting information at a certain point in time during the shooting period.

[0086] The information management device 20 may also sequentially transmit the photographing information generated based on the photographing by the photographing device 10 to the information processing device 30. That is, the photographing by the photographing device 10 and the reproduction by the reproduction device 40 may be performed in real time. In a form in which the reproduction data 61 is generated from the photographing data 60 stored in advance in the information management device 20, control such as stopping and repeating the reproduction is easily possible, and the reproduction image can be reproduced at each of a plurality of reproduction opportunities. On the other hand, in a form in which the information management device 20 sequentially relays the photographing information from the photographing device 10 to the information processing device 30, the situation of the photographed object is immediately conveyed to the user who views the reproduced image, so that the three-dimensional reconstruction system 100 can be suitably used for applications such as monitoring and guidance of the photographed object.

[0087] Moreover, the exchange of information between the photographing device 10 and the information management device 20 is not limited to being carried out through a network such as the Internet. The photographing device 10 and the information management device 20 may be connected by wire, or information may be exchanged by short-range wireless communication. Moreover, the exchange of information between the information management device 20 and the information processing device 30 is not limited to being carried out through a network such as the Internet. The information management device 20 and the information processing device 30 may be connected by wire, or information may be exchanged by short-range wireless communication.

[0088] The functions of the information processing device 30 may be realized by a plurality of information processing devices. One information processing device is a single computer device. That is, the information processing device 30 may be configured by one or a plurality of information processing devices.

[0089] As described above, according to the above embodiment, the following effects can be obtained. (1) Since the reproduction data 61 is generated by synthesizing a plurality of pieces of data based on shooting from different viewpoints, the reconstructed image based on the reproduction data 61 has a three-dimensional structure that can be confirmed from a plurality of viewpoints. Therefore, the reconstructed image can reproduce the photographed subject to a high degree. This allows the user to intuitively grasp the structure and color of the photographed subject because the reconstructed image can be confirmed from any viewpoint.

[0090] (2) The reconstructed image includes a mesh model generated from a point cloud. This makes it easier for the user to grasp the detailed structure and color of the reconstructed image compared to when the reconstructed image is represented by a point cloud. In addition, the process of generating a mesh model from shooting information via a point cloud has a smaller processing load than the process of generating a three-dimensional model by extracting feature points from multiple videos and still images. Therefore, it is easy to generate reproduction data 61 sequentially from the acquired shooting data 60 and reproduce the reconstructed image. In addition, a three-dimensional model generated from a video or still image that does not include depth information does not guarantee the size of the photographed object, such as the total length, whereas a three-dimensional model generated from shooting information that includes depth information makes it possible to reproduce the size of the photographed object in the reconstructed image.

[0091] (3) When generating the reproduction data 61, the reference positions of multiple point cloud data or mesh data based on images taken from different viewpoints are unified and these data are synthesized. Therefore, the positional deviation of the superimposed point cloud or mesh models is suppressed, and the structure of the photographed object is accurately reproduced in the reproduced image.

[0092] (4) Reproduction data 61 is generated from the photographic information at a common point in time specified based on the time stamp information among a plurality of pieces of photographic information from different viewpoints and for a predetermined photographing period. Therefore, in the reproduced image, the structure of the photographic subject at each point in time is accurately reproduced.

[0093] (5) The reproduction data 61 generated for each time point indicated by the time stamp information from multiple pieces of shooting information from different viewpoints and for a specific shooting period is output so as to correspond to the time series of the shooting information based on the time stamp information. This makes it possible to reproduce the reproduced image according to the time series of shooting, so that the movement of the shooting subject is accurately reproduced in the reproduced image. Therefore, it is possible to reproduce the shooting subject to a higher degree in the reproduced image.

[0094] (6) If the reproduction data 61 includes data of a mesh model corresponding to a part of the composite point cloud, then by making only the characteristic parts of the reproduced image into a mesh model, it is possible to reduce the processing load of generating the mesh model while enabling the user to accurately grasp the structure and color characteristics of the reproduced image. [Explanation of symbols]

[0095] 10, 10A~10D...Photographic equipment 11...Photographic equipment group 20…Information management device 30…Information processing equipment 31…Communications Department 32...Control section 33...Storage section 34...Point cloud generation section 35…Mesh generation section 36…Synthetic section 40...Playback device 50…Playback system 60,60A~60D...Shooting data 61...Playback data 65...Location management data 100...3D reconstruction system

Claims

1. an acquisition unit that acquires a plurality of pieces of shooting information including color information and depth information generated by shooting a target including a moving object, the shooting information being based on shooting from different viewpoints; a point cloud generating unit configured to generate point cloud data for each of the pieces of shooting information from the color information and the depth information, thereby generating a plurality of pieces of point cloud data from the plurality of pieces of shooting information; a mesh generating unit that generates a mesh model from a point cloud, and a reproduction data generating unit that generates reproduction data including data of a mesh model corresponding to at least a part of a composite point cloud that is a point cloud obtained by combining the plurality of point cloud data. Information processing equipment.

2. The point cloud data or mesh data that is data of a mesh model generated from the point cloud data is synthesis target data, The process of generating the reproduction data by the reproduction data generating unit includes unifying reference positions of position information included in each of the plurality of compositing target data corresponding to the plurality of point cloud data, and compositing the plurality of compositing target data.

2. An information processing device according to claim 1.

3. the shooting information includes time stamp information indicating time stamps added to the color information and the depth information, The acquisition unit acquires a plurality of pieces of shooting data including a series of the shooting information generated by shooting the object from one viewpoint during a predetermined shooting period, the shooting data being based on shooting from different viewpoints; In the plurality of pieces of shooting data, the shooting information including the time stamp information indicating a common point in the shooting period is target shooting information, and the composite point cloud is a point cloud obtained by combining the plurality of point cloud data generated from the plurality of pieces of shooting information including the target shooting information of each piece of shooting data.

2. An information processing device according to claim 1.

4. The reproduction data generating section outputs the reproduction data generated by corresponding each piece of the series of shooting information to the target shooting information in a time series based on the time stamp information.

4. An information processing device according to claim 3.

5. The reproduction data generating unit generates the reproduction data including data of a mesh model corresponding to a part of the composite point group.

2. An information processing device according to claim 1.

6. On one or more computers, Acquiring a plurality of pieces of shooting information including color information and depth information generated by shooting a target including a moving object, the shooting information being based on shooting from different viewpoints; generating point cloud data for each of the pieces of shooting information from the color information and the depth information, thereby generating a plurality of pieces of the point cloud data from the plurality of pieces of shooting information; generating a mesh model from the point cloud, and generating reproduction data including data of a mesh model corresponding to at least a part of a composite point cloud, which is a point cloud obtained by combining the plurality of point cloud data; Information processing program.

7. One or more computers Acquiring a plurality of pieces of shooting information including color information and depth information generated by shooting a target including a moving object, the shooting information being based on shooting from different viewpoints; generating point cloud data for each of the pieces of shooting information from the color information and the depth information, thereby generating a plurality of pieces of the point cloud data from the plurality of pieces of shooting information; generating a mesh model from the point cloud, and generating reproduction data including data of a mesh model corresponding to at least a part of a composite point cloud, which is a point cloud obtained by combining the plurality of point cloud data; Information processing method.

Images