Information processing device, information processing method

Abstract

To provide a technique for generating mesh group data according to the characteristics of an imaging device and the characteristics of a lens.SOLUTION: An information processing device determines an arrangement of a vertex group based on both first information of characteristics of an imaging device and second information of characteristics of a lens mounted on the imaging device; and generates mesh group data based on the vertex group in the determined arrangement.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a technique for generating data of a mesh group.

Background Art

[0002] As a display device used for viewers to enjoy immersive videos, a display device has been proposed that can be worn on the head to detect the viewing direction of the viewer and view a wide-angle video while looking around. In such a display device, if distortion remains in the video, there is a risk of causing physical discomfort to the viewer due to motion sickness or a sense of discomfort. As a method of correcting and displaying distortion and projection of a wide-angle video, there is a method of reproducing using mesh data.

[0003] Patent Document 1 discloses a technique for generating hemispherical mesh data for correcting and displaying a fisheye image captured by an imaging device by capturing a hemisphere as a real object generated by a 3D printer or the like based on the shape data of the hemisphere and associating the captured image with the vertices on the shape data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In recent years, head-mounted displays (HMDs), which are worn on the head to allow viewers to enjoy immersive visuals, have become widely used. When displaying images on an HMD, the input image captured by an imaging device equipped with a fisheye lens is mapped to a mesh and placed in a 3D CG space. The mesh is then rendered based on a virtual camera to generate the image for display. In the case of interchangeable lens imaging devices, switching lenses changes the distortion of the captured image and the imaging range. Therefore, if one mesh data is used with different lens and camera combinations, distortion may occur in the displayed image during viewing, or the entire imaging range may not be displayed depending on the lens and mesh combination. As a result, there have been problems such as viewers experiencing motion sickness or discomfort when viewing images displayed on an HMD. This invention provides a technology for generating mesh data according to the characteristics of the imaging device and the characteristics of the lens. [Means for solving the problem]

[0006] One aspect of the present invention is a determination means for determining the arrangement of a group of vertices based on first information relating to the characteristics of an imaging device and second information relating to the characteristics of a lens attached to the imaging device. The determination means generates data for a mesh group based on the vertex group in the arrangement determined by the determination means. Equipped with 、 The determination means determines the arrangement such that the spacing between vertices is the spacing corresponding to the projection method of the lens. It is characterized by the following. [Effects of the Invention]

[0007] According to the configuration of the present invention, it is possible to provide a technology for generating mesh group data according to the characteristics of the imaging device and the characteristics of the lens. [Brief explanation of the drawing]

[0008] [Figure 1] A block diagram showing an example of the hardware configuration of the information processing device 100. [Figure 2] A block diagram showing an example of the functional configuration of the information processing device 100. [Figure 3] A flowchart of the process performed by the information processing device 100 to generate mesh data. [Figure 4] A diagram illustrating mesh data. [Figure 5] A diagram showing an example of a mesh object. [Figure 6] A diagram showing an example of a mesh object. [Figure 7] A diagram showing an example of a mesh object. [Figure 8] A diagram showing an example of a mesh object. [Figure 9] A diagram showing an example of a mesh object. [Figure 10] A diagram illustrating how to generate a list of vertex coordinates. [Figure 11] A diagram illustrating how to generate a triangular list. [Figure 12] A diagram showing an example of UV coordinates. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] [First Embodiment] In this embodiment, an example of an information processing device that determines the arrangement of vertex groups based on first information relating to the characteristics of an imaging device and second information relating to the characteristics of a lens attached to the imaging device, and generates mesh group data based on the vertex groups in that arrangement, will be described.

[0011] First, a hardware configuration example of the information processing apparatus 100 according to this embodiment will be described using the block diagram of FIG. 1. As the information processing apparatus 100 according to this embodiment, computer apparatuses such as a PC (personal computer), a smartphone, and a tablet terminal device can be applied.

[0012] The CPU 101 executes various processes using computer programs and data stored in the RAM 102 and the ROM 103. Thereby, the CPU 101 controls the operation of the entire information processing apparatus 100 and executes or controls various processes described as being performed by the information processing apparatus 100.

[0013] The RAM 102 has areas for storing computer programs and data loaded from the ROM 103 and the HDD (hard disk drive) 105. Further, the RAM 102 has an area for storing computer programs and data received from the external server 111 via the network I / F 110. Further, the RAM 102 has an area for storing various data input from the input device 107 There is an area for storing. Further, the RAM 102 has a work area used when the CPU 101 executes various processes. Thus, the RAM 102 can appropriately provide various areas.

[0014] In the ROM 103, setting data of the information processing apparatus 100, computer programs and data related to the startup of the information processing apparatus 100, computer programs and data related to the basic operation of the information processing apparatus 100, and the like are stored.

[0015] The HDD 105 is connected to the information processing apparatus 100 via the HDD I / F 104, and the CPU 101 can access the HDD 105 via the HDD I / F 104. The HDD I / F 104 is, for example, an I / F such as Serial ATA (SATA).

[0016] The HDD 105 stores the OS (operating system), computer programs and data that cause the CPU 101 to execute or control various processes described as being performed by the information processing device 100. The data stored in the HDD 105 includes information that will be treated as known information in the following description. The computer programs and data stored in the HDD 105 are loaded into the RAM 102 as appropriate according to the control of the CPU 101 and become the target of processing by the CPU 101. Note that the HDD 105 is an example of a secondary storage device.

[0017] The input device 107 is connected to the information processing device 100 via the input interface 106, and the CPU 101 can communicate with the input device 107 via the input interface 106. The input interface 106 is, for example, a serial bus interface such as USB or IEEE1394.

[0018] The input device 107 is a user interface such as a keyboard, mouse, or touch panel screen; an imaging device such as a camera capable of capturing moving and / or still images; a scanner device; and other such equipment.

[0019] The output device 109 is connected to the information processing device 100 via the output interface 108, and the CPU 101 can communicate with the output device 109 via the output interface 108. The output interface 108 is, for example, a video output interface such as DVI or HDMI (registered trademark).

[0020] The output device 109 is a device having a display screen that displays the processing results from the CPU 101 as images, characters, etc., and may have, for example, an LCD screen or a touch panel screen. The output device 109 may also be a projection device such as a projector that projects images or characters.

[0021] CPU101, RAM102, ROM103, HDD I / F104, input I / F106, output I / F108, and network I / F110 are all connected to system bus 112.

[0022] Next, an example of the functional configuration of the information processing device 100 according to this embodiment is shown in the block diagram of Figure 2. In this embodiment, we will describe a case in which all the functional units shown in Figure 2 are implemented by computer programs. In the following, we may describe the functional units shown in Figure 2 as the main processing units, but in reality, the functions of the functional units are realized by the CPU 101 executing the computer program corresponding to the functional unit. Note that one or more of the functional units shown in Figure 2 may be implemented in hardware (hardware such as ASICs or electronic circuits).

[0023] The information processing device 100 according to this embodiment generates mesh object data (mesh data) used when displaying captured images (still images captured periodically or irregularly, or images of each frame in a moving image) on a viewing device such as an HMD. Here, mesh data will be explained with reference to Figure 4.

[0024] As shown in Figure 4, the mesh object according to this embodiment is a hemispherical object composed of multiple meshes (mesh groups), where each mesh is a triangular patch. In other words, the mesh data is the data of the mesh group that constitutes the mesh object.

[0025] The mesh data relating to such a mesh object includes information for positioning it in a three-dimensional virtual space such that the angle from the origin of the mesh object (the center of the hemisphere, a single point equal in distance from each mesh vertex) to the vertices of the mesh in the mesh object is equal to the angle of the incident light ray recorded in the pixels of the captured image mapped to the mesh object.

[0026] The captured image is mapped as a texture to a mesh object, and the mapped mesh object is placed in a 3D virtual space. Then, a rendering image obtained by rendering the texture from the origin of the placed mesh object is generated as a display image to be shown on the output device 109. If the output device 109 is an HMD, the generated display image is further transformed to take into account the characteristics of the device, such as eyepiece distortion.

[0027] Mesh data includes a list of the 3D coordinates of each mesh vertex (vertex coordinate list), coordinates for identifying the 2D coordinates corresponding to the 3D coordinates of the mesh vertices in the captured image mapped to the mesh object (UV coordinates), and a list of the indices of each mesh vertex (triangle list). This data contains topological information represented by this information.

[0028] When mapping captured images to a mesh object, the accuracy of distortion correction during image display changes depending on the arrangement of vertices in the mesh object. The higher the density of vertices in the mesh object, the better the accuracy of distortion correction during image display. Also, the wider the range of vertex generation, the larger the size of the mesh object, and the larger the size of the captured image mapped to such a mesh object. Therefore, displaying such an captured image allows viewers to see captured images with a wider field of view. On the other hand, the more vertices there are, the higher the load on mesh object generation, mapping, and image display, and the larger the data size of the mesh object. Therefore, it is necessary to place many vertices in areas with large distortion where correction accuracy is required, and to place vertices within an appropriate range according to the field of view.

[0029] In this embodiment, the arrangement of vertex groups is determined according to the characteristics of the imaging device that captured the image to be mapped to the mesh object, and the characteristics of the lens attached to the imaging device, and a mesh object is generated consisting of the vertex groups with the determined arrangement. Various lenses, such as fisheye lenses, can be used as the lens.

[0030] An example of a mesh object that maps images captured by an imaging device equipped with a long focal length lens, and an example of a mesh object that maps images captured by an imaging device equipped with a short focal length lens, will be explained using Figure 5.

[0031] Figure 5(a) shows an example of a mesh object 501 that maps images captured by an imaging device equipped with a lens with a focal length of f1. Figure 5(b) also shows an example of a mesh object 502 that maps images captured by an imaging device equipped with a lens with a focal length of f2 (>f1).

[0032] When the focal length is short, the imaging angle of view becomes wider, so it is necessary to widen the vertex placement range (i.e., the mesh generation range) in order to display the entire imaging area. Therefore, in this embodiment, the shorter the focal length of the lens, the wider the area over which vertices are placed to generate a mesh object with a wider mesh area.

[0033] Furthermore, the wider the light-receiving area of ​​the imaging sensor in the imaging device, the wider the imaging field of view. Therefore, in order to display the entire imaging area, it is necessary to widen the vertex placement range (i.e., the mesh generation range). For this reason, in this embodiment, the wider the light-receiving area of ​​the imaging sensor in the imaging device, the wider the area over which vertices are placed to generate a mesh object with a wider mesh area.

[0034] In Figure 5(a), point 503 is the origin of mesh object 501 (the origin of the hemisphere represented by mesh object 501), and point 504 is the origin of mesh object 502 (the origin of the hemisphere represented by mesh object 502). In Figure 5(a), the captured image mapped to mesh object 501, as seen from a virtual viewpoint placed at point 503, is rendered as the display image. In Figure 5(b), the captured image mapped to mesh object 502, as seen from a virtual viewpoint placed at point 504, is rendered as the display image.

[0035] In the case of a hemispherical mesh object, the display area can be represented by an angle. As shown in Figure 5, the field of view of the displayed image can be controlled by controlling the position of each vertex of the mesh in the mesh object based on the focal length.

[0036] Next, Figure 6(a) shows an example of a mesh object 601 that maps images captured by an imaging device equipped with an equidistant projection lens. Figure 6(b) also shows an example of a mesh object 602 that maps images captured by an imaging device equipped with an orthogonal projection lens.

[0037] Depending on the lens projection method, the spacing of incident light rays differs, resulting in different spacings for the incident angles of light rays recorded in adjacent pixels of the captured image. In equidistant projection, the distance from the projection center on the captured image is proportional to the incident angle of light rays relative to the front. Therefore, in this embodiment, each vertex is arranged so that the angles from the origin of the mesh object are equally spaced. On the other hand, in orthogonal projection, the spacing of the incident angles of light rays recorded in adjacent pixels is narrower at the periphery of the captured image than at the center of the image. Therefore, in this embodiment, each vertex is arranged so that the spacing between vertices becomes narrower the closer it is to the periphery of the captured image.

[0038] By calculating the spacing between vertices in the radial direction of a hemisphere based on the lens projection method, it is possible to generate the desired mesh object even when switching to a different lens with a different projection method attached to the imaging device. This allows for a display with less distortion, reducing discomfort during viewing.

[0039] Figure 7(a) shows an example of a mesh object 701 that maps images captured by an imaging device equipped with a distortion-free lens. Figure 7(b) shows an example of a mesh object 702 that maps images captured by an imaging device equipped with a distortion-causing lens.

[0040] Due to distortion, the angle at which light rays enter the image changes, resulting in different spacings between light rays recorded in adjacent pixels of the captured image. In this embodiment, when distortion is present, vertices are placed at positions adjusted to match the distortion rate, and the density of these vertices is adjusted. This improves the accuracy of distortion correction and allows for the display of captured images that reduce discomfort during viewing.

[0041] Figures 8(a) and 9(a) show examples of mesh objects generated when a user operates the imaging device to set a large number of imaging pixels. Figures 8(b) and 9(b) show examples of mesh objects generated when a user operates the imaging device to set a small number of imaging pixels.

[0042] In this embodiment, when a larger number of imaging pixels is set, the number of vertices is increased to construct a mesh object with more vertices, as shown in Figures 8(a) and 9(a). On the other hand, in this embodiment, when a smaller number of imaging pixels is set, the number of vertices is decreased to construct a mesh object with fewer vertices, as shown in Figures 8(b) and 9(b).

[0043] In this way, by changing the number of vertices according to the number of captured pixels, it is possible to control the accuracy of distortion correction when displaying the image, the time required to generate the mesh, the data size of the mesh object, and the load when playing back the mesh object.

[0044] Next, the process performed by the information processing device 100 to generate mesh data will be explained according to the flowchart in Figure 3. In step S301, the acquisition unit 202 acquires lens parameters (lens information) that represent the characteristics of the lens attached to the imaging device. Lens parameters are parameters that include information representing the characteristics of the lens, such as the projection method of the lens, the focal length of the lens, and the distortion information of the lens (including the presence or absence of distortion and the distortion rate). Note that the method of acquiring lens parameters is not limited to a specific acquisition method; for example, they may be acquired from the imaging device, or they may be acquired from the metadata of the captured image stored in the HDD 105.

[0045] In step S302, the acquisition unit 201 acquires camera parameters (camera information) that represent the characteristics of the imaging device. Camera parameters are parameters that include information representing the characteristics of the imaging device, such as the size of the imaging sensor, the number of pixels of the imaging sensor, and the principal point position. Note that the method of acquiring camera parameters is not limited to a specific acquisition method; for example, they may be acquired from the imaging device, or they may be acquired from the metadata of the captured images stored in the HDD 105.

[0046] In step S303, the acquisition unit 203 acquires mesh generation parameters, which are parameters required to generate a mesh object. The mesh generation parameters include, for example, the number of vertices per pixel, k,p. The method of acquiring the mesh generation parameters is not limited to a specific method; for example, they may be acquired from the HDD 105.

[0047] In step S304, the generation unit 204 generates a list of reference vertices using the lens parameters obtained in step S301, the camera parameters obtained in step S302, and the mesh generation parameters obtained in step S303.

[0048] In step S305, the generation unit 205, Step S304 Using the generated reference vertex list, a list containing the 3D coordinates of each mesh vertex in the mesh object is generated as a vertex coordinate list.

[0049] This embodiment describes a case where mesh data for a dome-shaped mesh object, where the vertices are arranged on concentric circles, is generated. The mesh data will be explained using Figure 4 as an example.

[0050] Mesh data is generated based on the number of sectors c, frequency a, field of view θ, lens parameters, and camera parameters. A sector refers to the region of one sector when a hemisphere (circle) viewed from the front is divided into multiple sectors, as shown in the front view of Figure 4. The number of sectors c is equal to the number of vertices on the circumference one circle outside the central vertex 401 in the front view of Figure 4. In a mesh object, vertices are arranged concentrically, and the number of vertices on the circle C' one circle outside a given circle C is equal to the number of sectors compared to the number of vertices on circle C. The frequency a is the number of divisions of the arc from vertex 401 to vertex 402 (the vertex at the other end when vertex 401 is one end of the sector boundary), as shown in the top view of Figure 4. The field of view θ is the field of view of the captured image mapped to the mesh object. In Figure 4, the number of sectors c=4, the frequency a=5, and the field of view θ=180°.

[0051] Lens parameters and camera parameters are used to calculate the coordinates of the corresponding position (projected position) in the captured image when the imaging device captures a position (3D position) in real space (3D space). Based on the lens parameters and camera parameters, a projection function x=P(VC) is defined to calculate the "UV coordinate x" from the "3D coordinate VC in the camera coordinate system (a coordinate system based on the position and orientation of the imaging device)". In the case of an imaging device equipped with a fisheye lens, the projection function takes into account the projection method, focal length, and distortion of the fisheye lens, projects the 3D coordinate VC onto the 2D coordinate on the captured image, converts the 2D coordinate into the UV coordinate x in the UV coordinate system, and calculates the UV coordinate x.

[0052] Next, we will explain how to generate a list (vertex coordinate list) in which the 3D coordinates of each mesh vertex in such a mesh object are registered, using Figure 10 as an example.

[0053] In step S304, as shown in Figure 10(a), a list is generated as the reference vertex list, which contains the 3D coordinates of each vertex (from vertex 1002 to vertex 1003) that are arranged on the arc.

[0054] In step S305, as shown in Figure 10(b), the vertices in the mesh object are arranged by placing each vertex, whose 3D coordinates are registered in the reference vertex list, at equal intervals around the origin of the mesh object in the forward direction. Then, a list containing the 3D coordinates of each of these placed vertices is generated as a vertex coordinate list.

[0055] The reference vertex list is generated as follows: The reference vertex list is a list in which the 3D coordinates of each vertex placed on the arc are registered based on frequency a, field of view θ, and hemispherical radius R. The radius R can be any value because the angle from the origin of the mesh object is important (default value).

[0056] As shown in Figure 10(a), vertex 1002, which is the starting point of the arc, is placed at a distance of radius R from a suitable position (the origin of the mesh object). Then, the position of vertex 1003, which is the endpoint of the arc, is calculated such that the angle 1001 formed by the line segment connecting vertex 1002 and the origin and the line segment connecting vertex 1003 and the origin is half the field of view θ. In an equidistant projection fisheye lens, the field of view θ can be calculated by calculating θ = s / f from the focal length f and the width s of the image sensor. Here, the field of view θ was calculated based on the width of the image sensor, but it is also acceptable to calculate the field of view θ based on the height or diagonal length of the image sensor. Next, vertices are placed at each position that divides the arc, with vertex 1002 at one end and vertex 1003 at the other, into equal parts a. A list is then generated as the reference vertex list, which registers the sequence of vertices on the arc excluding vertex 1002 (vertex 1003 and the vertices placed at each position that divides the arc into equal parts a). Here, the frequency a can be calculated by a = wxk from the number of horizontal pixels w and the number of vertices k per pixel in the captured image mapped to the mesh object.

[0057] The reference vertex list registers the 3D coordinates of a vertex Sj (j=1,…,a) in order of proximity to the central vertex 401 in the front view of Figure 4. If the lens projection is equidistant projection, the vertices are placed at equal intervals. In the case of orthogonal projection, the spacing of the vertices may be changed depending on the projection method, for example, by arranging the vertices so that the spacing becomes narrower as j increases. If there is lens distortion, the position of the vertices may be adjusted according to the distortion rate with respect to the vertex position calculated according to the projection method. For example, in the case of equidistant projection, if the distortion rate of vertex Sj, which represents the ratio of ideal image height to actual image height, is dj, then the angle φ of vertex Sj j If D is the sum of the curvatures of all reference vertices (vertices whose 3D coordinates are registered in the reference vertex list), then φ j =φ j-1 +d j It can be calculated as θ / (2D).

[0058] In this way, by generating a reference vertex list based on camera parameters and lens parameters, mesh data can be generated with a vertex generation range and vertex density suitable for the characteristics of the lens and imaging device.

[0059] The number of sectors c can be calculated by calculating c = pπw / a from the number of vertices p per pixel, where π is the ratio of a circle's circumference to its diameter. Then, as shown in Figure 10(b), a list is generated as a vertex coordinate list containing the 3D coordinates of the central vertex v0 and the vertices (v1, v2, v3, v4) that are arranged by rotating each reference vertex Sj by 2π / (j×c)[°] at equal intervals around the origin in the forward direction. The vertex coordinate list contains the 3D coordinates of N vertices, where N = 1 + c × a × (a + 1) / 2. The frequency a and the number of sectors c can both be controlled, or one can be fixed and only the other controlled.

[0060] In this way, by generating a list of vertex coordinates based on camera parameters and lens parameters, mesh data can be generated with a vertex generation range and vertex density suitable for the characteristics of the lens and imaging device.

[0061] Returning to Figure 3, in step S306, the generation unit 206 uses the vertex coordinate list generated in step S305 to generate a list of the indices of the three vertices of each mesh (triangle) that makes up the mesh object, as a triangle list. The method for generating the triangle list will be explained using Figure 11 as an example.

[0062] Each vertex of a mesh is identified by its vertex index, so a single mesh can be represented as a mesh(i,j,k) with the vertex indices of three vertices as a set. In other words, a mesh is represented by a set of multiple vertex indices. As shown in Figure 11(a), the vertex index of the central vertex is 0, the vertex indices of the vertices on the innermost concentric circle are 1, 2, 3, 4 clockwise, and the vertex indices of the vertices on the second innermost concentric circle are 5, 6, ... clockwise.

[0063] As shown in Figure 11(b), the mesh indices (face numbers) are 0, 1, 2, 3 clockwise for the innermost concentric circles, and 4, 5, ..., 15 clockwise for the second innermost concentric circles.

[0064] Then, as shown in Figure 11(c), a triangle list is generated for each mesh, with the face number of the mesh and the vertex index of each vertex constituting the mesh registered in association.

[0065] In step S307, the calculation unit 207 performs the following: Step S305 The process involves determining the 2D coordinates on the captured image that are mapped to the mesh object, corresponding to each 3D coordinate registered in the generated vertex coordinate list, identifying the corresponding coordinates (UV coordinates) in the UV coordinate system, and generating a list of these identified UV coordinates as a UV coordinate list.

[0066] An example of UV coordinates is shown in Figure 12. UV coordinates represent the correspondence between the vertices of a mesh object and the 2D coordinates on the captured image (in Figure 12, the captured image was taken using a fisheye lens), which is the texture. For example, if the 3D coordinates of a mesh object's vertex are (x,y,z) and the corresponding UV coordinates are (u,v), then the 3D coordinates (x,y,z) are associated with a point on the texture corresponding to the UV coordinates (u,v). When displaying the mesh (i.e., the partial image in the texture corresponding to the mesh), the color at the UV coordinates (u,v) of the texture is displayed at the pixel position corresponding to the 3D coordinates (x,y,z). In this embodiment, UV coordinates are expressed in a coordinate system where the horizontal axis is u, the vertical axis is v, the origin is the lower left of the captured image (0,0), the right end is u=1, and the top end is v=1. The range of possible UV coordinates is [0~1] for both u and v.

[0067] The UV coordinates for the 3D coordinates VI of the vertices registered in the vertex coordinate list are calculated, and a UV coordinate list is generated, which is a list in which these calculated UV coordinates are registered. The UV coordinate list contains the UV coordinates corresponding to N vertices. The 3D coordinates Vci of the vertices in the camera coordinate system are projected onto the captured image using the projection function P of the imaging device, and the UV coordinates Xi = P(Vci) are calculated.

[0068] In step S308, the output unit 209 generates mesh data including the vertex coordinate list generated in step S305, the triangle list generated in step S306, and the UV coordinate list generated in step S307. The output unit 209 then outputs the generated mesh data as a file. The output destination of the file by the output unit 209 is not limited to a specific destination. For example, the output unit 209 may output (save) the file to the HDD 105, or it may output (transmit) it to an external server 111 via the network I / F 110. Alternatively, the output unit 209 may use the generated mesh data as metadata for an image image mapped to a mesh object, and generate and output a file containing the image image and the mesh data.

[0069] The process shown in the flowchart of Figure 3 is performed when the characteristics of the imaging device or the characteristics of the lens change, such as when various settings in the imaging device are changed or when the lens attached to the imaging device is replaced with a lens having different characteristics.

[0070] Alternatively, mesh data may be saved to HDD105, and when displaying the captured image, a mesh object based on the saved mesh data may be generated, and the captured image mapped to the generated mesh object may be used as the display target.

[0071] Thus, according to this embodiment, mesh data can be generated based on the characteristics of the lens and the imaging device even when the characteristics of the imaging device and / or the characteristics of the lens change. Therefore, for example, even if the lens is changed in an interchangeable lens imaging device, mesh data can be generated that takes into account the distortion and differences in imaging range caused by the combination of lens and imaging device. This makes it possible to generate mesh data that can reduce the sense of discomfort when viewing the captured image displayed on a display device such as an HMD.

[0072] The information processing device 100 according to this embodiment can be used in, for example, the following system. The information processing device 100 generates a mapped image by mapping the image captured by the imaging device provided in the HMD onto a mesh object. The information processing device 100 then transmits the generated mapped image to the display screen of the HMD, the output device 109, or other devices.

[0073] The information processing device 100 may also display the captured image and the mapped image obtained by mapping the captured image to a mesh object side by side on the output device 109. In this case, the user can compare the captured image and the mapped image and adjust various parameters for generating mesh data, or make various adjustments / settings related to the generation of mesh data. In addition to or instead of the captured image and the mapped image, the information processing device 100 may also display a mesh object generated based on the current mesh data on the output device 109 or the like.

[0074] [Second Embodiment] In the first embodiment, the mesh object was a hemispherical object. However, the shape of the mesh object is not limited to a hemisphere, and therefore, the method for generating the mesh data is not limited to a specific method, as long as it is a method that results in a vertex arrangement that takes into account the above properties according to the camera parameters and lens parameters.

[0075] Furthermore, although the first embodiment described the mesh as consisting entirely of triangular patches, it is not limited to this; it may also consist of polygonal patches with quadrilaterals or more, or multiple types of patches may be used, such as using both triangular and quadrilateral patches. In other words, while the first embodiment used a list of triangles for the topology information, it may also be represented by a list of polygons represented by other polygons, such as a list of quadrilaterals.

[0076] Furthermore, the information processing device 100 may also accept a parallax fisheye video as the target for mapping. A parallax fisheye video is a video in which two parallax video images, captured by a parallax imaging device equipped with two fisheye lenses, are placed side by side. In this case, the first embodiment can be applied to capture images of each frame constituting the fisheye video. Note that instead of video images, still images captured periodically or irregularly may also be used.

[0077] Furthermore, the numerical values, processing timing, processing order, processing entity, data (information) destination / source / storage location, etc., used in each of the above embodiments are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0078] Furthermore, some or all of the embodiments described above may be used in appropriate combinations. Alternatively, some or all of the embodiments described above may be used selectively.

[0079] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0080] The disclosures herein include the following information processing devices, information processing methods, and computer programs.

[0081] (Item 1) A determination means for determining the arrangement of a group of vertices based on first information relating to the characteristics of an imaging device and second information relating to the characteristics of a lens attached to the imaging device, The determination means generates data for a mesh group based on the vertex group in the arrangement determined by the determination means. An information processing device characterized by comprising:

[0082] (Item 2) The information processing device according to item 1, characterized in that the determination means determines the arrangement of the vertex group using a sequence of vertices arranged on an arc corresponding to the first information and the second information.

[0083] (Item 3) The information processing apparatus according to item 1 or 2, characterized in that the determination means determines the arrangement within a range corresponding to the focal length of the lens.

[0084] (Item 4) The information processing apparatus according to item 1 or 2, characterized in that the determination means determines the arrangement within a range corresponding to the size of the imaging sensor of the imaging device.

[0085] (Item 5) The information processing apparatus according to item 1 or 2, characterized in that the determination means determines the arrangement such that the spacing between vertices is an interval corresponding to the projection method of the lens.

[0086] (Item 6) The information processing apparatus according to item 1 or 2, characterized in that the determination means determines the arrangement according to the distortion rate of the lens.

[0087] (Item 7) The information processing apparatus according to item 1 or 2, characterized in that the determination means determines the arrangement of a number of vertices corresponding to the number of pixels captured by the imaging device.

[0088] (Item 8) The information processing device according to any one of items 1 to 7, characterized in that the generation means generates data including a list of the three-dimensional coordinates of each of the vertex group, a list of coordinates for identifying the two-dimensional coordinates corresponding to the three-dimensional coordinates of the vertices of the mesh in an image mapped to a mesh object having the vertex group, and a list of indices of each mesh defined by the vertex group.

[0089] (Item 9) The information processing device according to any one of items 1 to 8, characterized in that the determination means and the generation means operate when the characteristics of the imaging device and / or the characteristics of the lens change.

[0090] (Item 10) moreover, The information processing device according to any one of items 1 to 9, characterized in that it includes output means for generating and outputting a file containing the captured image and the data, with the aforementioned data used as metadata for an image captured and mapped to a mesh object composed of the mesh group.

[0091] (Item 11) moreover, An information processing device according to any one of items 1 to 10, characterized by comprising means for displaying an captured image and a mapped captured image obtained by mapping the captured image onto a mesh object composed of the mesh group.

[0092] (Item 12) moreover, An information processing device according to any one of items 1 to 11, characterized by comprising means for displaying mesh objects composed of the aforementioned mesh group.

[0093] (Item 13) The information processing device according to item 1, characterized in that the captured image mapped to the mesh object composed of the mesh group is an image captured using a fisheye lens.

[0094] (Item 14) The information processing device according to item 1, characterized in that the captured image mapped to the mesh object composed of the mesh group is an image of each frame of a moving image obtained by arranging two moving images with parallax, captured by an imaging device equipped with two fisheye lenses, left and right.

[0095] (Item 15) An information processing method performed by an information processing device, The determination means of the information processing device includes a determination step of determining the arrangement of a group of vertices based on first information relating to the characteristics of an imaging device and second information relating to the characteristics of a lens attached to the imaging device. The generation means of the information processing device includes a generation step that generates data of a mesh group based on the vertex group in the arrangement determined in the determination step, and An information processing method characterized by comprising:

[0096] (Item 16) A computer program that causes a computer to function as one of the means of an information processing device described in any one of items 1 through 14.

[0097] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0098] 201: Acquisition unit 202: Acquisition unit 203: Acquisition unit 204: Generation unit 205: Generation unit 206: Generation unit 207: Calculation unit 209: Output unit

Claims

1. A determination means for determining the arrangement of a group of vertices based on first information relating to the characteristics of an imaging device and second information relating to the characteristics of a lens attached to the imaging device, The determination means generates data for a mesh group based on the vertex group in the arrangement determined by the determination means. Equipped with, The information processing apparatus is characterized in that the determination means determines the arrangement such that the spacing between vertices is an interval corresponding to the projection method of the lens.

2. The information processing device according to claim 1, characterized in that the determination means determines the arrangement of the vertex group using a sequence of vertices arranged on an arc corresponding to the first information and the second information.

3. The information processing apparatus according to claim 1, characterized in that the generation means generates data including a list of the three-dimensional coordinates of each of the vertex group, a list of coordinates for identifying the two-dimensional coordinates corresponding to the three-dimensional coordinates of the vertices of the mesh in an image mapped to a mesh object having the vertex group, and a list of indices of each mesh defined by the vertex group.

4. The information processing apparatus according to claim 1, characterized in that the determination means and the generation means operate when the characteristics of the imaging device and / or the characteristics of the lens change.

5. moreover, The information processing apparatus according to claim 1, characterized in that it includes output means for generating and outputting a file containing the captured image and the data, with the aforementioned data used as metadata for an image captured and mapped to a mesh object composed of the mesh group.

6. moreover, The information processing apparatus according to claim 1, comprising means for displaying an captured image and a mapped captured image obtained by mapping the captured image onto a mesh object composed of the mesh group.

7. moreover, The information processing apparatus according to claim 1, further comprising means for displaying mesh objects composed of the aforementioned mesh group.

8. The information processing device according to claim 1, characterized in that the captured image mapped to the mesh object composed of the mesh group is an image captured using a fisheye lens.

9. The information processing device according to claim 1, characterized in that the captured image mapped to the mesh object composed of the mesh group is an image of each frame of a moving image obtained by arranging two moving images with parallax, captured by an imaging device equipped with two fisheye lenses, left and right.

10. An information processing method performed by an information processing device, The determination means of the information processing device includes a determination step of determining the arrangement of a group of vertices based on first information relating to the characteristics of an imaging device and second information relating to the characteristics of a lens attached to the imaging device. The generation means of the information processing device includes a generation step that generates data of a mesh group based on the vertex group in the arrangement determined in the determination step, and Equipped with, The information processing method is characterized in that the determination step determines the arrangement such that the spacing between vertices is an interval corresponding to the projection method of the lens.

11. A computer program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 9.

Images