Multi-angle projection method, device, equipment and storage medium

Abstract

The present invention provides a multi-angle projection method, device, equipment and storage medium. The method determines the three-dimensional coordinates of multiple key points of the target object based on at least one image of the target object. For any target projection angle among multiple projection angles, the method determines the projection coordinates of each key point on the projection surface at the target projection angle based on the target projection angle and the three-dimensional coordinates of each key point. The method generates an image of the target object at the target projection angle based on each projection coordinate. The method extracts the three-dimensional coordinates of multiple key points of the target object, projects the multiple key points, and finally generates the image at the projection angle based on the projection coordinates of the multiple key points. Compared with generating a three-dimensional model of the target object and performing a projection transformation on the three-dimensional model, the method greatly reduces the coordinate points to be processed, thereby reducing the amount of calculation.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional display technology, and in particular to a multi-angle projection method, device, equipment and storage medium. Background Art

[0002] With the continuous iterative development of 3D display technology, multi-angle projection of objects in images has become a common problem in 3D display. For example, when an image at one angle is input, images at other angles can be output.

[0003] In existing technologies, an object is typically first modeled in three dimensions, and then the 3D model is projected and rendered from multiple angles. When performing multi-angle projection and rendering on a 3D model, the model is first projected according to a projection matrix, and then the projection result is rendered. Due to the large number of points in the model that need to be processed and the large number of parameters involved in the projection matrix, the computational complexity of projection and subsequent rendering is high. Therefore, projection and rendering methods are typically deployed on a GPU.

[0004] In the process of multi-angle projection and rendering, how to reduce the amount of calculation without reducing the projection transformation effect is an urgent problem to be solved. Summary of the Invention

[0005] The present invention provides a multi-angle projection method, device, equipment and storage medium, which are used to solve the problem of large calculation amount without reducing the projection transformation effect.

[0006] In a first aspect, the present invention provides a multi-angle projection method, the method comprising:

[0007] determining three-dimensional coordinates of a plurality of key points of the target object based on at least one image of the target object;

[0008] For any target projection angle among the multiple projection angles, determining the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point;

[0009] An image of the target object at the target projection angle is generated according to each of the projection coordinates.

[0010] Optionally, determining the three-dimensional coordinates of a plurality of key points of the target object based on at least one image of the target object includes:

[0011] The at least one image is input into a lightweight neural network model, and a plurality of three-dimensional coordinates output by the lightweight neural network model are determined as the three-dimensional coordinates of a plurality of key points of the target object.

[0012] Optionally, determining the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point includes:

[0013] Determine the three-dimensional coordinates of the camera;

[0014] For any target key point among the multiple key points, determining a projection light according to the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point;

[0015] An intersection point between the projection light and the projection surface is determined, and the projection coordinates of the target key point on the projection surface are determined based on the intersection point.

[0016] Optionally, the target key point coordinates and the camera coordinates are both coordinates in a world coordinate system; and determining the projection coordinates of the target key point on the projection surface according to the intersection point includes:

[0017] The shape parameters of the projection surface are acquired, and the coordinates of the intersection point in the world coordinate system are converted into projection coordinates in the projection surface coordinate system according to the shape parameters of the projection surface.

[0018] Optionally, generating an image of the target object at the target projection angle according to each of the projection coordinates includes:

[0019] constructing a plurality of Delaunay triangles according to the respective projection coordinates;

[0020] Generate and store corresponding interpolation coefficients based on basic information corresponding to the at least one image; the basic information includes: the Delaunay triangle to which each pixel point belongs, the vertex coordinates of the Delaunay triangle to which each pixel point belongs, and the texture coordinate values corresponding to the vertices of the Delaunay triangle;

[0021] For any target image to be processed of the target object, determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit;

[0022] The pixel value of each pixel point is determined according to the texture coordinate value of each pixel point and the texture map corresponding to the target image, so as to generate an image of the target object in the target image at the target projection angle.

[0023] Optionally, the interpolation coefficient includes the basic information; and determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit includes:

[0024] For any target pixel in the projected image, determine the weight information of the target pixel according to the position of the target pixel and the vertex coordinates of the Delaunay triangle to which the target pixel belongs in the interpolation coefficient;

[0025] The texture coordinate value of the target pixel point is determined according to the weight information and the texture coordinate values corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

[0026] Optionally, the interpolation coefficients include: a first array and a second array corresponding to each Delaunay triangle, as well as texture coordinate values corresponding to each vertex of each Delaunay triangle, and the Delaunay triangle to which each pixel belongs; and determining the texture coordinate value of each pixel in the projected image based on the interpolation coefficients stored in the storage unit includes:

[0027] For any target pixel in the projected image, a first weight is determined based on the first array and the target pixel coordinates, a second weight is determined based on the second array and the target pixel coordinates, and a third weight is determined based on the first weight and the second weight; the first array and the second array are determined based on the vertex coordinates of the Delaunay triangle to which each pixel belongs; the first weight and the second weight are functions of the target pixel coordinates, respectively, and the first array and the second array are multiple constants in the corresponding functions;

[0028] The texture coordinate value of the target pixel point is determined according to the first weight, the second weight, the third weight, and the texture coordinate values corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

[0029] Optionally, constructing a plurality of Delaunay triangles according to the projection coordinates includes:

[0030] determining remaining storage space information of the storage unit, and determining an upper limit on the number of constructed Delaunay triangles according to the remaining storage space information;

[0031] The Delaunay triangle is constructed according to the upper limit of quantity and each of the projection coordinates.

[0032] In a second aspect, the present invention provides a multi-angle projection device, comprising:

[0033] a first determining module, configured to determine three-dimensional coordinates of a plurality of key points of the target object based on at least one image of the target object;

[0034] a second determining module, configured to determine, for any one target projection angle among the plurality of projection angles, the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point;

[0035] A generating module is used to generate an image of the target object at the target projection angle according to each of the projection coordinates.

[0036] In a third aspect, the present invention provides a multi-angle projection system, comprising: a computing unit and a storage unit;

[0037] The computing unit is configured to execute the method according to any one of the first aspects;

[0038] The storage unit is used to store the intermediate results calculated by the calculation unit.

[0039] In a fourth aspect, the present invention provides an electronic device comprising: at least one processor and a memory;

[0040] The memory stores computer-executable instructions;

[0041] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method as described in any one of the first aspects.

[0042] In a fifth aspect, the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any one of the first aspects is implemented.

[0043] In a sixth aspect, the present invention provides a computer program product, comprising a computer program, which implements the method described in any one of the first aspects when executed by a processor.

[0044] The present invention provides a multi-angle projection method, device, equipment and storage medium. The method determines the three-dimensional coordinates of multiple key points of the target object based on at least one image of the target object. For any target projection angle among multiple projection angles, the method determines the projection coordinates of each key point on the projection surface at the target projection angle based on the target projection angle and the three-dimensional coordinates of each key point. The method generates an image of the target object at the target projection angle based on each projection coordinate. The method extracts the three-dimensional coordinates of multiple key points of the target object, projects the multiple key points, and finally generates the image at the projection angle based on the projection coordinates of the multiple key points. Compared with generating a three-dimensional model of the target object and performing a projection transformation on the three-dimensional model, the method greatly reduces the coordinate points to be processed, thereby reducing the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0046] Figure 1 A diagram illustrating an application scenario of a multi-angle projection method provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a flow chart of a multi-angle projection method provided by an embodiment of the present invention;

[0048] Figure 3 A schematic flow chart of another multi-angle projection method provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of the three-dimensional coordinates of multiple key points in a face provided by an embodiment of the present invention;

[0050] Figure 5 A schematic structural diagram of a multi-angle projection device provided by an embodiment of the present invention;

[0051] Figure 6 A schematic diagram of the architecture of a multi-angle projection system provided by an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention.

[0053] The above drawings illustrate specific embodiments of the present invention, which will be described in more detail below. These drawings and the accompanying description are not intended to limit the scope of the present invention in any way, but rather to illustrate the concept of the present invention to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0054] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present invention.

[0055] Multi-angle projection refers to the process of outputting a different projection angle for an image, based on the user's input. For example, in some VR applications, multi-angle projection allows users to view and analyze objects from different angles, enhancing the user experience. Figure 1 This is an application scenario diagram of a multi-angle projection method provided by an embodiment of the present invention, such as Figure 1 As shown, the projection surface can be a plane or a curved surface. The input image contains a target object. When the target object is viewed from three different projection angles, different projection images can be obtained at different positions on the projection surface.

[0056] When performing multi-angle projection, existing technologies typically first model the target object in the image to obtain a three-dimensional model of the target object, and then determine the projected image of the three-dimensional model. When building a three-dimensional model, detailed modeling methods (such as 3DMM and 3D Morphable Model) are typically used. However, these detailed modeling methods require a large number of model parameters and are computationally intensive. When projecting a three-dimensional model, a projection matrix is used to obtain a projected image. When calculating the projected image using the projection matrix method, every point on a surface must be processed, resulting in a high computational load. This load is even greater when the projection surface is curved.

[0057] Based on the above problems, this application considers obtaining only the three-dimensional coordinates of multiple key points in the target object during three-dimensional modeling. When calculating the projection coordinates of the target object on the projection surface, it is also only necessary to calculate the projection coordinates of multiple key points. Finally, all pixel information of the entire image is generated based on the projection coordinates of the multiple key points to obtain the image of the target object at the target projection angle. During the three-dimensional modeling and projection transformation stages, the amount of calculation is greatly reduced by calculating multiple key points. Finally, by generating all pixel information of the entire image based on the projection coordinates of multiple key points, it is possible to generate a projection image at any target projection angle without losing image transformation accuracy.

[0058] Figure 2 This is a flow chart of a multi-angle projection method provided by an embodiment of the present invention. The method is applied to a multi-angle projection device, which is provided on an electronic device. The method includes steps S201 to S203:

[0059] Step S201: Determine the three-dimensional coordinates of multiple key points of the target object based on at least one image of the target object.

[0060] The electronic device can capture images or receive images transmitted by other devices, and there is no limit on the number of images captured or received. When it is necessary to display an image at a target projection angle to a user, the three-dimensional coordinates of multiple key points of the target object can be determined first.

[0061] Determining the three-dimensional coordinates of multiple key points of a target object refers to determining the three-dimensional coordinates of the key points in three-dimensional space based on the plane coordinates of the key points in two-dimensional image space. This does not involve performing three-dimensional modeling of the target object, but rather determining the three-dimensional coordinates of the key points of the target object.

[0062] For example, when the target object is a human face, the 3D coordinates of multiple key points on the face may be determined. For example, when a face is represented by 68 key points, 6 key points may be used to represent the eyes and 5 key points may be used to represent the eyebrows. Other numbers of key points may also be used, such as 201 key points. Conventional 3D modeling methods determine the 3D coordinates of every point on the face. Unlike conventional 3D modeling methods, the number of 3D coordinates obtained here is significantly reduced.

[0063] When determining the three-dimensional coordinates of multiple key points of a target object, the determination may be based on a single image containing the target object, or may be based on multiple images containing the target object. For example, when the input is multiple images containing the target object, the three-dimensional coordinates of the multiple key points of the target object may be determined based on the multiple images; or, when the input is a video stream containing the target object, one or more images containing the target object may be obtained from the video stream and the three-dimensional coordinates of the multiple key points of the target object may be determined.

[0064] Step S202: for any target projection angle among the multiple projection angles, determine the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point.

[0065] After obtaining the three-dimensional coordinates of each key point in the target object, each key point can be mapped and transformed according to the needs of the projection transformation, that is, the corresponding projection coordinates of each key point on the projection surface are determined. When determining the projection coordinates, the target projection angle must be determined first.

[0066] like Figure 1 As shown in the figure, when the target projection angle is different, the projection coordinates of the target object on the projection surface are also different. For example, when the target projection angle is projection angle 1, the projection coordinates of each key point are located slightly to the right of the projection surface. When the target projection angle is projection angle 2, the projection coordinates of each key point are located slightly to the left of the projection surface.

[0067] The target projection angle can be determined according to the needs of the user. For example, when the user wants to view the target object from projection angle 1, projection angle 1 is determined as the target projection angle.

[0068] After the target projection angle is determined, the projection coordinates of the three-dimensional coordinates of each key point on the projection surface under the target projection angle can be determined.

[0069] Since the three-dimensional coordinates of each key point are projected, the amount of calculation can be greatly reduced compared to processing the three-dimensional model through the projection matrix.

[0070] Step S203: Generate an image of the target object at the target projection angle according to each of the projection coordinates.

[0071] After obtaining the projection coordinates of each key point on the projection surface, it is also necessary to restore the pixel information of each point in the target object according to these projection coordinates to obtain the image of the target object at the target projection angle.

[0072] Optionally, rasterization can be used to achieve this. Rasterization is a basic method of graphics rendering. We use rasterization to obtain pixel information of each point in the target object.

[0073] The present invention provides a multi-angle projection method, which determines the three-dimensional coordinates of multiple key points of the target object based on at least one image of the target object, and for any target projection angle among multiple projection angles, determines the projection coordinates of each key point on the projection surface at the target projection angle based on the target projection angle and the three-dimensional coordinates of each key point, and generates an image of the target object at the target projection angle based on each projection coordinate. By extracting the three-dimensional coordinates of multiple key points of the target object and projecting the multiple key points, and finally generating the image at the projection angle based on the projection coordinates of the multiple key points, this method greatly reduces the number of coordinate points to be processed compared to generating a three-dimensional model of the target object and performing a projection transformation on the three-dimensional model, thereby reducing the amount of calculation.

[0074] Figure 3 A schematic flow chart of another multi-angle projection method provided by an embodiment of the present invention, the method comprising steps S301 to S303:

[0075] Step S301: input the at least one image into a lightweight neural network model, and determine the multiple three-dimensional coordinates output by the lightweight neural network model as the three-dimensional coordinates of multiple key points of the target object.

[0076] Deep learning methods can be used to determine the 3D coordinates of multiple key points of a target object. Specifically, an image can be input into a lightweight neural network model, which outputs the 3D coordinates of the key points corresponding to the target object. The lightweight neural network model is a pre-trained network model that can determine the 3D coordinates of multiple key points of a target object in an image.

[0077] Optionally, based on the type of the target object, an image containing an object of the same type as the target object can be pre-input into the lightweight neural network model, and the three-dimensional coordinates of multiple key points corresponding to the object can be used as the output of the lightweight neural network model to train the lightweight neural network model.

[0078] For example, when the target object is a human face, multiple images containing human faces can be input into the lightweight neural network model in advance, and the three-dimensional coordinates of multiple key points of each face can be obtained through a three-dimensional scanning device. The obtained three-dimensional coordinates of multiple key points of each face are used as the output of the neural network model, thereby obtaining a trained lightweight neural network model.

[0079] Figure 4 A schematic diagram of the three-dimensional coordinates of multiple key points in a face provided by an embodiment of the present invention, such as Figure 4 As shown in FIG, for the input face image, the three-dimensional coordinates of multiple key points are obtained.

[0080] When determining the three-dimensional coordinates of multiple key points of a target object, a single image containing the target object can be input into the lightweight neural network to improve the efficiency of determining the three-dimensional coordinates of multiple key points of the target object.

[0081] The pre-trained lightweight neural network model can quickly map the two-dimensional key points on the image to the three-dimensional space.

[0082] Step S302: Determine the three-dimensional coordinates of the camera.

[0083] The camera represents the user's location, and the camera position represents the position from which the user views the target object. After determining the target projection angle, the camera position can be determined based on the positional relationship between the user and the electronic device. Optionally, the three-dimensional coordinates of the camera determined here can be three-dimensional coordinates in a world coordinate system.

[0084] Step S303: for any target key point among the multiple key points, determine a projection ray according to the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point.

[0085] For any target key point, the projection light can be determined according to the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point, that is, the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point are connected to obtain the projection light.

[0086] This method can be described as determining the projected coordinates of each key point on the projection surface through ray tracing. Since the three-dimensional coordinates of each key point of the target object are obtained, a point-based method can be used to obtain the corresponding coordinates on the projection surface. Ray tracing refers to tracing a ray from the target projection angle to determine the coordinates of the key point on the projection surface under the projection of the ray.

[0087] Before determining the projection ray, the target key points can be converted to coordinates in the world coordinate system. The coordinates of the target key points output by the lightweight neural network model may be based on a certain 3D coordinate point in the target object as the coordinate origin. Since the 3D coordinates of the camera are in the world coordinate system, each target key point also needs to be converted to coordinates in the world coordinate system.

[0088] Step S304: Determine the intersection point of the projection light and the projection surface, and determine the projection coordinates of the target key point on the projection surface according to the intersection point.

[0089] After the projection light is determined, the intersection of the projection light and the projection surface can be calculated based on the intersection of the light rays. The coordinates of the intersection point are also the coordinates in the world coordinate system. The projection surface is the position of the screen in the electronic device that receives the projection. The projection surface can be a flat surface or a curved surface.

[0090] By traversing each key point, the projection coordinates of each key point on the projection surface at the target projection angle can be calculated.

[0091] Since the intersection point of the projection light and the projection surface is calculated by ray intersection, the projection coordinates of the key points on the projection surface can be quickly calculated regardless of whether the projection surface is a plane or a curved surface.

[0092] By calculating the projection light, the projection coordinates of the key points on the projection surface can be quickly calculated based on the intersection of the projection light and the projection surface. Compared with the existing projection matrix method, this method is also applicable to scenes where the projection surface is a curved surface, and does not increase the amount of calculation due to the curved projection surface.

[0093] Optionally, determining the projection coordinates of the target key point on the projection surface according to the intersection point includes:

[0094] The shape parameters of the projection surface are acquired, and the coordinates of the intersection point in the world coordinate system are converted into projection coordinates in the projection surface coordinate system according to the shape parameters of the projection surface.

[0095] Optionally, the target key point coordinates and the camera coordinates are both coordinates in the world coordinate system. In step S304, the projection coordinates determined are coordinates in the world coordinate system. Since the final projected image needs to be displayed on the projection surface, it is necessary to determine the position of each projection coordinate relative to the projection surface, that is, to convert the projection coordinates from the world coordinate system to the projection surface's own coordinate system.

[0096] The projection surface can be either a flat surface or a curved surface. When the projection surface is flat, the projection coordinates in the world coordinate system can be directly converted to the projection coordinates in the projection surface coordinate system. When the projection surface is curved, the coordinates of the projection coordinates in the projection surface coordinate system need to be determined based on the shape parameters of the projection surface. In other words, the curved surface needs to be unfolded into a flat surface and the coordinates of the projection coordinates on the unfolded plane need to be determined.

[0097] Optionally, when the projection surface is a curved surface, a curve integral can be first performed based on the shape parameters of the projection surface to obtain the line length of the curve. For example, when the curved surface is curved in the horizontal direction and not curved in the vertical direction, the line length corresponding to the curved surface in the horizontal direction can be calculated by curve integral.

[0098] For the coordinates of the intersection point in the world coordinate system, the coordinates of the coordinates relative to the surface can be calculated, and the coordinates of the coordinates on the plane after the curve is unfolded can be calculated by curve integration of the converted coordinates.

[0099] When the projection surface is a curved surface, it is unfolded into a plane and the projection coordinates relative to the projection plane are calculated, which facilitates the subsequent calculation of the pixel values of other points on the target object on the projection plane and the final display of the projected image.

[0100] Rendering through rasterization means that when an image is projected at multiple angles, a triangle can be constructed based on the projection coordinates of each key point corresponding to the image. For each pixel in the projected image, the texture coordinate value corresponding to the pixel can be calculated based on the vertex coordinates and texture coordinate values corresponding to the Delaunay triangle to which the pixel belongs. Because the texture coordinate value of the pixel is related to the texture coordinate values of each vertex of the Delaunay triangle, after calculating the texture coordinate value corresponding to the pixel, RGB information can be extracted from the input image based on the texture coordinate value to obtain the projected image.

[0101] However, in practice, we may need to project multiple consecutively input images that are related. Correlation means that the position of a local part of the target object changes, that is, the target object makes a relatively small movement. For example, when it is necessary to perform multi-angle projection of multiple images of a person speaking, there is no need to calculate the three-dimensional coordinates of multiple key points of each image, and the projection coordinates of these three-dimensional coordinates on the projection surface. For any target image, the image of the target image at the target projection angle can be directly calculated based on the multiple projection coordinates corresponding to one of the multiple images.

[0102] When it is necessary to calculate the image of at least one image at the target projection angle, in order to reduce the amount of calculation, the interpolation coefficients can be stored, and the pixel value corresponding to each pixel point in the projected image corresponding to the target image can be quickly calculated based on the interpolation coefficients.

[0103] Step S305: construct multiple Delaunay triangles according to the projection coordinates.

[0104] In order to render the pixel information of each pixel in the projected image using a rasterization method, multiple Delaunay triangles can be constructed based on the projection coordinates of each key point. The Bowyer-Watson algorithm can be used to construct the Delaunay triangles.

[0105] Optionally, constructing a plurality of Delaunay triangles according to the projection coordinates includes:

[0106] Determine the remaining storage space information of the storage unit, and determine the upper limit of the number of constructed Delaunay triangles according to the remaining storage space information; and construct the Delaunay triangle according to the upper limit and each of the projection coordinates.

[0107] Since the stored interpolation coefficients are related to the number of constructed Delaunay triangles, the more interpolation coefficients are needed to be stored when the number of constructed Delaunay triangles is greater; conversely, the fewer interpolation coefficients are needed to be stored when the number of constructed Delaunay triangles is smaller. Since the interpolation coefficients need to be stored in a storage unit, the number of interpolation coefficients stored depends on the remaining storage space of the storage unit. When the remaining storage space is larger, more Delaunay triangles can be constructed; when the remaining storage space is smaller, fewer Delaunay triangles can be constructed.

[0108] Optionally, the remaining storage space information of the storage unit can be determined first, and an upper limit on the number of Delaunay triangles that can be constructed can be determined based on the remaining storage space information, so that when Delaunay triangles are constructed based on each projection coordinate, the number of constructed Delaunay triangles needs to be less than the upper limit.

[0109] By determining the upper limit of the number of Delaunay triangles, the interpolation coefficients corresponding to each Delaunay triangle can be successfully stored after the Delaunay triangle is constructed, avoiding the need to reconstruct the Delaunay triangle again after the interpolation coefficients of each Delaunay triangle cannot be successfully stored, thereby improving processing efficiency.

[0110] Step S306: Generate and store corresponding interpolation coefficients according to the basic information corresponding to the at least one image.

[0111] The basic information includes: the Delaunay triangle to which each pixel point belongs, the vertex coordinates of the Delaunay triangle to which each pixel point belongs, and the texture coordinate values corresponding to the vertices of the Delaunay triangle.

[0112] When storing the interpolation coefficient, the interpolation coefficient may be basic information corresponding to at least one image, and illustratively, may be basic information corresponding to the first image of the at least one input image.

[0113] Since the interpolation coefficients are related to the constructed Delaunay triangle, and the Delaunay triangle is related to the projection coordinates, and the projection coordinates are related to the target projection angle, in order to obtain projection images at multiple projection angles, it is necessary to determine the projection coordinates at multiple projection angles and calculate the interpolation coefficients based on the projection coordinates for storage. For example, when there are three projection angles, the interpolation coefficients corresponding to the three projection angles can be determined and stored.

[0114] For each pixel in the projected image, the texture coordinate value corresponding to the pixel can be calculated based on the vertex coordinates and texture coordinate values corresponding to the Delaunay triangle to which the pixel belongs. In this process, the Delaunay triangle to which the pixel belongs can be determined. If this information can be stored after calculating the Delaunay triangle to which each pixel belongs, then when subsequently calculating the projected image for other input images, there is no need to recalculate the Delaunay triangle to which each pixel belongs, which greatly reduces the amount of calculation and allows the calculation process to be implemented on a CPU (Central Processing Unit) without relying on a GPU (Graphics Processing Unit).

[0115] For a pixel point, the texture coordinate value corresponding to the pixel point is related to the texture coordinate value of the Delaunay triangle to which the pixel point belongs. The Delaunay triangle has three vertices, and there is a corresponding weight for each vertex. The weight represents the degree of influence of the texture coordinate value of the vertex on the texture coordinate value of the pixel point. Therefore, the texture coordinate value of the pixel point can be determined based on the weight and the texture coordinate values of the three vertices. The weight here is related to the position of the pixel point and each vertex. Therefore, it is also necessary to store the texture coordinate values corresponding to the vertices of each Delaunay triangle, as well as information related to the vertex coordinates of each Delaunay triangle.

[0116] Step S307 : for any target image to be processed of the target object, determine the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit.

[0117] Optionally, the target object can be any image in at least one input graphic, for example, the first image. In this case, since the interpolation coefficients have been stored in advance, the texture coordinate values of each pixel point in the projected image corresponding to the first image can be determined based on the interpolation coefficients.

[0118] Optionally, the target object can also be any image after the first image. When processing the first image, the interpolation coefficient is determined according to the projection coordinates corresponding to the first image and stored. When processing any image after the first image, the texture coordinate value of each pixel point in the target image can be calculated according to the interpolation coefficient.

[0119] Step S308: determining the pixel value of each pixel point according to the texture coordinate value of each pixel point and the texture map corresponding to the target image, so as to generate an image of the target object in the target image at the target projection angle.

[0120] For the target image, after determining the texture coordinate values of each pixel in the projected image, the pixel value of each pixel can be obtained based on the target image. The pixel value of any pixel can be obtained by sampling the texture map corresponding to the target image based on the texture coordinate value of the pixel. The pixel value can be the RGB value of the pixel. Based on the pixel value, the pixel can be displayed, thereby obtaining the image of the target image at the target projection angle.

[0121] Alternatively, the texture coordinate values of each pixel can be directly stored. However, for an image, the number of pixels is large, and storing the texture coordinate value of each pixel requires a large amount of storage space. Therefore, to balance the amount of computation and storage usage, this is achieved by storing interpolation coefficients, which does not require high storage space. In other words, by balancing the amount of computation and storage usage during the image rendering process, the most performant image transformation effect is achieved.

[0122] By pre-storing the interpolation coefficients, the projection image corresponding to the input target image can be quickly determined based on the interpolation coefficients, without having to perform steps such as determining the three-dimensional coordinates of multiple key points of the target image, and determining the projection coordinates corresponding to the three-dimensional coordinates. In addition, by storing the Delaunay triangle to which each pixel point belongs, when processing the target image, there is no need to recalculate the Delaunay triangle to which each pixel point belongs, thereby reducing the amount of calculation.

[0123] When storing the interpolation coefficients in the above step S306, the interpolation coefficients can be in two forms. One is to directly store the basic information, and the other is to calculate the first array and the second array based on the vertex coordinates of each Delaunay triangle in the basic information. The weight information of the target pixel point can be directly calculated based on the first array and the second array and the coordinates of each pixel point.

[0124] The following describes in detail how to calculate the weight information of the target pixel point based on the two interpolation coefficients and then calculate the texture coordinate value of the target pixel point.

[0125] Optionally, the interpolation coefficient includes the basic information; and determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit includes:

[0126] For any target pixel in the projected image, determine the weight information of the target pixel according to the position of the target pixel and the vertex coordinates of the Delaunay triangle to which the target pixel belongs in the interpolation coefficient;

[0127] The texture coordinate value of the target pixel point is determined according to the weight information and the texture coordinate values corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

[0128] When the stored interpolation coefficients are the basic information, the pixel points in the projected image can be traversed, and for the target pixel point (the target pixel point is a point within a certain Delaunay triangle), the target Delaunay triangle is obtained based on the stored basic information. The target Delaunay triangle is the Delaunay triangle to which the target pixel point belongs, and the vertex coordinates of the target Delaunay triangle are obtained. The weight is determined based on the coordinates of the target pixel point and the three vertex coordinates of the target Delaunay triangle. Among them, the vertices of the Delaunay triangle include vertex A, vertex B and vertex C, and the weight can be three values, namely the first weight, the second weight and the third weight.

[0129] For example, the weights can be calculated according to the following interpolation calculation formula of the Delaunay triangle, where α is the first weight, β is the second weight, and γ is the third weight. a ,y a ), (x b ,y b ), (x c ,y c ) are the coordinates of the three vertices of the Delaunay triangle, and (x, y) are the coordinates of the target pixel.

[0130]

[0131]

[0132] γ=1-α-β Formula 3

[0133] When calculating the texture coordinate value of the target pixel, the first weight is multiplied by the texture coordinate value of vertex A, the second weight is multiplied by the texture coordinate value of vertex B, and the third weight is multiplied by the texture coordinate value of vertex C. The three multiplication results are added together to obtain the texture coordinate value of the pixel. The texture coordinate value of the pixel contains two values, and the above formula can be used to calculate the two texture coordinate values separately.

[0134] By storing the basic information as the interpolation coefficient, the basic information can be directly stored in the storage unit without being deformed, which makes the operation simpler.

[0135] Optionally, the interpolation coefficients include: a first array and a second array corresponding to each Delaunay triangle, as well as texture coordinate values corresponding to each vertex of each Delaunay triangle, and the Delaunay triangle to which each pixel belongs; and determining the texture coordinate value of each pixel in the projected image based on the interpolation coefficients stored in the storage unit includes:

[0136] For any target pixel in the projected image, a first weight is determined based on the first array and the target pixel coordinates, a second weight is determined based on the second array and the target pixel coordinates, and a third weight is determined based on the first weight and the second weight; the first array and the second array are determined based on the vertex coordinates of the Delaunay triangle to which each pixel belongs; the first weight and the second weight are functions of the target pixel coordinates, respectively, and the first array and the second array are multiple constants in the corresponding functions;

[0137] The texture coordinate value of the target pixel point is determined according to the first weight, the second weight, the third weight, and the texture coordinate values corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

[0138] When storing interpolation coefficients, the obtained basic information can also be transformed to obtain the first array and the second array, and the first array and the second array can be stored, which can further simplify the process of calculating weights, thereby further reducing the amount of calculation when calculating the first weight and the second weight in real time.

[0139] Among them, the first array and the second array are only related to the three vertex coordinates of the Delaunay triangle, and the first weight and the second weight are respectively a binary linear function about the target pixel point, and the first array and the second array are respectively constants in the binary linear function. Exemplarily, the above formula 1 can be deformed to obtain a binary linear function of α about x and y, and the constant in the binary linear function is the first array. By storing the first array and the second array, after determining the value of the target pixel point (x, y), the first weight and the second weight can be obtained by simple multiplication and addition operations.

[0140] Compared with the above formula 1 and formula 2, by storing the first array and the second array, the process of calculating the first weight and the second weight is simplified.

[0141] After determining the first weight and the second weight, the third weight can be calculated, and then the texture coordinate value of the target pixel point can be determined based on the three weights and the texture coordinate values corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

[0142] By storing the first array and the second array, the weight calculation process can be further simplified, saving the computing power burden of the real-time calculation process.

[0143] Figure 5 This is a schematic structural diagram of a multi-angle projection device provided by an embodiment of the present invention. The device 50 includes:

[0144] A first determining module 501 is configured to determine the three-dimensional coordinates of a plurality of key points of a target object based on at least one image of the target object;

[0145] A second determining module 502 is configured to determine, for any one target projection angle among the plurality of projection angles, the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point;

[0146] The generating module 503 is configured to generate an image of the target object at the target projection angle according to each of the projection coordinates.

[0147] Optionally, the first determining module 501 is specifically configured to:

[0148] The at least one image is input into a lightweight neural network model, and a plurality of three-dimensional coordinates output by the lightweight neural network model are determined as the three-dimensional coordinates of a plurality of key points of the target object.

[0149] Optionally, the second determining module 502 is specifically configured to:

[0150] Determine the three-dimensional coordinates of the camera;

[0151] For any target key point among the multiple key points, determining a projection light according to the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point;

[0152] An intersection point between the projection light and the projection surface is determined, and the projection coordinates of the target key point on the projection surface are determined based on the intersection point.

[0153] Optionally, the target key point coordinates and the camera coordinates are both coordinates in a world coordinate system; when the second determining module 502 determines the projection coordinates of the target key point on the projection surface according to the intersection point, it is specifically configured to:

[0154] The shape parameters of the projection surface are acquired, and the coordinates of the intersection point in the world coordinate system are converted into projection coordinates in the projection surface coordinate system according to the shape parameters of the projection surface.

[0155] Optionally, the generating module 503 is specifically configured to:

[0156] constructing a plurality of Delaunay triangles according to the respective projection coordinates;

[0157] Generate and store corresponding interpolation coefficients based on basic information corresponding to the at least one image; the basic information includes: the Delaunay triangle to which each pixel point belongs, the vertex coordinates of the Delaunay triangle to which each pixel point belongs, and the texture coordinate values corresponding to the vertices of the Delaunay triangle;

[0158] For any target image to be processed of the target object, determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit;

[0159] The pixel value of each pixel point is determined according to the texture coordinate value of each pixel point and the texture map corresponding to the target image, so as to generate an image of the target object in the target image at the target projection angle.

[0160] Optionally, the interpolation coefficient includes the basic information; when the generation module 503 determines the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit, it is specifically used to:

[0161] For any target pixel in the projected image, determine the weight information of the target pixel according to the position of the target pixel and the vertex coordinates of the Delaunay triangle to which the target pixel belongs in the interpolation coefficient;

[0162] The texture coordinate value of the target pixel point is determined according to the weight information and the texture coordinate values corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

[0163] Optionally, the interpolation coefficients include: a first array and a second array corresponding to each Delaunay triangle, as well as texture coordinate values corresponding to each vertex of each Delaunay triangle, and the Delaunay triangle to which each pixel belongs; when the generation module 503 determines the texture coordinate value of each pixel in the projected image based on the interpolation coefficients stored in the storage unit, it is specifically used to:

[0164] For any target pixel in the projected image, a first weight is determined based on the first array and the target pixel coordinates, a second weight is determined based on the second array and the target pixel coordinates, and a third weight is determined based on the first weight and the second weight; the first array and the second array are determined based on the vertex coordinates of the Delaunay triangle to which each pixel belongs; the first weight and the second weight are functions of the target pixel coordinates, respectively, and the first array and the second array are multiple constants in the corresponding functions;

[0165] The texture coordinate value of the target pixel point is determined according to the first weight, the second weight, the third weight, and the texture coordinate values corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

[0166] Optionally, when constructing a plurality of Delaunay triangles according to the projection coordinates, the generating module 503 is specifically configured to:

[0167] determining remaining storage space information of the storage unit, and determining an upper limit on the number of constructed Delaunay triangles according to the remaining storage space information;

[0168] The Delaunay triangle is constructed according to the upper limit of quantity and each of the projection coordinates.

[0169] The multi-angle projection device provided by the embodiment of the present invention can achieve the above-mentioned Figure 2 and Figure 3 The multi-angle projection method of the embodiment shown has similar implementation principles and technical effects, which will not be described again here.

[0170] Figure 6 A schematic diagram of the architecture of a multi-angle projection system provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the multi-angle projection system provided by this embodiment includes: a computing unit and a storage unit;

[0171] The calculation unit is used to execute the method in the above method embodiment; wherein, the storage unit is used to store intermediate results calculated by the calculation unit.

[0172] like Figure 6As shown, the multi-angle projection system includes a computing unit and a storage unit. The computing unit can be a CPU. For the input first image, the computing unit can be used to extract the three-dimensional coordinates of multiple key points through a lightweight neural network model, and then perform ray tracing through the camera position and the projection surface to obtain the projection coordinates of each key point under the target projection angle (corresponding to the camera position). After calculating the projection coordinates, the interpolation coefficients can be determined and stored in the storage unit. The interpolation coefficients are used to calculate the projection images corresponding to other images (such as the second image). Since the interpolation coefficients are stored, when calculating the projection image corresponding to the second image, the amount of calculation is small, and the CPU can be used for calculation.

[0173] The intermediate results calculated by the computing unit can be interpolation coefficients. By storing them, the amount of calculation can be reduced, the requirements for computing power resources can be reduced, and the projection rendering of the target object in the image can be achieved by relying solely on the CPU.

[0174] Figure 7 Schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. Figure 7 As shown, the electronic device provided by this embodiment includes: at least one processor 701 and a memory 702. The processor 701 and the memory 702 are connected via a bus 703.

[0175] During the specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that the at least one processor 701 executes the method in the above method embodiment.

[0176] The specific implementation process of the processor 701 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0177] In the above Figure 7 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0178] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0179] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0180] An embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of the above method embodiment is implemented.

[0181] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of the above method embodiment when executed by a processor.

[0182] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. The computer-readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0183] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0184] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0185] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0187] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-angle projection method, characterized in that: The method comprises: determining three-dimensional coordinates of a plurality of key points of the target object based on at least one image of the target object; For any target projection angle among the multiple projection angles, determining the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point; An image of the target object at the target projection angle is generated according to each of the projection coordinates.

2. The method according to claim 1, characterized in that Determining three-dimensional coordinates of a plurality of key points of a target object according to at least one image of the target object includes: The at least one image is input into a lightweight neural network model, and a plurality of three-dimensional coordinates output by the lightweight neural network model are determined as the three-dimensional coordinates of a plurality of key points of the target object.

3. The method according to claim 1, characterized in that Determining the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point includes: Determine the three-dimensional coordinates of the camera; For any target key point among the multiple key points, determining a projection light according to the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point; An intersection point between the projection light and the projection surface is determined, and the projection coordinates of the target key point on the projection surface are determined based on the intersection point.

4. The method according to claim 3, characterized in that The target key point coordinates and the camera coordinates are both coordinates in the world coordinate system; Determining the projection coordinates of the target key point on the projection surface according to the intersection point includes: The shape parameters of the projection surface are acquired, and the coordinates of the intersection point in the world coordinate system are converted into projection coordinates in the projection surface coordinate system according to the shape parameters of the projection surface.

5. The method according to any one of claims 1 to 4, characterized in that Generating an image of the target object at the target projection angle according to each of the projection coordinates includes: constructing a plurality of Delaunay triangles according to the respective projection coordinates; Generate and store corresponding interpolation coefficients based on basic information corresponding to the at least one image; the basic information includes: the Delaunay triangle to which each pixel point belongs, the vertex coordinates of the Delaunay triangle to which each pixel point belongs, and the texture coordinate values corresponding to the vertices of the Delaunay triangle; For any target image to be processed of the target object, determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit; The pixel value of each pixel point is determined according to the texture coordinate value of each pixel point and the texture map corresponding to the target image, so as to generate an image of the target object in the target image at the target projection angle.

6. The method according to claim 5, characterized in that The interpolation coefficient includes the basic information; and determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit includes: For any target pixel in the projected image, determine the weight information of the target pixel according to the position of the target pixel and the vertex coordinates of the Delaunay triangle to which the target pixel belongs in the interpolation coefficient; The texture coordinate value of the target pixel point is determined according to the weight information and the texture coordinate values corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

7. The method according to claim 5, characterized in that The interpolation coefficients include: a first array and a second array corresponding to each Delaunay triangle, as well as texture coordinate values corresponding to each vertex of each Delaunay triangle, and the Delaunay triangle to which each pixel belongs; determining the texture coordinate value of each pixel in the projected image based on the interpolation coefficients stored in the storage unit includes: For any target pixel in the projected image, a first weight is determined based on the first array and the target pixel coordinates, a second weight is determined based on the second array and the target pixel coordinates, and a third weight is determined based on the first weight and the second weight; the first array and the second array are determined based on the vertex coordinates of the Delaunay triangle to which each pixel belongs; the first weight and the second weight are functions of the target pixel coordinates, respectively, and the first array and the second array are multiple constants in the corresponding functions; The texture coordinate value of the target pixel point is determined according to the first weight, the second weight, the third weight, and the texture coordinate values corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

8. The method according to claim 5, characterized in that Constructing a plurality of Delaunay triangles according to the respective projection coordinates, including: determining remaining storage space information of the storage unit, and determining an upper limit on the number of constructed Delaunay triangles according to the remaining storage space information; The Delaunay triangle is constructed according to the upper limit of quantity and each of the projection coordinates.

9. A multi-angle projection device, characterized in that: The device comprises: a first determining module, configured to determine three-dimensional coordinates of a plurality of key points of the target object based on at least one image of the target object; a second determining module, configured to determine, for any one target projection angle among the plurality of projection angles, the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point; A generating module is used to generate an image of the target object at the target projection angle according to each of the projection coordinates.

10. A multi-angle projection system, characterized in that: include: Computing unit and storage unit; The computing unit is configured to execute the method according to any one of claims 1 to 8; The storage unit is used to store the intermediate results calculated by the calculation unit.

11. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 8 is implemented.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images